2024-04-05T02:39:21.043 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-05T02:39:21.043 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241 --verbose --description upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/orig.config.yaml 2024-04-05T02:39:21.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T02:39:21.190 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241 branch: squid description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7641241' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8639 sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 sleep_before_teardown: 0 subset: 27/32 suite: upgrade suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqvfTydc8eNoNP7JJcUTv+zFEoz6LliN1iHyZNDmR6Z56IJThqPrsYzps+muW4nmbyg/TQe1ZvQ12NWLLvh3pU= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWj/nYva+qMAnCZelK9sstMdPhQ3++tTCxh4dzIMfL5e/3TDnp+FxE8FQghwVMwdb5hxv70bFCvu3OQ0HjZ/Qc= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_02:08:13 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-05T02:39:21.190 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa; will attempt to use it 2024-04-05T02:39:21.191 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks 2024-04-05T02:39:21.191 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-05T02:39:21.192 INFO:teuthology.task.internal:Checking packages... 2024-04-05T02:39:21.216 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0028525c15e19309c20289c609e4fb9d0b12dd06' 2024-04-05T02:39:21.216 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:39:21.216 INFO:teuthology.packaging:ref: None 2024-04-05T02:39:21.216 INFO:teuthology.packaging:tag: None 2024-04-05T02:39:21.216 INFO:teuthology.packaging:branch: squid 2024-04-05T02:39:21.216 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:39:21.216 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-05T02:39:21.603 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2013.g0028525c 2024-04-05T02:39:21.604 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-05T02:39:21.611 INFO:teuthology.task.internal:no buildpackages task found 2024-04-05T02:39:21.611 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-05T02:39:21.618 INFO:teuthology.task.internal:Saving configuration 2024-04-05T02:39:21.639 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-05T02:39:21.645 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T02:39:21.664 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 02:34:04.049151', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqvfTydc8eNoNP7JJcUTv+zFEoz6LliN1iHyZNDmR6Z56IJThqPrsYzps+muW4nmbyg/TQe1ZvQ12NWLLvh3pU='} 2024-04-05T02:39:21.681 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 02:34:04.047720', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWj/nYva+qMAnCZelK9sstMdPhQ3++tTCxh4dzIMfL5e/3TDnp+FxE8FQghwVMwdb5hxv70bFCvu3OQ0HjZ/Qc='} 2024-04-05T02:39:21.681 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-05T02:39:21.687 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-05T02:39:21.687 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-05T02:39:21.687 INFO:teuthology.run_tasks:Running task console_log... 2024-04-05T02:39:21.748 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd55e5140d0>, signals=[15]) 2024-04-05T02:39:21.748 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-05T02:39:21.755 INFO:teuthology.task.internal:Opening connections... 2024-04-05T02:39:21.755 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:39:21.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:39:21.848 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-05T02:39:21.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:39:21.937 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-05T02:39:21.943 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-05T02:39:21.960 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-05T02:39:21.961 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-05T02:39:22.017 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-05T02:39:22.018 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-05T02:39:22.019 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-05T02:39:22.041 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-05T02:39:22.058 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-05T02:39:22.059 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-05T02:39:22.116 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-05T02:39:22.117 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-05T02:39:22.117 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T02:39:22.117 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-05T02:39:22.117 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-05T02:39:22.117 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-05T02:39:22.133 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-05T02:39:22.142 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-05T02:39:22.147 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-05T02:39:22.147 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-05T02:39:22.150 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-05T02:39:22.171 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-05T02:39:22.177 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-05T02:39:22.177 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-05T02:39:22.206 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-05T02:39:22.296 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-05T02:39:22.347 INFO:teuthology.run_tasks:Running task kernel... 2024-04-05T02:39:22.363 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-05T02:39:22.363 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-05T02:39:22.363 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T02:39:22.364 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T02:39:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T02:39:22.381 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-05T02:39:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T02:39:22.382 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-05T02:39:22.437 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-04-05T02:39:22.437 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-04-05T02:39:22.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-04-05T02:39:22.439 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-04-05T02:39:22.439 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-04-05T02:39:22.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-04-05T02:39:25.139 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.7 MB 00:00 2024-04-05T02:39:25.175 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.7 MB 00:00 2024-04-05T02:39:27.050 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-04-05T02:39:27.206 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 2.1 MB/s | 2.0 MB 00:00 2024-04-05T02:39:28.199 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-04-05T02:39:30.924 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-05T02:39:33.671 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.0 MB 00:06 2024-04-05T02:39:34.875 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-04-05T02:39:35.472 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-05T02:39:37.159 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-05T02:39:38.468 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-05T02:39:39.523 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-04-05T02:39:41.757 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-05T02:39:43.410 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-05T02:39:43.643 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-05T02:39:45.892 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-05T02:39:45.892 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-05T02:39:45.973 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-435.el9 CentOS-BaseOS 212 k 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-435.el9 CentOS-BaseOS 15 M 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-435.el9 CentOS-BaseOS 34 M 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-435.el9 CentOS-BaseOS 28 M 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:39:45.985 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:39:45.986 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-04-05T02:39:45.986 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:39:45.986 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 77 M 2024-04-05T02:39:45.986 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-04-05T02:39:45.986 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:39:46.136 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.4 MB/s | 212 kB 00:00 2024-04-05T02:39:46.620 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-04-05T02:39:47.220 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-05T02:39:47.604 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-05T02:39:47.607 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:39:47.607 INFO:teuthology.orchestra.run.smithi153.stdout:Total 47 MB/s | 77 MB 00:01 2024-04-05T02:39:47.607 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:39:47.972 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:39:47.972 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:39:48.402 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-05T02:39:48.797 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:39:48.797 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:39:50.609 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-05T02:39:50.609 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-05T02:39:50.706 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:39:50.717 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:39:50.717 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-435.el9 CentOS-BaseOS 212 k 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-435.el9 CentOS-BaseOS 15 M 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-435.el9 CentOS-BaseOS 34 M 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-435.el9 CentOS-BaseOS 28 M 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-04-05T02:39:50.718 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:39:50.719 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 77 M 2024-04-05T02:39:50.719 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-04-05T02:39:50.719 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:39:50.870 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.4 MB/s | 212 kB 00:00 2024-04-05T02:39:51.061 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:39:51.353 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-04-05T02:39:51.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-05T02:39:51.728 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:39:51.945 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-05T02:39:52.345 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-05T02:39:52.348 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:39:52.349 INFO:teuthology.orchestra.run.smithi203.stdout:Total 47 MB/s | 77 MB 00:01 2024-04-05T02:39:52.349 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:39:52.754 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:39:52.755 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:39:53.601 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:39:53.602 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:39:54.480 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:39:54.631 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:39:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:39:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-05T02:39:56.731 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:39:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:39:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:39:59.496 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:39:59.645 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:40:04.116 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:04.388 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:40:04.502 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:09.253 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:33.492 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:33.492 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-05T02:40:33.492 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:40:33.531 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-05T02:40:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:40:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:40:34.270 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:34.270 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:40:34.270 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:40:34.270 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.271 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.271 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.271 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:40:34.271 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:40:34.590 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T02:40:34.729 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.729 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-05T02:40:34.729 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-05T02:40:34.729 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-05T02:40:34.729 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-05T02:40:34.729 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-05T02:40:34.730 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-04-05T02:40:34.730 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T02:40:34.810 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:34.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-05T02:40:34.810 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-04-05T02:40:34.844 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-04-05T02:40:34.844 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-04-05T02:40:34.844 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-05T02:40:34.845 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:40:34.846 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-04-05T02:40:34.847 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-05T02:40:34.847 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-435.el9.x86_64 2024-04-05T02:40:34.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-05T02:40:35.453 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-04-05T02:40:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-05T02:40:36.875 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-04-05T02:40:36.877 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-04-05T02:40:36.892 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.lmQwKkDx0M 2024-04-05T02:40:36.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lmQwKkDx0M 2024-04-05T02:40:36.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.lmQwKkDx0M 2024-04-05T02:40:37.103 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.lmQwKkDx0M is 6KB 2024-04-05T02:40:37.152 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.lmQwKkDx0M 2024-04-05T02:40:37.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-04-05T02:40:37.231 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-05T02:40:37.231 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-05T02:40:37.231 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-05T02:40:37.231 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-05T02:40:37.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-05T02:40:37.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-04-05T02:40:38.806 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:38.807 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-05T02:40:38.807 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:40:38.850 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:39.245 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:39.246 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-05T02:40:39.246 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-05T02:40:39.246 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-05T02:40:39.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-05T02:40:39.690 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:40:39.690 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:40:39.690 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:39.691 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-05T02:40:39.691 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-05T02:40:39.691 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-05T02:40:39.691 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:40:39.691 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:40:40.048 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T02:40:40.147 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:40.147 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-05T02:40:40.147 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-05T02:40:40.147 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-05T02:40:40.147 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-05T02:40:40.147 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-05T02:40:40.147 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-04-05T02:40:40.147 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T02:40:40.228 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-05T02:40:40.228 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-05T02:40:40.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-04-05T02:40:40.271 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:40:40.272 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-05T02:40:40.273 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-05T02:40:40.273 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-05T02:40:40.273 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-04-05T02:40:40.273 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:40:40.273 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-04-05T02:40:40.274 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-05T02:40:40.275 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-435.el9.x86_64 2024-04-05T02:40:40.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-05T02:40:40.859 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-04-05T02:40:42.217 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-05T02:40:42.254 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-04-05T02:40:42.257 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-05T02:40:42.271 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.NNRx8NHgqv 2024-04-05T02:40:42.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NNRx8NHgqv 2024-04-05T02:40:42.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.NNRx8NHgqv 2024-04-05T02:40:42.476 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.NNRx8NHgqv is 6KB 2024-04-05T02:40:42.524 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.NNRx8NHgqv 2024-04-05T02:40:42.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-04-05T02:40:42.603 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-05T02:40:42.603 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-05T02:40:42.603 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-05T02:40:42.603 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-05T02:40:42.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-05T02:40:42.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-04-05T02:41:07.392 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-05T02:41:07.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:41:07.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:41:12.772 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-05T02:41:12.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:41:12.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:41:25.795 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-05T02:41:31.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-05T02:41:34.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:41:34.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:41:37.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-05T02:41:40.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:41:40.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:41:43.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-05T02:41:49.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:41:49.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:41:55.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:41:55.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:42:49.919 DEBUG:teuthology.orchestra.remote:timed out 2024-04-05T02:42:55.270 DEBUG:teuthology.orchestra.remote:timed out 2024-04-05T02:43:04.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:43:04.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:43:05.354 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-05T02:43:05.728 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:43:05.728 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-435.el9.x86_64"... 2024-04-05T02:43:05.728 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-05T02:43:05.743 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-435.el9.x86_64 2024-04-05T02:43:05.743 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-05T02:43:05.744 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-05T02:43:05.744 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-05T02:43:06.744 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-05T02:43:06.744 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-05T02:43:06.837 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-04-05T02:43:06.879 DEBUG:teuthology.parallel:result is None 2024-04-05T02:43:10.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:43:10.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:43:10.793 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-05T02:43:11.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:43:11.189 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-435.el9.x86_64"... 2024-04-05T02:43:11.189 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-05T02:43:11.204 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-435.el9.x86_64 2024-04-05T02:43:11.204 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-05T02:43:11.205 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-05T02:43:11.205 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-05T02:43:12.206 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-05T02:43:12.206 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-05T02:43:12.273 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-04-05T02:43:12.306 DEBUG:teuthology.parallel:result is None 2024-04-05T02:43:12.306 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-05T02:43:12.314 INFO:teuthology.task.internal:Creating test directory... 2024-04-05T02:43:12.314 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T02:43:12.317 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T02:43:12.335 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-05T02:43:12.341 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-05T02:43:12.347 INFO:teuthology.task.internal:Creating archive directory... 2024-04-05T02:43:12.348 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T02:43:12.376 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T02:43:12.452 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-05T02:43:12.458 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-05T02:43:12.458 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T02:43:12.461 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T02:43:12.497 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T02:43:12.510 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T02:43:12.511 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T02:43:12.525 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T02:43:12.526 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-05T02:43:12.533 INFO:teuthology.task.internal:Configuring sudo... 2024-04-05T02:43:12.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T02:43:12.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T02:43:12.585 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-05T02:43:12.594 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-05T02:43:12.595 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T02:43:12.626 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T02:43:12.642 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T02:43:12.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T02:43:12.776 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T02:43:12.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T02:43:12.901 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:43:12.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T02:43:12.969 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T02:43:13.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T02:43:13.066 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T02:43:13.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T02:43:13.197 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:43:13.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T02:43:13.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-05T02:43:13.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-05T02:43:13.319 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T02:43:13.351 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T02:43:13.838 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-05T02:43:13.849 INFO:teuthology.task.internal:Starting timer... 2024-04-05T02:43:13.849 INFO:teuthology.run_tasks:Running task pcp... 2024-04-05T02:43:13.859 INFO:teuthology.run_tasks:Running task selinux... 2024-04-05T02:43:13.867 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-05T02:43:13.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-05T02:43:13.924 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-05T02:43:13.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-05T02:43:13.981 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-05T02:43:13.983 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-05T02:43:13.983 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-05T02:43:14.014 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-05T02:43:14.014 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-05T02:43:14.038 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-05T02:43:14.038 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-05T02:43:14.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-05T02:43:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T02:43:14.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-05T02:43:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T02:43:14.112 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-05T02:43:14.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-05T02:43:14.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-05T02:43:14.189 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-05T02:43:14.199 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-05T02:43:14.216 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-05T02:43:14.217 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-05T02:47:40.641 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-05T02:47:40.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:47:40.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:47:40.732 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-05T02:47:40.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-05T02:47:40.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:47:40.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T02:47:40.891 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-05T02:47:40.968 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-05T02:47:40.969 INFO:teuthology.run_tasks:Running task clock... 2024-04-05T02:47:40.979 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-05T02:47:40.979 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T02:47:40.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T02:47:40.982 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T02:47:40.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T02:47:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T02:47:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T02:47:41.039 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T02:47:41.056 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T02:47:41.057 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-05T02:47:41.070 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-05T02:47:41.085 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-05T02:47:41.087 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T02:47:41.099 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-05T02:47:41.104 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T02:47:41.116 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T02:47:41.134 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T02:47:41.156 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-05T02:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T02:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-05T02:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.160 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.161 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.183 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-05T02:47:41.186 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T02:47:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-05T02:47:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T02:47:41.187 INFO:teuthology.run_tasks:Running task install... 2024-04-05T02:47:41.195 DEBUG:teuthology.task.install:project ceph 2024-04-05T02:47:41.195 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0028525c15e19309c20289c609e4fb9d0b12dd06'}} 2024-04-05T02:47:41.195 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '0028525c15e19309c20289c609e4fb9d0b12dd06'} 2024-04-05T02:47:41.195 INFO:teuthology.task.install:Using flavor: default 2024-04-05T02:47:41.200 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-05T02:47:41.201 INFO:teuthology.task.install:extra packages: [] 2024-04-05T02:47:41.201 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0028525c15e19309c20289c609e4fb9d0b12dd06', 'tag': None, 'wait_for_package': False} 2024-04-05T02:47:41.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:47:41.201 INFO:teuthology.packaging:ref: None 2024-04-05T02:47:41.201 INFO:teuthology.packaging:tag: None 2024-04-05T02:47:41.201 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:47:41.201 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:47:41.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-05T02:47:41.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0028525c15e19309c20289c609e4fb9d0b12dd06', 'tag': None, 'wait_for_package': False} 2024-04-05T02:47:41.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:47:41.204 INFO:teuthology.packaging:ref: None 2024-04-05T02:47:41.204 INFO:teuthology.packaging:tag: None 2024-04-05T02:47:41.204 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:47:41.204 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:47:41.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-05T02:47:41.447 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/ 2024-04-05T02:47:41.447 INFO:teuthology.task.install.rpm:Package version is 17.2.7-815.g93c790e1 2024-04-05T02:47:41.542 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/ 2024-04-05T02:47:41.542 INFO:teuthology.task.install.rpm:Package version is 17.2.7-815.g93c790e1 2024-04-05T02:47:41.653 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T02:47:41.653 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:47:41.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T02:47:41.681 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/93c790e135cc94d629f832b908e3f6c70360bfd5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T02:47:41.681 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:47:41.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T02:47:41.682 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-05T02:47:41.682 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:47:41.683 INFO:teuthology.packaging:ref: None 2024-04-05T02:47:41.683 INFO:teuthology.packaging:tag: None 2024-04-05T02:47:41.683 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:47:41.683 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:47:41.683 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T02:47:41.712 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-05T02:47:41.712 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:47:41.712 INFO:teuthology.packaging:ref: None 2024-04-05T02:47:41.712 INFO:teuthology.packaging:tag: None 2024-04-05T02:47:41.712 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:47:41.712 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:47:41.712 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T02:47:41.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T02:47:41.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T02:47:41.832 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T02:47:41.860 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T02:47:41.898 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-05T02:47:41.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-05T02:47:41.927 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-05T02:47:41.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-05T02:47:42.146 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-04-05T02:47:42.170 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-04-05T02:47:42.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-04-05T02:47:42.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-04-05T02:47:42.932 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 170 kB/s | 77 kB 00:00 2024-04-05T02:47:42.939 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 177 kB/s | 77 kB 00:00 2024-04-05T02:47:43.374 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 23 kB/s | 9.2 kB 00:00 2024-04-05T02:47:43.381 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 23 kB/s | 9.2 kB 00:00 2024-04-05T02:47:43.798 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 4.7 kB/s | 1.8 kB 00:00 2024-04-05T02:47:43.933 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 3.4 kB/s | 1.8 kB 00:00 2024-04-05T02:47:44.685 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-04-05T02:47:44.708 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-04-05T02:47:46.284 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-04-05T02:47:46.436 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-04-05T02:47:47.543 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-05T02:47:48.308 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-04-05T02:47:50.111 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-05T02:47:54.621 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-04-05T02:47:56.480 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-05T02:47:58.731 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-05T02:48:03.673 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-04-05T02:48:06.284 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:17.2.7-815.g93c790e1.el9 ceph 14 M 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-05T02:48:06.287 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 3.1 M 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.9 M 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:17.2.7-815.g93c790e1.el9 ceph 5.0 M 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:17.2.7-815.g93c790e1.el9 ceph 22 M 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:17.2.7-815.g93c790e1.el9 ceph 25 k 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 637 k 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-05T02:48:06.288 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 458 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 4.9 M 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 45 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:17.2.7-815.g93c790e1.el9 ceph 117 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:17.2.7-815.g93c790e1.el9 ceph 156 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:17.2.7-815.g93c790e1.el9 ceph 320 k 2024-04-05T02:48:06.289 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 293 k 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:17.2.7-815.g93c790e1.el9 ceph 99 k 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout:Install 22 Packages 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-04-05T02:48:06.290 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:06.291 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-04-05T02:48:06.291 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:48:06.982 INFO:teuthology.orchestra.run.smithi153.stdout:(1/24): ceph-base-17.2.7-815.g93c790e1.el9.x86_ 10 MB/s | 5.0 MB 00:00 2024-04-05T02:48:07.015 INFO:teuthology.orchestra.run.smithi153.stdout:(2/24): ceph-selinux-17.2.7-815.g93c790e1.el9.x 759 kB/s | 25 kB 00:00 2024-04-05T02:48:07.074 INFO:teuthology.orchestra.run.smithi153.stdout:(3/24): libcephfs2-17.2.7-815.g93c790e1.el9.x86 11 MB/s | 637 kB 00:00 2024-04-05T02:48:07.133 INFO:teuthology.orchestra.run.smithi153.stdout:(4/24): libradosstriper1-17.2.7-815.g93c790e1.e 7.7 MB/s | 458 kB 00:00 2024-04-05T02:48:07.425 INFO:teuthology.orchestra.run.smithi153.stdout:(5/24): librgw2-17.2.7-815.g93c790e1.el9.x86_64 17 MB/s | 4.9 MB 00:00 2024-04-05T02:48:07.467 INFO:teuthology.orchestra.run.smithi153.stdout:(6/24): python3-ceph-argparse-17.2.7-815.g93c79 1.1 MB/s | 45 kB 00:00 2024-04-05T02:48:07.517 INFO:teuthology.orchestra.run.smithi153.stdout:(7/24): python3-ceph-common-17.2.7-815.g93c790e 2.3 MB/s | 117 kB 00:00 2024-04-05T02:48:07.576 INFO:teuthology.orchestra.run.smithi153.stdout:(8/24): python3-cephfs-17.2.7-815.g93c790e1.el9 2.6 MB/s | 156 kB 00:00 2024-04-05T02:48:07.643 INFO:teuthology.orchestra.run.smithi153.stdout:(9/24): python3-rados-17.2.7-815.g93c790e1.el9. 4.7 MB/s | 320 kB 00:00 2024-04-05T02:48:07.693 INFO:teuthology.orchestra.run.smithi153.stdout:(10/24): python3-rbd-17.2.7-815.g93c790e1.el9.x 5.7 MB/s | 293 kB 00:00 2024-04-05T02:48:07.743 INFO:teuthology.orchestra.run.smithi153.stdout:(11/24): python3-rgw-17.2.7-815.g93c790e1.el9.x 1.9 MB/s | 99 kB 00:00 2024-04-05T02:48:07.893 INFO:teuthology.orchestra.run.smithi153.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-04-05T02:48:07.952 INFO:teuthology.orchestra.run.smithi153.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-04-05T02:48:08.061 INFO:teuthology.orchestra.run.smithi153.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-04-05T02:48:08.102 INFO:teuthology.orchestra.run.smithi153.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-04-05T02:48:08.145 INFO:teuthology.orchestra.run.smithi153.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 984 kB/s | 41 kB 00:00 2024-04-05T02:48:08.462 INFO:teuthology.orchestra.run.smithi153.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 976 kB/s | 309 kB 00:00 2024-04-05T02:48:08.537 INFO:teuthology.orchestra.run.smithi153.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-04-05T02:48:08.604 INFO:teuthology.orchestra.run.smithi153.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-05T02:48:08.904 INFO:teuthology.orchestra.run.smithi153.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-04-05T02:48:09.246 INFO:teuthology.orchestra.run.smithi153.stdout:(21/24): ceph-common-17.2.7-815.g93c790e1.el9.x 8.2 MB/s | 22 MB 00:02 2024-04-05T02:48:09.388 INFO:teuthology.orchestra.run.smithi153.stdout:(22/24): ceph-radosgw-17.2.7-815.g93c790e1.el9. 4.9 MB/s | 14 MB 00:02 2024-04-05T02:48:09.446 INFO:teuthology.orchestra.run.smithi153.stdout:(23/24): librados2-17.2.7-815.g93c790e1.el9.x86 5.7 MB/s | 3.1 MB 00:00 2024-04-05T02:48:09.538 INFO:teuthology.orchestra.run.smithi153.stdout:(24/24): librbd1-17.2.7-815.g93c790e1.el9.x86_6 10 MB/s | 2.9 MB 00:00 2024-04-05T02:48:09.542 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:48:09.542 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 58 MB 00:03 2024-04-05T02:48:09.542 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:48:09.572 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:48:09.572 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:48:10.024 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:48:10.024 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:48:11.296 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:48:11.627 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-05T02:48:11.690 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:11.862 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 3/26 2024-04-05T02:48:11.892 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:11.980 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-05T02:48:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-05T02:48:12.499 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-05T02:48:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 8/26 2024-04-05T02:48:12.591 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 8/26 2024-04-05T02:48:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-05T02:48:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x 10/26 2024-04-05T02:48:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 11/26 2024-04-05T02:48:12.778 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 12/26 2024-04-05T02:48:12.806 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:12.872 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:12.911 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-05T02:48:12.947 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-05T02:48:12.980 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-05T02:48:13.282 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-05T02:48:13.310 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 18/26 2024-04-05T02:48:13.386 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 18/26 2024-04-05T02:48:13.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 19/26 2024-04-05T02:48:13.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86 20/26 2024-04-05T02:48:16.233 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:16.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:16.845 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 22/26 2024-04-05T02:48:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 22/26 2024-04-05T02:48:16.924 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-05T02:48:16.924 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-05T02:48:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:18.906 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 655 kB/s | 19 MB 00:29 2024-04-05T02:48:22.957 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-04-05T02:48:24.366 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-05T02:48:26.362 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:26.370 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-05T02:48:27.601 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 24/26 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 24/26 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-05T02:48:27.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:27.707 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-05T02:48:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-05T02:48:27.796 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:28.966 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:28.966 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 1/26 2024-04-05T02:48:28.966 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:28.966 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 3/26 2024-04-05T02:48:28.966 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 5/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 6/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 7/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x 8/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86 9/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 10/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 11/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 12/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-05T02:48:28.967 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-05T02:48:28.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 25/26 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-05T02:48:30.009 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.010 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:30.011 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:48:30.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-04-05T02:48:30.909 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:27 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:48:31.059 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:17.2.7-815.g93c790e1.el9 ceph 48 M 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:48:31.060 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:17.2.7-815.g93c790e1.el9 ceph 163 k 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-04-05T02:48:31.061 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:31.062 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 49 M 2024-04-05T02:48:31.062 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 193 M 2024-04-05T02:48:31.062 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:48:31.310 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-05T02:48:31.377 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-04-05T02:48:31.419 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-05T02:48:31.442 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 6.9 kB/s | 1.7 kB 00:00 2024-04-05T02:48:31.494 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-04-05T02:48:31.536 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): libcephsqlite-17.2.7-815.g93c790e1.el9.x 416 kB/s | 163 kB 00:00 2024-04-05T02:48:34.073 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:17.2.7-815.g93c790e1.el9 ceph 14 M 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 3.1 M 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.9 M 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:17.2.7-815.g93c790e1.el9 ceph 5.0 M 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:17.2.7-815.g93c790e1.el9 ceph 22 M 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:17.2.7-815.g93c790e1.el9 ceph 25 k 2024-04-05T02:48:34.092 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 637 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 458 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:17.2.7-815.g93c790e1.el9 ceph 4.9 M 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 45 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:17.2.7-815.g93c790e1.el9 ceph 117 k 2024-04-05T02:48:34.093 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:17.2.7-815.g93c790e1.el9 ceph 156 k 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:17.2.7-815.g93c790e1.el9 ceph 320 k 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 293 k 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:17.2.7-815.g93c790e1.el9 ceph 99 k 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout:Install 22 Packages 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-04-05T02:48:34.094 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:34.095 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 58 M 2024-04-05T02:48:34.095 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:48:34.670 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-17.2.7-815.g93c790e1.el9.x86_6 14 MB/s | 48 MB 00:03 2024-04-05T02:48:34.673 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:48:34.673 INFO:teuthology.orchestra.run.smithi153.stdout:Total 14 MB/s | 49 MB 00:03 2024-04-05T02:48:34.688 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:48:34.709 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:48:34.709 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:48:35.093 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:48:35.093 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:48:35.668 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:48:35.737 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-05T02:48:35.781 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-05T02:48:35.811 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-05T02:48:35.861 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-05T02:48:35.889 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 5/6 2024-04-05T02:48:36.018 INFO:teuthology.orchestra.run.smithi203.stdout:(1/24): ceph-base-17.2.7-815.g93c790e1.el9.x86_ 3.2 MB/s | 5.0 MB 00:01 2024-04-05T02:48:36.068 INFO:teuthology.orchestra.run.smithi203.stdout:(2/24): ceph-selinux-17.2.7-815.g93c790e1.el9.x 506 kB/s | 25 kB 00:00 2024-04-05T02:48:36.185 INFO:teuthology.orchestra.run.smithi203.stdout:(3/24): libcephfs2-17.2.7-815.g93c790e1.el9.x86 5.3 MB/s | 637 kB 00:00 2024-04-05T02:48:36.293 INFO:teuthology.orchestra.run.smithi203.stdout:(4/24): libradosstriper1-17.2.7-815.g93c790e1.e 4.1 MB/s | 458 kB 00:00 2024-04-05T02:48:37.010 INFO:teuthology.orchestra.run.smithi203.stdout:(5/24): ceph-radosgw-17.2.7-815.g93c790e1.el9.x 5.5 MB/s | 14 MB 00:02 2024-04-05T02:48:37.086 INFO:teuthology.orchestra.run.smithi203.stdout:(6/24): librgw2-17.2.7-815.g93c790e1.el9.x86_64 6.2 MB/s | 4.9 MB 00:00 2024-04-05T02:48:37.111 INFO:teuthology.orchestra.run.smithi203.stdout:(7/24): python3-ceph-argparse-17.2.7-815.g93c79 450 kB/s | 45 kB 00:00 2024-04-05T02:48:37.136 INFO:teuthology.orchestra.run.smithi203.stdout:(8/24): python3-ceph-common-17.2.7-815.g93c790e 2.3 MB/s | 117 kB 00:00 2024-04-05T02:48:37.161 INFO:teuthology.orchestra.run.smithi203.stdout:(9/24): python3-cephfs-17.2.7-815.g93c790e1.el9 3.1 MB/s | 156 kB 00:00 2024-04-05T02:48:37.187 INFO:teuthology.orchestra.run.smithi203.stdout:(10/24): python3-rados-17.2.7-815.g93c790e1.el9 6.3 MB/s | 320 kB 00:00 2024-04-05T02:48:37.212 INFO:teuthology.orchestra.run.smithi203.stdout:(11/24): python3-rbd-17.2.7-815.g93c790e1.el9.x 5.7 MB/s | 293 kB 00:00 2024-04-05T02:48:37.237 INFO:teuthology.orchestra.run.smithi203.stdout:(12/24): python3-rgw-17.2.7-815.g93c790e1.el9.x 1.9 MB/s | 99 kB 00:00 2024-04-05T02:48:37.546 INFO:teuthology.orchestra.run.smithi203.stdout:(13/24): ceph-common-17.2.7-815.g93c790e1.el9.x 7.3 MB/s | 22 MB 00:03 2024-04-05T02:48:37.613 INFO:teuthology.orchestra.run.smithi203.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 397 kB/s | 159 kB 00:00 2024-04-05T02:48:37.638 INFO:teuthology.orchestra.run.smithi203.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 118 kB/s | 44 kB 00:00 2024-04-05T02:48:37.705 INFO:teuthology.orchestra.run.smithi203.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 661 kB 00:00 2024-04-05T02:48:37.730 INFO:teuthology.orchestra.run.smithi203.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-04-05T02:48:37.747 INFO:teuthology.orchestra.run.smithi203.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2024-04-05T02:48:37.914 INFO:teuthology.orchestra.run.smithi203.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 292 kB/s | 49 kB 00:00 2024-04-05T02:48:37.939 INFO:teuthology.orchestra.run.smithi203.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-04-05T02:48:38.006 INFO:teuthology.orchestra.run.smithi203.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-05T02:48:38.165 INFO:teuthology.orchestra.run.smithi203.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-04-05T02:48:38.240 INFO:teuthology.orchestra.run.smithi203.stdout:(23/24): librados2-17.2.7-815.g93c790e1.el9.x86 10 MB/s | 3.1 MB 00:00 2024-04-05T02:48:38.290 INFO:teuthology.orchestra.run.smithi203.stdout:(24/24): librbd1-17.2.7-815.g93c790e1.el9.x86_6 10 MB/s | 2.9 MB 00:00 2024-04-05T02:48:38.295 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:48:38.295 INFO:teuthology.orchestra.run.smithi203.stdout:Total 14 MB/s | 58 MB 00:04 2024-04-05T02:48:38.296 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:48:38.330 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:48:38.330 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:48:38.786 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:48:38.786 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:48:40.018 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:48:40.264 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 5/6 2024-04-05T02:48:40.339 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 6/6 2024-04-05T02:48:40.349 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-05T02:48:40.415 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:40.635 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:40.721 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 3/26 2024-04-05T02:48:40.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:40.857 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:40.911 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-05T02:48:40.940 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-05T02:48:41.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-05T02:48:41.397 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 8/26 2024-04-05T02:48:41.459 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 8/26 2024-04-05T02:48:41.488 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-05T02:48:41.531 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x 10/26 2024-04-05T02:48:41.582 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 11/26 2024-04-05T02:48:41.646 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 12/26 2024-04-05T02:48:41.674 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:41.723 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:41.763 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-05T02:48:41.824 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-05T02:48:41.873 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-05T02:48:42.167 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-05T02:48:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 18/26 2024-04-05T02:48:42.310 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 18/26 2024-04-05T02:48:42.382 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 19/26 2024-04-05T02:48:42.406 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86 20/26 2024-04-05T02:48:42.556 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 6/6 2024-04-05T02:48:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 1/6 2024-04-05T02:48:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 2/6 2024-04-05T02:48:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-05T02:48:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-05T02:48:42.557 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:42.932 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:48:43.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-04-05T02:48:43.804 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:40 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:48:43.957 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:48:43.959 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-04-05T02:48:43.959 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:17.2.7-815.g93c790e1.el9 ceph 6.5 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.1 M 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:17.2.7-815.g93c790e1.el9 ceph 1.4 M 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 240 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:17.2.7-815.g93c790e1.el9 ceph 3.9 M 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 15 M 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-05T02:48:43.960 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-05T02:48:43.961 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout:Install 33 Packages 2024-04-05T02:48:43.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:43.963 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2024-04-05T02:48:43.963 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2024-04-05T02:48:43.963 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:48:44.793 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:44.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:44.973 INFO:teuthology.orchestra.run.smithi153.stdout:(1/33): ceph-17.2.7-815.g93c790e1.el9.x86_64.rp 20 kB/s | 6.5 kB 00:00 2024-04-05T02:48:45.353 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 21/26 2024-04-05T02:48:45.389 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 22/26 2024-04-05T02:48:45.480 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 22/26 2024-04-05T02:48:45.481 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-05T02:48:45.481 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-05T02:48:45.481 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:45.532 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:46.015 INFO:teuthology.orchestra.run.smithi153.stdout:(2/33): ceph-mon-17.2.7-815.g93c790e1.el9.x86_6 3.8 MB/s | 3.9 MB 00:01 2024-04-05T02:48:46.257 INFO:teuthology.orchestra.run.smithi153.stdout:(3/33): ceph-mgr-17.2.7-815.g93c790e1.el9.x86_6 894 kB/s | 1.4 MB 00:01 2024-04-05T02:48:46.315 INFO:teuthology.orchestra.run.smithi153.stdout:(4/33): ceph-mds-17.2.7-815.g93c790e1.el9.x86_6 1.2 MB/s | 2.1 MB 00:01 2024-04-05T02:48:46.465 INFO:teuthology.orchestra.run.smithi153.stdout:(5/33): ceph-mgr-modules-core-17.2.7-815.g93c79 1.1 MB/s | 240 kB 00:00 2024-04-05T02:48:46.490 INFO:teuthology.orchestra.run.smithi153.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 823 kB/s | 144 kB 00:00 2024-04-05T02:48:46.516 INFO:teuthology.orchestra.run.smithi153.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-04-05T02:48:46.541 INFO:teuthology.orchestra.run.smithi153.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-04-05T02:48:46.566 INFO:teuthology.orchestra.run.smithi153.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-04-05T02:48:46.609 INFO:teuthology.orchestra.run.smithi153.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-05T02:48:46.684 INFO:teuthology.orchestra.run.smithi153.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-04-05T02:48:46.709 INFO:teuthology.orchestra.run.smithi153.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-05T02:48:46.734 INFO:teuthology.orchestra.run.smithi153.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-05T02:48:46.759 INFO:teuthology.orchestra.run.smithi153.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-05T02:48:46.860 INFO:teuthology.orchestra.run.smithi153.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-04-05T02:48:46.902 INFO:teuthology.orchestra.run.smithi153.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-04-05T02:48:46.952 INFO:teuthology.orchestra.run.smithi153.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-05T02:48:46.985 INFO:teuthology.orchestra.run.smithi153.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-04-05T02:48:47.019 INFO:teuthology.orchestra.run.smithi153.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2024-04-05T02:48:47.078 INFO:teuthology.orchestra.run.smithi153.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-04-05T02:48:47.095 INFO:teuthology.orchestra.run.smithi153.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-04-05T02:48:47.120 INFO:teuthology.orchestra.run.smithi153.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 465 kB/s | 19 kB 00:00 2024-04-05T02:48:47.145 INFO:teuthology.orchestra.run.smithi153.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-04-05T02:48:47.171 INFO:teuthology.orchestra.run.smithi153.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-04-05T02:48:47.205 INFO:teuthology.orchestra.run.smithi153.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-04-05T02:48:47.230 INFO:teuthology.orchestra.run.smithi153.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-05T02:48:47.272 INFO:teuthology.orchestra.run.smithi153.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-04-05T02:48:47.314 INFO:teuthology.orchestra.run.smithi153.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-05T02:48:47.339 INFO:teuthology.orchestra.run.smithi153.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-04-05T02:48:47.364 INFO:teuthology.orchestra.run.smithi153.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-05T02:48:47.423 INFO:teuthology.orchestra.run.smithi153.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-04-05T02:48:47.448 INFO:teuthology.orchestra.run.smithi153.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-04-05T02:48:47.665 INFO:teuthology.orchestra.run.smithi153.stdout:(33/33): ceph-osd-17.2.7-815.g93c790e1.el9.x86_ 9.3 MB/s | 15 MB 00:01 2024-04-05T02:48:47.669 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:48:47.669 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.3 MB/s | 27 MB 00:03 2024-04-05T02:48:47.682 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:48:47.721 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:48:47.721 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:48:48.065 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:48:48.065 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:48:49.349 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:48:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-05T02:48:49.434 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-05T02:48:49.466 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-05T02:48:49.557 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-05T02:48:49.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-05T02:48:49.678 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-05T02:48:49.749 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-05T02:48:49.804 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-05T02:48:49.844 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-05T02:48:49.902 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-05T02:48:49.956 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-05T02:48:50.156 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-05T02:48:50.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-05T02:48:50.244 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-05T02:48:50.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-05T02:48:50.309 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-05T02:48:50.386 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-05T02:48:50.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-05T02:48:50.453 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-05T02:48:50.484 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-05T02:48:50.519 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-05T02:48:50.554 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-05T02:48:50.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-05T02:48:50.684 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-05T02:48:50.757 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-05T02:48:50.901 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-05T02:48:51.018 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-05T02:48:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.n 28/33 2024-04-05T02:48:51.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 29/33 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 29/33 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-05T02:48:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:52.565 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 30/33 2024-04-05T02:48:52.591 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 30/33 2024-04-05T02:48:52.591 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:52.592 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-05T02:48:52.592 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-05T02:48:52.592 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-05T02:48:52.592 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:52.966 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 31/33 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 31/33 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-05T02:48:52.989 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:53.192 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 32/33 2024-04-05T02:48:53.214 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 32/33 2024-04-05T02:48:53.215 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:53.215 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-05T02:48:53.215 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-05T02:48:53.215 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-05T02:48:53.215 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:53.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:17.2.7-815.g93c790e1.el9.x86_64 33/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:17.2.7-815.g93c790e1.el9.x86_64 33/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:17.2.7-815.g93c790e1.el9.x86_64 1/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 2/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 3/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 4/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 5/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.n 6/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-05T02:48:54.761 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-05T02:48:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-05T02:48:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-05T02:48:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-05T02:48:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-05T02:48:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-05T02:48:55.303 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-05T02:48:55.304 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:55.675 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-05T02:48:55.676 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-05T02:48:55.677 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:55.678 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:48:56.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-04-05T02:48:56.578 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 24/26 2024-04-05T02:48:56.597 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 24/26 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-05T02:48:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-05T02:48:56.694 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:48:56.723 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-05T02:48:56.723 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:56.745 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:48:56.746 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:48:56.746 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:48:56.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-04-05T02:48:57.367 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:54 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:48:57.518 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:48:57.518 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 82 k 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 372 k 2024-04-05T02:48:57.519 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 1/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 3/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 4/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 5/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 6/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 7/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x 8/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86 9/26 2024-04-05T02:48:58.000 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 10/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 11/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 12/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 13/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:17.2.7-815.g93c790e1.el9.x86_64 23/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-05T02:48:58.001 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 25/26 2024-04-05T02:48:58.090 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-17.2.7-815.g93c790e1.el9.noarch.rpm 144 kB/s | 82 kB 00:00 2024-04-05T02:48:58.090 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:48:58.091 INFO:teuthology.orchestra.run.smithi153.stdout:Total 143 kB/s | 82 kB 00:00 2024-04-05T02:48:58.091 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:48:58.096 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:48:58.096 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:48:58.097 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:48:58.098 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:48:58.174 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:48:58.545 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:48:58.652 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.054 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-05T02:48:59.055 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:48:59.056 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:48:59.057 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:48:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:48:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:59.364 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:48:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:48:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:48:59.364 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:48:59.440 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-04-05T02:48:59.586 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-04-05T02:49:00.007 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:29 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:00.160 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:00.160 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:17.2.7-815.g93c790e1.el9 ceph 48 M 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:17.2.7-815.g93c790e1.el9 ceph 163 k 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-05T02:49:00.161 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-05T02:49:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:00.162 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:00.162 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:00.162 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-04-05T02:49:00.162 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:00.163 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 49 M 2024-04-05T02:49:00.163 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 193 M 2024-04-05T02:49:00.163 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:00.165 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:57 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:00.318 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-815.g93c790e1.el9 ceph 137 k 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 137 k 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 379 k 2024-04-05T02:49:00.319 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:00.383 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-05T02:49:00.434 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-04-05T02:49:00.467 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-05T02:49:00.501 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): libcephsqlite-17.2.7-815.g93c790e1.el9.x 588 kB/s | 163 kB 00:00 2024-04-05T02:49:00.519 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-17.2.7-815.g93c790e 688 kB/s | 137 kB 00:00 2024-04-05T02:49:00.520 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:00.520 INFO:teuthology.orchestra.run.smithi153.stdout:Total 683 kB/s | 137 kB 00:00 2024-04-05T02:49:00.520 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:00.527 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:00.528 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:00.579 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:00.579 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:00.715 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:00.778 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:00.803 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:00.803 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:00.803 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-05T02:49:00.803 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:00.968 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:01.753 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:02.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-04-05T02:49:02.598 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:59 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:02.700 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:02.753 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:02.754 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:02.755 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:02.768 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-17.2.7-815.g93c790e1.el9.x86_6 19 MB/s | 48 MB 00:02 2024-04-05T02:49:02.771 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:02.772 INFO:teuthology.orchestra.run.smithi203.stdout:Total 19 MB/s | 49 MB 00:02 2024-04-05T02:49:02.787 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:02.808 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:02.808 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:02.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-04-05T02:49:03.201 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:03.202 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:03.379 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:00 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:03.531 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:03.532 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:03.532 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:49:03.532 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:03.532 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 1.7 M 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 24 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 15 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:03.533 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-04-05T02:49:03.534 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:03.534 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-04-05T02:49:03.534 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 8.1 M 2024-04-05T02:49:03.534 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:03.773 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:03.855 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-05T02:49:03.900 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-05T02:49:03.934 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-05T02:49:03.977 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-05T02:49:04.005 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 5/6 2024-04-05T02:49:04.139 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-prometheus-alerts-17.2.7-815.g93c79 61 kB/s | 15 kB 00:00 2024-04-05T02:49:04.156 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-grafana-dashboards-17.2.7-815.g93c7 92 kB/s | 24 kB 00:00 2024-04-05T02:49:04.282 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): ceph-mgr-dashboard-17.2.7-815.g93c790e1. 4.4 MB/s | 1.7 MB 00:00 2024-04-05T02:49:04.298 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2024-04-05T02:49:04.315 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 257 kB/s | 41 kB 00:00 2024-04-05T02:49:04.374 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 334 kB/s | 31 kB 00:00 2024-04-05T02:49:04.499 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2024-04-05T02:49:04.500 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:04.501 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-05T02:49:04.501 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:04.509 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:04.509 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:04.569 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:04.569 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:04.869 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:04.917 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-05T02:49:04.951 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-05T02:49:04.997 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-05T02:49:05.029 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-05T02:49:05.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.no 5/7 2024-04-05T02:49:05.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.n 6/7 2024-04-05T02:49:05.319 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 7/7 2024-04-05T02:49:05.823 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 7/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.n 1/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 2/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.no 3/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-05T02:49:05.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-05T02:49:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:06.118 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:06.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-05T02:49:06.932 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:03 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:07.085 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 7.4 M 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-05T02:49:07.086 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-04-05T02:49:07.087 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:07.088 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-04-05T02:49:07.088 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-04-05T02:49:07.088 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:07.306 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-04-05T02:49:07.431 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 138 kB/s | 17 kB 00:00 2024-04-05T02:49:07.481 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.5 MB/s | 3.0 MB 00:00 2024-04-05T02:49:07.506 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 462 kB/s | 34 kB 00:00 2024-04-05T02:49:07.740 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-05T02:49:07.807 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-04-05T02:49:07.832 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-04-05T02:49:07.924 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-81 9.8 MB/s | 7.4 MB 00:00 2024-04-05T02:49:07.999 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-04-05T02:49:08.041 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-05T02:49:08.317 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-04-05T02:49:08.402 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 5/6 2024-04-05T02:49:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 603 kB/s | 245 kB 00:00 2024-04-05T02:49:08.411 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:08.411 INFO:teuthology.orchestra.run.smithi153.stdout:Total 28 MB/s | 37 MB 00:01 2024-04-05T02:49:08.452 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:08.476 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:08.476 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:08.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 6/6 2024-04-05T02:49:08.954 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:08.954 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:09.748 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-05T02:49:09.779 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-05T02:49:09.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-05T02:49:09.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-05T02:49:10.103 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-05T02:49:10.131 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-05T02:49:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 6/6 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 1/6 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 2/6 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-05T02:49:10.795 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-05T02:49:10.923 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-05T02:49:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-05T02:49:11.175 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-05T02:49:11.175 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:11.175 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:11.175 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:11.176 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:11.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-04-05T02:49:12.017 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:41 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:12.180 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:17.2.7-815.g93c790e1.el9 ceph 6.5 k 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.1 M 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:17.2.7-815.g93c790e1.el9 ceph 1.4 M 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 240 k 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:17.2.7-815.g93c790e1.el9 ceph 3.9 M 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 15 M 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-05T02:49:12.183 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-05T02:49:12.184 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-05T02:49:12.185 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-05T02:49:12.186 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:12.186 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:12.186 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-05T02:49:12.186 INFO:teuthology.orchestra.run.smithi203.stdout:Install 33 Packages 2024-04-05T02:49:12.186 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:12.187 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 27 M 2024-04-05T02:49:12.187 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 96 M 2024-04-05T02:49:12.187 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:12.317 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-05T02:49:12.489 INFO:teuthology.orchestra.run.smithi203.stdout:(1/33): ceph-17.2.7-815.g93c790e1.el9.x86_64.rp 31 kB/s | 6.5 kB 00:00 2024-04-05T02:49:12.606 INFO:teuthology.orchestra.run.smithi203.stdout:(2/33): ceph-mgr-17.2.7-815.g93c790e1.el9.x86_6 4.3 MB/s | 1.4 MB 00:00 2024-04-05T02:49:13.065 INFO:teuthology.orchestra.run.smithi203.stdout:(3/33): ceph-mds-17.2.7-815.g93c790e1.el9.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-04-05T02:49:13.359 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-05T02:49:13.380 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 12/12 2024-04-05T02:49:13.541 INFO:teuthology.orchestra.run.smithi203.stdout:(4/33): ceph-osd-17.2.7-815.g93c790e1.el9.x86_6 16 MB/s | 15 MB 00:00 2024-04-05T02:49:13.641 INFO:teuthology.orchestra.run.smithi203.stdout:(5/33): ceph-mon-17.2.7-815.g93c790e1.el9.x86_6 3.4 MB/s | 3.9 MB 00:01 2024-04-05T02:49:13.683 INFO:teuthology.orchestra.run.smithi203.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 785 kB/s | 32 kB 00:00 2024-04-05T02:49:13.708 INFO:teuthology.orchestra.run.smithi203.stdout:(7/33): ceph-mgr-modules-core-17.2.7-815.g93c79 373 kB/s | 240 kB 00:00 2024-04-05T02:49:13.733 INFO:teuthology.orchestra.run.smithi203.stdout:(8/33): python3-mako-1.1.4-6.el9.noarch.rpm 750 kB/s | 144 kB 00:00 2024-04-05T02:49:13.758 INFO:teuthology.orchestra.run.smithi203.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 628 kB/s | 47 kB 00:00 2024-04-05T02:49:13.784 INFO:teuthology.orchestra.run.smithi203.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 512 kB/s | 39 kB 00:00 2024-04-05T02:49:13.809 INFO:teuthology.orchestra.run.smithi203.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-05T02:49:13.859 INFO:teuthology.orchestra.run.smithi203.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-04-05T02:49:13.884 INFO:teuthology.orchestra.run.smithi203.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-05T02:49:13.910 INFO:teuthology.orchestra.run.smithi203.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 124 kB 00:00 2024-04-05T02:49:13.943 INFO:teuthology.orchestra.run.smithi203.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-04-05T02:49:14.094 INFO:teuthology.orchestra.run.smithi203.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-04-05T02:49:14.244 INFO:teuthology.orchestra.run.smithi203.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 130 kB/s | 43 kB 00:00 2024-04-05T02:49:14.319 INFO:teuthology.orchestra.run.smithi203.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 144 kB/s | 11 kB 00:00 2024-04-05T02:49:14.361 INFO:teuthology.orchestra.run.smithi203.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 411 kB/s | 172 kB 00:00 2024-04-05T02:49:14.403 INFO:teuthology.orchestra.run.smithi203.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-04-05T02:49:14.437 INFO:teuthology.orchestra.run.smithi203.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-04-05T02:49:14.487 INFO:teuthology.orchestra.run.smithi203.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-04-05T02:49:14.512 INFO:teuthology.orchestra.run.smithi203.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-04-05T02:49:14.546 INFO:teuthology.orchestra.run.smithi203.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 794 kB/s | 358 kB 00:00 2024-04-05T02:49:14.590 INFO:teuthology.orchestra.run.smithi203.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-04-05T02:49:14.613 INFO:teuthology.orchestra.run.smithi203.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 783 kB/s | 79 kB 00:00 2024-04-05T02:49:14.639 INFO:teuthology.orchestra.run.smithi203.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-05T02:49:14.664 INFO:teuthology.orchestra.run.smithi203.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-04-05T02:49:14.706 INFO:teuthology.orchestra.run.smithi203.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 975 kB/s | 90 kB 00:00 2024-04-05T02:49:14.731 INFO:teuthology.orchestra.run.smithi203.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 387 kB/s | 36 kB 00:00 2024-04-05T02:49:14.798 INFO:teuthology.orchestra.run.smithi203.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-04-05T02:49:14.840 INFO:teuthology.orchestra.run.smithi203.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-05T02:49:14.865 INFO:teuthology.orchestra.run.smithi203.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-04-05T02:49:14.871 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:14.871 INFO:teuthology.orchestra.run.smithi203.stdout:Total 10 MB/s | 27 MB 00:02 2024-04-05T02:49:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:14.921 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:14.921 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:15.264 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:15.264 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:15.968 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 12/12 2024-04-05T02:49:15.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 1/12 2024-04-05T02:49:15.968 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-05T02:49:15.969 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-05T02:49:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-05T02:49:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-05T02:49:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-05T02:49:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:16.471 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:16.502 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:16.577 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-05T02:49:16.617 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-05T02:49:16.650 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-05T02:49:16.741 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-05T02:49:16.812 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-05T02:49:16.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-04-05T02:49:16.853 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-05T02:49:16.925 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-05T02:49:16.979 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-05T02:49:17.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-05T02:49:17.066 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-05T02:49:17.122 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-05T02:49:17.312 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-05T02:49:17.364 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-05T02:49:17.403 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-05T02:49:17.413 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:14 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:17.438 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-05T02:49:17.468 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-05T02:49:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-05T02:49:17.575 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:17.576 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:17.576 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 47 k 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-05T02:49:17.577 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-04-05T02:49:17.578 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:17.579 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-05T02:49:17.579 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-04-05T02:49:17.579 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-04-05T02:49:17.580 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:17.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-05T02:49:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-05T02:49:17.687 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-05T02:49:17.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-05T02:49:17.811 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-05T02:49:17.844 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-05T02:49:17.897 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-04-05T02:49:17.916 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-05T02:49:17.922 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-04-05T02:49:17.981 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-04-05T02:49:18.006 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-04-05T02:49:18.023 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-04-05T02:49:18.048 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-04-05T02:49:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-05T02:49:18.073 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-17.2.7-815.g93c790e1.el9. 157 kB/s | 47 kB 00:00 2024-04-05T02:49:18.157 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-04-05T02:49:18.169 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-05T02:49:18.182 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-04-05T02:49:18.283 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 970 kB/s | 203 kB 00:00 2024-04-05T02:49:18.308 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-04-05T02:49:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.n 28/33 2024-04-05T02:49:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 29/33 2024-04-05T02:49:18.366 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 29/33 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-05T02:49:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:18.400 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-04-05T02:49:18.402 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:18.402 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-04-05T02:49:18.403 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:18.416 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:18.416 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:18.505 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:18.506 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:19.016 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:19.181 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-05T02:49:19.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-05T02:49:19.306 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-05T02:49:19.344 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-05T02:49:19.385 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-05T02:49:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-05T02:49:19.588 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-05T02:49:19.661 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-05T02:49:19.701 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 30/33 2024-04-05T02:49:19.727 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 30/33 2024-04-05T02:49:19.727 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:19.727 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-05T02:49:19.727 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-05T02:49:19.728 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-05T02:49:19.728 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:20.102 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 31/33 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 31/33 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-05T02:49:20.125 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:20.328 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 32/33 2024-04-05T02:49:20.350 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 32/33 2024-04-05T02:49:20.351 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:20.351 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-05T02:49:20.351 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-05T02:49:20.351 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-05T02:49:20.351 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:20.389 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-05T02:49:20.416 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:17.2.7-815.g93c790e1.el9.x86_64 33/33 2024-04-05T02:49:20.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-05T02:49:20.466 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-05T02:49:20.513 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-05T02:49:20.536 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 13/13 2024-04-05T02:49:21.289 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 13/13 2024-04-05T02:49:21.289 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 1/13 2024-04-05T02:49:21.289 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-05T02:49:21.289 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-05T02:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-05T02:49:21.633 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:21.634 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:21.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:17.2.7-815.g93c790e1.el9.x86_64 33/33 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:17.2.7-815.g93c790e1.el9.x86_64 1/33 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 2/33 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 3/33 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 4/33 2024-04-05T02:49:22.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 5/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.n 6/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-05T02:49:22.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-05T02:49:22.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-05T02:49:22.439 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 123 k 2024-04-05T02:49:22.592 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-04-05T02:49:22.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:22.594 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-04-05T02:49:22.594 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-04-05T02:49:22.594 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:22.934 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-04-05T02:49:23.045 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-05T02:49:23.046 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-05T02:49:23.047 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:23.048 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:23.101 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-04-05T02:49:23.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-04-05T02:49:23.585 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 125 kB/s | 58 kB 00:00 2024-04-05T02:49:23.635 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): ceph-mgr-cephadm-17.2.7-815.g93c790e1.el 148 kB/s | 123 kB 00:00 2024-04-05T02:49:23.868 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 588 kB/s | 548 kB 00:00 2024-04-05T02:49:23.975 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:52 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:24.069 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 105 kB/s | 51 kB 00:00 2024-04-05T02:49:24.071 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:24.071 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2024-04-05T02:49:24.072 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:24.085 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:24.097 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:24.097 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:24.124 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:24.125 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:49:24.125 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:24.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-04-05T02:49:24.187 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:24.191 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:24.486 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:24.650 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-05T02:49:24.695 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-05T02:49:24.746 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:53 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:24.896 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:24.896 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 82 k 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 82 k 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 372 k 2024-04-05T02:49:24.897 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:25.212 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-05T02:49:25.298 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-05T02:49:25.353 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-05T02:49:25.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 6/6 2024-04-05T02:49:25.712 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-17.2.7-815.g93c790e1.el9.noarch.rpm 101 kB/s | 82 kB 00:00 2024-04-05T02:49:25.712 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:25.713 INFO:teuthology.orchestra.run.smithi203.stdout:Total 100 kB/s | 82 kB 00:00 2024-04-05T02:49:25.713 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:25.718 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:25.718 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:25.720 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:25.720 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:25.795 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:26.125 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 6/6 2024-04-05T02:49:26.126 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/6 2024-04-05T02:49:26.126 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-05T02:49:26.126 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-05T02:49:26.126 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-05T02:49:26.126 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-05T02:49:26.261 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:49:26.346 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:49:26.413 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:26.414 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:26.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-04-05T02:49:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:49:27.210 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/1 2024-04-05T02:49:27.211 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:27.211 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:27.211 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:27.211 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:27.211 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:27.407 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:27.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 812 k 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-04-05T02:49:27.408 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:27.409 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 891 k 2024-04-05T02:49:27.409 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.6 M 2024-04-05T02:49:27.409 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:27.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-04-05T02:49:27.537 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 616 kB/s | 79 kB 00:00 2024-04-05T02:49:28.006 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:57 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:28.159 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:28.159 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:28.159 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-815.g93c790e1.el9 ceph 137 k 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 137 k 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 379 k 2024-04-05T02:49:28.160 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:28.187 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-17.2.7-815.g93c790e1.el9.x86_6 1.0 MB/s | 812 kB 00:00 2024-04-05T02:49:28.188 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:28.189 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.1 MB/s | 891 kB 00:00 2024-04-05T02:49:28.189 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:28.210 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:28.210 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:28.283 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:28.284 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:28.475 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:28.479 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-17.2.7-815.g93c790e 431 kB/s | 137 kB 00:00 2024-04-05T02:49:28.480 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:28.480 INFO:teuthology.orchestra.run.smithi203.stdout:Total 429 kB/s | 137 kB 00:00 2024-04-05T02:49:28.480 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:28.488 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:28.488 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:28.540 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:28.540 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:28.607 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-05T02:49:28.668 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2/2 2024-04-05T02:49:28.677 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:28.755 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:28.781 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:28.781 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:28.781 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-05T02:49:28.781 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:29.297 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2/2 2024-04-05T02:49:29.298 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/2 2024-04-05T02:49:29.568 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-05T02:49:29.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:29.569 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:29.569 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-05T02:49:29.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:29.569 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.e 1/1 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:29.799 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:29.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-04-05T02:49:30.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-04-05T02:49:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:27 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:30.563 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:17.2.7-815.g93c790e1.el9 ceph 126 k 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:30.564 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 126 k 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 448 k 2024-04-05T02:49:30.565 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:30.604 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:59 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:30.703 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:30.757 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:30.758 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:49:30.758 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:30.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-04-05T02:49:30.882 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-17.2.7-815.g93c790e1.el9.x86_64. 398 kB/s | 126 kB 00:00 2024-04-05T02:49:30.883 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:30.883 INFO:teuthology.orchestra.run.smithi153.stdout:Total 396 kB/s | 126 kB 00:00 2024-04-05T02:49:30.883 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:30.889 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:30.889 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:30.946 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:30.946 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:31.102 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:31.197 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:31.386 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:00 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:31.538 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:31.539 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:31.539 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:31.539 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:31.539 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 1.7 M 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 24 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 15 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout:Install 7 Packages 2024-04-05T02:49:31.540 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:31.541 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-04-05T02:49:31.541 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 8.1 M 2024-04-05T02:49:31.541 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:31.638 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:31.901 INFO:teuthology.orchestra.run.smithi203.stdout:(1/7): ceph-prometheus-alerts-17.2.7-815.g93c79 51 kB/s | 15 kB 00:00 2024-04-05T02:49:31.951 INFO:teuthology.orchestra.run.smithi203.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 181 kB/s | 9.0 kB 00:00 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:31.953 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:31.976 INFO:teuthology.orchestra.run.smithi203.stdout:(3/7): ceph-grafana-dashboards-17.2.7-815.g93c7 66 kB/s | 24 kB 00:00 2024-04-05T02:49:32.002 INFO:teuthology.orchestra.run.smithi203.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 811 kB/s | 41 kB 00:00 2024-04-05T02:49:32.094 INFO:teuthology.orchestra.run.smithi203.stdout:(5/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-05T02:49:32.111 INFO:teuthology.orchestra.run.smithi203.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 229 kB/s | 31 kB 00:00 2024-04-05T02:49:32.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-04-05T02:49:32.767 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:32.870 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:32.924 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:32.926 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:32.926 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:32.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-04-05T02:49:33.228 INFO:teuthology.orchestra.run.smithi203.stdout:(7/7): ceph-mgr-dashboard-17.2.7-815.g93c790e1. 1.1 MB/s | 1.7 MB 00:01 2024-04-05T02:49:33.230 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:33.230 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-04-05T02:49:33.230 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:33.240 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:33.240 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:33.308 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:33.308 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:33.554 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:30 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-05T02:49:33.705 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:17.2.7-815.g93c790e1.el9 ceph 25 k 2024-04-05T02:49:33.706 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 110 k 2024-04-05T02:49:33.707 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-05T02:49:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-05T02:49:33.800 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-05T02:49:33.820 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.no 5/7 2024-04-05T02:49:34.037 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-17.2.7-815.g93c790e1.el9.x86_64 75 kB/s | 25 kB 00:00 2024-04-05T02:49:34.038 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:34.038 INFO:teuthology.orchestra.run.smithi153.stdout:Total 75 kB/s | 25 kB 00:00 2024-04-05T02:49:34.038 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:34.041 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:34.041 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:34.043 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:34.043 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:34.053 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.n 6/7 2024-04-05T02:49:34.082 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 7/7 2024-04-05T02:49:34.127 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:34.236 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:34.585 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:34.622 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 7/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.n 1/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 2/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.no 3/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-05T02:49:34.623 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:34.827 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:34.932 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:34.933 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:34.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-04-05T02:49:35.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-05T02:49:35.558 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:32 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:35.657 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:35.710 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:35.711 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:35.711 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:04 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:35.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-04-05T02:49:35.875 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 7.4 M 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-05T02:49:35.876 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2024-04-05T02:49:35.877 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-04-05T02:49:35.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:35.878 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-04-05T02:49:35.878 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-04-05T02:49:35.878 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:36.341 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:33 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:36.410 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 229 kB/s | 32 kB 00:00 2024-04-05T02:49:36.439 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:36.444 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-04-05T02:49:36.478 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-05T02:49:36.492 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:36.493 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:36.493 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:36.536 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-05T02:49:36.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-04-05T02:49:36.736 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-05T02:49:36.795 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-04-05T02:49:36.820 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-04-05T02:49:37.123 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:37.146 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-05T02:49:37.171 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-05T02:49:37.221 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:37.273 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:37.273 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:37.273 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:37.279 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-05T02:49:37.330 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-04-05T02:49:37.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-04-05T02:49:37.901 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:34 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:38.001 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:38.054 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:38.055 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:38.056 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:38.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-04-05T02:49:38.678 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:35 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:38.777 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:38.829 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:38.830 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:38.830 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:38.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-04-05T02:49:39.321 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 2.4 MB/s | 7.4 MB 00:03 2024-04-05T02:49:39.324 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:39.324 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 37 MB 00:03 2024-04-05T02:49:39.367 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:39.391 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:39.392 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:39.453 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:39.552 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:49:39.604 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:39.605 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-04-05T02:49:39.605 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:39.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-04-05T02:49:39.826 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:39.826 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:40.231 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:40.381 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:40.382 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:40.382 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-05T02:49:40.382 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:40.382 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 82 k 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 222 k 2024-04-05T02:49:40.383 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:40.564 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:40.636 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-05T02:49:40.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-05T02:49:40.757 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-17.2.7-815.g93c790e1.el9.x86_64.rpm 219 kB/s | 82 kB 00:00 2024-04-05T02:49:40.758 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:40.758 INFO:teuthology.orchestra.run.smithi153.stdout:Total 218 kB/s | 82 kB 00:00 2024-04-05T02:49:40.758 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:40.765 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:40.765 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:40.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-05T02:49:40.819 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:40.820 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:40.821 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-05T02:49:40.971 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:41.000 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-05T02:49:41.029 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-05T02:49:41.079 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:41.148 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-05T02:49:41.444 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:41.729 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:41.730 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:41.730 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:41.730 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:41.730 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:41.730 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:41.812 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-05T02:49:41.925 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-05T02:49:41.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-04-05T02:49:42.570 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:39 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:42.726 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.9 M 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:42.727 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:42.728 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:42.728 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.9 M 2024-04-05T02:49:42.728 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-04-05T02:49:42.728 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:43.160 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-17.2.7-815.g93c790e1.el9.x86_64.rpm 6.8 MB/s | 2.9 MB 00:00 2024-04-05T02:49:43.160 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:43.161 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.8 MB/s | 2.9 MB 00:00 2024-04-05T02:49:43.161 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:43.170 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:43.170 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:43.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-05T02:49:43.242 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:43.243 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:43.728 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:49:43.728 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-05T02:49:43.728 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-05T02:49:43.728 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-05T02:49:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:44.208 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-05T02:49:44.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 12/12 2024-04-05T02:49:44.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:44.824 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:44.825 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:44.825 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:44.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:44.825 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:45.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-04-05T02:49:45.616 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:42 ago on Fri 05 Apr 2024 02:48:03 AM UTC. 2024-04-05T02:49:45.768 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 165 k 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-04-05T02:49:45.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:45.770 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 165 k 2024-04-05T02:49:45.770 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 477 k 2024-04-05T02:49:45.770 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-05T02:49:46.121 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-17.2.7-815.g93c790e1.el9.x86_64.rpm 470 kB/s | 165 kB 00:00 2024-04-05T02:49:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:Total 468 kB/s | 165 kB 00:00 2024-04-05T02:49:46.122 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-05T02:49:46.129 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-05T02:49:46.130 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-05T02:49:46.185 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-05T02:49:46.185 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-05T02:49:46.332 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-05T02:49:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 12/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790 1/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-05T02:49:46.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-05T02:49:46.840 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-05T02:49:46.840 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-05T02:49:46.840 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-05T02:49:46.840 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-05T02:49:46.868 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:47.135 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:49:47.135 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:47.136 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-05T02:49:47.136 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:49:47.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:49:47.136 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-05T02:49:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-05T02:49:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:47.248 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:47.248 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-05T02:49:47.249 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:47.250 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:47.382 DEBUG:teuthology.parallel:result is None 2024-04-05T02:49:47.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-04-05T02:49:48.197 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:17 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:48.349 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:48.350 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:48.350 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 47 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-05T02:49:48.351 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:48.352 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-04-05T02:49:48.353 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-04-05T02:49:48.353 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:48.690 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 24 kB 00:00 2024-04-05T02:49:48.716 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2024-04-05T02:49:48.766 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-05T02:49:48.791 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-05T02:49:48.817 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-05T02:49:48.842 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-04-05T02:49:49.059 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): ceph-mgr-rook-17.2.7-815.g93c790e1.el9. 94 kB/s | 47 kB 00:00 2024-04-05T02:49:49.101 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-04-05T02:49:49.118 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-04-05T02:49:49.276 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 438 kB/s | 59 kB 00:00 2024-04-05T02:49:49.393 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-04-05T02:49:49.427 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-04-05T02:49:49.494 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 466 kB/s | 203 kB 00:00 2024-04-05T02:49:49.498 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:49.498 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-04-05T02:49:49.499 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:49.512 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:49.512 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:49.605 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:49.605 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:50.285 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-05T02:49:50.337 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-05T02:49:50.401 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-05T02:49:50.438 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-05T02:49:50.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-05T02:49:50.577 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-05T02:49:50.694 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-05T02:49:50.750 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-05T02:49:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-05T02:49:51.539 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-05T02:49:51.573 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-05T02:49:51.621 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-05T02:49:51.643 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 13/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 13/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 1/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-05T02:49:52.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-05T02:49:52.432 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-05T02:49:52.432 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-05T02:49:52.432 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-05T02:49:52.432 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-05T02:49:52.432 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-05T02:49:52.789 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:52.790 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:53.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-04-05T02:49:53.591 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:22 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:53.742 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:53.742 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:17.2.7-815.g93c790e1.el9 ceph-noarch 123 k 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-05T02:49:53.743 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-04-05T02:49:53.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:53.744 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-04-05T02:49:53.744 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-04-05T02:49:53.744 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:54.033 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-05T02:49:54.108 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-05T02:49:54.317 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-04-05T02:49:54.410 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 556 kB/s | 51 kB 00:00 2024-04-05T02:49:54.460 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-05T02:49:54.693 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-mgr-cephadm-17.2.7-815.g93c790e1.el 144 kB/s | 123 kB 00:00 2024-04-05T02:49:54.695 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:54.695 INFO:teuthology.orchestra.run.smithi203.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-04-05T02:49:54.709 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:54.719 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:54.719 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:54.808 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:54.809 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:55.104 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:55.278 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-05T02:49:55.328 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-05T02:49:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-05T02:49:55.949 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-05T02:49:56.003 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-05T02:49:56.032 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 6/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 6/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 1/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-05T02:49:56.829 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-05T02:49:57.147 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:17.2.7-815.g93c790e1.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:57.148 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:49:57.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-04-05T02:49:57.988 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:26 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:49:58.138 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 812 k 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-05T02:49:58.139 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:49:58.140 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 891 k 2024-04-05T02:49:58.140 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.6 M 2024-04-05T02:49:58.140 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:49:58.272 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 599 kB/s | 79 kB 00:00 2024-04-05T02:49:58.922 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-17.2.7-815.g93c790e1.el9.x86_6 1.0 MB/s | 812 kB 00:00 2024-04-05T02:49:58.924 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:49:58.924 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.1 MB/s | 891 kB 00:00 2024-04-05T02:49:58.924 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:49:58.946 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:49:58.946 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:49:59.020 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:49:59.020 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:49:59.213 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:49:59.343 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-05T02:49:59.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2/2 2024-04-05T02:50:00.091 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2/2 2024-04-05T02:50:00.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/2 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:00.387 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:00.646 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-04-05T02:50:01.232 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:30 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:01.387 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:01.388 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:01.388 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-05T02:50:01.388 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:17.2.7-815.g93c790e1.el9 ceph 126 k 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 126 k 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 448 k 2024-04-05T02:50:01.389 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:50:01.770 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-17.2.7-815.g93c790e1.el9.x86_64. 332 kB/s | 126 kB 00:00 2024-04-05T02:50:01.771 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:50:01.771 INFO:teuthology.orchestra.run.smithi203.stdout:Total 330 kB/s | 126 kB 00:00 2024-04-05T02:50:01.771 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:50:01.776 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:50:01.777 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:50:01.833 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:50:01.834 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:50:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:50:02.085 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:02.575 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:02.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:02.854 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:02.855 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:02.855 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:50:02.855 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:02.855 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:03.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-04-05T02:50:03.685 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:32 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:03.788 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:03.841 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:03.842 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:03.842 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:03.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-04-05T02:50:04.470 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:33 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:04.621 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:17.2.7-815.g93c790e1.el9 ceph 25 k 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:50:04.622 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:04.623 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 25 k 2024-04-05T02:50:04.623 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 110 k 2024-04-05T02:50:04.623 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:50:04.859 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-17.2.7-815.g93c790e1.el9.x86_64 106 kB/s | 25 kB 00:00 2024-04-05T02:50:04.860 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:50:04.860 INFO:teuthology.orchestra.run.smithi203.stdout:Total 105 kB/s | 25 kB 00:00 2024-04-05T02:50:04.860 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:50:04.863 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:50:04.863 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:50:04.865 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:50:04.865 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:50:04.951 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:50:05.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:05.465 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:05.735 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:05.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-04-05T02:50:06.470 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:35 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:06.570 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:06.623 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:06.624 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:06.624 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:06.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-04-05T02:50:07.262 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:07.361 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:07.413 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:07.414 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:07.414 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:07.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-04-05T02:50:08.050 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:37 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:08.149 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:08.201 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:08.202 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:08.202 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:08.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-04-05T02:50:08.829 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:37 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:08.927 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:08.980 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:08.981 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:08.981 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:09.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-04-05T02:50:09.614 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:09.712 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:09.764 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:09.766 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:09.766 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:09.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-04-05T02:50:10.395 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:39 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:10.493 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:17.2.7-815.g93c790e1.el9.x86_64 is already installed. 2024-04-05T02:50:10.545 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:10.546 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-05T02:50:10.546 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:10.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-04-05T02:50:11.172 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:40 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:11.322 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:11.323 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:11.323 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-05T02:50:11.323 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:11.323 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:17.2.7-815.g93c790e1.el9 ceph 82 k 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 82 k 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 222 k 2024-04-05T02:50:11.324 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:50:12.029 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-17.2.7-815.g93c790e1.el9.x86_64.rpm 116 kB/s | 82 kB 00:00 2024-04-05T02:50:12.029 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:50:12.029 INFO:teuthology.orchestra.run.smithi203.stdout:Total 116 kB/s | 82 kB 00:00 2024-04-05T02:50:12.030 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:50:12.037 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:50:12.037 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:50:12.095 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:50:12.095 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:50:12.243 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:50:12.343 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:12.757 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:13.076 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:13.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-04-05T02:50:13.884 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:42 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:14.043 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:14.043 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:17.2.7-815.g93c790e1.el9 ceph 2.9 M 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.9 M 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 12 M 2024-04-05T02:50:14.044 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:50:17.408 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-17.2.7-815.g93c790e1.el9.x86_64.rpm 894 kB/s | 2.9 MB 00:03 2024-04-05T02:50:17.409 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:50:17.409 INFO:teuthology.orchestra.run.smithi203.stdout:Total 893 kB/s | 2.9 MB 00:03 2024-04-05T02:50:17.409 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:50:17.420 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:50:17.420 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:50:17.496 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:50:17.496 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:50:17.900 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:50:17.959 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:17.982 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-05T02:50:17.983 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-05T02:50:17.983 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-05T02:50:17.983 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-05T02:50:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:19.117 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:19.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-04-05T02:50:19.954 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:48 ago on Fri 05 Apr 2024 02:48:31 AM UTC. 2024-04-05T02:50:20.109 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-05T02:50:20.110 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:20.110 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-05T02:50:20.110 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:20.110 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:17.2.7-815.g93c790e1.el9 ceph 165 k 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:20.111 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 165 k 2024-04-05T02:50:20.112 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 477 k 2024-04-05T02:50:20.112 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-05T02:50:20.838 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-17.2.7-815.g93c790e1.el9.x86_64.rpm 227 kB/s | 165 kB 00:00 2024-04-05T02:50:20.838 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:50:20.839 INFO:teuthology.orchestra.run.smithi203.stdout:Total 226 kB/s | 165 kB 00:00 2024-04-05T02:50:20.839 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-05T02:50:20.847 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-05T02:50:20.847 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-05T02:50:20.905 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-05T02:50:20.905 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-05T02:50:21.060 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-05T02:50:21.170 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:21.603 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:21.907 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 1/1 2024-04-05T02:50:21.908 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:21.908 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-05T02:50:21.908 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:17.2.7-815.g93c790e1.el9.x86_64 2024-04-05T02:50:21.908 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:50:21.908 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-05T02:50:22.133 DEBUG:teuthology.parallel:result is None 2024-04-05T02:50:22.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:50:22.133 INFO:teuthology.packaging:ref: None 2024-04-05T02:50:22.133 INFO:teuthology.packaging:tag: None 2024-04-05T02:50:22.133 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:50:22.133 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:50:22.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-05T02:50:22.357 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T02:50:22.383 INFO:teuthology.orchestra.run.smithi153.stdout:17.2.7-815.g93c790e1.el9 2024-04-05T02:50:22.384 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-815.g93c790e1.el9 2024-04-05T02:50:22.384 INFO:teuthology.task.install:The correct ceph version 17.2.7-815.g93c790e1 is installed. 2024-04-05T02:50:22.386 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T02:50:22.386 INFO:teuthology.packaging:ref: None 2024-04-05T02:50:22.386 INFO:teuthology.packaging:tag: None 2024-04-05T02:50:22.386 INFO:teuthology.packaging:branch: quincy 2024-04-05T02:50:22.386 INFO:teuthology.packaging:sha1: 0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T02:50:22.387 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-04-05T02:50:22.592 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T02:50:22.613 INFO:teuthology.orchestra.run.smithi203.stdout:17.2.7-815.g93c790e1.el9 2024-04-05T02:50:22.614 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-815.g93c790e1.el9 2024-04-05T02:50:22.614 INFO:teuthology.task.install:The correct ceph version 17.2.7-815.g93c790e1 is installed. 2024-04-05T02:50:22.615 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-05T02:50:22.616 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:50:22.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T02:50:22.650 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:50:22.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T02:50:22.682 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-05T02:50:22.683 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:50:22.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T02:50:22.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T02:50:22.789 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:50:22.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T02:50:22.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T02:50:22.887 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-05T02:50:22.888 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:50:22.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T02:50:22.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T02:50:22.984 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:50:22.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T02:50:23.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T02:50:23.078 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-05T02:50:23.079 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:50:23.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T02:50:23.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T02:50:23.174 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:50:23.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T02:50:23.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T02:50:23.268 INFO:teuthology.run_tasks:Running task print... 2024-04-05T02:50:23.278 INFO:teuthology.task.print:**** done install task... 2024-04-05T02:50:23.278 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-05T02:50:23.378 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '0028525c15e19309c20289c609e4fb9d0b12dd06'} 2024-04-05T02:50:23.378 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-04-05T02:50:23.379 INFO:tasks.cephadm:Cluster fsid is 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:50:23.379 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-05T02:50:23.379 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-05T02:50:23.379 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi153': '172.21.15.153', 'mon.smithi203': '172.21.15.203'} 2024-04-05T02:50:23.379 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-05T02:50:23.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-05T02:50:23.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-04-05T02:50:23.434 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-05T02:50:23.434 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.658 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 5 02:50 /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.659 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.847 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 5 02:50 /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.847 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.862 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T02:50:23.890 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-04-05T02:50:23.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-05T02:50:23.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-04-05T02:50:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-05T02:50:24.774 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-05T02:51:39.515 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)", 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48", 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030" 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-04-05T02:51:39.516 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)", 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48", 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030" 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-05T02:51:43.198 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T02:51:43.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-05T02:51:43.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-04-05T02:51:43.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-05T02:51:43.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-04-05T02:51:43.376 INFO:tasks.cephadm:Writing seed config... 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-05T02:51:43.378 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-05T02:51:43.379 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-05T02:51:43.380 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-05T02:51:43.381 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-05T02:51:43.381 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-05T02:51:43.381 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-05T02:51:43.381 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-05T02:51:43.381 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-05T02:51:43.382 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:51:43.382 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-05T02:51:43.396 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-05T02:51:43.397 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service 2024-04-05T02:51:43.439 INFO:tasks.cephadm:Bootstrapping... 2024-04-05T02:51:43.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.153 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-05T02:51:43.643 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-05T02:51:43.643 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '3fa69b9e-f2f7-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.153', '--skip-admin-label'] 2024-04-05T02:51:43.670 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T02:51:43.670 INFO:teuthology.orchestra.run.smithi153.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-05T02:51:43.670 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying podman|docker is present... 2024-04-05T02:51:43.693 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T02:51:43.693 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying lvm2 is present... 2024-04-05T02:51:43.693 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying time synchronization is in place... 2024-04-05T02:51:43.701 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T02:51:43.701 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T02:51:43.707 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T02:51:43.707 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:51:43.714 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-04-05T02:51:43.720 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-04-05T02:51:43.720 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-04-05T02:51:43.720 INFO:teuthology.orchestra.run.smithi153.stdout:Repeating the final host check... 2024-04-05T02:51:43.744 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T02:51:43.744 INFO:teuthology.orchestra.run.smithi153.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-05T02:51:43.744 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl is present 2024-04-05T02:51:43.744 INFO:teuthology.orchestra.run.smithi153.stdout:lvcreate is present 2024-04-05T02:51:43.751 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T02:51:43.751 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T02:51:43.757 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T02:51:43.757 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:51:43.763 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-04-05T02:51:43.771 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-04-05T02:51:43.771 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-04-05T02:51:43.771 INFO:teuthology.orchestra.run.smithi153.stdout:Host looks OK 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Cluster fsid: 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Acquiring lock 139963451270240 on /run/cephadm/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.lock 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Lock 139963451270240 acquired on /run/cephadm/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.lock 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 3300 ... 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 6789 ... 2024-04-05T02:51:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Base mon IP(s) is [172.21.15.153:3300, 172.21.15.153:6789], mon addrv is [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-04-05T02:51:43.774 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.153 metric 100 2024-04-05T02:51:43.774 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.153 metric 100 2024-04-05T02:51:43.777 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-05T02:51:43.777 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-05T02:51:43.777 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T02:51:43.780 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-04-05T02:51:43.781 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-04-05T02:51:43.781 INFO:teuthology.orchestra.run.smithi153.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-05T02:51:43.781 INFO:teuthology.orchestra.run.smithi153.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-05T02:51:43.782 INFO:teuthology.orchestra.run.smithi153.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-05T02:51:44.230 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Getting image source signatures 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:7c8e66990dd5fdc46cc6d5e1ff7927cb8fbc9e77ba63bb5152631196a3d52311 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:77070339c81bcb5c53bbbf3f31a47f7187e5393cb0e096452852a31192d97570 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying config sha256:a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48 2024-04-05T02:51:44.231 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-05T02:51:45.122 INFO:teuthology.orchestra.run.smithi153.stdout:ceph: stdout ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable) 2024-04-05T02:51:45.122 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph version: ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable) 2024-04-05T02:51:45.122 INFO:teuthology.orchestra.run.smithi153.stdout:Extracting ceph user uid/gid from container image... 2024-04-05T02:51:46.023 INFO:teuthology.orchestra.run.smithi153.stdout:stat: stdout 167 167 2024-04-05T02:51:46.024 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial keys... 2024-04-05T02:51:46.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBCZw9mQQdnFRAAZvC2ZRlpvy2veSqwWNa3Qw== 2024-04-05T02:51:47.761 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBDZw9m8rcjDhAAiZcR0WOu32rm4ezRvEmCWg== 2024-04-05T02:51:48.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBEZw9mZyOHBRAAAoFTrh2HbkHYcCa7QGMA7Q== 2024-04-05T02:51:48.650 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial monmap... 2024-04-05T02:51:49.592 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T02:51:49.592 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool for smithi153 [v2:172.21.15.153:3300,v1:172.21.15.153:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = octopus 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: set fsid to 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T02:51:49.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:51:49.594 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mon... 2024-04-05T02:51:50.473 INFO:teuthology.orchestra.run.smithi153.stdout:create mon.smithi153 on 2024-04-05T02:51:50.675 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-05T02:51:50.814 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-05T02:51:50.960 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target. 2024-04-05T02:51:50.960 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target. 2024-04-05T02:51:51.135 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153 2024-04-05T02:51:51.135 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service: Unit ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service not loaded. 2024-04-05T02:51:51.304 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target.wants/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service → /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@.service. 2024-04-05T02:51:51.839 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T02:51:51.839 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-04-05T02:51:51.846 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T02:51:51.846 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:51:51.846 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-04-05T02:51:51.846 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T02:51:51.846 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon to start... 2024-04-05T02:51:51.847 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon... 2024-04-05T02:51:52.947 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:52 smithi153 ceph-mon[29002]: from='client.? 172.21.15.153:0/463243340' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-05T02:51:52.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout cluster: 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout id: 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout services: 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi153 (age 0.636447s) 2024-04-05T02:51:52.956 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout data: 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pgs: 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.333+0000 7fa247ef3700 1 Processor -- start 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.333+0000 7fa247ef3700 1 -- start start 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.333+0000 7fa247ef3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.333+0000 7fa247ef3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2401021b0 con 0x7fa240101860 2024-04-05T02:51:52.957 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa245c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa245c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52130/0 (socket says 172.21.15.153:52130) 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa245c8f700 1 -- 172.21.15.153:0/2364083329 learned_addr learned my addr 172.21.15.153:0/2364083329 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa245c8f700 1 -- 172.21.15.153:0/2364083329 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2401022f0 con 0x7fa240101860 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa245c8f700 1 --2- 172.21.15.153:0/2364083329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00bb80 tx=0x7fa23c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d303a0c9dee20 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa244c8d700 1 -- 172.21.15.153:0/2364083329 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c00dd90 con 0x7fa240101860 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa244c8d700 1 -- 172.21.15.153:0/2364083329 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c010470 con 0x7fa240101860 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.334+0000 7fa244c8d700 1 -- 172.21.15.153:0/2364083329 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c010760 con 0x7fa240101860 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 -- 172.21.15.153:0/2364083329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 msgr2=0x7fa240101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 --2- 172.21.15.153:0/2364083329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00bb80 tx=0x7fa23c00b4f0 comp rx=0 tx=0).stop 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 -- 172.21.15.153:0/2364083329 shutdown_connections 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 --2- 172.21.15.153:0/2364083329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa240101c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 -- 172.21.15.153:0/2364083329 >> 172.21.15.153:0/2364083329 conn(0x7fa2400fcfa0 msgr2=0x7fa2400ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:52.958 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 -- 172.21.15.153:0/2364083329 shutdown_connections 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.335+0000 7fa247ef3700 1 -- 172.21.15.153:0/2364083329 wait complete. 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa247ef3700 1 Processor -- start 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa247ef3700 1 -- start start 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa247ef3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa247ef3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2401135e0 con 0x7fa240101860 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa245c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa245c8f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52144/0 (socket says 172.21.15.153:52144) 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa245c8f700 1 -- 172.21.15.153:0/463243340 learned_addr learned my addr 172.21.15.153:0/463243340 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa245c8f700 1 -- 172.21.15.153:0/463243340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c00b790 con 0x7fa240101860 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.336+0000 7fa245c8f700 1 --2- 172.21.15.153:0/463243340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000c00 tx=0x7fa23c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c0189e0 con 0x7fa240101860 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c018b40 con 0x7fa240101860 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2401137e0 con 0x7fa240101860 2024-04-05T02:51:52.959 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa240114350 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa23c018e30 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa23c022450 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.337+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa23c014070 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.338+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24005f5f0 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.340+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa23c027070 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.424+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa240114650 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.424+0000 7fa2367fc700 1 -- 172.21.15.153:0/463243340 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fa23c0228e0 con 0x7fa240101860 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 msgr2=0x7fa2401130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 --2- 172.21.15.153:0/463243340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000c00 tx=0x7fa23c01a000 comp rx=0 tx=0).stop 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 shutdown_connections 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 --2- 172.21.15.153:0/463243340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa240101860 0x7fa2401130a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 >> 172.21.15.153:0/463243340 conn(0x7fa2400fcfa0 msgr2=0x7fa2400fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 shutdown_connections 2024-04-05T02:51:52.960 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:52.425+0000 7fa247ef3700 1 -- 172.21.15.153:0/463243340 wait complete. 2024-04-05T02:51:52.961 INFO:teuthology.orchestra.run.smithi153.stdout:mon is available 2024-04-05T02:51:52.961 INFO:teuthology.orchestra.run.smithi153.stdout:Assimilating anything we can from ceph.conf... 2024-04-05T02:51:53.971 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:53.971 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T02:51:53.971 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:53.971 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T02:51:53.972 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8f59d9700 1 Processor -- start 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8f59d9700 1 -- start start 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8f59d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8f59d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0103270 con 0x7fc8f0102920 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52152/0 (socket says 172.21.15.153:52152) 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.353+0000 7fc8eeffd700 1 -- 172.21.15.153:0/3738373443 learned_addr learned my addr 172.21.15.153:0/3738373443 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8eeffd700 1 -- 172.21.15.153:0/3738373443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f01033b0 con 0x7fc8f0102920 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8eeffd700 1 --2- 172.21.15.153:0/3738373443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00098e0 tx=0x7fc8e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae12d81bf616d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8edffb700 1 -- 172.21.15.153:0/3738373443 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e000b860 con 0x7fc8f0102920 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8edffb700 1 -- 172.21.15.153:0/3738373443 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e000b9c0 con 0x7fc8f0102920 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8edffb700 1 -- 172.21.15.153:0/3738373443 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e000bc90 con 0x7fc8f0102920 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8f59d9700 1 -- 172.21.15.153:0/3738373443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 msgr2=0x7fc8f0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8f59d9700 1 --2- 172.21.15.153:0/3738373443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00098e0 tx=0x7fc8e0009250 comp rx=0 tx=0).stop 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8f59d9700 1 -- 172.21.15.153:0/3738373443 shutdown_connections 2024-04-05T02:51:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.354+0000 7fc8f59d9700 1 --2- 172.21.15.153:0/3738373443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0102d30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 -- 172.21.15.153:0/3738373443 >> 172.21.15.153:0/3738373443 conn(0x7fc8f00fe0a0 msgr2=0x7fc8f01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 -- 172.21.15.153:0/3738373443 shutdown_connections 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 -- 172.21.15.153:0/3738373443 wait complete. 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 Processor -- start 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 -- start start 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8f59d9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0111460 con 0x7fc8f0102920 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52168/0 (socket says 172.21.15.153:52168) 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.355+0000 7fc8eeffd700 1 -- 172.21.15.153:0/1179593124 learned_addr learned my addr 172.21.15.153:0/1179593124 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8eeffd700 1 -- 172.21.15.153:0/1179593124 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e00094f0 con 0x7fc8f0102920 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8eeffd700 1 --2- 172.21.15.153:0/1179593124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0003080 tx=0x7fc8e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:53.974 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e001c5c0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e001c720 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e001ca30 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f0111660 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f01120f0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc8e001a450 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.356+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc8e002e790 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.357+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f005f5f0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.360+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc8e001a890 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.426+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc8f004bed0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.429+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc8e002ede0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.429+0000 7fc8d77fe700 1 -- 172.21.15.153:0/1179593124 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fc8f004bed0 con 0x7fc8f0102920 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 msgr2=0x7fc8f0110f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 --2- 172.21.15.153:0/1179593124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0003080 tx=0x7fc8e0016000 comp rx=0 tx=0).stop 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 shutdown_connections 2024-04-05T02:51:53.975 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 --2- 172.21.15.153:0/1179593124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8f0102920 0x7fc8f0110f20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:53.976 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 >> 172.21.15.153:0/1179593124 conn(0x7fc8f00fe0a0 msgr2=0x7fc8f0105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:53.976 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 shutdown_connections 2024-04-05T02:51:53.976 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:53.430+0000 7fc8f59d9700 1 -- 172.21.15.153:0/1179593124 wait complete. 2024-04-05T02:51:53.976 INFO:teuthology.orchestra.run.smithi153.stdout:Generating new minimal ceph.conf... 2024-04-05T02:51:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:53 smithi153 ceph-mon[29002]: from='client.? 172.21.15.153:0/1179593124' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T02:51:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:53 smithi153 ceph-mon[29002]: from='client.? 172.21.15.153:0/1179593124' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-05T02:51:54.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:54 smithi153 ceph-mon[29002]: from='client.? 172.21.15.153:0/3543260529' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.342+0000 7fd6b5a5b700 1 Processor -- start 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.342+0000 7fd6b5a5b700 1 -- start start 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6b5a5b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6b5a5b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0103240 con 0x7fd6b01028f0 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52174/0 (socket says 172.21.15.153:52174) 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6aeffd700 1 -- 172.21.15.153:0/4091823743 learned_addr learned my addr 172.21.15.153:0/4091823743 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6aeffd700 1 -- 172.21.15.153:0/4091823743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b0103380 con 0x7fd6b01028f0 2024-04-05T02:51:55.001 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6aeffd700 1 --2- 172.21.15.153:0/4091823743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098e0 tx=0x7fd6a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9413e944fe48d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6adffb700 1 -- 172.21.15.153:0/4091823743 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6a800b860 con 0x7fd6b01028f0 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6adffb700 1 -- 172.21.15.153:0/4091823743 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd6a800be50 con 0x7fd6b01028f0 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.343+0000 7fd6adffb700 1 -- 172.21.15.153:0/4091823743 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6a8010380 con 0x7fd6b01028f0 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/4091823743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 msgr2=0x7fd6b0102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 --2- 172.21.15.153:0/4091823743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098e0 tx=0x7fd6a8009250 comp rx=0 tx=0).stop 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/4091823743 shutdown_connections 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 --2- 172.21.15.153:0/4091823743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b0102d00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/4091823743 >> 172.21.15.153:0/4091823743 conn(0x7fd6b00fe070 msgr2=0x7fd6b0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/4091823743 shutdown_connections 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.344+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/4091823743 wait complete. 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6b5a5b700 1 Processor -- start 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6b5a5b700 1 -- start start 2024-04-05T02:51:55.002 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6b5a5b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6b5a5b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0111c00 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52188/0 (socket says 172.21.15.153:52188) 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6aeffd700 1 -- 172.21.15.153:0/3543260529 learned_addr learned my addr 172.21.15.153:0/3543260529 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.345+0000 7fd6aeffd700 1 -- 172.21.15.153:0/3543260529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80094f0 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6aeffd700 1 --2- 172.21.15.153:0/3543260529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800b440 tx=0x7fd6a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6a8010760 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd6a8010d50 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b0111e00 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6a801ae20 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b0112970 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd6a802e6c0 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.346+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd6a8024900 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.347+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b005f5f0 con 0x7fd6b01028f0 2024-04-05T02:51:55.003 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.349+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd6a8029070 con 0x7fd6b01028f0 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.416+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd6b0104670 con 0x7fd6b01028f0 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.416+0000 7fd6977fe700 1 -- 172.21.15.153:0/3543260529 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fd6a801a6a0 con 0x7fd6b01028f0 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 msgr2=0x7fd6b01116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 --2- 172.21.15.153:0/3543260529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800b440 tx=0x7fd6a8009250 comp rx=0 tx=0).stop 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 shutdown_connections 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 --2- 172.21.15.153:0/3543260529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6b01028f0 0x7fd6b01116c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 >> 172.21.15.153:0/3543260529 conn(0x7fd6b00fe070 msgr2=0x7fd6b00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 shutdown_connections 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:54.417+0000 7fd6b5a5b700 1 -- 172.21.15.153:0/3543260529 wait complete. 2024-04-05T02:51:55.004 INFO:teuthology.orchestra.run.smithi153.stdout:Restarting the monitor... 2024-04-05T02:51:55.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T02:51:55.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[28998]: 2024-04-05T02:51:55.289+0000 7f4b53520700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T02:51:55.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[28998]: 2024-04-05T02:51:55.289+0000 7f4b53520700 -1 mon.smithi153@0(leader) e1 *** Got Signal Terminated *** 2024-04-05T02:51:55.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 podman[29251]: 2024-04-05 02:51:55.402285939 +0000 UTC m=+0.242339803 container died 534a8df4311dd1ba67a501fa6a34bc6da29d324b4526830beb15823c5dc69a3e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, RELEASE=quincy-6fda41f, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-05T02:51:56.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 podman[29251]: 2024-04-05 02:51:55.747197234 +0000 UTC m=+0.587251096 container cleanup 534a8df4311dd1ba67a501fa6a34bc6da29d324b4526830beb15823c5dc69a3e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-6fda41f, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-05T02:51:56.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 bash[29251]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153 2024-04-05T02:51:56.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:55 smithi153 podman[29265]: 2024-04-05 02:51:55.898184099 +0000 UTC m=+0.492110645 container remove 534a8df4311dd1ba67a501fa6a34bc6da29d324b4526830beb15823c5dc69a3e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, RELEASE=quincy-6fda41f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-05T02:51:56.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service: Deactivated successfully. 2024-04-05T02:51:56.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T02:51:56.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 systemd[1]: Starting Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T02:51:56.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 podman[29361]: 2024-04-05 02:51:56.315736298 +0000 UTC m=+0.105639810 container create 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-6fda41f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-05T02:51:56.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 podman[29361]: 2024-04-05 02:51:56.227979682 +0000 UTC m=+0.017883194 image pull a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48 quay.ceph.io/ceph-ci/ceph:quincy 2024-04-05T02:51:56.580 INFO:teuthology.orchestra.run.smithi153.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 podman[29361]: 2024-04-05 02:51:56.50266342 +0000 UTC m=+0.292566933 container init 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=quincy-6fda41f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 podman[29361]: 2024-04-05 02:51:56.505572669 +0000 UTC m=+0.295476183 container start 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=quincy-6fda41f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable), process ceph-mon, pid 2 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: pidfile_write: ignore empty --pid-file 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: load: jerasure load: lrc 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: RocksDB version: 6.15.5 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Compile date Apr 3 2024 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: DB SUMMARY 2024-04-05T02:51:56.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: DB Session ID: YC3OV6CH5UYXK7D6TKOW 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: CURRENT file: CURRENT 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000008.sst 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000010.log size: 98174 ; 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.error_if_exists: 0 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.create_if_missing: 0 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.paranoid_checks: 1 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.env: 0x5602fdc4ea80 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.fs: Posix File System 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.info_log: 0x5602ff799c00 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.statistics: (nil) 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.use_fsync: 0 2024-04-05T02:51:56.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_log_file_size: 0 2024-04-05T02:51:56.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T02:51:56.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T02:51:56.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_fallocate: 1 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.use_direct_reads: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.db_log_dir: 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi153/store.db 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T02:51:56.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.write_buffer_manager: 0x56030050e990 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T02:51:56.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.unordered_write: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.row_cache: None 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.wal_filter: None 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.preserve_deletes: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.two_write_queues: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.atomic_flush: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T02:51:56.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.log_readahead_size: 0 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_background_jobs: 2 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_background_compactions: -1 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_subcompactions: 1 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T02:51:56.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_open_files: -1 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_background_flushes: -1 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Compression algorithms supported: 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kZSTD supported: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kXpressCompression supported: 0 2024-04-05T02:51:56.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kZlibCompression supported: 1 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kSnappyCompression supported: 1 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kLZ4Compression supported: 1 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: kBZip2Compression supported: 0 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.merge_operator: 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_filter: None 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T02:51:56.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5602ff7eef58) 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: hash_index_allow_collision: 1 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 1 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x5602ff821090 2024-04-05T02:51:56.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-04-05T02:51:56.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 4 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression: NoCompression 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.num_levels: 7 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T02:51:56.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T02:51:56.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T02:51:56.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.arena_block_size: 4194304 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T02:51:56.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T02:51:56.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.table_properties_collectors: 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.inplace_update_support: 0 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T02:51:56.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.bloom_locality: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.max_successive_merges: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.ttl: 2592000 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.enable_blob_files: false 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.min_blob_size: 0 2024-04-05T02:51:56.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285516531976, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285516532821, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93380, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91492, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84056, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712285516, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4b3f8851-23b6-48a7-8cef-115510c9f836", "db_session_id": "YC3OV6CH5UYXK7D6TKOW"}} 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285516533212, "job": 1, "event": "recovery_finished"} 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T02:51:56.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5602ff7b5180 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: DB pointer 0x5602ff88a000 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:51:56.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 92.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 92.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T02:51:56.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 29.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 29.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 92.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 92.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:51:56.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.2 0.00 0.00 1 0.001 0 0 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 28.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: starting mon.smithi153 rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???) e1 preinit fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).mds e1 new map 2024-04-05T02:51:56.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).mds e1 print_map 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e1 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: -1 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: No filesystems configured 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 bash[29361]: 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 systemd[1]: Started Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: monmap e1: 1 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: fsmap 2024-04-05T02:51:56.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: osdmap e1: 0 total, 0 up, 0 in 2024-04-05T02:51:56.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:56 smithi153 ceph-mon[29381]: mgrmap e1: no daemons active 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.078+0000 7f17f9897700 1 Processor -- start 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.078+0000 7f17f9897700 1 -- start start 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.078+0000 7f17f9897700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.078+0000 7f17f9897700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4102160 con 0x7f17f4101810 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52204/0 (socket says 172.21.15.153:52204) 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f2ffd700 1 -- 172.21.15.153:0/1062111492 learned_addr learned my addr 172.21.15.153:0/1062111492 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:57.673 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f2ffd700 1 -- 172.21.15.153:0/1062111492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f41022a0 con 0x7f17f4101810 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f2ffd700 1 --2- 172.21.15.153:0/1062111492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e36e988f5614f4e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f27fc700 1 -- 172.21.15.153:0/1062111492 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e400b860 con 0x7f17f4101810 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f27fc700 1 -- 172.21.15.153:0/1062111492 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17e400be50 con 0x7f17f4101810 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.079+0000 7f17f27fc700 1 -- 172.21.15.153:0/1062111492 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4010380 con 0x7f17f4101810 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 -- 172.21.15.153:0/1062111492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 msgr2=0x7f17f4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 --2- 172.21.15.153:0/1062111492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 -- 172.21.15.153:0/1062111492 shutdown_connections 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 --2- 172.21.15.153:0/1062111492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4101c20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 -- 172.21.15.153:0/1062111492 >> 172.21.15.153:0/1062111492 conn(0x7f17f40fcfd0 msgr2=0x7f17f40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 -- 172.21.15.153:0/1062111492 shutdown_connections 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 -- 172.21.15.153:0/1062111492 wait complete. 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.080+0000 7f17f9897700 1 Processor -- start 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f9897700 1 -- start start 2024-04-05T02:51:57.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f9897700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f9897700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4111c90 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52220/0 (socket says 172.21.15.153:52220) 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f2ffd700 1 -- 172.21.15.153:0/2901299369 learned_addr learned my addr 172.21.15.153:0/2901299369 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f2ffd700 1 -- 172.21.15.153:0/2901299369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40094f0 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f2ffd700 1 --2- 172.21.15.153:0/2901299369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006b00 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4010760 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17e4010d50 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f4111e90 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e401ad40 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.081+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f41129b0 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.082+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f17e402e6c0 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.082+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f17e4024810 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.082+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f405f5f0 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.085+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f17e4029070 con 0x7f17f4101810 2024-04-05T02:51:57.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.150+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f17f4103590 con 0x7f17f4101810 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.152+0000 7f17ebfff700 1 -- 172.21.15.153:0/2901299369 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f17e401a5c0 con 0x7f17f4101810 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 msgr2=0x7f17f4111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 --2- 172.21.15.153:0/2901299369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006b00 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 shutdown_connections 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 --2- 172.21.15.153:0/2901299369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17f4101810 0x7f17f4111750 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 >> 172.21.15.153:0/2901299369 conn(0x7f17f40fcfd0 msgr2=0x7f17f40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 shutdown_connections 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:57.153+0000 7f17f9897700 1 -- 172.21.15.153:0/2901299369 wait complete. 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mgr... 2024-04-05T02:51:57.676 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 9283 ... 2024-04-05T02:51:57.851 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mgr.smithi153.npzyqy 2024-04-05T02:51:57.851 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mgr.smithi153.npzyqy.service: Unit ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mgr.smithi153.npzyqy.service not loaded. 2024-04-05T02:51:58.005 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4.target.wants/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mgr.smithi153.npzyqy.service → /etc/systemd/system/ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@.service. 2024-04-05T02:51:58.436 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:58 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2901299369' entity='client.admin' 2024-04-05T02:51:58.551 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T02:51:58.551 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-04-05T02:51:58.558 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T02:51:58.558 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:51:58.559 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-04-05T02:51:58.559 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T02:51:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T02:51:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr to start... 2024-04-05T02:51:58.575 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr... 2024-04-05T02:51:59.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:51:59 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/268885206' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T02:51:59.664 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "3fa69b9e-f2f7-11ee-b649-cb9ed24678a4", 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:51:59.665 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T02:51:59.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T02:51:59.674 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T02:51:59.675 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T02:51:51.790179+0000", 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff522499700 1 Processor -- start 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff522499700 1 -- start start 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff522499700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff522499700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c103230 con 0x7ff51c1028e0 2024-04-05T02:51:59.676 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff51bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff51bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41768/0 (socket says 172.21.15.153:41768) 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.044+0000 7ff51bfff700 1 -- 172.21.15.153:0/4033058707 learned_addr learned my addr 172.21.15.153:0/4033058707 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.045+0000 7ff51bfff700 1 -- 172.21.15.153:0/4033058707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c103370 con 0x7ff51c1028e0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.045+0000 7ff51bfff700 1 --2- 172.21.15.153:0/4033058707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098e0 tx=0x7ff50c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567db144520caa9e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.045+0000 7ff51affd700 1 -- 172.21.15.153:0/4033058707 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c00b860 con 0x7ff51c1028e0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.045+0000 7ff51affd700 1 -- 172.21.15.153:0/4033058707 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff50c00be50 con 0x7ff51c1028e0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.045+0000 7ff51affd700 1 -- 172.21.15.153:0/4033058707 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c010380 con 0x7ff51c1028e0 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- 172.21.15.153:0/4033058707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 msgr2=0x7ff51c102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 --2- 172.21.15.153:0/4033058707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098e0 tx=0x7ff50c009250 comp rx=0 tx=0).stop 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- 172.21.15.153:0/4033058707 shutdown_connections 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 --2- 172.21.15.153:0/4033058707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c102cf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- 172.21.15.153:0/4033058707 >> 172.21.15.153:0/4033058707 conn(0x7ff51c0fe040 msgr2=0x7ff51c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- 172.21.15.153:0/4033058707 shutdown_connections 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- 172.21.15.153:0/4033058707 wait complete. 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 Processor -- start 2024-04-05T02:51:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- start start 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.046+0000 7ff522499700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c111bc0 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff51bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff51bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41770/0 (socket says 172.21.15.153:41770) 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff51bfff700 1 -- 172.21.15.153:0/268885206 learned_addr learned my addr 172.21.15.153:0/268885206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff51bfff700 1 -- 172.21.15.153:0/268885206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c0094f0 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff51bfff700 1 --2- 172.21.15.153:0/268885206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b440 tx=0x7ff50c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c010760 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff50c010d50 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c01ae20 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c111dc0 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c112930 con 0x7ff51c1028e0 2024-04-05T02:51:59.678 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff50c02e6c0 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.047+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff50c024900 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.048+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c05f5f0 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.050+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff50c029070 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.122+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff51c104660 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.123+0000 7ff518ff9700 1 -- 172.21.15.153:0/268885206 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7ff50c01a6a0 con 0x7ff51c1028e0 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 msgr2=0x7ff51c111680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 --2- 172.21.15.153:0/268885206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00b440 tx=0x7ff50c009250 comp rx=0 tx=0).stop 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 shutdown_connections 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 --2- 172.21.15.153:0/268885206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c1028e0 0x7ff51c111680 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 >> 172.21.15.153:0/268885206 conn(0x7ff51c0fe040 msgr2=0x7ff51c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 shutdown_connections 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:51:59.124+0000 7ff522499700 1 -- 172.21.15.153:0/268885206 wait complete. 2024-04-05T02:51:59.679 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (1/15)... 2024-04-05T02:52:02.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:02 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1866158994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "3fa69b9e-f2f7-11ee-b649-cb9ed24678a4", 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T02:52:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T02:52:02.850 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T02:52:02.851 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:02.852 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T02:51:51.790179+0000", 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.134+0000 7fb3fd948700 1 Processor -- start 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3fd948700 1 -- start start 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3fd948700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3fd948700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8103270 con 0x7fb3f8102920 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41786/0 (socket says 172.21.15.153:41786) 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f6ffd700 1 -- 172.21.15.153:0/347303207 learned_addr learned my addr 172.21.15.153:0/347303207 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f6ffd700 1 -- 172.21.15.153:0/347303207 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f81033b0 con 0x7fb3f8102920 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f6ffd700 1 --2- 172.21.15.153:0/347303207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80098e0 tx=0x7fb3e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93d0f014a56e622 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f5ffb700 1 -- 172.21.15.153:0/347303207 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e800b860 con 0x7fb3f8102920 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f5ffb700 1 -- 172.21.15.153:0/347303207 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3e800be50 con 0x7fb3f8102920 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.135+0000 7fb3f5ffb700 1 -- 172.21.15.153:0/347303207 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8010380 con 0x7fb3f8102920 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 -- 172.21.15.153:0/347303207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 msgr2=0x7fb3f8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 --2- 172.21.15.153:0/347303207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80098e0 tx=0x7fb3e8009250 comp rx=0 tx=0).stop 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 -- 172.21.15.153:0/347303207 shutdown_connections 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 --2- 172.21.15.153:0/347303207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f8102d30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 -- 172.21.15.153:0/347303207 >> 172.21.15.153:0/347303207 conn(0x7fb3f80fe0a0 msgr2=0x7fb3f81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 -- 172.21.15.153:0/347303207 shutdown_connections 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.136+0000 7fb3fd948700 1 -- 172.21.15.153:0/347303207 wait complete. 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3fd948700 1 Processor -- start 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3fd948700 1 -- start start 2024-04-05T02:52:02.854 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3fd948700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3fd948700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8111be0 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41788/0 (socket says 172.21.15.153:41788) 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3f6ffd700 1 -- 172.21.15.153:0/1866158994 learned_addr learned my addr 172.21.15.153:0/1866158994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.137+0000 7fb3f6ffd700 1 -- 172.21.15.153:0/1866158994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e80094f0 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3f6ffd700 1 --2- 172.21.15.153:0/1866158994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800b440 tx=0x7fb3e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8010760 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb3e8010d50 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f8111de0 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e801ae20 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f8112950 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb3e802e6c0 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.138+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb3e8024970 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.139+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f805f5f0 con 0x7fb3f8102920 2024-04-05T02:52:02.855 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.141+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb3e8029070 con 0x7fb3f8102920 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.213+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb3f81046a0 con 0x7fb3f8102920 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.213+0000 7fb3ef7fe700 1 -- 172.21.15.153:0/1866158994 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fb3e801a6a0 con 0x7fb3f8102920 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 msgr2=0x7fb3f81116a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 --2- 172.21.15.153:0/1866158994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800b440 tx=0x7fb3e8009250 comp rx=0 tx=0).stop 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 shutdown_connections 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 --2- 172.21.15.153:0/1866158994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3f8102920 0x7fb3f81116a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 >> 172.21.15.153:0/1866158994 conn(0x7fb3f80fe0a0 msgr2=0x7fb3f80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 shutdown_connections 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:02.214+0000 7fb3fd948700 1 -- 172.21.15.153:0/1866158994 wait complete. 2024-04-05T02:52:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (2/15)... 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: Activating manager daemon smithi153.npzyqy 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: mgrmap e2: smithi153.npzyqy(active, starting, since 0.00293002s) 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: Manager daemon smithi153.npzyqy is now available 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T02:52:05.262 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T02:52:05.263 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:05.263 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:05.263 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:04 smithi153 ceph-mon[29381]: from='mgr.14100 172.21.15.153:0/2919291411' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "3fa69b9e-f2f7-11ee-b649-cb9ed24678a4", 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T02:52:05.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-05T02:52:05.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T02:52:05.946 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T02:52:05.947 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T02:51:51.790179+0000", 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f747b033700 1 Processor -- start 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f747b033700 1 -- start start 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f747b033700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f747b033700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7474103270 con 0x7f7474102920 2024-04-05T02:52:05.948 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f7478dcf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f7478dcf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41838/0 (socket says 172.21.15.153:41838) 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f7478dcf700 1 -- 172.21.15.153:0/3707014418 learned_addr learned my addr 172.21.15.153:0/3707014418 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.280+0000 7f7478dcf700 1 -- 172.21.15.153:0/3707014418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74741033b0 con 0x7f7474102920 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f7478dcf700 1 --2- 172.21.15.153:0/3707014418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f74680098e0 tx=0x7f7468009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13ef86bd5558429 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f74737fe700 1 -- 172.21.15.153:0/3707014418 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746800b860 con 0x7f7474102920 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f74737fe700 1 -- 172.21.15.153:0/3707014418 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f746800be50 con 0x7f7474102920 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f74737fe700 1 -- 172.21.15.153:0/3707014418 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7468010380 con 0x7f7474102920 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 -- 172.21.15.153:0/3707014418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 msgr2=0x7f7474102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 --2- 172.21.15.153:0/3707014418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f74680098e0 tx=0x7f7468009250 comp rx=0 tx=0).stop 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 -- 172.21.15.153:0/3707014418 shutdown_connections 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 --2- 172.21.15.153:0/3707014418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f7474102d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 -- 172.21.15.153:0/3707014418 >> 172.21.15.153:0/3707014418 conn(0x7f74740fe0a0 msgr2=0x7f74741004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 -- 172.21.15.153:0/3707014418 shutdown_connections 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.281+0000 7f747b033700 1 -- 172.21.15.153:0/3707014418 wait complete. 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.282+0000 7f747b033700 1 Processor -- start 2024-04-05T02:52:05.949 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.282+0000 7f747b033700 1 -- start start 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.282+0000 7f747b033700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.282+0000 7f747b033700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7474111be0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.282+0000 7f7478dcf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f7478dcf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41852/0 (socket says 172.21.15.153:41852) 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f7478dcf700 1 -- 172.21.15.153:0/2969680064 learned_addr learned my addr 172.21.15.153:0/2969680064 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f7478dcf700 1 -- 172.21.15.153:0/2969680064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74680094f0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f7478dcf700 1 --2- 172.21.15.153:0/2969680064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7468000c00 tx=0x7f7468009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74680107c0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7468010db0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7474111de0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746801ad00 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.283+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7474112950 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.284+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f74680193c0 con 0x7f7474102920 2024-04-05T02:52:05.950 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.284+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747405f5f0 con 0x7f7474102920 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.284+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f746804a1e0 con 0x7f7474102920 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.286+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7468019670 con 0x7f7474102920 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.355+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f74741046a0 con 0x7f7474102920 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f74717fa700 1 -- 172.21.15.153:0/2969680064 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f7468022340 con 0x7f7474102920 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 msgr2=0x7f74741116a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 --2- 172.21.15.153:0/2969680064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7468000c00 tx=0x7f7468009250 comp rx=0 tx=0).stop 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 shutdown_connections 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 --2- 172.21.15.153:0/2969680064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7474102920 0x7f74741116a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 >> 172.21.15.153:0/2969680064 conn(0x7f74740fe0a0 msgr2=0x7f74740fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 shutdown_connections 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:05.356+0000 7f747b033700 1 -- 172.21.15.153:0/2969680064 wait complete. 2024-04-05T02:52:05.951 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (3/15)... 2024-04-05T02:52:06.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:05 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2969680064' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T02:52:06.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:05 smithi153 ceph-mon[29381]: mgrmap e3: smithi153.npzyqy(active, since 1.00637s) 2024-04-05T02:52:08.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:07 smithi153 ceph-mon[29381]: mgrmap e4: smithi153.npzyqy(active, since 2s) 2024-04-05T02:52:12.036 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:11 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/671830112' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "3fa69b9e-f2f7-11ee-b649-cb9ed24678a4", 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T02:52:16.169 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.170 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T02:52:16.171 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:16.171 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T02:52:16.171 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T02:52:16.171 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T02:52:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T02:52:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T02:51:51.790179+0000", 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T02:52:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.372+0000 7facc77f2700 1 Processor -- start 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.372+0000 7facc77f2700 1 -- start start 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc77f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc77f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00716a0 con 0x7facc0071290 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc558e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc558e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33764/0 (socket says 172.21.15.153:33764) 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc558e700 1 -- 172.21.15.153:0/2208711772 learned_addr learned my addr 172.21.15.153:0/2208711772 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc558e700 1 -- 172.21.15.153:0/2208711772 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0105a40 con 0x7facc0071290 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facc558e700 1 --2- 172.21.15.153:0/2208711772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7facb00098e0 tx=0x7facb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbfa44c4a7c699f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facb7fff700 1 -- 172.21.15.153:0/2208711772 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb000b860 con 0x7facc0071290 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facb7fff700 1 -- 172.21.15.153:0/2208711772 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7facb000be50 con 0x7facc0071290 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.373+0000 7facb7fff700 1 -- 172.21.15.153:0/2208711772 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb0010380 con 0x7facc0071290 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- 172.21.15.153:0/2208711772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 msgr2=0x7facc0105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 --2- 172.21.15.153:0/2208711772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7facb00098e0 tx=0x7facb0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- 172.21.15.153:0/2208711772 shutdown_connections 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 --2- 172.21.15.153:0/2208711772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0105500 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- 172.21.15.153:0/2208711772 >> 172.21.15.153:0/2208711772 conn(0x7facc00fcfb0 msgr2=0x7facc00ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- 172.21.15.153:0/2208711772 shutdown_connections 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- 172.21.15.153:0/2208711772 wait complete. 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 Processor -- start 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.374+0000 7facc77f2700 1 -- start start 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc77f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc77f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc0111c80 con 0x7facc0071290 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc558e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc558e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33780/0 (socket says 172.21.15.153:33780) 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc558e700 1 -- 172.21.15.153:0/671830112 learned_addr learned my addr 172.21.15.153:0/671830112 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc558e700 1 -- 172.21.15.153:0/671830112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00094f0 con 0x7facc0071290 2024-04-05T02:52:16.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc558e700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7facb0011bd0 tx=0x7facb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb00107c0 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7facb0010db0 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc0111e80 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.375+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facb001ad60 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.376+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01129f0 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.376+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7facb002e440 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.376+0000 7facb5ffb700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 0x7facac038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.376+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7facb004ab30 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.376+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc005f5f0 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.377+0000 7facc4d8d700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 0x7facac038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.377+0000 7facc4d8d700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 0x7facac038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7facbc009990 tx=0x7facbc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.381+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7facb0023980 con 0x7facc0071290 2024-04-05T02:52:16.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.644+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7facc0112cc0 con 0x7facc0071290 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.645+0000 7facb5ffb700 1 -- 172.21.15.153:0/671830112 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7facb0048020 con 0x7facc0071290 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 msgr2=0x7facac038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 0x7facac038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7facbc009990 tx=0x7facbc006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 msgr2=0x7facc0111740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7facb0011bd0 tx=0x7facb0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 shutdown_connections 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7facac036790 0x7facac038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 --2- 172.21.15.153:0/671830112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7facc0071290 0x7facc0111740 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 >> 172.21.15.153:0/671830112 conn(0x7facc00fcfb0 msgr2=0x7facc00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:16.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 shutdown_connections 2024-04-05T02:52:16.187 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:11.647+0000 7facc77f2700 1 -- 172.21.15.153:0/671830112 wait complete. 2024-04-05T02:52:16.187 INFO:teuthology.orchestra.run.smithi153.stdout:mgr is available 2024-04-05T02:52:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:16 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/4083878998' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T02:52:17.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db6ccb700 1 Processor -- start 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db6ccb700 1 -- start start 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db6ccb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db6ccb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db0103270 con 0x7f3db0102920 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db4a67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db4a67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33786/0 (socket says 172.21.15.153:33786) 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.594+0000 7f3db4a67700 1 -- 172.21.15.153:0/2222394924 learned_addr learned my addr 172.21.15.153:0/2222394924 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db4a67700 1 -- 172.21.15.153:0/2222394924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db01033b0 con 0x7f3db0102920 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db4a67700 1 --2- 172.21.15.153:0/2222394924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3da40098e0 tx=0x7f3da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3117e2b36a17ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3daf7fe700 1 -- 172.21.15.153:0/2222394924 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3da400b860 con 0x7f3db0102920 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3daf7fe700 1 -- 172.21.15.153:0/2222394924 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3da400be50 con 0x7f3db0102920 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3daf7fe700 1 -- 172.21.15.153:0/2222394924 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3da4010380 con 0x7f3db0102920 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db6ccb700 1 -- 172.21.15.153:0/2222394924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 msgr2=0x7f3db0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/2222394924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3da40098e0 tx=0x7f3da4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db6ccb700 1 -- 172.21.15.153:0/2222394924 shutdown_connections 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/2222394924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0102d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.595+0000 7f3db6ccb700 1 -- 172.21.15.153:0/2222394924 >> 172.21.15.153:0/2222394924 conn(0x7f3db00fe0a0 msgr2=0x7f3db01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 -- 172.21.15.153:0/2222394924 shutdown_connections 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 -- 172.21.15.153:0/2222394924 wait complete. 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 Processor -- start 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 -- start start 2024-04-05T02:52:17.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.596+0000 7f3db6ccb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db0079180 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db4a67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db4a67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33788/0 (socket says 172.21.15.153:33788) 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db4a67700 1 -- 172.21.15.153:0/4083878998 learned_addr learned my addr 172.21.15.153:0/4083878998 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db4a67700 1 -- 172.21.15.153:0/4083878998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da40094f0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db4a67700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000c00 tx=0x7f3da4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3da40107c0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3da4010db0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db00771f0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3da401ad00 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.597+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db00776e0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.598+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f3da40193c0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.598+0000 7f3dad7fa700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 0x7f3da0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.598+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db004bed0 con 0x7f3db0102920 2024-04-05T02:52:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.598+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3da404a830 con 0x7f3db0102920 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.598+0000 7f3daffff700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 0x7f3da0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.599+0000 7f3daffff700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 0x7f3da0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009990 tx=0x7f3d9c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.603+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3da4019670 con 0x7f3db0102920 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.785+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3db0077b60 con 0x7f3db0102920 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.786+0000 7f3dad7fa700 1 -- 172.21.15.153:0/4083878998 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f3da4022340 con 0x7f3db0102920 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 msgr2=0x7f3da0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 0x7f3da0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009990 tx=0x7f3d9c006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 msgr2=0x7f3db0078c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000c00 tx=0x7f3da4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 shutdown_connections 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f3da0036790 0x7f3da0038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 --2- 172.21.15.153:0/4083878998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3db0102920 0x7f3db0078c40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 >> 172.21.15.153:0/4083878998 conn(0x7f3db00fe0a0 msgr2=0x7f3db00fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 shutdown_connections 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:16.789+0000 7f3db6ccb700 1 -- 172.21.15.153:0/4083878998 wait complete. 2024-04-05T02:52:17.341 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling cephadm module... 2024-04-05T02:52:18.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:18 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/400184791' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45d0e9f700 1 Processor -- start 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45d0e9f700 1 -- start start 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45d0e9f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45d0e9f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc1021a0 con 0x7f45cc101850 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33804/0 (socket says 172.21.15.153:33804) 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45ca59c700 1 -- 172.21.15.153:0/105542987 learned_addr learned my addr 172.21.15.153:0/105542987 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.720+0000 7f45ca59c700 1 -- 172.21.15.153:0/105542987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc1022e0 con 0x7f45cc101850 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45ca59c700 1 --2- 172.21.15.153:0/105542987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098e0 tx=0x7f45bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cda25f2f87ab07d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45c959a700 1 -- 172.21.15.153:0/105542987 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc00b860 con 0x7f45cc101850 2024-04-05T02:52:18.652 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45c959a700 1 -- 172.21.15.153:0/105542987 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f45bc00be50 con 0x7f45cc101850 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45c959a700 1 -- 172.21.15.153:0/105542987 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc010380 con 0x7f45cc101850 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 -- 172.21.15.153:0/105542987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 msgr2=0x7f45cc101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/105542987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098e0 tx=0x7f45bc009250 comp rx=0 tx=0).stop 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 -- 172.21.15.153:0/105542987 shutdown_connections 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/105542987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc101c60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 -- 172.21.15.153:0/105542987 >> 172.21.15.153:0/105542987 conn(0x7f45cc0fcf90 msgr2=0x7f45cc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.721+0000 7f45d0e9f700 1 -- 172.21.15.153:0/105542987 shutdown_connections 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45d0e9f700 1 -- 172.21.15.153:0/105542987 wait complete. 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45d0e9f700 1 Processor -- start 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45d0e9f700 1 -- start start 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45d0e9f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45d0e9f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc111c10 con 0x7f45cc101850 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45ca59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33810/0 (socket says 172.21.15.153:33810) 2024-04-05T02:52:18.653 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.722+0000 7f45ca59c700 1 -- 172.21.15.153:0/400184791 learned_addr learned my addr 172.21.15.153:0/400184791 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45ca59c700 1 -- 172.21.15.153:0/400184791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0094f0 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45ca59c700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc0107c0 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f45bc010db0 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc111e10 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45bc01ad00 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.723+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc112980 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f45bc0193c0 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45cc05f5f0 con 0x7f45cc101850 2024-04-05T02:52:18.654 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45baffd700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 0x7f45b4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f45bc04a7c0 con 0x7f45cc101850 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45c9d9b700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 0x7f45b4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.724+0000 7f45c9d9b700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 0x7f45b4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009990 tx=0x7f45c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.728+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f45bc019670 con 0x7f45cc101850 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:17.961+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f45cc04bed0 con 0x7f45cc101850 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.014+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f45bc01a530 con 0x7f45cc101850 2024-04-05T02:52:18.655 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.016+0000 7f45baffd700 1 -- 172.21.15.153:0/400184791 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f45bc022340 con 0x7f45cc101850 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 msgr2=0x7f45b4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 0x7f45b4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009990 tx=0x7f45c0006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 msgr2=0x7f45cc1116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc009250 comp rx=0 tx=0).stop 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 shutdown_connections 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f45b4036790 0x7f45b4038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 --2- 172.21.15.153:0/400184791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45cc101850 0x7f45cc1116d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 >> 172.21.15.153:0/400184791 conn(0x7f45cc0fcf90 msgr2=0x7f45cc0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 shutdown_connections 2024-04-05T02:52:18.656 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:18.018+0000 7f45d0e9f700 1 -- 172.21.15.153:0/400184791 wait complete. 2024-04-05T02:52:19.266 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:19 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/400184791' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-05T02:52:19.266 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:19 smithi153 ceph-mon[29381]: mgrmap e5: smithi153.npzyqy(active, since 13s) 2024-04-05T02:52:19.926 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:19.926 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-05T02:52:19.926 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T02:52:19.926 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.npzyqy", 2024-04-05T02:52:19.926 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac89f49700 1 Processor -- start 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac89f49700 1 -- start start 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac89f49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac89f49700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84076ee0 con 0x7fac84076ad0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac837fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac837fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33852/0 (socket says 172.21.15.153:33852) 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac837fe700 1 -- 172.21.15.153:0/3185388391 learned_addr learned my addr 172.21.15.153:0/3185388391 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.060+0000 7fac837fe700 1 -- 172.21.15.153:0/3185388391 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84105b00 con 0x7fac84076ad0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac837fe700 1 --2- 172.21.15.153:0/3185388391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac740098e0 tx=0x7fac74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323a6ce3ea9cdcba server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac827fc700 1 -- 172.21.15.153:0/3185388391 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac7400b8f0 con 0x7fac84076ad0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac827fc700 1 -- 172.21.15.153:0/3185388391 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fac74011420 con 0x7fac84076ad0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac827fc700 1 -- 172.21.15.153:0/3185388391 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac74010540 con 0x7fac84076ad0 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac89f49700 1 -- 172.21.15.153:0/3185388391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 msgr2=0x7fac841055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac89f49700 1 --2- 172.21.15.153:0/3185388391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fac740098e0 tx=0x7fac74009250 comp rx=0 tx=0).stop 2024-04-05T02:52:19.927 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac89f49700 1 -- 172.21.15.153:0/3185388391 shutdown_connections 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac89f49700 1 --2- 172.21.15.153:0/3185388391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac841055c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.061+0000 7fac89f49700 1 -- 172.21.15.153:0/3185388391 >> 172.21.15.153:0/3185388391 conn(0x7fac840fd010 msgr2=0x7fac840ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 -- 172.21.15.153:0/3185388391 shutdown_connections 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 -- 172.21.15.153:0/3185388391 wait complete. 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 Processor -- start 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 -- start start 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.062+0000 7fac89f49700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84111d40 con 0x7fac84076ad0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac837fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac837fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33856/0 (socket says 172.21.15.153:33856) 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac837fe700 1 -- 172.21.15.153:0/3072210757 learned_addr learned my addr 172.21.15.153:0/3072210757 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac837fe700 1 -- 172.21.15.153:0/3072210757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740094f0 con 0x7fac84076ad0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac837fe700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fac74011db0 tx=0x7fac74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac740107b0 con 0x7fac84076ad0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fac74010da0 con 0x7fac84076ad0 2024-04-05T02:52:19.928 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac84111f40 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac7401ad60 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.063+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac84112a60 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.064+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fac740244a0 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.064+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8405f5f0 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.065+0000 7fac6bfff700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 0x7fac6c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.065+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fac7404ac70 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.065+0000 7fac82ffd700 1 -- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 msgr2=0x7fac6c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.065+0000 7fac82ffd700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 0x7fac6c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.073+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fac7402e440 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.266+0000 7fac82ffd700 1 -- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 msgr2=0x7fac6c038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.266+0000 7fac82ffd700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 0x7fac6c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.322+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fac84113570 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.323+0000 7fac6bfff700 1 -- 172.21.15.153:0/3072210757 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fac7402cb80 con 0x7fac84076ad0 2024-04-05T02:52:19.929 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 msgr2=0x7fac6c038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 0x7fac6c038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 msgr2=0x7fac84111800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fac74011db0 tx=0x7fac74009250 comp rx=0 tx=0).stop 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 shutdown_connections 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7fac6c036790 0x7fac6c038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 --2- 172.21.15.153:0/3072210757 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac84076ad0 0x7fac84111800 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 >> 172.21.15.153:0/3072210757 conn(0x7fac840fd010 msgr2=0x7fac840fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.325+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 shutdown_connections 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:19.326+0000 7fac89f49700 1 -- 172.21.15.153:0/3072210757 wait complete. 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-04-05T02:52:19.930 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 5... 2024-04-05T02:52:20.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:20 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3072210757' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T02:52:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: Activating manager daemon smithi153.npzyqy 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: osdmap e2: 0 total, 0 up, 0 in 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: mgrmap e6: smithi153.npzyqy(active, starting, since 0.00314304s) 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: Manager daemon smithi153.npzyqy is now available 2024-04-05T02:52:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T02:52:25.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:24 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T02:52:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:25 smithi153 ceph-mon[29381]: Found migration_current of "None". Setting to last migration. 2024-04-05T02:52:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:25 smithi153 ceph-mon[29381]: mgrmap e7: smithi153.npzyqy(active, since 1.00512s) 2024-04-05T02:52:26.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:26.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-05T02:52:26.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a698ec700 1 Processor -- start 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a698ec700 1 -- start start 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a698ec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a698ec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640716a0 con 0x7f9a64071290 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33864/0 (socket says 172.21.15.153:33864) 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.376+0000 7f9a62ffd700 1 -- 172.21.15.153:0/3214847893 learned_addr learned my addr 172.21.15.153:0/3214847893 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a62ffd700 1 -- 172.21.15.153:0/3214847893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64105a40 con 0x7f9a64071290 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a62ffd700 1 --2- 172.21.15.153:0/3214847893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a540098e0 tx=0x7f9a54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7a246dd9f39635 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a61ffb700 1 -- 172.21.15.153:0/3214847893 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a5400b860 con 0x7f9a64071290 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a61ffb700 1 -- 172.21.15.153:0/3214847893 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a5400be50 con 0x7f9a64071290 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a61ffb700 1 -- 172.21.15.153:0/3214847893 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a54010380 con 0x7f9a64071290 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.377+0000 7f9a698ec700 1 -- 172.21.15.153:0/3214847893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 msgr2=0x7f9a64105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:26.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 --2- 172.21.15.153:0/3214847893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a540098e0 tx=0x7f9a54009250 comp rx=0 tx=0).stop 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 -- 172.21.15.153:0/3214847893 shutdown_connections 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 --2- 172.21.15.153:0/3214847893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64105500 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 -- 172.21.15.153:0/3214847893 >> 172.21.15.153:0/3214847893 conn(0x7f9a640fcfb0 msgr2=0x7f9a640ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 -- 172.21.15.153:0/3214847893 shutdown_connections 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 -- 172.21.15.153:0/3214847893 wait complete. 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 Processor -- start 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 -- start start 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.378+0000 7f9a698ec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a698ec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64111c80 con 0x7f9a64071290 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a62ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33876/0 (socket says 172.21.15.153:33876) 2024-04-05T02:52:26.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a62ffd700 1 -- 172.21.15.153:0/4146844268 learned_addr learned my addr 172.21.15.153:0/4146844268 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a62ffd700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a540094f0 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a62ffd700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a540090b0 tx=0x7f9a54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a54010760 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9a54010d50 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64111e80 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a5401ace0 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.379+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a641129f0 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9a5402d3f0 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a4b7fe700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a4c0393a0 con 0x7f9a4c0367e0 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9a5404a0b0 con 0x7f9a64071290 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a627fc700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.380+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.581+0000 7f9a627fc700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:26.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.581+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.982+0000 7f9a627fc700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:20.982+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:21.783+0000 7f9a627fc700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:21.783+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:23.384+0000 7f9a627fc700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/458993476 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:23.384+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:24.550+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f9a5402d6f0 con 0x7f9a64071290 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:24.550+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 msgr2=0x7f9a4c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:24.550+0000 7f9a4b7fe700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.551+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f9a5402da70 con 0x7f9a64071290 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.551+0000 7f9a4b7fe700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 0x7f9a4c03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.551+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a4c0393a0 con 0x7f9a4c039fc0 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.552+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 0x7f9a4c03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:26.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.552+0000 7f9a627fc700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 0x7f9a4c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9a580039d0 tx=0x7f9a58009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.553+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f9a4c0393a0 con 0x7f9a4c039fc0 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9a64112310 con 0x7f9a4c039fc0 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/4146844268 <== mgr.14120 v2:172.21.15.153:6800/1535236796 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9a64112310 con 0x7f9a4c039fc0 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 msgr2=0x7f9a4c03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 0x7f9a4c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9a580039d0 tx=0x7f9a58009350 comp rx=0 tx=0).stop 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 msgr2=0x7f9a64111740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a540090b0 tx=0x7f9a54009250 comp rx=0 tx=0).stop 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 shutdown_connections 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f9a4c039fc0 0x7f9a4c03c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a64071290 0x7f9a64111740 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 --2- 172.21.15.153:0/4146844268 >> [v2:172.21.15.153:6800/458993476,v1:172.21.15.153:6801/458993476] conn(0x7f9a4c0367e0 0x7f9a4c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 >> 172.21.15.153:0/4146844268 conn(0x7f9a640fcfb0 msgr2=0x7f9a640fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.559+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 shutdown_connections 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:25.560+0000 7f9a698ec700 1 -- 172.21.15.153:0/4146844268 wait complete. 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 5 is available 2024-04-05T02:52:26.113 INFO:teuthology.orchestra.run.smithi153.stdout:Setting orchestrator backend to cephadm... 2024-04-05T02:52:27.313 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:25] ENGINE Bus STARTING 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:25] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:25] ENGINE Bus STARTED 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:27.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:26 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb9db92700 1 Processor -- start 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb9db92700 1 -- start start 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb9db92700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb9db92700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98103270 con 0x7fcb98102920 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb977fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb977fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33968/0 (socket says 172.21.15.153:33968) 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb977fe700 1 -- 172.21.15.153:0/1249615528 learned_addr learned my addr 172.21.15.153:0/1249615528 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.514+0000 7fcb977fe700 1 -- 172.21.15.153:0/1249615528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb981033b0 con 0x7fcb98102920 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb977fe700 1 --2- 172.21.15.153:0/1249615528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ed8c8f059a85ba server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:27.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb967fc700 1 -- 172.21.15.153:0/1249615528 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8800b860 con 0x7fcb98102920 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb967fc700 1 -- 172.21.15.153:0/1249615528 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcb8800be50 con 0x7fcb98102920 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb967fc700 1 -- 172.21.15.153:0/1249615528 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb88010380 con 0x7fcb98102920 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 -- 172.21.15.153:0/1249615528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 msgr2=0x7fcb98102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1249615528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb88009250 comp rx=0 tx=0).stop 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 -- 172.21.15.153:0/1249615528 shutdown_connections 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1249615528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb98102d30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 -- 172.21.15.153:0/1249615528 >> 172.21.15.153:0/1249615528 conn(0x7fcb980fe0a0 msgr2=0x7fcb981004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.515+0000 7fcb9db92700 1 -- 172.21.15.153:0/1249615528 shutdown_connections 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb9db92700 1 -- 172.21.15.153:0/1249615528 wait complete. 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb9db92700 1 Processor -- start 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb9db92700 1 -- start start 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb9db92700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb9db92700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98111c00 con 0x7fcb98102920 2024-04-05T02:52:27.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.516+0000 7fcb977fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb977fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33984/0 (socket says 172.21.15.153:33984) 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb977fe700 1 -- 172.21.15.153:0/1319359669 learned_addr learned my addr 172.21.15.153:0/1319359669 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb977fe700 1 -- 172.21.15.153:0/1319359669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880094f0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb977fe700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb880107c0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcb88010db0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb98111e00 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb8801ad00 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.517+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb98112970 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.518+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fcb880193c0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.518+0000 7fcb7ffff700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 0x7fcb80038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.518+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9805f5f0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.518+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcb8804a4c0 con 0x7fcb98102920 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.518+0000 7fcb96ffd700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 0x7fcb80038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:27.323 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.519+0000 7fcb96ffd700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 0x7fcb80038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009990 tx=0x7fcb8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.523+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcb88022930 con 0x7fcb98102920 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.718+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb980ff350 con 0x7fcb800366b0 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.728+0000 7fcb7ffff700 1 -- 172.21.15.153:0/1319359669 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcb980ff350 con 0x7fcb800366b0 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 msgr2=0x7fcb80038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 0x7fcb80038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009990 tx=0x7fcb8c006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 msgr2=0x7fcb981116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000c00 tx=0x7fcb88009250 comp rx=0 tx=0).stop 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 shutdown_connections 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcb800366b0 0x7fcb80038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 --2- 172.21.15.153:0/1319359669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb98102920 0x7fcb981116c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 >> 172.21.15.153:0/1319359669 conn(0x7fcb980fe0a0 msgr2=0x7fcb980fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 shutdown_connections 2024-04-05T02:52:27.324 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:26.731+0000 7fcb9db92700 1 -- 172.21.15.153:0/1319359669 wait complete. 2024-04-05T02:52:28.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:27 smithi153 ceph-mon[29381]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:28.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:27 smithi153 ceph-mon[29381]: mgrmap e8: smithi153.npzyqy(active, since 2s) 2024-04-05T02:52:28.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-05T02:52:28.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.703+0000 7f68e03ee700 1 Processor -- start 2024-04-05T02:52:28.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.703+0000 7f68e03ee700 1 -- start start 2024-04-05T02:52:28.453 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.703+0000 7f68e03ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.703+0000 7f68e03ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d81032c0 con 0x7f68d8102970 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68de18a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68de18a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34000/0 (socket says 172.21.15.153:34000) 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68de18a700 1 -- 172.21.15.153:0/2332408006 learned_addr learned my addr 172.21.15.153:0/2332408006 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68de18a700 1 -- 172.21.15.153:0/2332408006 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d8103400 con 0x7f68d8102970 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68de18a700 1 --2- 172.21.15.153:0/2332408006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68d40098e0 tx=0x7f68d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede0a1bcb5b5fc20 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68dd188700 1 -- 172.21.15.153:0/2332408006 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d400b860 con 0x7f68d8102970 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68dd188700 1 -- 172.21.15.153:0/2332408006 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f68d400be50 con 0x7f68d8102970 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.704+0000 7f68dd188700 1 -- 172.21.15.153:0/2332408006 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d4010380 con 0x7f68d8102970 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- 172.21.15.153:0/2332408006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 msgr2=0x7f68d8102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:28.454 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 --2- 172.21.15.153:0/2332408006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68d40098e0 tx=0x7f68d4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- 172.21.15.153:0/2332408006 shutdown_connections 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 --2- 172.21.15.153:0/2332408006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8102d80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- 172.21.15.153:0/2332408006 >> 172.21.15.153:0/2332408006 conn(0x7f68d80fe090 msgr2=0x7f68d8100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- 172.21.15.153:0/2332408006 shutdown_connections 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- 172.21.15.153:0/2332408006 wait complete. 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 Processor -- start 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.705+0000 7f68e03ee700 1 -- start start 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68e03ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68e03ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8111c60 con 0x7f68d8102970 2024-04-05T02:52:28.455 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68de18a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68de18a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34004/0 (socket says 172.21.15.153:34004) 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68de18a700 1 -- 172.21.15.153:0/1185940223 learned_addr learned my addr 172.21.15.153:0/1185940223 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68de18a700 1 -- 172.21.15.153:0/1185940223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40094f0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68de18a700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68d400ba90 tx=0x7f68d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d40107c0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d8111e60 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f68d4010db0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d401add0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.706+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d81129d0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.707+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f68d402e440 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.707+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d805f5f0 con 0x7f68d8102970 2024-04-05T02:52:28.456 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.707+0000 7f68ceffd700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 0x7f68c4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.708+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f68d404a760 con 0x7f68d8102970 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.708+0000 7f68dd989700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 0x7f68c4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.708+0000 7f68dd989700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 0x7f68c4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009990 tx=0x7f68c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.712+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f68d4018350 con 0x7f68d8102970 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.903+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f68d80ff3e0 con 0x7f68c4036790 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.904+0000 7f68ceffd700 1 -- 172.21.15.153:0/1185940223 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f68d80ff3e0 con 0x7f68c4036790 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 msgr2=0x7f68c4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 0x7f68c4038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009990 tx=0x7f68c8006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 msgr2=0x7f68d8111720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68d400ba90 tx=0x7f68d4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 shutdown_connections 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f68c4036790 0x7f68c4038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 --2- 172.21.15.153:0/1185940223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68d8102970 0x7f68d8111720 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:28.458 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 >> 172.21.15.153:0/1185940223 conn(0x7f68d80fe090 msgr2=0x7f68d80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:28.458 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 shutdown_connections 2024-04-05T02:52:28.458 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:27.906+0000 7f68e03ee700 1 -- 172.21.15.153:0/1185940223 wait complete. 2024-04-05T02:52:28.458 INFO:teuthology.orchestra.run.smithi153.stdout:Generating ssh key... 2024-04-05T02:52:29.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:28 smithi153 ceph-mon[29381]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:29.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:29 smithi153 ceph-mon[29381]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:29.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:29 smithi153 ceph-mon[29381]: Generating ssh key... 2024-04-05T02:52:29.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:29 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:29.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:29 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f2940154700 1 Processor -- start 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f2940154700 1 -- start start 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f2940154700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f2940154700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2938105a30 con 0x7f2938076ba0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f293def0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f293def0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39588/0 (socket says 172.21.15.153:39588) 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.855+0000 7f293def0700 1 -- 172.21.15.153:0/1814413835 learned_addr learned my addr 172.21.15.153:0/1814413835 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.856+0000 7f293def0700 1 -- 172.21.15.153:0/1814413835 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2938105b70 con 0x7f2938076ba0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.856+0000 7f293def0700 1 --2- 172.21.15.153:0/1814413835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2930009e50 tx=0x7f2930009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f46bc953d31746 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.856+0000 7f293ceee700 1 -- 172.21.15.153:0/1814413835 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29300092e0 con 0x7f2938076ba0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.856+0000 7f293ceee700 1 -- 172.21.15.153:0/1814413835 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2930012430 con 0x7f2938076ba0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.856+0000 7f293ceee700 1 -- 172.21.15.153:0/1814413835 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29300075c0 con 0x7f2938076ba0 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- 172.21.15.153:0/1814413835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 msgr2=0x7f29381054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 --2- 172.21.15.153:0/1814413835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2930009e50 tx=0x7f2930009840 comp rx=0 tx=0).stop 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- 172.21.15.153:0/1814413835 shutdown_connections 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 --2- 172.21.15.153:0/1814413835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381054f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- 172.21.15.153:0/1814413835 >> 172.21.15.153:0/1814413835 conn(0x7f29380fcf40 msgr2=0x7f29380ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- 172.21.15.153:0/1814413835 shutdown_connections 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- 172.21.15.153:0/1814413835 wait complete. 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 Processor -- start 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- start start 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.857+0000 7f2940154700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2938111c30 con 0x7f2938076ba0 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f293def0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f293def0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39594/0 (socket says 172.21.15.153:39594) 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f293def0700 1 -- 172.21.15.153:0/867284627 learned_addr learned my addr 172.21.15.153:0/867284627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f293def0700 1 -- 172.21.15.153:0/867284627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930009ae0 con 0x7f2938076ba0 2024-04-05T02:52:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f293def0700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2930009fd0 tx=0x7f2930009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29300092e0 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2930007d10 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2938111e30 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f293001c880 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.858+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29381129a0 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.859+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2930019070 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.859+0000 7f292e7fc700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 0x7f2924038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.859+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2930049de0 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.859+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293805f5f0 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.859+0000 7f293d6ef700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 0x7f2924038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.860+0000 7f293d6ef700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 0x7f2924038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2928009990 tx=0x7f2928006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:28.864+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2930017080 con 0x7f2938076ba0 2024-04-05T02:52:29.979 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.051+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f29380fe2a0 con 0x7f29240367e0 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.384+0000 7f292e7fc700 1 -- 172.21.15.153:0/867284627 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f29380fe2a0 con 0x7f29240367e0 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.386+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 msgr2=0x7f2924038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.386+0000 7f2940154700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 0x7f2924038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2928009990 tx=0x7f2928006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.386+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 msgr2=0x7f29381116f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.386+0000 7f2940154700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2930009fd0 tx=0x7f2930009840 comp rx=0 tx=0).stop 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 shutdown_connections 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f29240367e0 0x7f2924038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 --2- 172.21.15.153:0/867284627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2938076ba0 0x7f29381116f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 >> 172.21.15.153:0/867284627 conn(0x7f29380fcf40 msgr2=0x7f29380fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 shutdown_connections 2024-04-05T02:52:29.980 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:29.387+0000 7f2940154700 1 -- 172.21.15.153:0/867284627 wait complete. 2024-04-05T02:52:31.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vrlDrO0CTiQbBZS1J5k8t+ZkoFsLpbnxPb5yE9PxUz0ArfVs+I2MsyJj7H3wVmApVsy80KiulGYjuT2G0kvm3K1pZbt/2ebUD6eqYAFhEANrn4R6iI36KNQwwNjGx+qhboYiZtv29hdn6jEZu6lccB+jOUls27KcNue/9mySVEwYOhTbzL1biNRZ3isB6AISpoZngxXdliCZWf68KOPwRpkUhJUw81T8ppBobGEX+HGXiUvnDsK9/nYIsxhrmCyb2aU+Q64LNF1RANz7M4fPZriURfLrDOR9CQ2lHw+SIhOV8oM79OogEdrOgjP71aiPIjBfc4+AYXLGYn5dQUBBIX5bBQ2vz1/ri2Z8M34JISdq8nJwImV9h6OeQxsL3HDj6/a5C5VomQ/Ss2IPeEpQ3tY2xORLUJ9EbvGGOj2daRrhu+LKAbgf3Cesj3gdAPw+OsSv7Ns6px8BCQoqai4P/9MK0egC6SnoTWiHbWnOYxYsNpf++IWjRvB/617tcis= ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.410+0000 7fae56b94700 1 Processor -- start 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.410+0000 7fae56b94700 1 -- start start 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae56b94700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae56b94700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50071640 con 0x7fae50071230 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae54930700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae54930700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39610/0 (socket says 172.21.15.153:39610) 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae54930700 1 -- 172.21.15.153:0/2989599994 learned_addr learned my addr 172.21.15.153:0/2989599994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae54930700 1 -- 172.21.15.153:0/2989599994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50105aa0 con 0x7fae50071230 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.411+0000 7fae54930700 1 --2- 172.21.15.153:0/2989599994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae440098e0 tx=0x7fae44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9295a60a733bc070 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae4f7fe700 1 -- 172.21.15.153:0/2989599994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4400b860 con 0x7fae50071230 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae4f7fe700 1 -- 172.21.15.153:0/2989599994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae4400be50 con 0x7fae50071230 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae4f7fe700 1 -- 172.21.15.153:0/2989599994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae44010380 con 0x7fae50071230 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 -- 172.21.15.153:0/2989599994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 msgr2=0x7fae50105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:31.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 --2- 172.21.15.153:0/2989599994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae440098e0 tx=0x7fae44009250 comp rx=0 tx=0).stop 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 -- 172.21.15.153:0/2989599994 shutdown_connections 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 --2- 172.21.15.153:0/2989599994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50105560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 -- 172.21.15.153:0/2989599994 >> 172.21.15.153:0/2989599994 conn(0x7fae500fcfb0 msgr2=0x7fae500ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 -- 172.21.15.153:0/2989599994 shutdown_connections 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.412+0000 7fae56b94700 1 -- 172.21.15.153:0/2989599994 wait complete. 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.413+0000 7fae56b94700 1 Processor -- start 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.413+0000 7fae56b94700 1 -- start start 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.413+0000 7fae56b94700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.413+0000 7fae56b94700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50111c70 con 0x7fae50071230 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.413+0000 7fae54930700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae54930700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39616/0 (socket says 172.21.15.153:39616) 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae54930700 1 -- 172.21.15.153:0/1408312968 learned_addr learned my addr 172.21.15.153:0/1408312968 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae54930700 1 -- 172.21.15.153:0/1408312968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae440094f0 con 0x7fae50071230 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae54930700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae44011bd0 tx=0x7fae44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:31.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae440107c0 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae50111e70 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fae44010db0 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4401ad60 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.414+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae501129e0 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.415+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fae4402e440 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.415+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5005f5f0 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.415+0000 7fae4d7fa700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 0x7fae40038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.415+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fae4404ac60 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.415+0000 7fae4ffff700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 0x7fae40038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.416+0000 7fae4ffff700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 0x7fae40038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009990 tx=0x7fae3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.420+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fae44023d60 con 0x7fae50071230 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.607+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fae500fe380 con 0x7fae40036790 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.607+0000 7fae4d7fa700 1 -- 172.21.15.153:0/1408312968 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fae500fe380 con 0x7fae40036790 2024-04-05T02:52:31.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 msgr2=0x7fae40038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 0x7fae40038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009990 tx=0x7fae3c006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 msgr2=0x7fae50111730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae44011bd0 tx=0x7fae44009250 comp rx=0 tx=0).stop 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 shutdown_connections 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fae40036790 0x7fae40038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 --2- 172.21.15.153:0/1408312968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae50071230 0x7fae50111730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 >> 172.21.15.153:0/1408312968 conn(0x7fae500fcfb0 msgr2=0x7fae500fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 shutdown_connections 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:30.610+0000 7fae56b94700 1 -- 172.21.15.153:0/1408312968 wait complete. 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:Adding key to root@localhost authorized_keys... 2024-04-05T02:52:31.179 INFO:teuthology.orchestra.run.smithi153.stdout:Adding host smithi153... 2024-04-05T02:52:31.366 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:31 smithi153 ceph-mon[29381]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:32.347 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:32 smithi153 ceph-mon[29381]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:33.287 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:33 smithi153 ceph-mon[29381]: Deploying cephadm binary to smithi153 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Added host 'smithi153' with addr '172.21.15.153' 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f11030c8700 1 Processor -- start 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f11030c8700 1 -- start start 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f11030c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f11030c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1032e0 con 0x7f10fc076ba0 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f1100e64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f1100e64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39624/0 (socket says 172.21.15.153:39624) 2024-04-05T02:52:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.552+0000 7f1100e64700 1 -- 172.21.15.153:0/2407984699 learned_addr learned my addr 172.21.15.153:0/2407984699 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f1100e64700 1 -- 172.21.15.153:0/2407984699 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc107e30 con 0x7f10fc076ba0 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f1100e64700 1 --2- 172.21.15.153:0/2407984699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10f00098e0 tx=0x7f10f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e55b87f62315698 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f10fb7fe700 1 -- 172.21.15.153:0/2407984699 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f000b860 con 0x7f10fc076ba0 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f10fb7fe700 1 -- 172.21.15.153:0/2407984699 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f10f000be50 con 0x7f10fc076ba0 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f10fb7fe700 1 -- 172.21.15.153:0/2407984699 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0010380 con 0x7f10fc076ba0 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f11030c8700 1 -- 172.21.15.153:0/2407984699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 msgr2=0x7f10fc1078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f11030c8700 1 --2- 172.21.15.153:0/2407984699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10f00098e0 tx=0x7f10f0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.553+0000 7f11030c8700 1 -- 172.21.15.153:0/2407984699 shutdown_connections 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 --2- 172.21.15.153:0/2407984699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1078f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 -- 172.21.15.153:0/2407984699 >> 172.21.15.153:0/2407984699 conn(0x7f10fc0fcf40 msgr2=0x7f10fc0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 -- 172.21.15.153:0/2407984699 shutdown_connections 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 -- 172.21.15.153:0/2407984699 wait complete. 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 Processor -- start 2024-04-05T02:52:34.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 -- start start 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f11030c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc113e00 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.554+0000 7f1100e64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f1100e64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39634/0 (socket says 172.21.15.153:39634) 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f1100e64700 1 -- 172.21.15.153:0/3198314970 learned_addr learned my addr 172.21.15.153:0/3198314970 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f1100e64700 1 -- 172.21.15.153:0/3198314970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00094f0 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f1100e64700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10f0006b00 tx=0x7f10f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f00107c0 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f10f0010db0 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f001acf0 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc114000 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.555+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc114b70 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.556+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f10f00193c0 con 0x7f10fc076ba0 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.556+0000 7f10f97fa700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 0x7f10ec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:34.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.556+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc05f5f0 con 0x7f10fc076ba0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.556+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f10f004a820 con 0x7f10fc076ba0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.556+0000 7f10fbfff700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 0x7f10ec038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.557+0000 7f10fbfff700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 0x7f10ec038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009990 tx=0x7f10e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.561+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f10f0010b00 con 0x7f10fc076ba0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:31.749+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}) v1 -- 0x7f10fc0fe2a0 con 0x7f10ec0367e0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.830+0000 7f10f97fa700 1 -- 172.21.15.153:0/3198314970 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f10fc0fe2a0 con 0x7f10ec0367e0 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 msgr2=0x7f10ec038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 0x7f10ec038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009990 tx=0x7f10e8006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 msgr2=0x7f10fc1138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10f0006b00 tx=0x7f10f0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 shutdown_connections 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f10ec0367e0 0x7f10ec038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 --2- 172.21.15.153:0/3198314970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10fc076ba0 0x7f10fc1138c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 >> 172.21.15.153:0/3198314970 conn(0x7f10fc0fcf40 msgr2=0x7f10fc0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:34.412 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 shutdown_connections 2024-04-05T02:52:34.413 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:33.833+0000 7f11030c8700 1 -- 172.21.15.153:0/3198314970 wait complete. 2024-04-05T02:52:34.413 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mon service with default placement... 2024-04-05T02:52:34.956 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:34 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:34.956 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:34 smithi153 ceph-mon[29381]: Added host smithi153 2024-04-05T02:52:34.956 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:34 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.911+0000 7f6ec0eec700 1 Processor -- start 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.911+0000 7f6ec0eec700 1 -- start start 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.911+0000 7f6ec0eec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.911+0000 7f6ec0eec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc103260 con 0x7f6ebc102910 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eba59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eba59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39648/0 (socket says 172.21.15.153:39648) 2024-04-05T02:52:36.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eba59c700 1 -- 172.21.15.153:0/4067702316 learned_addr learned my addr 172.21.15.153:0/4067702316 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eba59c700 1 -- 172.21.15.153:0/4067702316 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc1033a0 con 0x7f6ebc102910 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eba59c700 1 --2- 172.21.15.153:0/4067702316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0098e0 tx=0x7f6eac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9f6b1f6fe8705e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eb9d9b700 1 -- 172.21.15.153:0/4067702316 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac00b860 con 0x7f6ebc102910 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eb9d9b700 1 -- 172.21.15.153:0/4067702316 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6eac00be50 con 0x7f6ebc102910 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.912+0000 7f6eb9d9b700 1 -- 172.21.15.153:0/4067702316 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac010380 con 0x7f6ebc102910 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- 172.21.15.153:0/4067702316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 msgr2=0x7f6ebc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/4067702316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0098e0 tx=0x7f6eac009250 comp rx=0 tx=0).stop 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- 172.21.15.153:0/4067702316 shutdown_connections 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/4067702316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc102d20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- 172.21.15.153:0/4067702316 >> 172.21.15.153:0/4067702316 conn(0x7f6ebc0fe070 msgr2=0x7f6ebc1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- 172.21.15.153:0/4067702316 shutdown_connections 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- 172.21.15.153:0/4067702316 wait complete. 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 Processor -- start 2024-04-05T02:52:36.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.913+0000 7f6ec0eec700 1 -- start start 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6ec0eec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6ec0eec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc111c40 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eba59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eba59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39652/0 (socket says 172.21.15.153:39652) 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eba59c700 1 -- 172.21.15.153:0/64038588 learned_addr learned my addr 172.21.15.153:0/64038588 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eba59c700 1 -- 172.21.15.153:0/64038588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac0094f0 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eba59c700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6eac000c00 tx=0x7f6eac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac0107c0 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6eac010db0 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc111e40 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6eac01ad00 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.914+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc1129b0 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.915+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6eac0193c0 con 0x7f6ebc102910 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.915+0000 7f6eb37fe700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 0x7f6ea8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:36.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.915+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6eac04a6e0 con 0x7f6ebc102910 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.915+0000 7f6eb1d9b700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 0x7f6ea8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.915+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc05f5f0 con 0x7f6ebc102910 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.916+0000 7f6eb1d9b700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 0x7f6ea8038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009990 tx=0x7f6ea4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:34.920+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6eac0196c0 con 0x7f6ebc102910 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.125+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ebc0ff340 con 0x7f6ea80367e0 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.132+0000 7f6eb37fe700 1 -- 172.21.15.153:0/64038588 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ebc0ff340 con 0x7f6ea80367e0 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 msgr2=0x7f6ea8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 0x7f6ea8038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009990 tx=0x7f6ea4006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 msgr2=0x7f6ebc111700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6eac000c00 tx=0x7f6eac009250 comp rx=0 tx=0).stop 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 shutdown_connections 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f6ea80367e0 0x7f6ea8038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 --2- 172.21.15.153:0/64038588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ebc102910 0x7f6ebc111700 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 >> 172.21.15.153:0/64038588 conn(0x7f6ebc0fe070 msgr2=0x7f6ebc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 shutdown_connections 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:35.134+0000 7f6ec0eec700 1 -- 172.21.15.153:0/64038588 wait complete. 2024-04-05T02:52:36.251 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mgr service with default placement... 2024-04-05T02:52:36.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:36 smithi153 ceph-mon[29381]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:36.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:36 smithi153 ceph-mon[29381]: Saving service mon spec with placement count:5 2024-04-05T02:52:36.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:36 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:37.932 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:37 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:37.933 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:37 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:37.933 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:37 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1bb4b7700 1 Processor -- start 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1bb4b7700 1 -- start start 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1bb4b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1bb4b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4070db0 con 0x7fc1b4103f00 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1b9253700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1b9253700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39656/0 (socket says 172.21.15.153:39656) 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.099+0000 7fc1b9253700 1 -- 172.21.15.153:0/2043256237 learned_addr learned my addr 172.21.15.153:0/2043256237 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:37.941 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.100+0000 7fc1b9253700 1 -- 172.21.15.153:0/2043256237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4104850 con 0x7fc1b4103f00 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.100+0000 7fc1b9253700 1 --2- 172.21.15.153:0/2043256237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00098e0 tx=0x7fc1b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7468d2f34a425b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.100+0000 7fc1abfff700 1 -- 172.21.15.153:0/2043256237 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b000b860 con 0x7fc1b4103f00 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.100+0000 7fc1abfff700 1 -- 172.21.15.153:0/2043256237 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1b000be90 con 0x7fc1b4103f00 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.100+0000 7fc1abfff700 1 -- 172.21.15.153:0/2043256237 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b00103c0 con 0x7fc1b4103f00 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/2043256237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 msgr2=0x7fc1b4104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/2043256237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1b00098e0 tx=0x7fc1b0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/2043256237 shutdown_connections 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/2043256237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b4104310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/2043256237 >> 172.21.15.153:0/2043256237 conn(0x7fc1b40fc700 msgr2=0x7fc1b40feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/2043256237 shutdown_connections 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/2043256237 wait complete. 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 Processor -- start 2024-04-05T02:52:37.942 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.101+0000 7fc1bb4b7700 1 -- start start 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1bb4b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1bb4b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4111c00 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1b9253700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1b9253700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39658/0 (socket says 172.21.15.153:39658) 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1b9253700 1 -- 172.21.15.153:0/39900701 learned_addr learned my addr 172.21.15.153:0/39900701 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1b9253700 1 -- 172.21.15.153:0/39900701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00094f0 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1b9253700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0011680 tx=0x7fc1b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b0010890 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1b0010ec0 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b4111e00 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b00243f0 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.102+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b4112970 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.103+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc1b002d3f0 con 0x7fc1b4103f00 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.103+0000 7fc1a9ffb700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 0x7fc1a0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:37.943 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.104+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc1b004a250 con 0x7fc1b4103f00 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.104+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b405f5f0 con 0x7fc1b4103f00 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.104+0000 7fc1b8a52700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 0x7fc1a0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.104+0000 7fc1b8a52700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 0x7fc1a0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009990 tx=0x7fc1a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.108+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc1b002b8e0 con 0x7fc1b4103f00 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.334+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1b40fd960 con 0x7fc1a00367e0 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.342+0000 7fc1a9ffb700 1 -- 172.21.15.153:0/39900701 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc1b40fd960 con 0x7fc1a00367e0 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 msgr2=0x7fc1a0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 0x7fc1a0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009990 tx=0x7fc1a4006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 msgr2=0x7fc1b41116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0011680 tx=0x7fc1b0009250 comp rx=0 tx=0).stop 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 shutdown_connections 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc1a00367e0 0x7fc1a0038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:37.944 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 --2- 172.21.15.153:0/39900701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1b4103f00 0x7fc1b41116c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:37.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 >> 172.21.15.153:0/39900701 conn(0x7fc1b40fc700 msgr2=0x7fc1b40fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:37.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 shutdown_connections 2024-04-05T02:52:37.945 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:37.344+0000 7fc1bb4b7700 1 -- 172.21.15.153:0/39900701 wait complete. 2024-04-05T02:52:37.945 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying crash service with default placement... 2024-04-05T02:52:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:38 smithi153 ceph-mon[29381]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:38.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:38 smithi153 ceph-mon[29381]: Saving service mgr spec with placement count:2 2024-04-05T02:52:38.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:38 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:39.281 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-05T02:52:39.281 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.340+0000 7f44b7ed0700 1 Processor -- start 2024-04-05T02:52:39.281 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.340+0000 7f44b7ed0700 1 -- start start 2024-04-05T02:52:39.281 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b7ed0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b7ed0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0102180 con 0x7f44b0101830 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b5c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b5c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39662/0 (socket says 172.21.15.153:39662) 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b5c6c700 1 -- 172.21.15.153:0/3294419542 learned_addr learned my addr 172.21.15.153:0/3294419542 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b5c6c700 1 -- 172.21.15.153:0/3294419542 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b01022c0 con 0x7f44b0101830 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b5c6c700 1 --2- 172.21.15.153:0/3294419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00bb80 tx=0x7f44ac00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71a75c017eaa638 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b4c6a700 1 -- 172.21.15.153:0/3294419542 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44ac00dd00 con 0x7f44b0101830 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b4c6a700 1 -- 172.21.15.153:0/3294419542 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ac00de60 con 0x7f44b0101830 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.341+0000 7f44b4c6a700 1 -- 172.21.15.153:0/3294419542 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44ac018920 con 0x7f44b0101830 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 -- 172.21.15.153:0/3294419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 msgr2=0x7f44b0101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/3294419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00bb80 tx=0x7f44ac00b4f0 comp rx=0 tx=0).stop 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 -- 172.21.15.153:0/3294419542 shutdown_connections 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/3294419542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b0101c40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 -- 172.21.15.153:0/3294419542 >> 172.21.15.153:0/3294419542 conn(0x7f44b00fcfb0 msgr2=0x7f44b00ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:39.282 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 -- 172.21.15.153:0/3294419542 shutdown_connections 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.342+0000 7f44b7ed0700 1 -- 172.21.15.153:0/3294419542 wait complete. 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b7ed0700 1 Processor -- start 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b7ed0700 1 -- start start 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b7ed0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b7ed0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b010fa40 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b5c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b5c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39674/0 (socket says 172.21.15.153:39674) 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b5c6c700 1 -- 172.21.15.153:0/2741252634 learned_addr learned my addr 172.21.15.153:0/2741252634 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b5c6c700 1 -- 172.21.15.153:0/2741252634 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac00b790 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b5c6c700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0106a0 tx=0x7f44ac00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44ac010c30 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ac01d050 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b010fc40 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.343+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44ac022590 con 0x7f44b0101830 2024-04-05T02:52:39.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.344+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b01107b0 con 0x7f44b0101830 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f44ac023470 con 0x7f44b0101830 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b005f5f0 con 0x7f44b0101830 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44a67fc700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 0x7f449c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f44ac014070 con 0x7f44b0101830 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44b546b700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 0x7f449c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.345+0000 7f44b546b700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 0x7f449c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009990 tx=0x7f44a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.349+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f44ac022810 con 0x7f44b0101830 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.568+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f44b0110bc0 con 0x7f449c036790 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.572+0000 7f44a67fc700 1 -- 172.21.15.153:0/2741252634 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f44b0110bc0 con 0x7f449c036790 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 msgr2=0x7f449c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 0x7f449c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009990 tx=0x7f44a0006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 msgr2=0x7f44b010f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0106a0 tx=0x7f44ac00b4f0 comp rx=0 tx=0).stop 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 shutdown_connections 2024-04-05T02:52:39.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f449c036790 0x7f449c038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:39.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 --2- 172.21.15.153:0/2741252634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44b0101830 0x7f44b010f500 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:39.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.574+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 >> 172.21.15.153:0/2741252634 conn(0x7f44b00fcfb0 msgr2=0x7f44b0105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:39.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.575+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 shutdown_connections 2024-04-05T02:52:39.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:38.575+0000 7f44b7ed0700 1 -- 172.21.15.153:0/2741252634 wait complete. 2024-04-05T02:52:39.285 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying prometheus service with default placement... 2024-04-05T02:52:39.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:39 smithi153 ceph-mon[29381]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:39.954 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:39 smithi153 ceph-mon[29381]: Saving service crash spec with placement * 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7ead0700 1 Processor -- start 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7ead0700 1 -- start start 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7ead0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7ead0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780790a0 con 0x7feb78078c90 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7c86c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7c86c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42404/0 (socket says 172.21.15.153:42404) 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.987+0000 7feb7c86c700 1 -- 172.21.15.153:0/926761706 learned_addr learned my addr 172.21.15.153:0/926761706 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.988+0000 7feb7c86c700 1 -- 172.21.15.153:0/926761706 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb780777d0 con 0x7feb78078c90 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.988+0000 7feb7c86c700 1 --2- 172.21.15.153:0/926761706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098e0 tx=0x7feb6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80375675613f2be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.988+0000 7feb777fe700 1 -- 172.21.15.153:0/926761706 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c00b860 con 0x7feb78078c90 2024-04-05T02:52:40.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.988+0000 7feb777fe700 1 -- 172.21.15.153:0/926761706 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6c00be90 con 0x7feb78078c90 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.988+0000 7feb777fe700 1 -- 172.21.15.153:0/926761706 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c0103c0 con 0x7feb78078c90 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- 172.21.15.153:0/926761706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 msgr2=0x7feb78077290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 --2- 172.21.15.153:0/926761706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098e0 tx=0x7feb6c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- 172.21.15.153:0/926761706 shutdown_connections 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 --2- 172.21.15.153:0/926761706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb78077290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- 172.21.15.153:0/926761706 >> 172.21.15.153:0/926761706 conn(0x7feb780fe080 msgr2=0x7feb78100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- 172.21.15.153:0/926761706 shutdown_connections 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- 172.21.15.153:0/926761706 wait complete. 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 Processor -- start 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.989+0000 7feb7ead0700 1 -- start start 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7ead0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7ead0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78113e10 con 0x7feb78078c90 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7c86c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7c86c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42416/0 (socket says 172.21.15.153:42416) 2024-04-05T02:52:40.834 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7c86c700 1 -- 172.21.15.153:0/4040864477 learned_addr learned my addr 172.21.15.153:0/4040864477 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7c86c700 1 -- 172.21.15.153:0/4040864477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c0094f0 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7c86c700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0094c0 tx=0x7feb6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c010680 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb78114010 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb6c010cb0 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c01ad60 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.990+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb78114b80 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.991+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7feb6c025400 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.992+0000 7feb757fa700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 0x7feb68038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.991+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7805f5f0 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.992+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7feb6c04a400 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.992+0000 7feb77fff700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 0x7feb68038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.992+0000 7feb77fff700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 0x7feb68038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feb64009990 tx=0x7feb64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:39.996+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7feb6c02c6c0 con 0x7feb78078c90 2024-04-05T02:52:40.835 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.212+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7feb780ff2e0 con 0x7feb680367e0 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.230+0000 7feb757fa700 1 -- 172.21.15.153:0/4040864477 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7feb780ff2e0 con 0x7feb680367e0 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 msgr2=0x7feb68038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 0x7feb68038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feb64009990 tx=0x7feb64006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 msgr2=0x7feb781138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0094c0 tx=0x7feb6c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 shutdown_connections 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7feb680367e0 0x7feb68038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 --2- 172.21.15.153:0/4040864477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb78078c90 0x7feb781138d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 >> 172.21.15.153:0/4040864477 conn(0x7feb780fe080 msgr2=0x7feb780febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 shutdown_connections 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:40.232+0000 7feb7ead0700 1 -- 172.21.15.153:0/4040864477 wait complete. 2024-04-05T02:52:40.836 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying grafana service with default placement... 2024-04-05T02:52:41.144 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:40 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:41.144 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:40 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:41.144 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:40 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:42.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:41 smithi153 ceph-mon[29381]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:42.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:41 smithi153 ceph-mon[29381]: Saving service prometheus spec with placement count:1 2024-04-05T02:52:42.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:41 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:42.862 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-05T02:52:42.862 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.243+0000 7f8532f15700 1 Processor -- start 2024-04-05T02:52:42.862 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.243+0000 7f8532f15700 1 -- start start 2024-04-05T02:52:42.862 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.243+0000 7f8532f15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:42.862 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.243+0000 7f8532f15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c1059f0 con 0x7f852c076ce0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f8530cb1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f8530cb1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42430/0 (socket says 172.21.15.153:42430) 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f8530cb1700 1 -- 172.21.15.153:0/2924850502 learned_addr learned my addr 172.21.15.153:0/2924850502 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f8530cb1700 1 -- 172.21.15.153:0/2924850502 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c105b30 con 0x7f852c076ce0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f8530cb1700 1 --2- 172.21.15.153:0/2924850502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85200098e0 tx=0x7f8520009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcce6c3e35131b05 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f852b7fe700 1 -- 172.21.15.153:0/2924850502 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f852000b860 con 0x7f852c076ce0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f852b7fe700 1 -- 172.21.15.153:0/2924850502 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f852000be90 con 0x7f852c076ce0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.244+0000 7f852b7fe700 1 -- 172.21.15.153:0/2924850502 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85200103c0 con 0x7f852c076ce0 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 -- 172.21.15.153:0/2924850502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 msgr2=0x7f852c1054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 --2- 172.21.15.153:0/2924850502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85200098e0 tx=0x7f8520009250 comp rx=0 tx=0).stop 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 -- 172.21.15.153:0/2924850502 shutdown_connections 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 --2- 172.21.15.153:0/2924850502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c1054b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 -- 172.21.15.153:0/2924850502 >> 172.21.15.153:0/2924850502 conn(0x7f852c0fcf40 msgr2=0x7f852c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 -- 172.21.15.153:0/2924850502 shutdown_connections 2024-04-05T02:52:42.863 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.245+0000 7f8532f15700 1 -- 172.21.15.153:0/2924850502 wait complete. 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.246+0000 7f8532f15700 1 Processor -- start 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.246+0000 7f8532f15700 1 -- start start 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.246+0000 7f8532f15700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.246+0000 7f8532f15700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c103340 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8530cb1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8530cb1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42444/0 (socket says 172.21.15.153:42444) 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8530cb1700 1 -- 172.21.15.153:0/3011594015 learned_addr learned my addr 172.21.15.153:0/3011594015 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8530cb1700 1 -- 172.21.15.153:0/3011594015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85200094f0 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8530cb1700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85200094c0 tx=0x7f8520009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8520010890 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8520010ec0 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c1013b0 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85200243f0 con 0x7f852c076ce0 2024-04-05T02:52:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.247+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c1018a0 con 0x7f852c076ce0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f852001a7e0 con 0x7f852c076ce0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c05f5f0 con 0x7f852c076ce0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f85297fa700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 0x7f851c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f852004a210 con 0x7f852c076ce0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f852bfff700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 0x7f851c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.249+0000 7f852bfff700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 0x7f851c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8518009990 tx=0x7f8518006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.254+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f852002b8e0 con 0x7f852c076ce0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.461+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f852c101df0 con 0x7f851c0367e0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.466+0000 7f85297fa700 1 -- 172.21.15.153:0/3011594015 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f852c101df0 con 0x7f851c0367e0 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 msgr2=0x7f851c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 0x7f851c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8518009990 tx=0x7f8518006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 msgr2=0x7f852c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85200094c0 tx=0x7f8520009250 comp rx=0 tx=0).stop 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 shutdown_connections 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f851c0367e0 0x7f851c038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:42.865 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 --2- 172.21.15.153:0/3011594015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f852c076ce0 0x7f852c102e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:42.866 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 >> 172.21.15.153:0/3011594015 conn(0x7f852c0fcf40 msgr2=0x7f852c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:42.866 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 shutdown_connections 2024-04-05T02:52:42.866 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:41.468+0000 7f8532f15700 1 -- 172.21.15.153:0/3011594015 wait complete. 2024-04-05T02:52:42.866 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying node-exporter service with default placement... 2024-04-05T02:52:43.148 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:42 smithi153 ceph-mon[29381]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:43.149 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:42 smithi153 ceph-mon[29381]: Saving service grafana spec with placement count:1 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.246+0000 7fc80f883700 1 Processor -- start 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.246+0000 7fc80f883700 1 -- start start 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80f883700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80f883700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8081032c0 con 0x7fc808102970 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80d61f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80d61f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42460/0 (socket says 172.21.15.153:42460) 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80d61f700 1 -- 172.21.15.153:0/3995934257 learned_addr learned my addr 172.21.15.153:0/3995934257 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:44.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80d61f700 1 -- 172.21.15.153:0/3995934257 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808103400 con 0x7fc808102970 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc80d61f700 1 --2- 172.21.15.153:0/3995934257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8040098e0 tx=0x7fc804009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88235e0e735cf2e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc7fffff700 1 -- 172.21.15.153:0/3995934257 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80400b860 con 0x7fc808102970 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc7fffff700 1 -- 172.21.15.153:0/3995934257 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc80400be90 con 0x7fc808102970 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.247+0000 7fc7fffff700 1 -- 172.21.15.153:0/3995934257 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8040103c0 con 0x7fc808102970 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 -- 172.21.15.153:0/3995934257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 msgr2=0x7fc808102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 --2- 172.21.15.153:0/3995934257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc8040098e0 tx=0x7fc804009250 comp rx=0 tx=0).stop 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 -- 172.21.15.153:0/3995934257 shutdown_connections 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 --2- 172.21.15.153:0/3995934257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808102d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 -- 172.21.15.153:0/3995934257 >> 172.21.15.153:0/3995934257 conn(0x7fc8080fe090 msgr2=0x7fc808100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 -- 172.21.15.153:0/3995934257 shutdown_connections 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.248+0000 7fc80f883700 1 -- 172.21.15.153:0/3995934257 wait complete. 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80f883700 1 Processor -- start 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80f883700 1 -- start start 2024-04-05T02:52:44.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80f883700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80f883700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808111ca0 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80d61f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80d61f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42476/0 (socket says 172.21.15.153:42476) 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80d61f700 1 -- 172.21.15.153:0/2128221708 learned_addr learned my addr 172.21.15.153:0/2128221708 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80d61f700 1 -- 172.21.15.153:0/2128221708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8040094f0 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc80d61f700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc804006b00 tx=0x7fc804009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.249+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc804010840 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.250+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc804010e70 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.250+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc808111ea0 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.250+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80401ae70 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.250+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808112a10 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.251+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc804024480 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.251+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80805f5f0 con 0x7fc808102970 2024-04-05T02:52:44.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.251+0000 7fc7fdffb700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 0x7fc7f4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.251+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc80404ac40 con 0x7fc808102970 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.251+0000 7fc80ce1e700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 0x7fc7f4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.252+0000 7fc80ce1e700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 0x7fc7f4038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009990 tx=0x7fc7f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.256+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc80402e440 con 0x7fc808102970 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.470+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8080ff490 con 0x7fc7f40367e0 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.476+0000 7fc7fdffb700 1 -- 172.21.15.153:0/2128221708 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc8080ff490 con 0x7fc7f40367e0 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 msgr2=0x7fc7f4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 0x7fc7f4038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009990 tx=0x7fc7f8006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 msgr2=0x7fc808111760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc804006b00 tx=0x7fc804009250 comp rx=0 tx=0).stop 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 shutdown_connections 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fc7f40367e0 0x7fc7f4038c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 --2- 172.21.15.153:0/2128221708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc808102970 0x7fc808111760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:44.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 >> 172.21.15.153:0/2128221708 conn(0x7fc8080fe090 msgr2=0x7fc8080fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:44.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 shutdown_connections 2024-04-05T02:52:44.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:43.478+0000 7fc80f883700 1 -- 172.21.15.153:0/2128221708 wait complete. 2024-04-05T02:52:44.089 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying alertmanager service with default placement... 2024-04-05T02:52:44.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:44 smithi153 ceph-mon[29381]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:44.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:44 smithi153 ceph-mon[29381]: Saving service node-exporter spec with placement * 2024-04-05T02:52:44.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:44 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a36642700 1 Processor -- start 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a36642700 1 -- start start 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a36642700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a36642700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a301026e0 con 0x7f2a30101d90 2024-04-05T02:52:45.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a2ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a2ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42490/0 (socket says 172.21.15.153:42490) 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a2ffff700 1 -- 172.21.15.153:0/561706385 learned_addr learned my addr 172.21.15.153:0/561706385 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a2ffff700 1 -- 172.21.15.153:0/561706385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a30102820 con 0x7f2a30101d90 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.493+0000 7f2a2ffff700 1 --2- 172.21.15.153:0/561706385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a200098e0 tx=0x7f2a20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253f8b30c898b706 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a2effd700 1 -- 172.21.15.153:0/561706385 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a2000b860 con 0x7f2a30101d90 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a2effd700 1 -- 172.21.15.153:0/561706385 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a2000be90 con 0x7f2a30101d90 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a2effd700 1 -- 172.21.15.153:0/561706385 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a200103c0 con 0x7f2a30101d90 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 -- 172.21.15.153:0/561706385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 msgr2=0x7f2a301021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:45.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 --2- 172.21.15.153:0/561706385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a200098e0 tx=0x7f2a20009250 comp rx=0 tx=0).stop 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 -- 172.21.15.153:0/561706385 shutdown_connections 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 --2- 172.21.15.153:0/561706385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a301021a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 -- 172.21.15.153:0/561706385 >> 172.21.15.153:0/561706385 conn(0x7f2a300fc700 msgr2=0x7f2a300feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 -- 172.21.15.153:0/561706385 shutdown_connections 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.494+0000 7f2a36642700 1 -- 172.21.15.153:0/561706385 wait complete. 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a36642700 1 Processor -- start 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a36642700 1 -- start start 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a36642700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a36642700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a30118220 con 0x7f2a30101d90 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a2ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a2ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42506/0 (socket says 172.21.15.153:42506) 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a2ffff700 1 -- 172.21.15.153:0/945878708 learned_addr learned my addr 172.21.15.153:0/945878708 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.495+0000 7f2a2ffff700 1 -- 172.21.15.153:0/945878708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a200094f0 con 0x7f2a30101d90 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a2ffff700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a200090b0 tx=0x7f2a20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:45.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a20010840 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a20010e70 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a30118420 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a20024e40 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.496+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a30118f90 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2a2001a4c0 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a2cff9700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 0x7f2a1803ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2a2004a780 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3005f5f0 con 0x7f2a30101d90 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a2f7fe700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 0x7f2a1803ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.497+0000 7f2a2f7fe700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 0x7f2a1803ae30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009990 tx=0x7f2a24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:45.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.501+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2a20018940 con 0x7f2a30101d90 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.698+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a301192e0 con 0x7f2a18038980 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.705+0000 7f2a2cff9700 1 -- 172.21.15.153:0/945878708 <== mgr.14120 v2:172.21.15.153:6800/1535236796 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f2a301192e0 con 0x7f2a18038980 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 msgr2=0x7f2a1803ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 0x7f2a1803ae30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009990 tx=0x7f2a24006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 msgr2=0x7f2a30117ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a200090b0 tx=0x7f2a20009250 comp rx=0 tx=0).stop 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 shutdown_connections 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f2a18038980 0x7f2a1803ae30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 --2- 172.21.15.153:0/945878708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a30101d90 0x7f2a30117ce0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 >> 172.21.15.153:0/945878708 conn(0x7f2a300fc700 msgr2=0x7f2a300699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 shutdown_connections 2024-04-05T02:52:45.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:44.707+0000 7f2a36642700 1 -- 172.21.15.153:0/945878708 wait complete. 2024-04-05T02:52:46.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:45 smithi153 ceph-mon[29381]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:52:46.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:45 smithi153 ceph-mon[29381]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:46.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:45 smithi153 ceph-mon[29381]: Saving service alertmanager spec with placement count:1 2024-04-05T02:52:46.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:45 smithi153 ceph-mon[29381]: from='mgr.14120 172.21.15.153:0/2720622591' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.838+0000 7f69e1462700 1 Processor -- start 2024-04-05T02:52:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.838+0000 7f69e1462700 1 -- start start 2024-04-05T02:52:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.838+0000 7f69e1462700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.838+0000 7f69e1462700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc071640 con 0x7f69dc071230 2024-04-05T02:52:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.838+0000 7f69daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42514/0 (socket says 172.21.15.153:42514) 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69daffd700 1 -- 172.21.15.153:0/3550639548 learned_addr learned my addr 172.21.15.153:0/3550639548 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69daffd700 1 -- 172.21.15.153:0/3550639548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc105aa0 con 0x7f69dc071230 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69daffd700 1 --2- 172.21.15.153:0/3550639548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0098e0 tx=0x7f69cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e4df7b948627d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69da7fc700 1 -- 172.21.15.153:0/3550639548 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc00b860 con 0x7f69dc071230 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69da7fc700 1 -- 172.21.15.153:0/3550639548 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69cc00be90 con 0x7f69dc071230 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.839+0000 7f69da7fc700 1 -- 172.21.15.153:0/3550639548 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc0103c0 con 0x7f69dc071230 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- 172.21.15.153:0/3550639548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 msgr2=0x7f69dc105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 --2- 172.21.15.153:0/3550639548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0098e0 tx=0x7f69cc009250 comp rx=0 tx=0).stop 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- 172.21.15.153:0/3550639548 shutdown_connections 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 --2- 172.21.15.153:0/3550639548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc105560 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- 172.21.15.153:0/3550639548 >> 172.21.15.153:0/3550639548 conn(0x7f69dc0fcfb0 msgr2=0x7f69dc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- 172.21.15.153:0/3550639548 shutdown_connections 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- 172.21.15.153:0/3550639548 wait complete. 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 Processor -- start 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- start start 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.840+0000 7f69e1462700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc111c70 con 0x7f69dc071230 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42524/0 (socket says 172.21.15.153:42524) 2024-04-05T02:52:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69daffd700 1 -- 172.21.15.153:0/3527340382 learned_addr learned my addr 172.21.15.153:0/3527340382 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69daffd700 1 -- 172.21.15.153:0/3527340382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0094f0 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69daffd700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0094c0 tx=0x7f69cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc0118f0 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69cc010d60 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc111e70 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc01a430 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.841+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc1129e0 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.842+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f69cc01a590 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.842+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc05f5f0 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.842+0000 7f69d3fff700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 0x7f69c8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.842+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f69cc04a4a0 con 0x7f69dc071230 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.842+0000 7f69d27fc700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 0x7f69c8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.843+0000 7f69d27fc700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 0x7f69c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009990 tx=0x7f69c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:46.584 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:45.847+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f69cc018db0 con 0x7f69dc071230 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.031+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f69dc04bed0 con 0x7f69dc071230 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.034+0000 7f69d3fff700 1 -- 172.21.15.153:0/3527340382 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f69cc010ed0 con 0x7f69dc071230 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 msgr2=0x7f69c8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 0x7f69c8038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009990 tx=0x7f69c4006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 msgr2=0x7f69dc111730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0094c0 tx=0x7f69cc009250 comp rx=0 tx=0).stop 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 shutdown_connections 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f69c8036790 0x7f69c8038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 --2- 172.21.15.153:0/3527340382 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc071230 0x7f69dc111730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 >> 172.21.15.153:0/3527340382 conn(0x7f69dc0fcfb0 msgr2=0x7f69dc0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 shutdown_connections 2024-04-05T02:52:46.585 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.037+0000 7f69e1462700 1 -- 172.21.15.153:0/3527340382 wait complete. 2024-04-05T02:52:47.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:47 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3527340382' entity='client.admin' 2024-04-05T02:52:47.754 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0fd14e700 1 Processor -- start 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0fd14e700 1 -- start start 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0fd14e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0fd14e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f81020f0 con 0x7fb0f81017a0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42532/0 (socket says 172.21.15.153:42532) 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.991+0000 7fb0f6d9d700 1 -- 172.21.15.153:0/4117601251 learned_addr learned my addr 172.21.15.153:0/4117601251 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0f6d9d700 1 -- 172.21.15.153:0/4117601251 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f8102230 con 0x7fb0f81017a0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0f6d9d700 1 --2- 172.21.15.153:0/4117601251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009e50 tx=0x7fb0e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d17488892dadb29 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0f5d9b700 1 -- 172.21.15.153:0/4117601251 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e80092e0 con 0x7fb0f81017a0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0f5d9b700 1 -- 172.21.15.153:0/4117601251 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0e8012430 con 0x7fb0f81017a0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0f5d9b700 1 -- 172.21.15.153:0/4117601251 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e80075a0 con 0x7fb0f81017a0 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4117601251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 msgr2=0x7fb0f8101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:47.755 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.992+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4117601251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009e50 tx=0x7fb0e8009840 comp rx=0 tx=0).stop 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4117601251 shutdown_connections 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4117601251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f8101bb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4117601251 >> 172.21.15.153:0/4117601251 conn(0x7fb0f80fcf40 msgr2=0x7fb0f80ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4117601251 shutdown_connections 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4117601251 wait complete. 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 Processor -- start 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- start start 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.993+0000 7fb0fd14e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f8110b30 con 0x7fb0f81017a0 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0f6d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42546/0 (socket says 172.21.15.153:42546) 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0f6d9d700 1 -- 172.21.15.153:0/4290724054 learned_addr learned my addr 172.21.15.153:0/4290724054 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0f6d9d700 1 -- 172.21.15.153:0/4290724054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e8009ae0 con 0x7fb0f81017a0 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0f6d9d700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009fd0 tx=0x7fb0e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e80092e0 con 0x7fb0f81017a0 2024-04-05T02:52:47.756 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0e8007ce0 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f8110d30 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e8025990 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.994+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81118a0 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.995+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb0e8019050 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.995+0000 7fb0df7fe700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 0x7fb0e0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.995+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f805f5f0 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.995+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb0e804a100 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.995+0000 7fb0f659c700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 0x7fb0e0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:46.996+0000 7fb0f659c700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 0x7fb0e0038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009990 tx=0x7fb0ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.000+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb0e807a190 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.188+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb0f804bed0 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.194+0000 7fb0df7fe700 1 -- 172.21.15.153:0/4290724054 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb0e8020070 con 0x7fb0f81017a0 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 msgr2=0x7fb0e0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 0x7fb0e0038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009990 tx=0x7fb0ec006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:47.757 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 msgr2=0x7fb0f81105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009fd0 tx=0x7fb0e8009840 comp rx=0 tx=0).stop 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 shutdown_connections 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fb0e00367e0 0x7fb0e0038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 --2- 172.21.15.153:0/4290724054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0f81017a0 0x7fb0f81105f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 >> 172.21.15.153:0/4290724054 conn(0x7fb0f80fcf40 msgr2=0x7fb0f80fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 shutdown_connections 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:47.196+0000 7fb0fd14e700 1 -- 172.21.15.153:0/4290724054 wait complete. 2024-04-05T02:52:47.758 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling the dashboard module... 2024-04-05T02:52:48.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:48 smithi153 ceph-mon[29381]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:52:48.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:48 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/4290724054' entity='client.admin' 2024-04-05T02:52:49.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:49 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1601035293' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.155+0000 7f899cd52700 1 Processor -- start 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899cd52700 1 -- start start 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899cd52700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899cd52700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89981021a0 con 0x7f8998101850 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42552/0 (socket says 172.21.15.153:42552) 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899659c700 1 -- 172.21.15.153:0/938877153 learned_addr learned my addr 172.21.15.153:0/938877153 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.156+0000 7f899659c700 1 -- 172.21.15.153:0/938877153 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89981022e0 con 0x7f8998101850 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899659c700 1 --2- 172.21.15.153:0/938877153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f898c0098e0 tx=0x7f898c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b543f4642bb6d9ad server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899559a700 1 -- 172.21.15.153:0/938877153 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c00b860 con 0x7f8998101850 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899559a700 1 -- 172.21.15.153:0/938877153 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f898c00be90 con 0x7f8998101850 2024-04-05T02:52:49.629 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899559a700 1 -- 172.21.15.153:0/938877153 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c0103c0 con 0x7f8998101850 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 -- 172.21.15.153:0/938877153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 msgr2=0x7f8998101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 --2- 172.21.15.153:0/938877153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f898c0098e0 tx=0x7f898c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 -- 172.21.15.153:0/938877153 shutdown_connections 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 --2- 172.21.15.153:0/938877153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998101c60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 -- 172.21.15.153:0/938877153 >> 172.21.15.153:0/938877153 conn(0x7f89980fcf90 msgr2=0x7f89980ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 -- 172.21.15.153:0/938877153 shutdown_connections 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.157+0000 7f899cd52700 1 -- 172.21.15.153:0/938877153 wait complete. 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899cd52700 1 Processor -- start 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899cd52700 1 -- start start 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899cd52700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899cd52700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8998114420 con 0x7f8998101850 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42560/0 (socket says 172.21.15.153:42560) 2024-04-05T02:52:49.630 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899659c700 1 -- 172.21.15.153:0/1601035293 learned_addr learned my addr 172.21.15.153:0/1601035293 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.158+0000 7f899659c700 1 -- 172.21.15.153:0/1601035293 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0094f0 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f899659c700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f898c0090b0 tx=0x7f898c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c010840 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f898c010e70 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8998114560 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c024e40 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.159+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89981149f0 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f898c01a530 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899804bed0 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f8986ffd700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 0x7f8980038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f898c04a960 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f8995d9b700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 0x7f8980038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.160+0000 7f8995d9b700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 0x7f8980038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8988009990 tx=0x7f8988006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.164+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f898c018350 con 0x7f8998101850 2024-04-05T02:52:49.631 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:48.406+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8998114f40 con 0x7f8998101850 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.044+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f898c024710 con 0x7f8998101850 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.045+0000 7f8986ffd700 1 -- 172.21.15.153:0/1601035293 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f898c04c690 con 0x7f8998101850 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 msgr2=0x7f8980038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 0x7f8980038bf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8988009990 tx=0x7f8988006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 msgr2=0x7f8998076b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f898c0090b0 tx=0x7f898c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 shutdown_connections 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f8980036740 0x7f8980038bf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 --2- 172.21.15.153:0/1601035293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8998101850 0x7f8998076b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 >> 172.21.15.153:0/1601035293 conn(0x7f89980fcf90 msgr2=0x7f89980fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 shutdown_connections 2024-04-05T02:52:49.632 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:49.048+0000 7f899cd52700 1 -- 172.21.15.153:0/1601035293 wait complete. 2024-04-05T02:52:50.259 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:50 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1601035293' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-05T02:52:50.259 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:50 smithi153 ceph-mon[29381]: mgrmap e9: smithi153.npzyqy(active, since 24s) 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.npzyqy", 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T02:52:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.053+0000 7fcc7146a700 1 Processor -- start 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.053+0000 7fcc7146a700 1 -- start start 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.053+0000 7fcc7146a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.053+0000 7fcc7146a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c1026a0 con 0x7fcc6c101d50 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41232/0 (socket says 172.21.15.153:41232) 2024-04-05T02:52:50.912 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc6affd700 1 -- 172.21.15.153:0/110561627 learned_addr learned my addr 172.21.15.153:0/110561627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc6affd700 1 -- 172.21.15.153:0/110561627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c1027e0 con 0x7fcc6c101d50 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc6affd700 1 --2- 172.21.15.153:0/110561627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0098e0 tx=0x7fcc5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7eb4ccdf267d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc69ffb700 1 -- 172.21.15.153:0/110561627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c00b860 con 0x7fcc6c101d50 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc69ffb700 1 -- 172.21.15.153:0/110561627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc5c00be90 con 0x7fcc6c101d50 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.054+0000 7fcc69ffb700 1 -- 172.21.15.153:0/110561627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c0103c0 con 0x7fcc6c101d50 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 -- 172.21.15.153:0/110561627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 msgr2=0x7fcc6c102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 --2- 172.21.15.153:0/110561627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0098e0 tx=0x7fcc5c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 -- 172.21.15.153:0/110561627 shutdown_connections 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 --2- 172.21.15.153:0/110561627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c102160 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 -- 172.21.15.153:0/110561627 >> 172.21.15.153:0/110561627 conn(0x7fcc6c0fc6c0 msgr2=0x7fcc6c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:50.913 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 -- 172.21.15.153:0/110561627 shutdown_connections 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.055+0000 7fcc7146a700 1 -- 172.21.15.153:0/110561627 wait complete. 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc7146a700 1 Processor -- start 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc7146a700 1 -- start start 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc7146a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc7146a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c1143b0 con 0x7fcc6c101d50 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41248/0 (socket says 172.21.15.153:41248) 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc6affd700 1 -- 172.21.15.153:0/506137989 learned_addr learned my addr 172.21.15.153:0/506137989 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.056+0000 7fcc6affd700 1 -- 172.21.15.153:0/506137989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0094f0 con 0x7fcc6c101d50 2024-04-05T02:52:50.914 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc6affd700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c011c10 tx=0x7fcc5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c010890 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c1144f0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc5c010ec0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc5c0243f0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.057+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c1149e0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c04bed0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fcc5c02d3f0 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc537fe700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 0x7fcc54038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcc5c04a430 con 0x7fcc6c101d50 2024-04-05T02:52:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc6a7fc700 1 -- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 msgr2=0x7fcc54038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.058+0000 7fcc6a7fc700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 0x7fcc54038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.063+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcc5c018b40 con 0x7fcc6c101d50 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.259+0000 7fcc6a7fc700 1 -- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 msgr2=0x7fcc54038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.259+0000 7fcc6a7fc700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 0x7fcc54038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.304+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcc6c114c20 con 0x7fcc6c101d50 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.304+0000 7fcc537fe700 1 -- 172.21.15.153:0/506137989 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fcc5c018b40 con 0x7fcc6c101d50 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 msgr2=0x7fcc54038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 0x7fcc54038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 msgr2=0x7fcc6c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c011c10 tx=0x7fcc5c009250 comp rx=0 tx=0).stop 2024-04-05T02:52:50.916 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 shutdown_connections 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7fcc540367e0 0x7fcc54038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 --2- 172.21.15.153:0/506137989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc6c101d50 0x7fcc6c1015b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 >> 172.21.15.153:0/506137989 conn(0x7fcc6c0fc6c0 msgr2=0x7fcc6c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 shutdown_connections 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:50.307+0000 7fcc7146a700 1 -- 172.21.15.153:0/506137989 wait complete. 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-04-05T02:52:50.917 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 9... 2024-04-05T02:52:51.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:51 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/506137989' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T02:52:55.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T02:52:55.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: Activating manager daemon smithi153.npzyqy 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: osdmap e3: 0 total, 0 up, 0 in 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: mgrmap e10: smithi153.npzyqy(active, starting, since 0.00402609s) 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: Manager daemon smithi153.npzyqy is now available 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T02:52:55.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:55 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T02:52:57.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:57 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:55] ENGINE Bus STARTING 2024-04-05T02:52:57.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:57 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:56] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T02:52:57.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:57 smithi153 ceph-mon[29381]: [05/Apr/2024:02:52:56] ENGINE Bus STARTED 2024-04-05T02:52:57.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:57 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:57.188 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:57 smithi153 ceph-mon[29381]: mgrmap e11: smithi153.npzyqy(active, since 1.00554s) 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90d02f2700 1 Processor -- start 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90d02f2700 1 -- start start 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90d02f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:57.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90d02f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8070db0 con 0x7f90c8103f30 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90ce08e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90ce08e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41262/0 (socket says 172.21.15.153:41262) 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.411+0000 7f90ce08e700 1 -- 172.21.15.153:0/1226443400 learned_addr learned my addr 172.21.15.153:0/1226443400 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.412+0000 7f90ce08e700 1 -- 172.21.15.153:0/1226443400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c8104880 con 0x7f90c8103f30 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.412+0000 7f90ce08e700 1 --2- 172.21.15.153:0/1226443400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90c40098e0 tx=0x7f90c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59ce0eb467554a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.412+0000 7f90cd08c700 1 -- 172.21.15.153:0/1226443400 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c400b860 con 0x7f90c8103f30 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.412+0000 7f90cd08c700 1 -- 172.21.15.153:0/1226443400 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c400be90 con 0x7f90c8103f30 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.412+0000 7f90cd08c700 1 -- 172.21.15.153:0/1226443400 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c40103c0 con 0x7f90c8103f30 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- 172.21.15.153:0/1226443400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 msgr2=0x7f90c8104340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 --2- 172.21.15.153:0/1226443400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f90c40098e0 tx=0x7f90c4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- 172.21.15.153:0/1226443400 shutdown_connections 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 --2- 172.21.15.153:0/1226443400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8104340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- 172.21.15.153:0/1226443400 >> 172.21.15.153:0/1226443400 conn(0x7f90c80fc730 msgr2=0x7f90c80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:57.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- 172.21.15.153:0/1226443400 shutdown_connections 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- 172.21.15.153:0/1226443400 wait complete. 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 Processor -- start 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- start start 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.413+0000 7f90d02f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8103b80 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90ce08e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90ce08e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41268/0 (socket says 172.21.15.153:41268) 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90ce08e700 1 -- 172.21.15.153:0/3506723561 learned_addr learned my addr 172.21.15.153:0/3506723561 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90ce08e700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40094f0 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90ce08e700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90c4011c10 tx=0x7f90c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c4010890 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90c4010ec0 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c8101bf0 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90c40243f0 con 0x7f90c8103f30 2024-04-05T02:52:57.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.414+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c81020e0 con 0x7f90c8103f30 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f90c402d3f0 con 0x7f90c8103f30 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90beffd700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90b40393a0 con 0x7f90b40367e0 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f90c404a400 con 0x7f90c8103f30 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90cd88d700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.415+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.616+0000 7f90cd88d700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:51.616+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:52.017+0000 7f90cd88d700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:52.017+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:52.818+0000 7f90cd88d700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:52.818+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:54.419+0000 7f90cd88d700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1535236796 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:54.419+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T02:52:57.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:55.614+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f90c402d740 con 0x7f90c8103f30 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:55.614+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 msgr2=0x7f90b4038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:55.614+0000 7f90beffd700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.615+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f90c404b780 con 0x7f90c8103f30 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.615+0000 7f90beffd700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 0x7f90b403c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.615+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90b40393a0 con 0x7f90b4039fd0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.616+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 0x7f90b403c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.616+0000 7f90cd88d700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 0x7f90b403c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f90b80039d0 tx=0x7f90b80092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.617+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f90b40393a0 con 0x7f90b4039fd0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f90c804bed0 con 0x7f90b4039fd0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90beffd700 1 -- 172.21.15.153:0/3506723561 <== mgr.14162 v2:172.21.15.153:6800/4288916206 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f90c804bed0 con 0x7f90b4039fd0 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 msgr2=0x7f90b403c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 0x7f90b403c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f90b80039d0 tx=0x7f90b80092f0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 msgr2=0x7f90c8103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:57.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90c4011c10 tx=0x7f90c4009250 comp rx=0 tx=0).stop 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 shutdown_connections 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f90b4039fd0 0x7f90b403c3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90c8103f30 0x7f90c8103640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 --2- 172.21.15.153:0/3506723561 >> [v2:172.21.15.153:6800/1535236796,v1:172.21.15.153:6801/1535236796] conn(0x7f90b40367e0 0x7f90b4038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 >> 172.21.15.153:0/3506723561 conn(0x7f90c80fc730 msgr2=0x7f90c80fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 shutdown_connections 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:56.623+0000 7f90d02f2700 1 -- 172.21.15.153:0/3506723561 wait complete. 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 9 is available 2024-04-05T02:52:57.204 INFO:teuthology.orchestra.run.smithi153.stdout:Generating a dashboard self-signed certificate... 2024-04-05T02:52:58.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:58 smithi153 ceph-mon[29381]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T02:52:58.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:58 smithi153 ceph-mon[29381]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T02:52:58.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:58 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:58.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:58 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:58.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:58 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:52:58.511 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-05T02:52:58.511 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.586+0000 7f96d2e75700 1 Processor -- start 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.586+0000 7f96d2e75700 1 -- start start 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.586+0000 7f96d2e75700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.586+0000 7f96d2e75700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc076d20 con 0x7f96cc103470 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96d0c11700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96d0c11700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41344/0 (socket says 172.21.15.153:41344) 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96d0c11700 1 -- 172.21.15.153:0/713175564 learned_addr learned my addr 172.21.15.153:0/713175564 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96d0c11700 1 -- 172.21.15.153:0/713175564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc076e60 con 0x7f96cc103470 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96d0c11700 1 --2- 172.21.15.153:0/713175564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009e50 tx=0x7f96c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee502dffeabb1296 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96cb7fe700 1 -- 172.21.15.153:0/713175564 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c40092e0 con 0x7f96cc103470 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96cb7fe700 1 -- 172.21.15.153:0/713175564 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96c4012430 con 0x7f96cc103470 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.587+0000 7f96cb7fe700 1 -- 172.21.15.153:0/713175564 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c40075a0 con 0x7f96cc103470 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- 172.21.15.153:0/713175564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 msgr2=0x7f96cc105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 --2- 172.21.15.153:0/713175564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009e50 tx=0x7f96c4009840 comp rx=0 tx=0).stop 2024-04-05T02:52:58.512 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- 172.21.15.153:0/713175564 shutdown_connections 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 --2- 172.21.15.153:0/713175564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc105850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- 172.21.15.153:0/713175564 >> 172.21.15.153:0/713175564 conn(0x7f96cc0fcf30 msgr2=0x7f96cc0ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- 172.21.15.153:0/713175564 shutdown_connections 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- 172.21.15.153:0/713175564 wait complete. 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 Processor -- start 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.588+0000 7f96d2e75700 1 -- start start 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d2e75700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d2e75700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc111c80 con 0x7f96cc103470 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d0c11700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d0c11700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41354/0 (socket says 172.21.15.153:41354) 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d0c11700 1 -- 172.21.15.153:0/562192588 learned_addr learned my addr 172.21.15.153:0/562192588 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d0c11700 1 -- 172.21.15.153:0/562192588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4009ae0 con 0x7f96cc103470 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d0c11700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009fd0 tx=0x7f96c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c40092e0 con 0x7f96cc103470 2024-04-05T02:52:58.513 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96c4007ce0 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc111e80 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96c40259c0 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.589+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc1129f0 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.590+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f96c4019050 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.590+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc112310 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.590+0000 7f96c97fa700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 0x7f96b8038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.590+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f96c4049f70 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.591+0000 7f96cbfff700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 0x7f96b8038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.591+0000 7f96cbfff700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 0x7f96b8038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009990 tx=0x7f96b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.595+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f96c402d9a0 con 0x7f96cc103470 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.810+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f96cc0fe2f0 con 0x7f96b8036710 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.958+0000 7f96c97fa700 1 -- 172.21.15.153:0/562192588 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f96cc0fe2f0 con 0x7f96b8036710 2024-04-05T02:52:58.514 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 msgr2=0x7f96b8038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 0x7f96b8038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009990 tx=0x7f96b4006e40 comp rx=0 tx=0).stop 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 msgr2=0x7f96cc111740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009fd0 tx=0x7f96c4009840 comp rx=0 tx=0).stop 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 shutdown_connections 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f96b8036710 0x7f96b8038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 --2- 172.21.15.153:0/562192588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96cc103470 0x7f96cc111740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 >> 172.21.15.153:0/562192588 conn(0x7f96cc0fcf30 msgr2=0x7f96cc0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:52:58.515 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 shutdown_connections 2024-04-05T02:52:58.516 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:57.961+0000 7f96d2e75700 1 -- 172.21.15.153:0/562192588 wait complete. 2024-04-05T02:52:58.516 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial admin user... 2024-04-05T02:52:59.106 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:59 smithi153 ceph-mon[29381]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:52:59.107 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:59 smithi153 ceph-mon[29381]: mgrmap e12: smithi153.npzyqy(active, since 2s) 2024-04-05T02:52:59.107 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:52:59 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zXDHqPaS96kQ2EUzPS7Kg.yu0qKpSNW3ZlPUkQzOtU87RqtnbC556", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712285579, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.948+0000 7f7a6e0f6700 1 Processor -- start 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.948+0000 7f7a6e0f6700 1 -- start start 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a6e0f6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a6e0f6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a681059f0 con 0x7f7a68076ce0 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35024/0 (socket says 172.21.15.153:35024) 2024-04-05T02:53:00.197 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a677fe700 1 -- 172.21.15.153:0/496380232 learned_addr learned my addr 172.21.15.153:0/496380232 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a677fe700 1 -- 172.21.15.153:0/496380232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68105b30 con 0x7f7a68076ce0 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.949+0000 7f7a677fe700 1 --2- 172.21.15.153:0/496380232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0098e0 tx=0x7f7a5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49741b8725a75bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a667fc700 1 -- 172.21.15.153:0/496380232 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c00b860 con 0x7f7a68076ce0 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a667fc700 1 -- 172.21.15.153:0/496380232 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a5c00be90 con 0x7f7a68076ce0 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a667fc700 1 -- 172.21.15.153:0/496380232 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c0103c0 con 0x7f7a68076ce0 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/496380232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 msgr2=0x7f7a681054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/496380232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0098e0 tx=0x7f7a5c009250 comp rx=0 tx=0).stop 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/496380232 shutdown_connections 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/496380232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a681054b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/496380232 >> 172.21.15.153:0/496380232 conn(0x7f7a680fcf40 msgr2=0x7f7a680ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/496380232 shutdown_connections 2024-04-05T02:53:00.198 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.950+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/496380232 wait complete. 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a6e0f6700 1 Processor -- start 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a6e0f6700 1 -- start start 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a6e0f6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a6e0f6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a68103340 con 0x7f7a68076ce0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35030/0 (socket says 172.21.15.153:35030) 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.951+0000 7f7a677fe700 1 -- 172.21.15.153:0/3479771122 learned_addr learned my addr 172.21.15.153:0/3479771122 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a677fe700 1 -- 172.21.15.153:0/3479771122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c0094f0 con 0x7f7a68076ce0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a677fe700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c006d40 tx=0x7f7a5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c010840 con 0x7f7a68076ce0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a5c010e70 con 0x7f7a68076ce0 2024-04-05T02:53:00.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c024ea0 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a681013b0 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.952+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a681018a0 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.953+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f7a5c01a4c0 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.953+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a68106970 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.953+0000 7f7a4ffff700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 0x7f7a50038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.954+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7a5c04a980 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.954+0000 7f7a66ffd700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 0x7f7a50038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.954+0000 7f7a66ffd700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 0x7f7a50038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009990 tx=0x7f7a58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:58.959+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7a5c010b80 con 0x7f7a68076ce0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.171+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7a6805f410 con 0x7f7a500367e0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.394+0000 7f7a4ffff700 1 -- 172.21.15.153:0/3479771122 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7a6805f410 con 0x7f7a500367e0 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 msgr2=0x7f7a50038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:00.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 0x7f7a50038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009990 tx=0x7f7a58006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 msgr2=0x7f7a68102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c006d40 tx=0x7f7a5c009250 comp rx=0 tx=0).stop 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 shutdown_connections 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7a500367e0 0x7f7a50038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 --2- 172.21.15.153:0/3479771122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a68076ce0 0x7f7a68102e00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 >> 172.21.15.153:0/3479771122 conn(0x7f7a680fcf40 msgr2=0x7f7a680fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 shutdown_connections 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:52:59.396+0000 7f7a6e0f6700 1 -- 172.21.15.153:0/3479771122 wait complete. 2024-04-05T02:53:00.201 INFO:teuthology.orchestra.run.smithi153.stdout:Fetching dashboard port number... 2024-04-05T02:53:00.474 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:00 smithi153 ceph-mon[29381]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:53:00.474 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:00 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:01.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 8443 2024-04-05T02:53:01.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8939479700 1 Processor -- start 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8939479700 1 -- start start 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8939479700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8939479700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89341021c0 con 0x7f8934101870 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35034/0 (socket says 172.21.15.153:35034) 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.575+0000 7f8932ffd700 1 -- 172.21.15.153:0/3467602555 learned_addr learned my addr 172.21.15.153:0/3467602555 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.576+0000 7f8932ffd700 1 -- 172.21.15.153:0/3467602555 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934102300 con 0x7f8934101870 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.576+0000 7f8932ffd700 1 --2- 172.21.15.153:0/3467602555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89240098e0 tx=0x7f8924009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c1357390376035 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.576+0000 7f8931ffb700 1 -- 172.21.15.153:0/3467602555 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892400b8f0 con 0x7f8934101870 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.576+0000 7f8931ffb700 1 -- 172.21.15.153:0/3467602555 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8924011420 con 0x7f8934101870 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.576+0000 7f8931ffb700 1 -- 172.21.15.153:0/3467602555 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8924010540 con 0x7f8934101870 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 -- 172.21.15.153:0/3467602555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 msgr2=0x7f8934101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 --2- 172.21.15.153:0/3467602555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89240098e0 tx=0x7f8924009250 comp rx=0 tx=0).stop 2024-04-05T02:53:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 -- 172.21.15.153:0/3467602555 shutdown_connections 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 --2- 172.21.15.153:0/3467602555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934101c80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 -- 172.21.15.153:0/3467602555 >> 172.21.15.153:0/3467602555 conn(0x7f89340fcfd0 msgr2=0x7f89340ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 -- 172.21.15.153:0/3467602555 shutdown_connections 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.577+0000 7f8939479700 1 -- 172.21.15.153:0/3467602555 wait complete. 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8939479700 1 Processor -- start 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8939479700 1 -- start start 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8939479700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8939479700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934111c50 con 0x7f8934101870 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8932ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35038/0 (socket says 172.21.15.153:35038) 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8932ffd700 1 -- 172.21.15.153:0/2751921912 learned_addr learned my addr 172.21.15.153:0/2751921912 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8932ffd700 1 -- 172.21.15.153:0/2751921912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89240094f0 con 0x7f8934101870 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.578+0000 7f8932ffd700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8924006b00 tx=0x7f8924009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.579+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8924010800 con 0x7f8934101870 2024-04-05T02:53:01.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.579+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8924010e30 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.579+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8934111e50 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.579+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892401ae40 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.579+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89341129c0 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f892402e440 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f891b7fe700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 0x7f891c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893405f5f0 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f892404a6e0 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f89327fc700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 0x7f891c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.580+0000 7f89327fc700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 0x7f891c038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8928009990 tx=0x7f8928006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.585+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8924018710 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.777+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f893404bed0 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.778+0000 7f891b7fe700 1 -- 172.21.15.153:0/2751921912 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8924018350 con 0x7f8934101870 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 msgr2=0x7f891c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 0x7f891c038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8928009990 tx=0x7f8928006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:01.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 msgr2=0x7f8934111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8924006b00 tx=0x7f8924009250 comp rx=0 tx=0).stop 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 shutdown_connections 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f891c0367e0 0x7f891c038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 --2- 172.21.15.153:0/2751921912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8934101870 0x7f8934111710 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 >> 172.21.15.153:0/2751921912 conn(0x7f89340fcfd0 msgr2=0x7f89340fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 shutdown_connections 2024-04-05T02:53:01.558 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:00.780+0000 7f8939479700 1 -- 172.21.15.153:0/2751921912 wait complete. 2024-04-05T02:53:01.561 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T02:53:01.561 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-04-05T02:53:01.569 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T02:53:01.569 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-04-05T02:53:01.569 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-04-05T02:53:01.569 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph Dashboard is now available at: 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout: URL: https://smithi153.front.sepia.ceph.com:8443/ 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout: User: admin 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout: Password: jnpdpclila 2024-04-05T02:53:01.571 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:01.572 INFO:teuthology.orchestra.run.smithi153.stdout:Saving cluster configuration to /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config directory 2024-04-05T02:53:01.572 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling autotune for osd_memory_target 2024-04-05T02:53:01.689 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:01 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2751921912' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-05T02:53:02.785 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.974+0000 7fbfea9a9700 1 Processor -- start 2024-04-05T02:53:02.785 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.974+0000 7fbfea9a9700 1 -- start start 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.974+0000 7fbfea9a9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.974+0000 7fbfea9a9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4103260 con 0x7fbfe4102910 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35042/0 (socket says 172.21.15.153:35042) 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe3fff700 1 -- 172.21.15.153:0/788872507 learned_addr learned my addr 172.21.15.153:0/788872507 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe3fff700 1 -- 172.21.15.153:0/788872507 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe41033a0 con 0x7fbfe4102910 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe3fff700 1 --2- 172.21.15.153:0/788872507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40098e0 tx=0x7fbfd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de1fbb40720a0ce server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe2ffd700 1 -- 172.21.15.153:0/788872507 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd400b860 con 0x7fbfe4102910 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe2ffd700 1 -- 172.21.15.153:0/788872507 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd400be90 con 0x7fbfe4102910 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.975+0000 7fbfe2ffd700 1 -- 172.21.15.153:0/788872507 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd40103c0 con 0x7fbfe4102910 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 -- 172.21.15.153:0/788872507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 msgr2=0x7fbfe4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:02.786 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/788872507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40098e0 tx=0x7fbfd4009250 comp rx=0 tx=0).stop 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 -- 172.21.15.153:0/788872507 shutdown_connections 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/788872507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4102d20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 -- 172.21.15.153:0/788872507 >> 172.21.15.153:0/788872507 conn(0x7fbfe40fe070 msgr2=0x7fbfe41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 -- 172.21.15.153:0/788872507 shutdown_connections 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 -- 172.21.15.153:0/788872507 wait complete. 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.976+0000 7fbfea9a9700 1 Processor -- start 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfea9a9700 1 -- start start 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfea9a9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfea9a9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4111c50 con 0x7fbfe4102910 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35044/0 (socket says 172.21.15.153:35044) 2024-04-05T02:53:02.787 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe3fff700 1 -- 172.21.15.153:0/3070680410 learned_addr learned my addr 172.21.15.153:0/3070680410 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe3fff700 1 -- 172.21.15.153:0/3070680410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40094f0 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe3fff700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40090b0 tx=0x7fbfd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd4010840 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.977+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd4010e70 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.978+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4111e50 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.978+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfd4024e90 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.978+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe41129c0 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.978+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fbfd401a4c0 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.978+0000 7fbfe0ff9700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 0x7fbfcc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.979+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbfd4024430 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.979+0000 7fbfe37fe700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 0x7fbfcc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.979+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe405f5f0 con 0x7fbfe4102910 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.979+0000 7fbfe37fe700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 0x7fbfcc038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009990 tx=0x7fbfd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:02.788 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:01.983+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbfd4018940 con 0x7fbfe4102910 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.172+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fbfe404bed0 con 0x7fbfe4102910 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.173+0000 7fbfe0ff9700 1 -- 172.21.15.153:0/3070680410 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fbfd4018350 con 0x7fbfe4102910 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 msgr2=0x7fbfcc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 0x7fbfcc038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009990 tx=0x7fbfd8006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 msgr2=0x7fbfe4111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40090b0 tx=0x7fbfd4009250 comp rx=0 tx=0).stop 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 shutdown_connections 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbfcc0367e0 0x7fbfcc038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 --2- 172.21.15.153:0/3070680410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfe4102910 0x7fbfe4111710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 >> 172.21.15.153:0/3070680410 conn(0x7fbfe40fe070 msgr2=0x7fbfe40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 shutdown_connections 2024-04-05T02:53:02.789 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:02.175+0000 7fbfea9a9700 1 -- 172.21.15.153:0/3070680410 wait complete. 2024-04-05T02:53:03.800 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:03 smithi153 ceph-mon[29381]: mgrmap e13: smithi153.npzyqy(active, since 6s) 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f983110f700 1 Processor -- start 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f983110f700 1 -- start start 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f983110f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f983110f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1020f0 con 0x7f982c1017a0 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f982ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f982ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35046/0 (socket says 172.21.15.153:35046) 2024-04-05T02:53:04.076 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f982ad9d700 1 -- 172.21.15.153:0/3223856717 learned_addr learned my addr 172.21.15.153:0/3223856717 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.196+0000 7f982ad9d700 1 -- 172.21.15.153:0/3223856717 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c102230 con 0x7f982c1017a0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f982ad9d700 1 --2- 172.21.15.153:0/3223856717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f981c00bb80 tx=0x7f981c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e3e42613a2cdcb server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f9829d9b700 1 -- 172.21.15.153:0/3223856717 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c00dd00 con 0x7f982c1017a0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f9829d9b700 1 -- 172.21.15.153:0/3223856717 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f981c00de60 con 0x7f982c1017a0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f9829d9b700 1 -- 172.21.15.153:0/3223856717 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c018920 con 0x7f982c1017a0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f983110f700 1 -- 172.21.15.153:0/3223856717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 msgr2=0x7f982c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f983110f700 1 --2- 172.21.15.153:0/3223856717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f981c00bb80 tx=0x7f981c00b4f0 comp rx=0 tx=0).stop 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f983110f700 1 -- 172.21.15.153:0/3223856717 shutdown_connections 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f983110f700 1 --2- 172.21.15.153:0/3223856717 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c101bb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.197+0000 7f983110f700 1 -- 172.21.15.153:0/3223856717 >> 172.21.15.153:0/3223856717 conn(0x7f982c0fcf40 msgr2=0x7f982c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 -- 172.21.15.153:0/3223856717 shutdown_connections 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 -- 172.21.15.153:0/3223856717 wait complete. 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 Processor -- start 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 -- start start 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f983110f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c113e00 con 0x7f982c1017a0 2024-04-05T02:53:04.077 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f982ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f982ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35050/0 (socket says 172.21.15.153:35050) 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.198+0000 7f982ad9d700 1 -- 172.21.15.153:0/2774856374 learned_addr learned my addr 172.21.15.153:0/2774856374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f982ad9d700 1 -- 172.21.15.153:0/2774856374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c00b790 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f982ad9d700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f981c0106a0 tx=0x7f981c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c018e90 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f981c010c30 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c114000 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981c021530 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.199+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c114b20 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.200+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f981c021720 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.200+0000 7f98137fe700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 0x7f9814038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.200+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c05f5f0 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.200+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f981c014070 con 0x7f982c1017a0 2024-04-05T02:53:04.078 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.200+0000 7f982a59c700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 0x7f9814038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.201+0000 7f982a59c700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 0x7f9814038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9820009990 tx=0x7f9820006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.205+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f981c01fd10 con 0x7f982c1017a0 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.499+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f982c04bed0 con 0x7f982c1017a0 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.504+0000 7f98137fe700 1 -- 172.21.15.153:0/2774856374 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f981c01fb30 con 0x7f982c1017a0 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 msgr2=0x7f9814038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 0x7f9814038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9820009990 tx=0x7f9820006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 msgr2=0x7f982c1138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f981c0106a0 tx=0x7f981c00b4f0 comp rx=0 tx=0).stop 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 shutdown_connections 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f9814036880 0x7f9814038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 --2- 172.21.15.153:0/2774856374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f982c1017a0 0x7f982c1138c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 >> 172.21.15.153:0/2774856374 conn(0x7f982c0fcf40 msgr2=0x7f982c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 shutdown_connections 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-04-05T02:53:03.506+0000 7f983110f700 1 -- 172.21.15.153:0/2774856374 wait complete. 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-05T02:53:04.079 INFO:teuthology.orchestra.run.smithi153.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout:Or, if you are only running a single cluster on this host: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: ceph telemetry on 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout:For more information see: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:04.080 INFO:teuthology.orchestra.run.smithi153.stdout:Bootstrap complete. 2024-04-05T02:53:04.112 INFO:tasks.cephadm:Fetching config... 2024-04-05T02:53:04.112 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:53:04.112 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-05T02:53:04.168 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-05T02:53:04.169 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:53:04.169 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-05T02:53:04.223 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-05T02:53:04.224 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:53:04.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/keyring of=/dev/stdout 2024-04-05T02:53:04.291 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-05T02:53:04.291 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:53:04.291 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-05T02:53:04.347 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-05T02:53:04.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vrlDrO0CTiQbBZS1J5k8t+ZkoFsLpbnxPb5yE9PxUz0ArfVs+I2MsyJj7H3wVmApVsy80KiulGYjuT2G0kvm3K1pZbt/2ebUD6eqYAFhEANrn4R6iI36KNQwwNjGx+qhboYiZtv29hdn6jEZu6lccB+jOUls27KcNue/9mySVEwYOhTbzL1biNRZ3isB6AISpoZngxXdliCZWf68KOPwRpkUhJUw81T8ppBobGEX+HGXiUvnDsK9/nYIsxhrmCyb2aU+Q64LNF1RANz7M4fPZriURfLrDOR9CQ2lHw+SIhOV8oM79OogEdrOgjP71aiPIjBfc4+AYXLGYn5dQUBBIX5bBQ2vz1/ri2Z8M34JISdq8nJwImV9h6OeQxsL3HDj6/a5C5VomQ/Ss2IPeEpQ3tY2xORLUJ9EbvGGOj2daRrhu+LKAbgf3Cesj3gdAPw+OsSv7Ns6px8BCQoqai4P/9MK0egC6SnoTWiHbWnOYxYsNpf++IWjRvB/617tcis= ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T02:53:04.428 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vrlDrO0CTiQbBZS1J5k8t+ZkoFsLpbnxPb5yE9PxUz0ArfVs+I2MsyJj7H3wVmApVsy80KiulGYjuT2G0kvm3K1pZbt/2ebUD6eqYAFhEANrn4R6iI36KNQwwNjGx+qhboYiZtv29hdn6jEZu6lccB+jOUls27KcNue/9mySVEwYOhTbzL1biNRZ3isB6AISpoZngxXdliCZWf68KOPwRpkUhJUw81T8ppBobGEX+HGXiUvnDsK9/nYIsxhrmCyb2aU+Q64LNF1RANz7M4fPZriURfLrDOR9CQ2lHw+SIhOV8oM79OogEdrOgjP71aiPIjBfc4+AYXLGYn5dQUBBIX5bBQ2vz1/ri2Z8M34JISdq8nJwImV9h6OeQxsL3HDj6/a5C5VomQ/Ss2IPeEpQ3tY2xORLUJ9EbvGGOj2daRrhu+LKAbgf3Cesj3gdAPw+OsSv7Ns6px8BCQoqai4P/9MK0egC6SnoTWiHbWnOYxYsNpf++IWjRvB/617tcis= ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:53:04.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vrlDrO0CTiQbBZS1J5k8t+ZkoFsLpbnxPb5yE9PxUz0ArfVs+I2MsyJj7H3wVmApVsy80KiulGYjuT2G0kvm3K1pZbt/2ebUD6eqYAFhEANrn4R6iI36KNQwwNjGx+qhboYiZtv29hdn6jEZu6lccB+jOUls27KcNue/9mySVEwYOhTbzL1biNRZ3isB6AISpoZngxXdliCZWf68KOPwRpkUhJUw81T8ppBobGEX+HGXiUvnDsK9/nYIsxhrmCyb2aU+Q64LNF1RANz7M4fPZriURfLrDOR9CQ2lHw+SIhOV8oM79OogEdrOgjP71aiPIjBfc4+AYXLGYn5dQUBBIX5bBQ2vz1/ri2Z8M34JISdq8nJwImV9h6OeQxsL3HDj6/a5C5VomQ/Ss2IPeEpQ3tY2xORLUJ9EbvGGOj2daRrhu+LKAbgf3Cesj3gdAPw+OsSv7Ns6px8BCQoqai4P/9MK0egC6SnoTWiHbWnOYxYsNpf++IWjRvB/617tcis= ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T02:53:04.488 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vrlDrO0CTiQbBZS1J5k8t+ZkoFsLpbnxPb5yE9PxUz0ArfVs+I2MsyJj7H3wVmApVsy80KiulGYjuT2G0kvm3K1pZbt/2ebUD6eqYAFhEANrn4R6iI36KNQwwNjGx+qhboYiZtv29hdn6jEZu6lccB+jOUls27KcNue/9mySVEwYOhTbzL1biNRZ3isB6AISpoZngxXdliCZWf68KOPwRpkUhJUw81T8ppBobGEX+HGXiUvnDsK9/nYIsxhrmCyb2aU+Q64LNF1RANz7M4fPZriURfLrDOR9CQ2lHw+SIhOV8oM79OogEdrOgjP71aiPIjBfc4+AYXLGYn5dQUBBIX5bBQ2vz1/ri2Z8M34JISdq8nJwImV9h6OeQxsL3HDj6/a5C5VomQ/Ss2IPeEpQ3tY2xORLUJ9EbvGGOj2daRrhu+LKAbgf3Cesj3gdAPw+OsSv7Ns6px8BCQoqai4P/9MK0egC6SnoTWiHbWnOYxYsNpf++IWjRvB/617tcis= ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:53:04.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-05T02:53:04.757 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:53:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:04 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2774856374' entity='client.admin' 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.970+0000 7f66ccb7b700 1 -- 172.21.15.153:0/1380613838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 msgr2=0x7f66c80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.970+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/1380613838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c80fd460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66b80048c0 tx=0x7f66b8009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- 172.21.15.153:0/1380613838 shutdown_connections 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/1380613838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c80fd460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- 172.21.15.153:0/1380613838 >> 172.21.15.153:0/1380613838 conn(0x7f66c80f70d0 msgr2=0x7f66c80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- 172.21.15.153:0/1380613838 shutdown_connections 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- 172.21.15.153:0/1380613838 wait complete. 2024-04-05T02:53:06.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 Processor -- start 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- start start 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66ccb7b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c810c500 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66c659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66c659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35054/0 (socket says 172.21.15.153:35054) 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.971+0000 7f66c659c700 1 -- 172.21.15.153:0/3483423442 learned_addr learned my addr 172.21.15.153:0/3483423442 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66c659c700 1 -- 172.21.15.153:0/3483423442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b8005040 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66c659c700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f66b80096b0 tx=0x7f66b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b800bed0 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66b800b5c0 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c810c700 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b801c5b0 con 0x7f66c80fd050 2024-04-05T02:53:06.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.972+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c810d270 con 0x7f66c80fd050 2024-04-05T02:53:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.973+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f66b801c710 con 0x7f66c80fd050 2024-04-05T02:53:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.973+0000 7f66b6ffd700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 0x7f66b0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.973+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f66b8049ad0 con 0x7f66c80fd050 2024-04-05T02:53:06.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.973+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c8101e70 con 0x7f66c80fd050 2024-04-05T02:53:06.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.973+0000 7f66c5d9b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 0x7f66b0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:06.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.974+0000 7f66c5d9b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 0x7f66b0038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009990 tx=0x7f66bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:06.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:06.978+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f66b8025400 con 0x7f66c80fd050 2024-04-05T02:53:07.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.174+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f66c804bed0 con 0x7f66c80fd050 2024-04-05T02:53:07.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.177+0000 7f66b6ffd700 1 -- 172.21.15.153:0/3483423442 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f66b80340c0 con 0x7f66c80fd050 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 msgr2=0x7f66b0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 0x7f66b0038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009990 tx=0x7f66bc006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 msgr2=0x7f66c810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f66b80096b0 tx=0x7f66b8009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 shutdown_connections 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f66b00368d0 0x7f66b0038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 --2- 172.21.15.153:0/3483423442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66c80fd050 0x7f66c810bfc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 >> 172.21.15.153:0/3483423442 conn(0x7f66c80f70d0 msgr2=0x7f66c8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 shutdown_connections 2024-04-05T02:53:07.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:07.180+0000 7f66ccb7b700 1 -- 172.21.15.153:0/3483423442 wait complete. 2024-04-05T02:53:07.891 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-05T02:53:07.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-05T02:53:08.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:08 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3483423442' entity='client.admin' 2024-04-05T02:53:08.696 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:53:09.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:09.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:09.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:53:09.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:09.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:53:09.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-05T02:53:09.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:53:09.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:09 smithi153 ceph-mon[29381]: Deploying daemon crash.smithi153 on smithi153 2024-04-05T02:53:11.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 -- 172.21.15.153:0/1394903337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 msgr2=0x7f33300fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:11.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 --2- 172.21.15.153:0/1394903337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f33300fce60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f33240048c0 tx=0x7f3324009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:11.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 -- 172.21.15.153:0/1394903337 shutdown_connections 2024-04-05T02:53:11.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 --2- 172.21.15.153:0/1394903337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f33300fce60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:11.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 -- 172.21.15.153:0/1394903337 >> 172.21.15.153:0/1394903337 conn(0x7f33300f8190 msgr2=0x7f33300fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 -- 172.21.15.153:0/1394903337 shutdown_connections 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.254+0000 7f3336b35700 1 -- 172.21.15.153:0/1394903337 wait complete. 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f3336b35700 1 Processor -- start 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f3336b35700 1 -- start start 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f3336b35700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f3336b35700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333010d5e0 con 0x7f33300fca50 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f33348d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f33348d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32950/0 (socket says 172.21.15.153:32950) 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f33348d1700 1 -- 172.21.15.153:0/157095881 learned_addr learned my addr 172.21.15.153:0/157095881 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.255+0000 7f33348d1700 1 -- 172.21.15.153:0/157095881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324005040 con 0x7f33300fca50 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f33348d1700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3324009680 tx=0x7f3324009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332400b5c0 con 0x7f33300fca50 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f332400be70 con 0x7f33300fca50 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333010d7e0 con 0x7f33300fca50 2024-04-05T02:53:11.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33240135a0 con 0x7f33300fca50 2024-04-05T02:53:11.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.256+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333010e350 con 0x7f33300fca50 2024-04-05T02:53:11.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f3324013860 con 0x7f33300fca50 2024-04-05T02:53:11.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f332d7fa700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 0x7f3320038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:11.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3330102f60 con 0x7f33300fca50 2024-04-05T02:53:11.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f33240493e0 con 0x7f33300fca50 2024-04-05T02:53:11.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f332ffff700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 0x7f3320038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:11.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.257+0000 7f332ffff700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 0x7f3320038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f331c009990 tx=0x7f331c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:11.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.262+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3324011960 con 0x7f33300fca50 2024-04-05T02:53:11.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.458+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f333010e590 con 0x7f3320036880 2024-04-05T02:53:11.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.462+0000 7f332d7fa700 1 -- 172.21.15.153:0/157095881 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f333010e590 con 0x7f3320036880 2024-04-05T02:53:11.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 msgr2=0x7f3320038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:11.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 0x7f3320038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f331c009990 tx=0x7f331c006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:11.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 msgr2=0x7f333010d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:11.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3324009680 tx=0x7f3324009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:11.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 shutdown_connections 2024-04-05T02:53:11.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3320036880 0x7f3320038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:11.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 --2- 172.21.15.153:0/157095881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33300fca50 0x7f333010d0a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:11.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 >> 172.21.15.153:0/157095881 conn(0x7f33300f8190 msgr2=0x7f3330103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:11.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 shutdown_connections 2024-04-05T02:53:11.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:11.464+0000 7f3336b35700 1 -- 172.21.15.153:0/157095881 wait complete. 2024-04-05T02:53:12.243 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-04-05T02:53:12.243 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:53:12.243 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-04-05T02:53:12.263 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:53:12.263 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:53:12.320 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-04-05T02:53:12.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi203 2024-04-05T02:53:12.699 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:12.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:12 smithi153 ceph-mon[29381]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-05T02:53:12.703 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 -- 172.21.15.153:0/2144427608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 msgr2=0x7f82a80fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 --2- 172.21.15.153:0/2144427608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a80fce60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009c60 tx=0x7f82a4004d10 comp rx=0 tx=0).stop 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 -- 172.21.15.153:0/2144427608 shutdown_connections 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 --2- 172.21.15.153:0/2144427608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a80fce60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 -- 172.21.15.153:0/2144427608 >> 172.21.15.153:0/2144427608 conn(0x7f82a80f8190 msgr2=0x7f82a80fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 -- 172.21.15.153:0/2144427608 shutdown_connections 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.018+0000 7f82afbc8700 1 -- 172.21.15.153:0/2144427608 wait complete. 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82afbc8700 1 Processor -- start 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82afbc8700 1 -- start start 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82afbc8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82afbc8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a80722a0 con 0x7f82a80fca50 2024-04-05T02:53:14.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82ad964700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82ad964700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32982/0 (socket says 172.21.15.153:32982) 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82ad964700 1 -- 172.21.15.153:0/1187639184 learned_addr learned my addr 172.21.15.153:0/1187639184 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82ad964700 1 -- 172.21.15.153:0/1187639184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a4009940 con 0x7f82a80fca50 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f82ad964700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82a4004e60 tx=0x7f82a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a4012410 con 0x7f82a80fca50 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82a4012a40 con 0x7f82a80fca50 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.019+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a401ba40 con 0x7f82a80fca50 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.020+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a8070370 con 0x7f82a80fca50 2024-04-05T02:53:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.020+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a8070860 con 0x7f82a80fca50 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.020+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f82a4019040 con 0x7f82a80fca50 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.020+0000 7f829e7fc700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 0x7f8294038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.020+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a805f5f0 con 0x7f82a80fca50 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.021+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f82a4049ba0 con 0x7f82a80fca50 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.021+0000 7f82ad163700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 0x7f8294038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:14.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.021+0000 7f82ad163700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 0x7f8294038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8298009990 tx=0x7f8298006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:14.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.025+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f82a401f080 con 0x7f82a80fca50 2024-04-05T02:53:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:14.224+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7f82a81064a0 con 0x7f82940368d0 2024-04-05T02:53:15.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:15 smithi153 ceph-mon[29381]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:53:15.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:15 smithi153 ceph-mon[29381]: Deploying cephadm binary to smithi203 2024-04-05T02:53:16.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.504+0000 7f829e7fc700 1 -- 172.21.15.153:0/1187639184 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f82a81064a0 con 0x7f82940368d0 2024-04-05T02:53:16.505 INFO:teuthology.orchestra.run.smithi153.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 msgr2=0x7f8294038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 0x7f8294038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8298009990 tx=0x7f8298006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 msgr2=0x7f82a8071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82a4004e60 tx=0x7f82a4004d10 comp rx=0 tx=0).stop 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 shutdown_connections 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f82940368d0 0x7f8294038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 --2- 172.21.15.153:0/1187639184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82a80fca50 0x7f82a8071d60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.506+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 >> 172.21.15.153:0/1187639184 conn(0x7f82a80f8190 msgr2=0x7f82a8103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.507+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 shutdown_connections 2024-04-05T02:53:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:16.507+0000 7f82afbc8700 1 -- 172.21.15.153:0/1187639184 wait complete. 2024-04-05T02:53:16.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:16 smithi153 ceph-mon[29381]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:16.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:16.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:17.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-05T02:53:17.331 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:53:17.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:17 smithi153 ceph-mon[29381]: Added host smithi203 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 -- 172.21.15.153:0/817558109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 msgr2=0x7fe2580fb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/817558109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe2580fb270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0048c0 tx=0x7fe24c009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 -- 172.21.15.153:0/817558109 shutdown_connections 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/817558109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe2580fb270 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 -- 172.21.15.153:0/817558109 >> 172.21.15.153:0/817558109 conn(0x7fe2580f6840 msgr2=0x7fe2580f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 -- 172.21.15.153:0/817558109 shutdown_connections 2024-04-05T02:53:18.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.921+0000 7fe25ec4e700 1 -- 172.21.15.153:0/817558109 wait complete. 2024-04-05T02:53:18.922 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:18 smithi153 ceph-mon[29381]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:18.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25ec4e700 1 Processor -- start 2024-04-05T02:53:18.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25ec4e700 1 -- start start 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25ec4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25ec4e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25810d810 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25c9ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25c9ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41744/0 (socket says 172.21.15.153:41744) 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.922+0000 7fe25c9ea700 1 -- 172.21.15.153:0/1783431831 learned_addr learned my addr 172.21.15.153:0/1783431831 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe25c9ea700 1 -- 172.21.15.153:0/1783431831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c005040 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe25c9ea700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0096b0 tx=0x7fe24c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe24c00bed0 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25810da10 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe24c00b5c0 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe24c01c5b0 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.923+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25810e580 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.924+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe24c01c820 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.924+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe258103190 con 0x7fe2580fae60 2024-04-05T02:53:18.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.924+0000 7fe2557fa700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 0x7fe248038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:18.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.924+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe24c049a40 con 0x7fe2580fae60 2024-04-05T02:53:18.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.924+0000 7fe257fff700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 0x7fe248038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:18.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.925+0000 7fe257fff700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 0x7fe248038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe244009990 tx=0x7fe244006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:18.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:18.929+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe24c023300 con 0x7fe2580fae60 2024-04-05T02:53:19.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.117+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2580008d0 con 0x7fe248036880 2024-04-05T02:53:19.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.118+0000 7fe2557fa700 1 -- 172.21.15.153:0/1783431831 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fe2580008d0 con 0x7fe248036880 2024-04-05T02:53:19.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:53:19.118 INFO:teuthology.orchestra.run.smithi153.stdout:[{"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-04-05T02:53:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 msgr2=0x7fe248038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 0x7fe248038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe244009990 tx=0x7fe244006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 msgr2=0x7fe25810d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0096b0 tx=0x7fe24c009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:19.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 shutdown_connections 2024-04-05T02:53:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe248036880 0x7fe248038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 --2- 172.21.15.153:0/1783431831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2580fae60 0x7fe25810d2d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 >> 172.21.15.153:0/1783431831 conn(0x7fe2580f6840 msgr2=0x7fe258103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 shutdown_connections 2024-04-05T02:53:19.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:19.120+0000 7fe25ec4e700 1 -- 172.21.15.153:0/1783431831 wait complete. 2024-04-05T02:53:19.684 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-05T02:53:19.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-05T02:53:19.933 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:53:20.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:20 smithi153 ceph-mon[29381]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T02:53:20.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:20 smithi153 ceph-mon[29381]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 -- 172.21.15.153:0/2547721585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 msgr2=0x7fbf6c0ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 --2- 172.21.15.153:0/2547721585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c0ff6d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbf600048c0 tx=0x7fbf60009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 -- 172.21.15.153:0/2547721585 shutdown_connections 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 --2- 172.21.15.153:0/2547721585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c0ff6d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 -- 172.21.15.153:0/2547721585 >> 172.21.15.153:0/2547721585 conn(0x7fbf6c0f70c0 msgr2=0x7fbf6c0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 -- 172.21.15.153:0/2547721585 shutdown_connections 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.242+0000 7fbf72deb700 1 -- 172.21.15.153:0/2547721585 wait complete. 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf72deb700 1 Processor -- start 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf72deb700 1 -- start start 2024-04-05T02:53:21.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf72deb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf72deb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c10d610 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf70b87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf70b87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41768/0 (socket says 172.21.15.153:41768) 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.243+0000 7fbf70b87700 1 -- 172.21.15.153:0/3261364567 learned_addr learned my addr 172.21.15.153:0/3261364567 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf70b87700 1 -- 172.21.15.153:0/3261364567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60005040 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf70b87700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009680 tx=0x7fbf60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6000bed0 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf6000b5c0 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c10d810 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6001c5b0 con 0x7fbf6c066960 2024-04-05T02:53:21.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.244+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c10e380 con 0x7fbf6c066960 2024-04-05T02:53:21.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fbf6001c710 con 0x7fbf6c066960 2024-04-05T02:53:21.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c102f60 con 0x7fbf6c066960 2024-04-05T02:53:21.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf697fa700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 0x7fbf5c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:21.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbf600499a0 con 0x7fbf6c066960 2024-04-05T02:53:21.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf6bfff700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 0x7fbf5c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:21.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.245+0000 7fbf6bfff700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 0x7fbf5c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009990 tx=0x7fbf58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:21.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.250+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbf60013420 con 0x7fbf6c066960 2024-04-05T02:53:21.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.439+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbf6c10ed90 con 0x7fbf6c066960 2024-04-05T02:53:21.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.678+0000 7fbf697fa700 1 -- 172.21.15.153:0/3261364567 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbf60011370 con 0x7fbf6c066960 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 msgr2=0x7fbf5c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 0x7fbf5c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009990 tx=0x7fbf58006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 msgr2=0x7fbf6c10d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009680 tx=0x7fbf60009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 shutdown_connections 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbf5c036880 0x7fbf5c038d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:21.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 --2- 172.21.15.153:0/3261364567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf6c066960 0x7fbf6c10d0d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:21.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 >> 172.21.15.153:0/3261364567 conn(0x7fbf6c0f70c0 msgr2=0x7fbf6c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:21.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 shutdown_connections 2024-04-05T02:53:21.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:53:21.680+0000 7fbf72deb700 1 -- 172.21.15.153:0/3261364567 wait complete. 2024-04-05T02:53:21.681 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-04-05T02:53:21.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:21 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3261364567' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-05T02:53:22.287 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-04-05T02:53:22.288 INFO:tasks.cephadm:Adding mon.smithi203 on smithi203 2024-04-05T02:53:22.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi153:172.21.15.153=smithi153;smithi203:172.21.15.203=smithi203' 2024-04-05T02:53:22.538 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:22.585 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:22 smithi153 ceph-mon[29381]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:22 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3261364567' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-05T02:53:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:22 smithi153 ceph-mon[29381]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T02:53:24.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 -- 172.21.15.203:0/2361892453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 msgr2=0x7f70f00fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:24.109 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 --2- 172.21.15.203:0/2361892453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f00fbd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0048c0 tx=0x7f70ec009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 -- 172.21.15.203:0/2361892453 shutdown_connections 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 --2- 172.21.15.203:0/2361892453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f00fbd70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 -- 172.21.15.203:0/2361892453 >> 172.21.15.203:0/2361892453 conn(0x7f70f00f70c0 msgr2=0x7f70f00f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 -- 172.21.15.203:0/2361892453 shutdown_connections 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.108+0000 7f70f7fad700 1 -- 172.21.15.203:0/2361892453 wait complete. 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f7fad700 1 Processor -- start 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f7fad700 1 -- start start 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f7fad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f7fad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f010dc40 con 0x7f70f00fb960 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f5d49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f5d49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:36708/0 (socket says 172.21.15.203:36708) 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.109+0000 7f70f5d49700 1 -- 172.21.15.203:0/3066331200 learned_addr learned my addr 172.21.15.203:0/3066331200 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70f5d49700 1 -- 172.21.15.203:0/3066331200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec005040 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70f5d49700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009300 tx=0x7f70ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ec004e60 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70ec009680 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f0112250 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ec01c3f0 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.110+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f0112680 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.111+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f70ec0097f0 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.111+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f0105920 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.111+0000 7f70e67fc700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 0x7f70dc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.111+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f70ec049430 con 0x7f70f00fb960 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.111+0000 7f70f5548700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 0x7f70dc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:24.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.112+0000 7f70f5548700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 0x7f70dc038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009990 tx=0x7f70e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:24.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.116+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f70ec011780 con 0x7f70f00fb960 2024-04-05T02:53:24.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.309+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 --> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi203:172.21.15.203=smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7f70f002b3c0 con 0x7f70dc036880 2024-04-05T02:53:24.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.316+0000 7f70e67fc700 1 -- 172.21.15.203:0/3066331200 <== mgr.14162 v2:172.21.15.153:6800/4288916206 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f70f002b3c0 con 0x7f70dc036880 2024-04-05T02:53:24.316 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 msgr2=0x7f70dc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 0x7f70dc038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009990 tx=0x7f70e0006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 msgr2=0x7f70f0066880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009300 tx=0x7f70ec009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 shutdown_connections 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f70dc036880 0x7f70dc038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 --2- 172.21.15.203:0/3066331200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70f00fb960 0x7f70f0066880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:24.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 >> 172.21.15.203:0/3066331200 conn(0x7f70f00f70c0 msgr2=0x7f70f0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:24.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 shutdown_connections 2024-04-05T02:53:24.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:24.318+0000 7f70f7fad700 1 -- 172.21.15.203:0/3066331200 wait complete. 2024-04-05T02:53:24.884 DEBUG:teuthology.orchestra.run.smithi203:mon.smithi203> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi203.service 2024-04-05T02:53:24.887 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:24.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:24 smithi153 ceph-mon[29381]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:25.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:24 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:53:25.164 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:25.211 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:25 smithi153 ceph-mon[29381]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi203:172.21.15.203=smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:53:26.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:25 smithi153 ceph-mon[29381]: Saving service mon spec with placement smithi153:172.21.15.153=smithi153;smithi203:172.21.15.203=smithi203;count:2 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 -- 172.21.15.203:0/2898690182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 msgr2=0x7f3e240fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/2898690182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240fbd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e200048c0 tx=0x7f3e20009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 -- 172.21.15.203:0/2898690182 shutdown_connections 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/2898690182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240fbd60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 -- 172.21.15.203:0/2898690182 >> 172.21.15.203:0/2898690182 conn(0x7f3e240f7090 msgr2=0x7f3e240f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 -- 172.21.15.203:0/2898690182 shutdown_connections 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.597+0000 7f3e2c41c700 1 -- 172.21.15.203:0/2898690182 wait complete. 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2c41c700 1 Processor -- start 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2c41c700 1 -- start start 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2c41c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:26.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2c41c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2410fd40 con 0x7f3e240fb950 2024-04-05T02:53:26.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2a1b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:26.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2a1b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53102/0 (socket says 172.21.15.203:53102) 2024-04-05T02:53:26.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2a1b8700 1 -- 172.21.15.203:0/3440335301 learned_addr learned my addr 172.21.15.203:0/3440335301 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:26.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.598+0000 7f3e2a1b8700 1 -- 172.21.15.203:0/3440335301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20005040 con 0x7f3e240fb950 2024-04-05T02:53:26.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e2a1b8700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e200096b0 tx=0x7f3e20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e2000bed0 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e2000b5c0 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e2001c5b0 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2410fee0 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.599+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e24110370 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.600+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f3e2001c820 con 0x7f3e240fb950 2024-04-05T02:53:26.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.600+0000 7f3e1affd700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 0x7f3e10038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:26.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.600+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e24105910 con 0x7f3e240fb950 2024-04-05T02:53:26.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.600+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3e200499e0 con 0x7f3e240fb950 2024-04-05T02:53:26.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.600+0000 7f3e299b7700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 0x7f3e10038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:26.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.601+0000 7f3e299b7700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 0x7f3e10038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009990 tx=0x7f3e14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:26.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.605+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3e20023300 con 0x7f3e240fb950 2024-04-05T02:53:26.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.864+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e2404bed0 con 0x7f3e240fb950 2024-04-05T02:53:26.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.864+0000 7f3e1affd700 1 -- 172.21.15.203:0/3440335301 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f3e20011b60 con 0x7f3e240fb950 2024-04-05T02:53:26.864 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:26.864 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:26.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 msgr2=0x7f3e10038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:26.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 0x7f3e10038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009990 tx=0x7f3e14006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 msgr2=0x7f3e240668a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e200096b0 tx=0x7f3e20009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 shutdown_connections 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f3e10036880 0x7f3e10038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 --2- 172.21.15.203:0/3440335301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e240fb950 0x7f3e240668a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 >> 172.21.15.203:0/3440335301 conn(0x7f3e240f7090 msgr2=0x7f3e24102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 shutdown_connections 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:26.866+0000 7f3e2c41c700 1 -- 172.21.15.203:0/3440335301 wait complete. 2024-04-05T02:53:26.867 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:26 smithi153 ceph-mon[29381]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:28.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:27 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3440335301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:28.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:28.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:28.720 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:28.767 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:28 smithi153 ceph-mon[29381]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 -- 172.21.15.203:0/3186523291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 msgr2=0x7f221c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 --2- 172.21.15.203:0/3186523291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c0ffa30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f22180048c0 tx=0x7f2218009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 -- 172.21.15.203:0/3186523291 shutdown_connections 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 --2- 172.21.15.203:0/3186523291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c0ffa30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 -- 172.21.15.203:0/3186523291 >> 172.21.15.203:0/3186523291 conn(0x7f221c0f70d0 msgr2=0x7f221c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 -- 172.21.15.203:0/3186523291 shutdown_connections 2024-04-05T02:53:30.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.085+0000 7f2223bb7700 1 -- 172.21.15.203:0/3186523291 wait complete. 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2223bb7700 1 Processor -- start 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2223bb7700 1 -- start start 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2223bb7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2223bb7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c10d5c0 con 0x7f221c0fd650 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2221953700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2221953700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53120/0 (socket says 172.21.15.203:53120) 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.086+0000 7f2221953700 1 -- 172.21.15.203:0/1955861022 learned_addr learned my addr 172.21.15.203:0/1955861022 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f2221953700 1 -- 172.21.15.203:0/1955861022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218005040 con 0x7f221c0fd650 2024-04-05T02:53:30.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f2221953700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2218009680 tx=0x7f2218009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:30.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221800bed0 con 0x7f221c0fd650 2024-04-05T02:53:30.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f221800b5c0 con 0x7f221c0fd650 2024-04-05T02:53:30.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c10d7c0 con 0x7f221c0fd650 2024-04-05T02:53:30.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221801c5b0 con 0x7f221c0fd650 2024-04-05T02:53:30.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.087+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c10e330 con 0x7f221c0fd650 2024-04-05T02:53:30.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.088+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f2218013420 con 0x7f221c0fd650 2024-04-05T02:53:30.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.088+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221c102f60 con 0x7f221c0fd650 2024-04-05T02:53:30.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.088+0000 7f22127fc700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 0x7f2208038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:30.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.088+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2218049980 con 0x7f221c0fd650 2024-04-05T02:53:30.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.088+0000 7f2221152700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 0x7f2208038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:30.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.089+0000 7f2221152700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 0x7f2208038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f220c009990 tx=0x7f220c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:30.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.093+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2218013a80 con 0x7f221c0fd650 2024-04-05T02:53:30.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.354+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f221c10ed40 con 0x7f221c0fd650 2024-04-05T02:53:30.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.355+0000 7f22127fc700 1 -- 172.21.15.203:0/1955861022 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2218011370 con 0x7f221c0fd650 2024-04-05T02:53:30.355 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:30.356 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:30.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 msgr2=0x7f2208038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:30.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 0x7f2208038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f220c009990 tx=0x7f220c006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:30.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 msgr2=0x7f221c10d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:30.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2218009680 tx=0x7f2218009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 shutdown_connections 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f2208036880 0x7f2208038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 --2- 172.21.15.203:0/1955861022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221c0fd650 0x7f221c10d080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 >> 172.21.15.203:0/1955861022 conn(0x7f221c0f70d0 msgr2=0x7f221c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 shutdown_connections 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:30.357+0000 7f2223bb7700 1 -- 172.21.15.203:0/1955861022 wait complete. 2024-04-05T02:53:30.359 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:30 smithi153 ceph-mon[29381]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:31.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:30 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1955861022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:31.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:31.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:32.090 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:32.138 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:33.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:32 smithi153 ceph-mon[29381]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:33.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.579+0000 7fb437658700 1 -- 172.21.15.203:0/946438746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 msgr2=0x7fb4300fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:33.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.579+0000 7fb437658700 1 --2- 172.21.15.203:0/946438746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb4300fd450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4200048c0 tx=0x7fb420009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:33.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.579+0000 7fb437658700 1 -- 172.21.15.203:0/946438746 shutdown_connections 2024-04-05T02:53:33.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 --2- 172.21.15.203:0/946438746 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb4300fd450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:33.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 -- 172.21.15.203:0/946438746 >> 172.21.15.203:0/946438746 conn(0x7fb4300f70c0 msgr2=0x7fb4300f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 -- 172.21.15.203:0/946438746 shutdown_connections 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 -- 172.21.15.203:0/946438746 wait complete. 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 Processor -- start 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 -- start start 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb437658700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43010fd80 con 0x7fb4300fd040 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb4353f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.580+0000 7fb4353f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53136/0 (socket says 172.21.15.203:53136) 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb4353f4700 1 -- 172.21.15.203:0/1833066043 learned_addr learned my addr 172.21.15.203:0/1833066043 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb4353f4700 1 -- 172.21.15.203:0/1833066043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420005040 con 0x7fb4300fd040 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb4353f4700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb420009680 tx=0x7fb420009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb42000b5c0 con 0x7fb4300fd040 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb42000be70 con 0x7fb4300fd040 2024-04-05T02:53:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43010fec0 con 0x7fb4300fd040 2024-04-05T02:53:33.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4200135a0 con 0x7fb4300fd040 2024-04-05T02:53:33.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.581+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4301103b0 con 0x7fb4300fd040 2024-04-05T02:53:33.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.582+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fb42001c430 con 0x7fb4300fd040 2024-04-05T02:53:33.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.582+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb430105910 con 0x7fb4300fd040 2024-04-05T02:53:33.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.582+0000 7fb42dffb700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 0x7fb418038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:33.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.582+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb4200493a0 con 0x7fb4300fd040 2024-04-05T02:53:33.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.583+0000 7fb434bf3700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 0x7fb418038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:33.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.583+0000 7fb434bf3700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 0x7fb418038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb424009990 tx=0x7fb424006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:33.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.587+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb42001c7d0 con 0x7fb4300fd040 2024-04-05T02:53:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.851+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb43005f5f0 con 0x7fb4300fd040 2024-04-05T02:53:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.851+0000 7fb42dffb700 1 -- 172.21.15.203:0/1833066043 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb420021070 con 0x7fb4300fd040 2024-04-05T02:53:33.852 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:33.852 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.853+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 msgr2=0x7fb418038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.853+0000 7fb437658700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 0x7fb418038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb424009990 tx=0x7fb424006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.853+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 msgr2=0x7fb430066480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.853+0000 7fb437658700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb420009680 tx=0x7fb420009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 shutdown_connections 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb418036880 0x7fb418038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 --2- 172.21.15.203:0/1833066043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4300fd040 0x7fb430066480 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 >> 172.21.15.203:0/1833066043 conn(0x7fb4300f70c0 msgr2=0x7fb430102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 shutdown_connections 2024-04-05T02:53:33.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:33.854+0000 7fb437658700 1 -- 172.21.15.203:0/1833066043 wait complete. 2024-04-05T02:53:33.855 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:34 smithi153 ceph-mon[29381]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:34 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1833066043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:35.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:35.469 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:35.722 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:35.770 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:36 smithi153 ceph-mon[29381]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 -- 172.21.15.203:0/1598831323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 msgr2=0x7f171c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 --2- 172.21.15.203:0/1598831323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c0fd460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f17040048c0 tx=0x7f1704009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 -- 172.21.15.203:0/1598831323 shutdown_connections 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 --2- 172.21.15.203:0/1598831323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c0fd460 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 -- 172.21.15.203:0/1598831323 >> 172.21.15.203:0/1598831323 conn(0x7f171c0f70d0 msgr2=0x7f171c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 -- 172.21.15.203:0/1598831323 shutdown_connections 2024-04-05T02:53:37.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.166+0000 7f172294d700 1 -- 172.21.15.203:0/1598831323 wait complete. 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.167+0000 7f172294d700 1 Processor -- start 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.167+0000 7f172294d700 1 -- start start 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.167+0000 7f172294d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.167+0000 7f172294d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c10fd80 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f171bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f171bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53154/0 (socket says 172.21.15.203:53154) 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f171bfff700 1 -- 172.21.15.203:0/1324667575 learned_addr learned my addr 172.21.15.203:0/1324667575 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f171bfff700 1 -- 172.21.15.203:0/1324667575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704005040 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f171bfff700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17040096b0 tx=0x7f1704009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f170400bed0 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c10ff20 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f170400b5c0 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c1103b0 con 0x7f171c0fd050 2024-04-05T02:53:37.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.168+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1704025560 con 0x7f171c0fd050 2024-04-05T02:53:37.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.169+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1704013420 con 0x7f171c0fd050 2024-04-05T02:53:37.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.169+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c105920 con 0x7f171c0fd050 2024-04-05T02:53:37.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.169+0000 7f1718ff9700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 0x7f170803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:37.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.169+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f17040499a0 con 0x7f171c0fd050 2024-04-05T02:53:37.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.169+0000 7f171b7fe700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 0x7f170803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:37.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.170+0000 7f171b7fe700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 0x7f170803d140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f170c009990 tx=0x7f170c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:37.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.174+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1704013a80 con 0x7f171c0fd050 2024-04-05T02:53:37.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.434+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f171c02b2a0 con 0x7f171c0fd050 2024-04-05T02:53:37.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.435+0000 7f1718ff9700 1 -- 172.21.15.203:0/1324667575 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1704011370 con 0x7f171c0fd050 2024-04-05T02:53:37.435 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:37.435 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 msgr2=0x7f170803d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 0x7f170803d140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f170c009990 tx=0x7f170c006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 msgr2=0x7f171c065c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17040096b0 tx=0x7f1704009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 shutdown_connections 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f170803ac90 0x7f170803d140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 --2- 172.21.15.203:0/1324667575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f171c0fd050 0x7f171c065c30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 >> 172.21.15.203:0/1324667575 conn(0x7f171c0f70d0 msgr2=0x7f171c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 shutdown_connections 2024-04-05T02:53:37.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:37.437+0000 7f172294d700 1 -- 172.21.15.203:0/1324667575 wait complete. 2024-04-05T02:53:37.438 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:38.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:37 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1324667575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:38.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:38.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:38 smithi153 ceph-mon[29381]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:39.253 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:39.300 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:40.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.711+0000 7fefcec0e700 1 -- 172.21.15.203:0/4110735894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 msgr2=0x7fefc80fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.711+0000 7fefcec0e700 1 --2- 172.21.15.203:0/4110735894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc80fe4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0048c0 tx=0x7fefbc009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- 172.21.15.203:0/4110735894 shutdown_connections 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 --2- 172.21.15.203:0/4110735894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc80fe4b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- 172.21.15.203:0/4110735894 >> 172.21.15.203:0/4110735894 conn(0x7fefc80f6840 msgr2=0x7fefc80f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- 172.21.15.203:0/4110735894 shutdown_connections 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- 172.21.15.203:0/4110735894 wait complete. 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 Processor -- start 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- start start 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.712+0000 7fefcec0e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc810d5f0 con 0x7fefc80fe0a0 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcc9aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcc9aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53174/0 (socket says 172.21.15.203:53174) 2024-04-05T02:53:40.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcc9aa700 1 -- 172.21.15.203:0/814158243 learned_addr learned my addr 172.21.15.203:0/814158243 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcc9aa700 1 -- 172.21.15.203:0/814158243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc005040 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcc9aa700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009680 tx=0x7fefbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefbc00b5c0 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc810d7f0 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefbc00be70 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefbc0135a0 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.713+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc810e360 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.714+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fefbc01ca90 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.714+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc8102f70 con 0x7fefc80fe0a0 2024-04-05T02:53:40.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.714+0000 7fefc57fa700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 0x7fefb8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:40.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.714+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fefbc0493a0 con 0x7fefc80fe0a0 2024-04-05T02:53:40.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.715+0000 7fefc7fff700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 0x7fefb8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:40.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.715+0000 7fefc7fff700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 0x7fefb8038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009990 tx=0x7fefb4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:40.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.719+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fefbc011960 con 0x7fefc80fe0a0 2024-04-05T02:53:40.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.978+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefc810ed70 con 0x7fefc80fe0a0 2024-04-05T02:53:40.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.979+0000 7fefc57fa700 1 -- 172.21.15.203:0/814158243 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fefbc021070 con 0x7fefc80fe0a0 2024-04-05T02:53:40.979 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:40.979 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 msgr2=0x7fefb8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 0x7fefb8038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009990 tx=0x7fefb4006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 msgr2=0x7fefc810d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009680 tx=0x7fefbc009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 shutdown_connections 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fefb8036880 0x7fefb8038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 --2- 172.21.15.203:0/814158243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fefc80fe0a0 0x7fefc810d0b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 >> 172.21.15.203:0/814158243 conn(0x7fefc80f6840 msgr2=0x7fefc8103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 shutdown_connections 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:40.981+0000 7fefcec0e700 1 -- 172.21.15.203:0/814158243 wait complete. 2024-04-05T02:53:40.981 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:40 smithi153 ceph-mon[29381]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:42.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:41 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/814158243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:42.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:42.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:42.820 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:42.866 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:42.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:42 smithi153 ceph-mon[29381]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.173+0000 7f7391124700 1 -- 172.21.15.203:0/1120891654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 msgr2=0x7f738c0fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 --2- 172.21.15.203:0/1120891654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c0fce70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f737c0048c0 tx=0x7f737c009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 -- 172.21.15.203:0/1120891654 shutdown_connections 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 --2- 172.21.15.203:0/1120891654 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c0fce70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 -- 172.21.15.203:0/1120891654 >> 172.21.15.203:0/1120891654 conn(0x7f738c0f81c0 msgr2=0x7f738c0fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 -- 172.21.15.203:0/1120891654 shutdown_connections 2024-04-05T02:53:44.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 -- 172.21.15.203:0/1120891654 wait complete. 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 Processor -- start 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.174+0000 7f7391124700 1 -- start start 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f7391124700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f7391124700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c0722a0 con 0x7f738c0fca60 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f738ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f738ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53190/0 (socket says 172.21.15.203:53190) 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f738ad9d700 1 -- 172.21.15.203:0/1167210016 learned_addr learned my addr 172.21.15.203:0/1167210016 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f738ad9d700 1 -- 172.21.15.203:0/1167210016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c005040 con 0x7f738c0fca60 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f738ad9d700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f737c0096b0 tx=0x7f737c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f737c00bed0 con 0x7f738c0fca60 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c070370 con 0x7f738c0fca60 2024-04-05T02:53:44.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f737c00b5c0 con 0x7f738c0fca60 2024-04-05T02:53:44.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.175+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f737c01c5b0 con 0x7f738c0fca60 2024-04-05T02:53:44.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.176+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c070860 con 0x7f738c0fca60 2024-04-05T02:53:44.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.176+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f737c01c820 con 0x7f738c0fca60 2024-04-05T02:53:44.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.176+0000 7f73737fe700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 0x7f7374038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:44.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.176+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f737c049b30 con 0x7f738c0fca60 2024-04-05T02:53:44.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.176+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c05f5f0 con 0x7f738c0fca60 2024-04-05T02:53:44.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.177+0000 7f738a59c700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 0x7f7374038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:44.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.177+0000 7f738a59c700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 0x7f7374038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7380009990 tx=0x7f7380006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:44.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.181+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f737c023300 con 0x7f738c0fca60 2024-04-05T02:53:44.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.441+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f738c102940 con 0x7f738c0fca60 2024-04-05T02:53:44.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.442+0000 7f73737fe700 1 -- 172.21.15.203:0/1167210016 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f737c011b10 con 0x7f738c0fca60 2024-04-05T02:53:44.442 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:44.442 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 msgr2=0x7f7374038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 0x7f7374038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7380009990 tx=0x7f7380006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 msgr2=0x7f738c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f737c0096b0 tx=0x7f737c009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 shutdown_connections 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7374036880 0x7f7374038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 --2- 172.21.15.203:0/1167210016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f738c0fca60 0x7f738c071d60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:44.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 >> 172.21.15.203:0/1167210016 conn(0x7f738c0f81c0 msgr2=0x7f738c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:44.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 shutdown_connections 2024-04-05T02:53:44.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:44.444+0000 7f7391124700 1 -- 172.21.15.203:0/1167210016 wait complete. 2024-04-05T02:53:44.445 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:44 smithi153 ceph-mon[29381]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:44 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1167210016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:45.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:45.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:46.173 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:46.222 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:47.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:46 smithi153 ceph-mon[29381]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:47.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 -- 172.21.15.203:0/867712821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 msgr2=0x7f78880fe570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:47.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 --2- 172.21.15.203:0/867712821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f78880fe570 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78780048c0 tx=0x7f7878009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:47.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 -- 172.21.15.203:0/867712821 shutdown_connections 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 --2- 172.21.15.203:0/867712821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f78880fe570 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 -- 172.21.15.203:0/867712821 >> 172.21.15.203:0/867712821 conn(0x7f78880f6820 msgr2=0x7f78880f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 -- 172.21.15.203:0/867712821 shutdown_connections 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.652+0000 7f788c824700 1 -- 172.21.15.203:0/867712821 wait complete. 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788c824700 1 Processor -- start 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788c824700 1 -- start start 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788c824700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788c824700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810f810 con 0x7f78880fe160 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:58814/0 (socket says 172.21.15.203:58814) 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788659c700 1 -- 172.21.15.203:0/2996231344 learned_addr learned my addr 172.21.15.203:0/2996231344 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.653+0000 7f788659c700 1 -- 172.21.15.203:0/2996231344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878005040 con 0x7f78880fe160 2024-04-05T02:53:47.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f788659c700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78780096b0 tx=0x7f7878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f787800bed0 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f787800b5c0 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788810fa10 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f787801c5b0 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.654+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7888110580 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f7878004d30 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888102f70 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f7876ffd700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 0x7f7870038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7878049a50 con 0x7f78880fe160 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f7885d9b700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 0x7f7870038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.655+0000 7f7885d9b700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 0x7f7870038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f787c009990 tx=0x7f787c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:47.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.660+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7878011940 con 0x7f78880fe160 2024-04-05T02:53:47.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.916+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78881107c0 con 0x7f78880fe160 2024-04-05T02:53:47.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.917+0000 7f7876ffd700 1 -- 172.21.15.203:0/2996231344 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7878011370 con 0x7f78880fe160 2024-04-05T02:53:47.917 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:47.917 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 msgr2=0x7f7870038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 0x7f7870038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f787c009990 tx=0x7f787c006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 msgr2=0x7f788810f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78780096b0 tx=0x7f7878009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 shutdown_connections 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7870036880 0x7f7870038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 --2- 172.21.15.203:0/2996231344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78880fe160 0x7f788810f2d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:47.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 >> 172.21.15.203:0/2996231344 conn(0x7f78880f6820 msgr2=0x7f78880f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:47.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 shutdown_connections 2024-04-05T02:53:47.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:47.919+0000 7f788c824700 1 -- 172.21.15.203:0/2996231344 wait complete. 2024-04-05T02:53:47.920 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:48 smithi153 ceph-mon[29381]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:48 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2996231344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:49.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:49.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:49.812 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:49.859 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:50 smithi153 ceph-mon[29381]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 -- 172.21.15.203:0/130484242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 msgr2=0x7f5d400ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 --2- 172.21.15.203:0/130484242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d400ff630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5d300048c0 tx=0x7f5d30009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 -- 172.21.15.203:0/130484242 shutdown_connections 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 --2- 172.21.15.203:0/130484242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d400ff630 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 -- 172.21.15.203:0/130484242 >> 172.21.15.203:0/130484242 conn(0x7f5d400f70c0 msgr2=0x7f5d400f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 -- 172.21.15.203:0/130484242 shutdown_connections 2024-04-05T02:53:51.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 -- 172.21.15.203:0/130484242 wait complete. 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.190+0000 7f5d4630f700 1 Processor -- start 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d4630f700 1 -- start start 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d4630f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d4630f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4010d610 con 0x7f5d40066a40 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d3ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d3ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:58830/0 (socket says 172.21.15.203:58830) 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.191+0000 7f5d3ffff700 1 -- 172.21.15.203:0/732841264 learned_addr learned my addr 172.21.15.203:0/732841264 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d3ffff700 1 -- 172.21.15.203:0/732841264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30005040 con 0x7f5d40066a40 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d3ffff700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009680 tx=0x7f5d30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3000bed0 con 0x7f5d40066a40 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d3000b5c0 con 0x7f5d40066a40 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4010d810 con 0x7f5d40066a40 2024-04-05T02:53:51.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3001c5b0 con 0x7f5d40066a40 2024-04-05T02:53:51.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.192+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4010e380 con 0x7f5d40066a40 2024-04-05T02:53:51.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.193+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5d30013420 con 0x7f5d40066a40 2024-04-05T02:53:51.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.193+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d40102f40 con 0x7f5d40066a40 2024-04-05T02:53:51.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.193+0000 7f5d3cff9700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 0x7f5d28038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:51.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.193+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5d30049980 con 0x7f5d40066a40 2024-04-05T02:53:51.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.193+0000 7f5d3f7fe700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 0x7f5d28038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:51.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.194+0000 7f5d3f7fe700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 0x7f5d28038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009990 tx=0x7f5d34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:51.199 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.198+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5d30013a80 con 0x7f5d40066a40 2024-04-05T02:53:51.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.459+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d4010ed90 con 0x7f5d40066a40 2024-04-05T02:53:51.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.460+0000 7f5d3cff9700 1 -- 172.21.15.203:0/732841264 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5d30011370 con 0x7f5d40066a40 2024-04-05T02:53:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:51.461 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:51.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 msgr2=0x7f5d28038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:51.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 0x7f5d28038d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009990 tx=0x7f5d34006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:51.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 msgr2=0x7f5d4010d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:51.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009680 tx=0x7f5d30009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:51.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 shutdown_connections 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5d28036880 0x7f5d28038d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 --2- 172.21.15.203:0/732841264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5d40066a40 0x7f5d4010d0d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 >> 172.21.15.203:0/732841264 conn(0x7f5d400f70c0 msgr2=0x7f5d400f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 shutdown_connections 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:51.462+0000 7f5d4630f700 1 -- 172.21.15.203:0/732841264 wait complete. 2024-04-05T02:53:51.464 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:51 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/732841264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:52 smithi153 ceph-mon[29381]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:53.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:53.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:53.351 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:53.398 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 -- 172.21.15.203:0/3566846060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 msgr2=0x7f71600ff710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 --2- 172.21.15.203:0/3566846060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f71600ff710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f71500048c0 tx=0x7f7150009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 -- 172.21.15.203:0/3566846060 shutdown_connections 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 --2- 172.21.15.203:0/3566846060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f71600ff710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 -- 172.21.15.203:0/3566846060 >> 172.21.15.203:0/3566846060 conn(0x7f71600f70c0 msgr2=0x7f71600f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 -- 172.21.15.203:0/3566846060 shutdown_connections 2024-04-05T02:53:54.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.964+0000 7f71676b6700 1 -- 172.21.15.203:0/3566846060 wait complete. 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f71676b6700 1 Processor -- start 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f71676b6700 1 -- start start 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f71676b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f71676b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716010d5f0 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f7165452700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f7165452700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33746/0 (socket says 172.21.15.203:33746) 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f7165452700 1 -- 172.21.15.203:0/2281440828 learned_addr learned my addr 172.21.15.203:0/2281440828 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f7165452700 1 -- 172.21.15.203:0/2281440828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150005040 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.965+0000 7f7165452700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7150009680 tx=0x7f7150009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.966+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f715000bed0 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.966+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716010d7f0 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.966+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f715000b5c0 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.966+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f715001c5b0 con 0x7f7160066920 2024-04-05T02:53:54.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.966+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716010e360 con 0x7f7160066920 2024-04-05T02:53:54.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.967+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f7150004d30 con 0x7f7160066920 2024-04-05T02:53:54.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.967+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7160102f60 con 0x7f7160066920 2024-04-05T02:53:54.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.967+0000 7f7155ffb700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 0x7f714c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:54.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.967+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7150049a40 con 0x7f7160066920 2024-04-05T02:53:54.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.967+0000 7f7164c51700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 0x7f714c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:54.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.968+0000 7f7164c51700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 0x7f714c038d30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f715c009990 tx=0x7f715c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:54.972 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:54.972+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7150011960 con 0x7f7160066920 2024-04-05T02:53:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:54 smithi153 ceph-mon[29381]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:55.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.233+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f716010edd0 con 0x7f7160066920 2024-04-05T02:53:55.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.233+0000 7f7155ffb700 1 -- 172.21.15.203:0/2281440828 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7150011370 con 0x7f7160066920 2024-04-05T02:53:55.234 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:55.234 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.235+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 msgr2=0x7f714c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.235+0000 7f71676b6700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 0x7f714c038d30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f715c009990 tx=0x7f715c006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.235+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 msgr2=0x7f716010d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.235+0000 7f71676b6700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7150009680 tx=0x7f7150009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 shutdown_connections 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f714c036880 0x7f714c038d30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 --2- 172.21.15.203:0/2281440828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7160066920 0x7f716010d0b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 >> 172.21.15.203:0/2281440828 conn(0x7f71600f70c0 msgr2=0x7f7160103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:55.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 shutdown_connections 2024-04-05T02:53:55.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:55.236+0000 7f71676b6700 1 -- 172.21.15.203:0/2281440828 wait complete. 2024-04-05T02:53:55.237 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:56.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:55 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2281440828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:53:56.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:53:56.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:53:57.058 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:56 smithi153 ceph-mon[29381]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:53:57.104 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:53:58.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.557+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/1680212410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 msgr2=0x7f5b740fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:58.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.557+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/1680212410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b740fce50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5b640048c0 tx=0x7f5b64009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:58.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.557+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/1680212410 shutdown_connections 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.557+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/1680212410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b740fce50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.557+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/1680212410 >> 172.21.15.203:0/1680212410 conn(0x7f5b740f81c0 msgr2=0x7f5b740fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/1680212410 shutdown_connections 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/1680212410 wait complete. 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 Processor -- start 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 -- start start 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b7bc5e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7410d5d0 con 0x7f5b740fca40 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b799fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b799fa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33756/0 (socket says 172.21.15.203:33756) 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.558+0000 7f5b799fa700 1 -- 172.21.15.203:0/2319418573 learned_addr learned my addr 172.21.15.203:0/2319418573 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b799fa700 1 -- 172.21.15.203:0/2319418573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64005040 con 0x7f5b740fca40 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b799fa700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b640096b0 tx=0x7f5b64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b6400bed0 con 0x7f5b740fca40 2024-04-05T02:53:58.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7410d7d0 con 0x7f5b740fca40 2024-04-05T02:53:58.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b6400b5c0 con 0x7f5b740fca40 2024-04-05T02:53:58.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7410e340 con 0x7f5b740fca40 2024-04-05T02:53:58.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.559+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b64025560 con 0x7f5b740fca40 2024-04-05T02:53:58.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.560+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5b64013420 con 0x7f5b740fca40 2024-04-05T02:53:58.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.560+0000 7f5b6a7fc700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 0x7f5b60038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:53:58.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.560+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b74102f50 con 0x7f5b740fca40 2024-04-05T02:53:58.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.560+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5b64049ab0 con 0x7f5b740fca40 2024-04-05T02:53:58.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.560+0000 7f5b791f9700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 0x7f5b60038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:53:58.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.561+0000 7f5b791f9700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 0x7f5b60038d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009990 tx=0x7f5b70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:53:58.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.565+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5b640256c0 con 0x7f5b740fca40 2024-04-05T02:53:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.827+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b7404bed0 con 0x7f5b740fca40 2024-04-05T02:53:58.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.828+0000 7f5b6a7fc700 1 -- 172.21.15.203:0/2319418573 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5b64011b20 con 0x7f5b740fca40 2024-04-05T02:53:58.828 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:53:58.828 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 msgr2=0x7f5b60038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 0x7f5b60038d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009990 tx=0x7f5b70006e40 comp rx=0 tx=0).stop 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 msgr2=0x7f5b7410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b640096b0 tx=0x7f5b64009e30 comp rx=0 tx=0).stop 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 shutdown_connections 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f5b60036880 0x7f5b60038d30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:58.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 --2- 172.21.15.203:0/2319418573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b740fca40 0x7f5b7410d090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:53:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 >> 172.21.15.203:0/2319418573 conn(0x7f5b740f81c0 msgr2=0x7f5b74103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:53:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 shutdown_connections 2024-04-05T02:53:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:53:58.830+0000 7f5b7bc5e700 1 -- 172.21.15.203:0/2319418573 wait complete. 2024-04-05T02:53:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:53:59.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:58 smithi153 ceph-mon[29381]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:00.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:53:59 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2319418573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:00.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:00.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:00.685 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:00.731 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:00 smithi153 ceph-mon[29381]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:02.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 -- 172.21.15.203:0/2194275787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 msgr2=0x7f22e40ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:02.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 --2- 172.21.15.203:0/2194275787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e40ff640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f22d40048c0 tx=0x7f22d4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:02.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 -- 172.21.15.203:0/2194275787 shutdown_connections 2024-04-05T02:54:02.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 --2- 172.21.15.203:0/2194275787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e40ff640 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 -- 172.21.15.203:0/2194275787 >> 172.21.15.203:0/2194275787 conn(0x7f22e40f7090 msgr2=0x7f22e40f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 -- 172.21.15.203:0/2194275787 shutdown_connections 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.016+0000 7f22e9aed700 1 -- 172.21.15.203:0/2194275787 wait complete. 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e9aed700 1 Processor -- start 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e9aed700 1 -- start start 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e9aed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e9aed700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e410d620 con 0x7f22e40669c0 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33766/0 (socket says 172.21.15.203:33766) 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.017+0000 7f22e37fe700 1 -- 172.21.15.203:0/3097574646 learned_addr learned my addr 172.21.15.203:0/3097574646 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22e37fe700 1 -- 172.21.15.203:0/3097574646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4005040 con 0x7f22e40669c0 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22e37fe700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009680 tx=0x7f22d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:02.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d400bed0 con 0x7f22e40669c0 2024-04-05T02:54:02.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e410d820 con 0x7f22e40669c0 2024-04-05T02:54:02.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22d400b5c0 con 0x7f22e40669c0 2024-04-05T02:54:02.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22d401c5b0 con 0x7f22e40669c0 2024-04-05T02:54:02.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.018+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e410e390 con 0x7f22e40669c0 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.019+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f22d4013420 con 0x7f22e40669c0 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.019+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e4102f50 con 0x7f22e40669c0 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.019+0000 7f22cbfff700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 0x7f22cc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.019+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f22d4049980 con 0x7f22e40669c0 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.019+0000 7f22e2ffd700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 0x7f22cc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:02.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.020+0000 7f22e2ffd700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 0x7f22cc038d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009990 tx=0x7f22d8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:02.024 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.024+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f22d4013a80 con 0x7f22e40669c0 2024-04-05T02:54:02.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.288+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22e410eda0 con 0x7f22e40669c0 2024-04-05T02:54:02.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.288+0000 7f22cbfff700 1 -- 172.21.15.203:0/3097574646 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f22d4011370 con 0x7f22e40669c0 2024-04-05T02:54:02.289 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:02.289 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.291+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 msgr2=0x7f22cc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.291+0000 7f22e9aed700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 0x7f22cc038d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009990 tx=0x7f22d8006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.291+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 msgr2=0x7f22e410d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.291+0000 7f22e9aed700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009680 tx=0x7f22d4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 shutdown_connections 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f22cc036880 0x7f22cc038d30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 --2- 172.21.15.203:0/3097574646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22e40669c0 0x7f22e410d0e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 >> 172.21.15.203:0/3097574646 conn(0x7f22e40f7090 msgr2=0x7f22e4103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 shutdown_connections 2024-04-05T02:54:02.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:02.292+0000 7f22e9aed700 1 -- 172.21.15.203:0/3097574646 wait complete. 2024-04-05T02:54:02.293 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:03.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:02 smithi153 ceph-mon[29381]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:03.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:02 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3097574646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:03.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:03.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:04.029 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:04.076 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:05.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:04 smithi153 ceph-mon[29381]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 -- 172.21.15.203:0/1199194318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 msgr2=0x7f78640fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1199194318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f78640fce50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f78580048c0 tx=0x7f7858009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 -- 172.21.15.203:0/1199194318 shutdown_connections 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1199194318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f78640fce50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 -- 172.21.15.203:0/1199194318 >> 172.21.15.203:0/1199194318 conn(0x7f78640f81c0 msgr2=0x7f78640fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 -- 172.21.15.203:0/1199194318 shutdown_connections 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.389+0000 7f7868c6c700 1 -- 172.21.15.203:0/1199194318 wait complete. 2024-04-05T02:54:05.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f7868c6c700 1 Processor -- start 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f7868c6c700 1 -- start start 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f7868c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f7868c6c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78640722a0 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f786259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f786259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:60550/0 (socket says 172.21.15.203:60550) 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f786259c700 1 -- 172.21.15.203:0/1274656067 learned_addr learned my addr 172.21.15.203:0/1274656067 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.390+0000 7f786259c700 1 -- 172.21.15.203:0/1274656067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858005040 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f786259c700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78580096b0 tx=0x7f7858009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785800bed0 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7864070370 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f785800b5c0 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785801c5b0 con 0x7f78640fca40 2024-04-05T02:54:05.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.391+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7864070860 con 0x7f78640fca40 2024-04-05T02:54:05.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.392+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786405f5f0 con 0x7f78640fca40 2024-04-05T02:54:05.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.392+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f785801c820 con 0x7f78640fca40 2024-04-05T02:54:05.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.392+0000 7f7852ffd700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 0x7f784c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:05.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.392+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7858049a90 con 0x7f78640fca40 2024-04-05T02:54:05.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.392+0000 7f7861d9b700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 0x7f784c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:05.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.393+0000 7f7861d9b700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 0x7f784c038d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7854009990 tx=0x7f7854006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:05.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.397+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7858023300 con 0x7f78640fca40 2024-04-05T02:54:05.658 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.658+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7864106420 con 0x7f78640fca40 2024-04-05T02:54:05.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.659+0000 7f7852ffd700 1 -- 172.21.15.203:0/1274656067 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7858011b60 con 0x7f78640fca40 2024-04-05T02:54:05.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:05.659 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 msgr2=0x7f784c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 0x7f784c038d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7854009990 tx=0x7f7854006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 msgr2=0x7f7864071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f78580096b0 tx=0x7f7858009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 shutdown_connections 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f784c036880 0x7f784c038d30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 --2- 172.21.15.203:0/1274656067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78640fca40 0x7f7864071d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 >> 172.21.15.203:0/1274656067 conn(0x7f78640f81c0 msgr2=0x7f7864103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 shutdown_connections 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:05.661+0000 7f7868c6c700 1 -- 172.21.15.203:0/1274656067 wait complete. 2024-04-05T02:54:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:06.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:05 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1274656067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:07.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:06 smithi153 ceph-mon[29381]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:07.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:07.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:07.490 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:07.538 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: Deploying daemon alertmanager.smithi153 on smithi153 2024-04-05T02:54:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:08 smithi153 ceph-mon[29381]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 -- 172.21.15.203:0/2236382723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 msgr2=0x7f7ca80fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 --2- 172.21.15.203:0/2236382723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca80fce50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7c980048c0 tx=0x7f7c98009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 -- 172.21.15.203:0/2236382723 shutdown_connections 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 --2- 172.21.15.203:0/2236382723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca80fce50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 -- 172.21.15.203:0/2236382723 >> 172.21.15.203:0/2236382723 conn(0x7f7ca80f81c0 msgr2=0x7f7ca80fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 -- 172.21.15.203:0/2236382723 shutdown_connections 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 -- 172.21.15.203:0/2236382723 wait complete. 2024-04-05T02:54:08.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.958+0000 7f7cae911700 1 Processor -- start 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7cae911700 1 -- start start 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7cae911700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7cae911700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca810d5d0 con 0x7f7ca80fca40 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7ca7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7ca7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:60562/0 (socket says 172.21.15.203:60562) 2024-04-05T02:54:08.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7ca7fff700 1 -- 172.21.15.203:0/1496806380 learned_addr learned my addr 172.21.15.203:0/1496806380 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:08.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.959+0000 7f7ca7fff700 1 -- 172.21.15.203:0/1496806380 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98005040 con 0x7f7ca80fca40 2024-04-05T02:54:08.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7ca7fff700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7c980096b0 tx=0x7f7c98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:08.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9800bed0 con 0x7f7ca80fca40 2024-04-05T02:54:08.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca810d7d0 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c9800b5c0 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9801c5b0 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.960+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca810e340 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.961+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f7c98013a80 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.961+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca8102f50 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.961+0000 7f7ca4ff9700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 0x7f7c90038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.961+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7c98049ae0 con 0x7f7ca80fca40 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.961+0000 7f7ca77fe700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 0x7f7c90038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:08.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.962+0000 7f7ca77fe700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 0x7f7c90038d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009990 tx=0x7f7c9c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:08.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:08.966+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7c98011940 con 0x7f7ca80fca40 2024-04-05T02:54:09.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.224+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ca810ed50 con 0x7f7ca80fca40 2024-04-05T02:54:09.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.225+0000 7f7ca4ff9700 1 -- 172.21.15.203:0/1496806380 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7c98011370 con 0x7f7ca80fca40 2024-04-05T02:54:09.225 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:09.226 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 msgr2=0x7f7c90038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 0x7f7c90038d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009990 tx=0x7f7c9c006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 msgr2=0x7f7ca810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7c980096b0 tx=0x7f7c98009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 shutdown_connections 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7c90036880 0x7f7c90038d30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 --2- 172.21.15.203:0/1496806380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ca80fca40 0x7f7ca810d090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:09.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 >> 172.21.15.203:0/1496806380 conn(0x7f7ca80f81c0 msgr2=0x7f7ca8103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:09.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 shutdown_connections 2024-04-05T02:54:09.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:09.227+0000 7f7cae911700 1 -- 172.21.15.203:0/1496806380 wait complete. 2024-04-05T02:54:09.228 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1496806380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:10.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:10 smithi153 ceph-mon[29381]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:10.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:10.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:11.032 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:11.080 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:11.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:11 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.506+0000 7fdf2e860700 1 -- 172.21.15.203:0/1753558011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 msgr2=0x7fdf280fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.506+0000 7fdf2e860700 1 --2- 172.21.15.203:0/1753558011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf280fce50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf180048c0 tx=0x7fdf18009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 -- 172.21.15.203:0/1753558011 shutdown_connections 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 --2- 172.21.15.203:0/1753558011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf280fce50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 -- 172.21.15.203:0/1753558011 >> 172.21.15.203:0/1753558011 conn(0x7fdf280f81c0 msgr2=0x7fdf280fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 -- 172.21.15.203:0/1753558011 shutdown_connections 2024-04-05T02:54:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 -- 172.21.15.203:0/1753558011 wait complete. 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 Processor -- start 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.507+0000 7fdf2e860700 1 -- start start 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf2e860700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf2e860700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2810d5d0 con 0x7fdf280fca40 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf27fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf27fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:60588/0 (socket says 172.21.15.203:60588) 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf27fff700 1 -- 172.21.15.203:0/2863204296 learned_addr learned my addr 172.21.15.203:0/2863204296 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf27fff700 1 -- 172.21.15.203:0/2863204296 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf18005040 con 0x7fdf280fca40 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf27fff700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009680 tx=0x7fdf18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf1800b5c0 con 0x7fdf280fca40 2024-04-05T02:54:12.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2810d7d0 con 0x7fdf280fca40 2024-04-05T02:54:12.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf1800be70 con 0x7fdf280fca40 2024-04-05T02:54:12.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.508+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf180135a0 con 0x7fdf280fca40 2024-04-05T02:54:12.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.509+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2810e340 con 0x7fdf280fca40 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.509+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fdf18013860 con 0x7fdf280fca40 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.509+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf28102f50 con 0x7fdf280fca40 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.509+0000 7fdf24ff9700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 0x7fdf10038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.510+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdf18049460 con 0x7fdf280fca40 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.510+0000 7fdf277fe700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 0x7fdf10038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:12.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.510+0000 7fdf277fe700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 0x7fdf10038d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009990 tx=0x7fdf1c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:12.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.514+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdf18011960 con 0x7fdf280fca40 2024-04-05T02:54:12.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.773+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf2810ed50 con 0x7fdf280fca40 2024-04-05T02:54:12.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.774+0000 7fdf24ff9700 1 -- 172.21.15.203:0/2863204296 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fdf18021070 con 0x7fdf280fca40 2024-04-05T02:54:12.774 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:12.774 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 msgr2=0x7fdf10038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 0x7fdf10038d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009990 tx=0x7fdf1c006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 msgr2=0x7fdf2810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009680 tx=0x7fdf18009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 shutdown_connections 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fdf10036880 0x7fdf10038d30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 --2- 172.21.15.203:0/2863204296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf280fca40 0x7fdf2810d090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:12.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 >> 172.21.15.203:0/2863204296 conn(0x7fdf280f81c0 msgr2=0x7fdf28103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:12.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 shutdown_connections 2024-04-05T02:54:12.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:12.776+0000 7fdf2e860700 1 -- 172.21.15.203:0/2863204296 wait complete. 2024-04-05T02:54:12.777 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:12.955 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:12 smithi153 ceph-mon[29381]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:13 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2863204296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:14.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:14.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:14.594 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:14.640 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:15.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:14 smithi153 ceph-mon[29381]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2852959197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 msgr2=0x7fb5f00fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2852959197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fe4b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0048c0 tx=0x7fb5ec009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2852959197 shutdown_connections 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2852959197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fe4b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2852959197 >> 172.21.15.203:0/2852959197 conn(0x7fb5f00f6840 msgr2=0x7fb5f00f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2852959197 shutdown_connections 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.057+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2852959197 wait complete. 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f7a40700 1 Processor -- start 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f7a40700 1 -- start start 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f7a40700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f7a40700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00fdcf0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f57dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f57dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:36390/0 (socket says 172.21.15.203:36390) 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f57dc700 1 -- 172.21.15.203:0/2033258424 learned_addr learned my addr 172.21.15.203:0/2033258424 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f57dc700 1 -- 172.21.15.203:0/2033258424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec005040 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.058+0000 7fb5f57dc700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0096b0 tx=0x7fb5ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.059+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ec00bed0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.059+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5ec00b5c0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.059+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5ec01c5b0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.059+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f00fbdc0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.059+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f00fc2b0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fb5ec013a80 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5e5ffb700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 0x7fb5dc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb5ec049ac0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f005f5f0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5f4fdb700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 0x7fb5dc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:16.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.060+0000 7fb5f4fdb700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 0x7fb5dc038d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009990 tx=0x7fb5e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:16.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.065+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb5ec011960 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.325+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5f004bed0 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.325+0000 7fb5e5ffb700 1 -- 172.21.15.203:0/2033258424 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb5ec011370 con 0x7fb5f00fe0a0 2024-04-05T02:54:16.325 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:16.326 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 msgr2=0x7fb5dc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 0x7fb5dc038d30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009990 tx=0x7fb5e0006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 msgr2=0x7fb5f00fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0096b0 tx=0x7fb5ec009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 shutdown_connections 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fb5dc036880 0x7fb5dc038d30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 --2- 172.21.15.203:0/2033258424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f00fe0a0 0x7fb5f00fd7b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 >> 172.21.15.203:0/2033258424 conn(0x7fb5f00f6840 msgr2=0x7fb5f0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 shutdown_connections 2024-04-05T02:54:16.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:16.328+0000 7fb5f7a40700 1 -- 172.21.15.203:0/2033258424 wait complete. 2024-04-05T02:54:16.329 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: Deploying daemon grafana.smithi153 on smithi153 2024-04-05T02:54:16.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:16.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:16.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:16 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2033258424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:17.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:17.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:18.073 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:18.121 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:18 smithi153 ceph-mon[29381]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:19.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 -- 172.21.15.203:0/2698040710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 msgr2=0x7fce4c0fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:19.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 --2- 172.21.15.203:0/2698040710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c0fe4c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fce48009c60 tx=0x7fce48004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:19.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 -- 172.21.15.203:0/2698040710 shutdown_connections 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 --2- 172.21.15.203:0/2698040710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c0fe4c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 -- 172.21.15.203:0/2698040710 >> 172.21.15.203:0/2698040710 conn(0x7fce4c0f6850 msgr2=0x7fce4c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 -- 172.21.15.203:0/2698040710 shutdown_connections 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 -- 172.21.15.203:0/2698040710 wait complete. 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.469+0000 7fce52a84700 1 Processor -- start 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce52a84700 1 -- start start 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce52a84700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce52a84700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c10d600 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce50820700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:19.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce50820700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:36404/0 (socket says 172.21.15.203:36404) 2024-04-05T02:54:19.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce50820700 1 -- 172.21.15.203:0/3095027045 learned_addr learned my addr 172.21.15.203:0/3095027045 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:19.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce50820700 1 -- 172.21.15.203:0/3095027045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48009940 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.470+0000 7fce50820700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce48009500 tx=0x7fce48004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:19.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.471+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce48007c80 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.471+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce4801d910 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.471+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce480129d0 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.471+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c10d800 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.471+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4c10e370 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fce48019040 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4c102f80 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce417fa700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 0x7fce34038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fce4804a310 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce43fff700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 0x7fce34038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:19.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.472+0000 7fce43fff700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 0x7fce34038d80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009990 tx=0x7fce3c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:19.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.477+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fce4802eb40 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.735+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce4c10ecd0 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.735+0000 7fce417fa700 1 -- 172.21.15.203:0/3095027045 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fce48017020 con 0x7fce4c0fe0b0 2024-04-05T02:54:19.736 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:19.736 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:19.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 msgr2=0x7fce34038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:19.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 0x7fce34038d80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009990 tx=0x7fce3c006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:19.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 msgr2=0x7fce4c10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce48009500 tx=0x7fce48004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 shutdown_connections 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fce340368d0 0x7fce34038d80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 --2- 172.21.15.203:0/3095027045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce4c0fe0b0 0x7fce4c10d0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 >> 172.21.15.203:0/3095027045 conn(0x7fce4c0f6850 msgr2=0x7fce4c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 shutdown_connections 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:19.737+0000 7fce52a84700 1 -- 172.21.15.203:0/3095027045 wait complete. 2024-04-05T02:54:19.738 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:20 smithi153 ceph-mon[29381]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:20 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3095027045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:21.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:21.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:21.596 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:21.642 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/745673446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 msgr2=0x7f1b180fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/745673446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b180fd460 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009c60 tx=0x7f1b10004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/745673446 shutdown_connections 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/745673446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b180fd460 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/745673446 >> 172.21.15.203:0/745673446 conn(0x7f1b180f70d0 msgr2=0x7f1b180f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/745673446 shutdown_connections 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.062+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/745673446 wait complete. 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1fd7e700 1 Processor -- start 2024-04-05T02:54:23.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1fd7e700 1 -- start start 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1fd7e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1fd7e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1810c500 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1db1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1db1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:36422/0 (socket says 172.21.15.203:36422) 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1db1a700 1 -- 172.21.15.203:0/3519906315 learned_addr learned my addr 172.21.15.203:0/3519906315 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1db1a700 1 -- 172.21.15.203:0/3519906315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10009940 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.063+0000 7f1b1db1a700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009500 tx=0x7f1b10004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b10007c80 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b1001d910 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1810c700 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b100129d0 con 0x7f1b180fd050 2024-04-05T02:54:23.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1810d270 con 0x7f1b180fd050 2024-04-05T02:54:23.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1b10019040 con 0x7f1b180fd050 2024-04-05T02:54:23.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.064+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b18101e70 con 0x7f1b180fd050 2024-04-05T02:54:23.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.065+0000 7f1b167fc700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 0x7f1b00038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:23.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.065+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1b1004a510 con 0x7f1b180fd050 2024-04-05T02:54:23.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.065+0000 7f1b1d319700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 0x7f1b00038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:23.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.065+0000 7f1b1d319700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 0x7f1b00038d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009990 tx=0x7f1b0c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:23.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.070+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1b1001d440 con 0x7f1b180fd050 2024-04-05T02:54:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:22 smithi153 ceph-mon[29381]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:23.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.331+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b1810dbd0 con 0x7f1b180fd050 2024-04-05T02:54:23.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.331+0000 7f1b167fc700 1 -- 172.21.15.203:0/3519906315 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1b10017070 con 0x7f1b180fd050 2024-04-05T02:54:23.331 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:23.332 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:23.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 msgr2=0x7f1b00038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:23.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 0x7f1b00038d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009990 tx=0x7f1b0c006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:23.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 msgr2=0x7f1b1810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009500 tx=0x7f1b10004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 shutdown_connections 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b00036880 0x7f1b00038d30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 --2- 172.21.15.203:0/3519906315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b180fd050 0x7f1b1810bfc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 >> 172.21.15.203:0/3519906315 conn(0x7f1b180f70d0 msgr2=0x7f1b18102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 shutdown_connections 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:23.333+0000 7f1b1fd7e700 1 -- 172.21.15.203:0/3519906315 wait complete. 2024-04-05T02:54:23.334 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:24.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:23 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3519906315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:24.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:24.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:24 smithi153 ceph-mon[29381]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:25.157 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:25.206 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 -- 172.21.15.203:0/2646734829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 msgr2=0x7f68c80ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 --2- 172.21.15.203:0/2646734829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c80ffa80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f68b80048c0 tx=0x7f68b8009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 -- 172.21.15.203:0/2646734829 shutdown_connections 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 --2- 172.21.15.203:0/2646734829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c80ffa80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 -- 172.21.15.203:0/2646734829 >> 172.21.15.203:0/2646734829 conn(0x7f68c80f70c0 msgr2=0x7f68c80f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:26.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 -- 172.21.15.203:0/2646734829 shutdown_connections 2024-04-05T02:54:26.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.645+0000 7f68cd486700 1 -- 172.21.15.203:0/2646734829 wait complete. 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68cd486700 1 Processor -- start 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68cd486700 1 -- start start 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68cd486700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68cd486700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c810d600 con 0x7f68c80fd6a0 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68c6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68c6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:47242/0 (socket says 172.21.15.203:47242) 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68c6ffd700 1 -- 172.21.15.203:0/2747768561 learned_addr learned my addr 172.21.15.203:0/2747768561 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68c6ffd700 1 -- 172.21.15.203:0/2747768561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8005040 con 0x7f68c80fd6a0 2024-04-05T02:54:26.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.646+0000 7f68c6ffd700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68b800b900 tx=0x7f68b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b800b5c0 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68b800be70 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c810d800 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b801c590 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c810e370 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f68b8013420 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.647+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c8102f50 con 0x7f68c80fd6a0 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.648+0000 7f68af7fe700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 0x7f68b0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:26.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.648+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f68b8049400 con 0x7f68c80fd6a0 2024-04-05T02:54:26.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.648+0000 7f68c67fc700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 0x7f68b0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:26.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.648+0000 7f68c67fc700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 0x7f68b0038d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009990 tx=0x7f68bc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:26.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.653+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f68b8011550 con 0x7f68c80fd6a0 2024-04-05T02:54:26.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:26 smithi153 ceph-mon[29381]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:26.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.914+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68c810ee40 con 0x7f68c80fd6a0 2024-04-05T02:54:26.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.914+0000 7f68af7fe700 1 -- 172.21.15.203:0/2747768561 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f68b8021020 con 0x7f68c80fd6a0 2024-04-05T02:54:26.914 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:26.915 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:26.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 msgr2=0x7f68b0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:26.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 0x7f68b0038d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009990 tx=0x7f68bc006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:26.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 msgr2=0x7f68c810d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68b800b900 tx=0x7f68b8009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 shutdown_connections 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f68b0036880 0x7f68b0038d30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 --2- 172.21.15.203:0/2747768561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68c80fd6a0 0x7f68c810d0c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 >> 172.21.15.203:0/2747768561 conn(0x7f68c80f70c0 msgr2=0x7f68c8103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 shutdown_connections 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:26.916+0000 7f68cd486700 1 -- 172.21.15.203:0/2747768561 wait complete. 2024-04-05T02:54:26.917 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:27.985 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:27 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2747768561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:28.504 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:28.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:28.754 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:28.800 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:28.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:28 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:30.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:29 smithi153 ceph-mon[29381]: Deploying daemon prometheus.smithi153 on smithi153 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 -- 172.21.15.203:0/4037128679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 msgr2=0x7f87d00fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 --2- 172.21.15.203:0/4037128679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d00fe4b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0048c0 tx=0x7f87cc009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 -- 172.21.15.203:0/4037128679 shutdown_connections 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 --2- 172.21.15.203:0/4037128679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d00fe4b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 -- 172.21.15.203:0/4037128679 >> 172.21.15.203:0/4037128679 conn(0x7f87d00f6840 msgr2=0x7f87d00f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 -- 172.21.15.203:0/4037128679 shutdown_connections 2024-04-05T02:54:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.166+0000 7f87d854c700 1 -- 172.21.15.203:0/4037128679 wait complete. 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d854c700 1 Processor -- start 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d854c700 1 -- start start 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d854c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d854c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d010d620 con 0x7f87d00fe0a0 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d62e8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d62e8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:47260/0 (socket says 172.21.15.203:47260) 2024-04-05T02:54:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d62e8700 1 -- 172.21.15.203:0/2136882071 learned_addr learned my addr 172.21.15.203:0/2136882071 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.167+0000 7f87d62e8700 1 -- 172.21.15.203:0/2136882071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc005040 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87d62e8700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0096b0 tx=0x7f87cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc00bed0 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87cc00b5c0 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d010d820 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc01c5b0 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.168+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d010e390 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.169+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f87cc01c710 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.169+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d0102f70 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.169+0000 7f87c6ffd700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 0x7f87bc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.169+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f87cc049b90 con 0x7f87d00fe0a0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.169+0000 7f87d5ae7700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 0x7f87bc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:30.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.170+0000 7f87d5ae7700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 0x7f87bc038d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009990 tx=0x7f87c0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:30.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.174+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f87cc024e20 con 0x7f87d00fe0a0 2024-04-05T02:54:30.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.433+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87d0102f70 con 0x7f87d00fe0a0 2024-04-05T02:54:30.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.434+0000 7f87c6ffd700 1 -- 172.21.15.203:0/2136882071 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f87cc004d30 con 0x7f87d00fe0a0 2024-04-05T02:54:30.435 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:30.435 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 msgr2=0x7f87bc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 0x7f87bc038d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009990 tx=0x7f87c0006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 msgr2=0x7f87d010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0096b0 tx=0x7f87cc009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 shutdown_connections 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f87bc0368d0 0x7f87bc038d80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 --2- 172.21.15.203:0/2136882071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87d00fe0a0 0x7f87d010d0e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 >> 172.21.15.203:0/2136882071 conn(0x7f87d00f6840 msgr2=0x7f87d00f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 shutdown_connections 2024-04-05T02:54:30.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:30.436+0000 7f87d854c700 1 -- 172.21.15.203:0/2136882071 wait complete. 2024-04-05T02:54:30.438 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:30 smithi153 ceph-mon[29381]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:30 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2136882071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:30 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:54:32.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:32.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:32.277 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:32.324 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:33.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:32 smithi153 ceph-mon[29381]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:33.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/3487523904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 msgr2=0x7fcc480fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/3487523904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc480fce70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c0048c0 tx=0x7fcc3c009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/3487523904 shutdown_connections 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/3487523904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc480fce70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/3487523904 >> 172.21.15.203:0/3487523904 conn(0x7fcc480f81c0 msgr2=0x7fcc480fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/3487523904 shutdown_connections 2024-04-05T02:54:33.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.765+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/3487523904 wait complete. 2024-04-05T02:54:33.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.766+0000 7fcc4ea1d700 1 Processor -- start 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.766+0000 7fcc4ea1d700 1 -- start start 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.766+0000 7fcc4ea1d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.766+0000 7fcc4ea1d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4810d5d0 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.766+0000 7fcc47fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc47fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:47282/0 (socket says 172.21.15.203:47282) 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc47fff700 1 -- 172.21.15.203:0/1837307297 learned_addr learned my addr 172.21.15.203:0/1837307297 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc47fff700 1 -- 172.21.15.203:0/1837307297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c005040 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc47fff700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009680 tx=0x7fcc3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc3c00b5c0 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc3c00be70 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4810d7d0 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc3c0135a0 con 0x7fcc480fca60 2024-04-05T02:54:33.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.767+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4810e340 con 0x7fcc480fca60 2024-04-05T02:54:33.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.768+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fcc3c01c430 con 0x7fcc480fca60 2024-04-05T02:54:33.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.768+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc48102f70 con 0x7fcc480fca60 2024-04-05T02:54:33.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.768+0000 7fcc44ff9700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 0x7fcc30038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:33.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.768+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcc3c0493a0 con 0x7fcc480fca60 2024-04-05T02:54:33.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.769+0000 7fcc477fe700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 0x7fcc30038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:33.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.769+0000 7fcc477fe700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 0x7fcc30038d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc38009990 tx=0x7fcc38006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:33.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:33.774+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcc3c01c7d0 con 0x7fcc480fca60 2024-04-05T02:54:34.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.032+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc4810ed50 con 0x7fcc480fca60 2024-04-05T02:54:34.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.032+0000 7fcc44ff9700 1 -- 172.21.15.203:0/1837307297 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fcc3c021070 con 0x7fcc480fca60 2024-04-05T02:54:34.033 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:34.033 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 msgr2=0x7fcc30038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 0x7fcc30038d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcc38009990 tx=0x7fcc38006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 msgr2=0x7fcc4810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009680 tx=0x7fcc3c009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 shutdown_connections 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fcc30036880 0x7fcc30038d30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 --2- 172.21.15.203:0/1837307297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc480fca60 0x7fcc4810d090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:34.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 >> 172.21.15.203:0/1837307297 conn(0x7fcc480f81c0 msgr2=0x7fcc48103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:34.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 shutdown_connections 2024-04-05T02:54:34.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:34.035+0000 7fcc4ea1d700 1 -- 172.21.15.203:0/1837307297 wait complete. 2024-04-05T02:54:34.036 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:34 smithi153 ceph-mon[29381]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:34 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1837307297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:35.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:35.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:35.920 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:35.967 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:36 smithi153 ceph-mon[29381]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:37.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 -- 172.21.15.203:0/3510384170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 msgr2=0x7f44000fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:37.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 --2- 172.21.15.203:0/3510384170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f44000fd450 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f43f00048c0 tx=0x7f43f0009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:37.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 -- 172.21.15.203:0/3510384170 shutdown_connections 2024-04-05T02:54:37.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 --2- 172.21.15.203:0/3510384170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f44000fd450 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:37.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 -- 172.21.15.203:0/3510384170 >> 172.21.15.203:0/3510384170 conn(0x7f44000f70c0 msgr2=0x7f44000f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:37.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 -- 172.21.15.203:0/3510384170 shutdown_connections 2024-04-05T02:54:37.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 -- 172.21.15.203:0/3510384170 wait complete. 2024-04-05T02:54:37.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.479+0000 7f4408555700 1 Processor -- start 2024-04-05T02:54:37.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f4408555700 1 -- start start 2024-04-05T02:54:37.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f4408555700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:37.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f4408555700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440010d5c0 con 0x7f44000fd040 2024-04-05T02:54:37.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f44062f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:37.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f44062f1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:46204/0 (socket says 172.21.15.203:46204) 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f44062f1700 1 -- 172.21.15.203:0/3771736596 learned_addr learned my addr 172.21.15.203:0/3771736596 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.480+0000 7f44062f1700 1 -- 172.21.15.203:0/3771736596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0005040 con 0x7f44000fd040 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f44062f1700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f43f00096b0 tx=0x7f43f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f000bed0 con 0x7f44000fd040 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440010d7c0 con 0x7f44000fd040 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43f000b5c0 con 0x7f44000fd040 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f001c5b0 con 0x7f44000fd040 2024-04-05T02:54:37.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.481+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440010e330 con 0x7f44000fd040 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.482+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f43f0013a80 con 0x7f44000fd040 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.482+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400101e60 con 0x7f44000fd040 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.482+0000 7f43f6ffd700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 0x7f43ec038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.482+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f43f0049a20 con 0x7f44000fd040 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.482+0000 7f4405af0700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 0x7f43ec038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:37.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.483+0000 7f4405af0700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 0x7f43ec038d30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009990 tx=0x7f43fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:37.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.487+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f43f0011940 con 0x7f44000fd040 2024-04-05T02:54:37.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.749+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f440010e570 con 0x7f44000fd040 2024-04-05T02:54:37.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.750+0000 7f43f6ffd700 1 -- 172.21.15.203:0/3771736596 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f43f0011370 con 0x7f44000fd040 2024-04-05T02:54:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:37.750 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 msgr2=0x7f43ec038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 0x7f43ec038d30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009990 tx=0x7f43fc006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 msgr2=0x7f440010d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f43f00096b0 tx=0x7f43f0009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 shutdown_connections 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f43ec036880 0x7f43ec038d30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 --2- 172.21.15.203:0/3771736596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44000fd040 0x7f440010d080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 >> 172.21.15.203:0/3771736596 conn(0x7f44000f70c0 msgr2=0x7f4400065900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 shutdown_connections 2024-04-05T02:54:37.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:37.752+0000 7f4408555700 1 -- 172.21.15.203:0/3771736596 wait complete. 2024-04-05T02:54:37.753 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:38 smithi153 ceph-mon[29381]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:38 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3771736596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:39.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:39.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:39.543 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:39.590 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:41.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 -- 172.21.15.203:0/3208675960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 msgr2=0x7fa9dc0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 --2- 172.21.15.203:0/3208675960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc0fbd70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0048c0 tx=0x7fa9cc009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 -- 172.21.15.203:0/3208675960 shutdown_connections 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 --2- 172.21.15.203:0/3208675960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc0fbd70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 -- 172.21.15.203:0/3208675960 >> 172.21.15.203:0/3208675960 conn(0x7fa9dc0f70c0 msgr2=0x7fa9dc0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 -- 172.21.15.203:0/3208675960 shutdown_connections 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.009+0000 7fa9e1797700 1 -- 172.21.15.203:0/3208675960 wait complete. 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9e1797700 1 Processor -- start 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9e1797700 1 -- start start 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9e1797700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9e1797700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc10fdb0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:41.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9daffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:46222/0 (socket says 172.21.15.203:46222) 2024-04-05T02:54:41.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.010+0000 7fa9daffd700 1 -- 172.21.15.203:0/614223901 learned_addr learned my addr 172.21.15.203:0/614223901 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:41.012 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9daffd700 1 -- 172.21.15.203:0/614223901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc005040 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9daffd700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0096b0 tx=0x7fa9cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc00bed0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9cc00b5c0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc10fef0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9cc01c5b0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.011+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc1103e0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.012+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc105920 con 0x7fa9dc0fb960 2024-04-05T02:54:41.014 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.012+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fa9cc013a80 con 0x7fa9dc0fb960 2024-04-05T02:54:41.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.012+0000 7fa9d3fff700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 0x7fa9c8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:41.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.012+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa9cc049a20 con 0x7fa9dc0fb960 2024-04-05T02:54:41.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.012+0000 7fa9d27fc700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 0x7fa9c8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:41.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.013+0000 7fa9d27fc700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 0x7fa9c8038d30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009990 tx=0x7fa9c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:41.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.017+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa9cc011940 con 0x7fa9dc0fb960 2024-04-05T02:54:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:40 smithi153 ceph-mon[29381]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:41.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.272+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9dc05f5f0 con 0x7fa9dc0fb960 2024-04-05T02:54:41.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.273+0000 7fa9d3fff700 1 -- 172.21.15.203:0/614223901 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa9cc011370 con 0x7fa9dc0fb960 2024-04-05T02:54:41.273 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:41.274 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:41.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 msgr2=0x7fa9c8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:41.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 0x7fa9c8038d30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009990 tx=0x7fa9c4006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:41.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 msgr2=0x7fa9dc066460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0096b0 tx=0x7fa9cc009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 shutdown_connections 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa9c8036880 0x7fa9c8038d30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 --2- 172.21.15.203:0/614223901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc0fb960 0x7fa9dc066460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 >> 172.21.15.203:0/614223901 conn(0x7fa9dc0f70c0 msgr2=0x7fa9dc102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 shutdown_connections 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:41.276+0000 7fa9e1797700 1 -- 172.21.15.203:0/614223901 wait complete. 2024-04-05T02:54:41.277 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:42.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:41 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/614223901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:42.849 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:42.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:42.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:42 smithi153 ceph-mon[29381]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:43.146 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 -- 172.21.15.203:0/163582276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 msgr2=0x7fa1e00fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 --2- 172.21.15.203:0/163582276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fc370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009c60 tx=0x7fa1dc004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 -- 172.21.15.203:0/163582276 shutdown_connections 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 --2- 172.21.15.203:0/163582276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fc370 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 -- 172.21.15.203:0/163582276 >> 172.21.15.203:0/163582276 conn(0x7fa1e00f6850 msgr2=0x7fa1e00f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:44.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 -- 172.21.15.203:0/163582276 shutdown_connections 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.525+0000 7fa1e797a700 1 -- 172.21.15.203:0/163582276 wait complete. 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e797a700 1 Processor -- start 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e797a700 1 -- start start 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e797a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e797a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e010fdc0 con 0x7fa1e00fbf60 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e5716700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e5716700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:49888/0 (socket says 172.21.15.203:49888) 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.526+0000 7fa1e5716700 1 -- 172.21.15.203:0/2352699090 learned_addr learned my addr 172.21.15.203:0/2352699090 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1e5716700 1 -- 172.21.15.203:0/2352699090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009940 con 0x7fa1e00fbf60 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1e5716700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0097b0 tx=0x7fa1dc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc012410 con 0x7fa1e00fbf60 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1dc012a40 con 0x7fa1e00fbf60 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1dc01b980 con 0x7fa1e00fbf60 2024-04-05T02:54:44.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e010ff00 con 0x7fa1e00fbf60 2024-04-05T02:54:44.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.527+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e01103f0 con 0x7fa1e00fbf60 2024-04-05T02:54:44.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.528+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fa1dc019040 con 0x7fa1e00fbf60 2024-04-05T02:54:44.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.528+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e0105920 con 0x7fa1e00fbf60 2024-04-05T02:54:44.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.528+0000 7fa1d5ffb700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 0x7fa1cc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:44.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.528+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa1dc049d60 con 0x7fa1e00fbf60 2024-04-05T02:54:44.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.528+0000 7fa1e4f15700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 0x7fa1cc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:44.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.529+0000 7fa1e4f15700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 0x7fa1cc038d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009990 tx=0x7fa1d0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:44.533 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.533+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa1dc022ac0 con 0x7fa1e00fbf60 2024-04-05T02:54:44.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.796+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1e005f5f0 con 0x7fa1e00fbf60 2024-04-05T02:54:44.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.797+0000 7fa1d5ffb700 1 -- 172.21.15.203:0/2352699090 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa1dc017090 con 0x7fa1e00fbf60 2024-04-05T02:54:44.797 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:44.797 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:44.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 msgr2=0x7fa1cc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:44.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 0x7fa1cc038d30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009990 tx=0x7fa1d0006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:44.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 msgr2=0x7fa1e00fb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:44.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0097b0 tx=0x7fa1dc004d10 comp rx=0 tx=0).stop 2024-04-05T02:54:44.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 shutdown_connections 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fa1cc036880 0x7fa1cc038d30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 --2- 172.21.15.203:0/2352699090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1e00fbf60 0x7fa1e00fb780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 >> 172.21.15.203:0/2352699090 conn(0x7fa1e00f6850 msgr2=0x7fa1e0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 shutdown_connections 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:44.799+0000 7fa1e797a700 1 -- 172.21.15.203:0/2352699090 wait complete. 2024-04-05T02:54:44.800 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:44 smithi153 ceph-mon[29381]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:46.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:45 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2352699090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:46.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:46.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:46.615 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:46.662 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:47.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:46 smithi153 ceph-mon[29381]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:48.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 -- 172.21.15.203:0/4146674492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 msgr2=0x7f7d380fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 --2- 172.21.15.203:0/4146674492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d380fce80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d340048c0 tx=0x7f7d34009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 -- 172.21.15.203:0/4146674492 shutdown_connections 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 --2- 172.21.15.203:0/4146674492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d380fce80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 -- 172.21.15.203:0/4146674492 >> 172.21.15.203:0/4146674492 conn(0x7f7d380f81d0 msgr2=0x7f7d380fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.052+0000 7f7d3f990700 1 -- 172.21.15.203:0/4146674492 shutdown_connections 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3f990700 1 -- 172.21.15.203:0/4146674492 wait complete. 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3f990700 1 Processor -- start 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3f990700 1 -- start start 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3f990700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3f990700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380722a0 con 0x7f7d380fca70 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3d72c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3d72c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:49918/0 (socket says 172.21.15.203:49918) 2024-04-05T02:54:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.053+0000 7f7d3d72c700 1 -- 172.21.15.203:0/3313920418 learned_addr learned my addr 172.21.15.203:0/3313920418 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:48.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d3d72c700 1 -- 172.21.15.203:0/3313920418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34005040 con 0x7f7d380fca70 2024-04-05T02:54:48.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d3d72c700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d340096b0 tx=0x7f7d34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d3400bed0 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d38070370 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d3400b5c0 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d3401c5b0 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.054+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d38070860 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.055+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3805f5f0 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.055+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f7d34013a80 con 0x7f7d380fca70 2024-04-05T02:54:48.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.055+0000 7f7d2dffb700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 0x7f7d24038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:48.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.055+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7d34049a30 con 0x7f7d380fca70 2024-04-05T02:54:48.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.055+0000 7f7d3cf2b700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 0x7f7d24038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:48.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.056+0000 7f7d3cf2b700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 0x7f7d24038d30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009990 tx=0x7f7d28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:48.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.060+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7d34011960 con 0x7f7d380fca70 2024-04-05T02:54:48.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.321+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d38106450 con 0x7f7d380fca70 2024-04-05T02:54:48.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.322+0000 7f7d2dffb700 1 -- 172.21.15.203:0/3313920418 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7d34011370 con 0x7f7d380fca70 2024-04-05T02:54:48.322 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:48.322 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:48.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 msgr2=0x7f7d24038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:48.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 0x7f7d24038d30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009990 tx=0x7f7d28006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:48.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 msgr2=0x7f7d38071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:48.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d340096b0 tx=0x7f7d34009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:48.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 shutdown_connections 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f7d24036880 0x7f7d24038d30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 --2- 172.21.15.203:0/3313920418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d380fca70 0x7f7d38071d60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 >> 172.21.15.203:0/3313920418 conn(0x7f7d380f81d0 msgr2=0x7f7d38103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 shutdown_connections 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:48.324+0000 7f7d3f990700 1 -- 172.21.15.203:0/3313920418 wait complete. 2024-04-05T02:54:48.325 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:48 smithi153 ceph-mon[29381]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:49.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:48 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3313920418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:49.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:49.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:50.095 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:50.142 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:50 smithi153 ceph-mon[29381]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 -- 172.21.15.203:0/3462754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 msgr2=0x7fd6dc0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/3462754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc0fd460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40048c0 tx=0x7fd6c4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 -- 172.21.15.203:0/3462754279 shutdown_connections 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/3462754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc0fd460 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 -- 172.21.15.203:0/3462754279 >> 172.21.15.203:0/3462754279 conn(0x7fd6dc0f70d0 msgr2=0x7fd6dc0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 -- 172.21.15.203:0/3462754279 shutdown_connections 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.607+0000 7fd6e27b2700 1 -- 172.21.15.203:0/3462754279 wait complete. 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6e27b2700 1 Processor -- start 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6e27b2700 1 -- start start 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6e27b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:51.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6e27b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc10f3b0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:49932/0 (socket says 172.21.15.203:49932) 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.608+0000 7fd6dbfff700 1 -- 172.21.15.203:0/809145483 learned_addr learned my addr 172.21.15.203:0/809145483 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6dbfff700 1 -- 172.21.15.203:0/809145483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4005040 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6dbfff700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40096b0 tx=0x7fd6c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6c400bed0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc10b3a0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6c400b5c0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6c401c5b0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.609+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc10b890 con 0x7fd6dc0fd050 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.610+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc101e70 con 0x7fd6dc0fd050 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.610+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fd6c4004d30 con 0x7fd6dc0fd050 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.610+0000 7fd6d8ff9700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 0x7fd6c8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.610+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd6c4049a50 con 0x7fd6dc0fd050 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.610+0000 7fd6db7fe700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 0x7fd6c8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:51.611 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.611+0000 7fd6db7fe700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 0x7fd6c8038d30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009990 tx=0x7fd6cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:51.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.615+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd6c4011940 con 0x7fd6dc0fd050 2024-04-05T02:54:51.875 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.875+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6dc04bed0 con 0x7fd6dc0fd050 2024-04-05T02:54:51.876 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.876+0000 7fd6d8ff9700 1 -- 172.21.15.203:0/809145483 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd6c4011370 con 0x7fd6dc0fd050 2024-04-05T02:54:51.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:51.877 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:51.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 msgr2=0x7fd6c8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 0x7fd6c8038d30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009990 tx=0x7fd6cc006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 msgr2=0x7fd6dc10ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40096b0 tx=0x7fd6c4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 shutdown_connections 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fd6c8036880 0x7fd6c8038d30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:51.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 --2- 172.21.15.203:0/809145483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6dc0fd050 0x7fd6dc10ae60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:51.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 >> 172.21.15.203:0/809145483 conn(0x7fd6dc0f70d0 msgr2=0x7fd6dc0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:51.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 shutdown_connections 2024-04-05T02:54:51.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:51.878+0000 7fd6e27b2700 1 -- 172.21.15.203:0/809145483 wait complete. 2024-04-05T02:54:51.880 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:52 smithi153 ceph-mon[29381]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:52 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/809145483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:53.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:53.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:53.746 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:53.792 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:55.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:54 smithi153 ceph-mon[29381]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- 172.21.15.203:0/2778006597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 msgr2=0x7fe8f00fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2778006597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f00fce50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40048c0 tx=0x7fe8e4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- 172.21.15.203:0/2778006597 shutdown_connections 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2778006597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f00fce50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- 172.21.15.203:0/2778006597 >> 172.21.15.203:0/2778006597 conn(0x7fe8f00f81c0 msgr2=0x7fe8f00fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- 172.21.15.203:0/2778006597 shutdown_connections 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- 172.21.15.203:0/2778006597 wait complete. 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 Processor -- start 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.290+0000 7fe8f5214700 1 -- start start 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8f5214700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:55.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8f5214700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00722a0 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:43052/0 (socket says 172.21.15.203:43052) 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8eed9d700 1 -- 172.21.15.203:0/2689682241 learned_addr learned my addr 172.21.15.203:0/2689682241 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8eed9d700 1 -- 172.21.15.203:0/2689682241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e4005040 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.291+0000 7fe8eed9d700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40096b0 tx=0x7fe8e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e400bed0 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8e400b5c0 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f0070370 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e401c5b0 con 0x7fe8f00fca40 2024-04-05T02:54:55.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f0070860 con 0x7fe8f00fca40 2024-04-05T02:54:55.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.292+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f005f5f0 con 0x7fe8f00fca40 2024-04-05T02:54:55.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.293+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe8e4013a80 con 0x7fe8f00fca40 2024-04-05T02:54:55.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.293+0000 7fe8d77fe700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 0x7fe8d8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:55.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.293+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe8e40499e0 con 0x7fe8f00fca40 2024-04-05T02:54:55.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.293+0000 7fe8ee59c700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 0x7fe8d8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:55.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.294+0000 7fe8ee59c700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 0x7fe8d8038d80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009990 tx=0x7fe8e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:55.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.298+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe8e4011940 con 0x7fe8f00fca40 2024-04-05T02:54:55.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.555+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8f0106420 con 0x7fe8f00fca40 2024-04-05T02:54:55.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.556+0000 7fe8d77fe700 1 -- 172.21.15.203:0/2689682241 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fe8e4011370 con 0x7fe8f00fca40 2024-04-05T02:54:55.556 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:55.556 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 msgr2=0x7fe8d8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 0x7fe8d8038d80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009990 tx=0x7fe8e0006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 msgr2=0x7fe8f0071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40096b0 tx=0x7fe8e4009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 shutdown_connections 2024-04-05T02:54:55.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe8d80368d0 0x7fe8d8038d80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 --2- 172.21.15.203:0/2689682241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8f00fca40 0x7fe8f0071d60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 >> 172.21.15.203:0/2689682241 conn(0x7fe8f00f81c0 msgr2=0x7fe8f0103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 shutdown_connections 2024-04-05T02:54:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:55.558+0000 7fe8f5214700 1 -- 172.21.15.203:0/2689682241 wait complete. 2024-04-05T02:54:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:56.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:55 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2689682241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:54:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:56 smithi153 ceph-mon[29381]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:54:57.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:54:57.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:54:57.376 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:57.424 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4131196905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 msgr2=0x7f1b340fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4131196905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b340fce70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b240048c0 tx=0x7f1b24009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4131196905 shutdown_connections 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4131196905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b340fce70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4131196905 >> 172.21.15.203:0/4131196905 conn(0x7f1b340f81c0 msgr2=0x7f1b340fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4131196905 shutdown_connections 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4131196905 wait complete. 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 Processor -- start 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.737+0000 7f1b39b7a700 1 -- start start 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b39b7a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:58.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b39b7a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340722a0 con 0x7f1b340fca60 2024-04-05T02:54:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b337fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b337fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:43070/0 (socket says 172.21.15.203:43070) 2024-04-05T02:54:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b337fe700 1 -- 172.21.15.203:0/4057341467 learned_addr learned my addr 172.21.15.203:0/4057341467 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:54:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b337fe700 1 -- 172.21.15.203:0/4057341467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24005040 con 0x7f1b340fca60 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b337fe700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b240096b0 tx=0x7f1b24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2400bed0 con 0x7f1b340fca60 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b34070370 con 0x7f1b340fca60 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b2400b5c0 con 0x7f1b340fca60 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2401c5b0 con 0x7f1b340fca60 2024-04-05T02:54:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.738+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b34070860 con 0x7f1b340fca60 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.739+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3405f5f0 con 0x7f1b340fca60 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.739+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1b24004d30 con 0x7f1b340fca60 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.740+0000 7f1b1bfff700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 0x7f1b1c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.740+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1b24049a20 con 0x7f1b340fca60 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.740+0000 7f1b32ffd700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 0x7f1b1c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:54:58.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.740+0000 7f1b32ffd700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 0x7f1b1c038d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009990 tx=0x7f1b28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:54:58.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:58.745+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1b24011960 con 0x7f1b340fca60 2024-04-05T02:54:59.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.003+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b3404bed0 con 0x7f1b340fca60 2024-04-05T02:54:59.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.004+0000 7f1b1bfff700 1 -- 172.21.15.203:0/4057341467 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1b24011370 con 0x7f1b340fca60 2024-04-05T02:54:59.004 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:54:59.004 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 msgr2=0x7f1b1c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 0x7f1b1c038d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009990 tx=0x7f1b28006e40 comp rx=0 tx=0).stop 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 msgr2=0x7f1b34071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b240096b0 tx=0x7f1b24009e30 comp rx=0 tx=0).stop 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 shutdown_connections 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f1b1c036880 0x7f1b1c038d30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 --2- 172.21.15.203:0/4057341467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b340fca60 0x7f1b34071d60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 >> 172.21.15.203:0/4057341467 conn(0x7f1b340f81c0 msgr2=0x7f1b34103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 shutdown_connections 2024-04-05T02:54:59.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:54:59.006+0000 7f1b39b7a700 1 -- 172.21.15.203:0/4057341467 wait complete. 2024-04-05T02:54:59.007 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:54:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:58 smithi153 ceph-mon[29381]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:00.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:54:59 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/4057341467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:00.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:00.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:00.819 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:00.866 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:00 smithi153 ceph-mon[29381]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:02.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1918558546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 msgr2=0x7fbcf00ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:02.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1918558546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf00ffa80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009c60 tx=0x7fbce8004d10 comp rx=0 tx=0).stop 2024-04-05T02:55:02.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1918558546 shutdown_connections 2024-04-05T02:55:02.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1918558546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf00ffa80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1918558546 >> 172.21.15.203:0/1918558546 conn(0x7fbcf00f70c0 msgr2=0x7fbcf00f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1918558546 shutdown_connections 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.305+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1918558546 wait complete. 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcf5c35700 1 Processor -- start 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcf5c35700 1 -- start start 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcf5c35700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcf5c35700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf010d600 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcef7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcef7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:43090/0 (socket says 172.21.15.203:43090) 2024-04-05T02:55:02.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.306+0000 7fbcef7fe700 1 -- 172.21.15.203:0/1550246903 learned_addr learned my addr 172.21.15.203:0/1550246903 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbcef7fe700 1 -- 172.21.15.203:0/1550246903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce8009940 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbcef7fe700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbce80097b0 tx=0x7fbce8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce8012410 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbce8012a40 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf010d800 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce801ba20 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.307+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf010e370 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.308+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fbce8019040 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.308+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0102f50 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.308+0000 7fbce7fff700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 0x7fbcdc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.308+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbce8049c60 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.308+0000 7fbceeffd700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 0x7fbcdc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:02.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.309+0000 7fbceeffd700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 0x7fbcdc038d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009990 tx=0x7fbcd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:02.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.313+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbce8012bb0 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.571+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcf010ee40 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.571+0000 7fbce7fff700 1 -- 172.21.15.203:0/1550246903 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fbce8017300 con 0x7fbcf00fd6a0 2024-04-05T02:55:02.572 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:02.572 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:02.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 msgr2=0x7fbcdc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:02.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 0x7fbcdc038d30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009990 tx=0x7fbcd8006e40 comp rx=0 tx=0).stop 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 msgr2=0x7fbcf010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbce80097b0 tx=0x7fbce8004d10 comp rx=0 tx=0).stop 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 shutdown_connections 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fbcdc036880 0x7fbcdc038d30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 --2- 172.21.15.203:0/1550246903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcf00fd6a0 0x7fbcf010d0c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.573+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 >> 172.21.15.203:0/1550246903 conn(0x7fbcf00f70c0 msgr2=0x7fbcf0103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.574+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 shutdown_connections 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:02.574+0000 7fbcf5c35700 1 -- 172.21.15.203:0/1550246903 wait complete. 2024-04-05T02:55:02.574 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:03.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:02 smithi153 ceph-mon[29381]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:03.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:02 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1550246903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:04.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:04.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:04.431 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:04.486 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:05.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:04 smithi153 ceph-mon[29381]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:06.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 -- 172.21.15.203:0/1153950887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 msgr2=0x7fe26c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:06.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 --2- 172.21.15.203:0/1153950887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c0ffa30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0048c0 tx=0x7fe25c009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:06.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 -- 172.21.15.203:0/1153950887 shutdown_connections 2024-04-05T02:55:06.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 --2- 172.21.15.203:0/1153950887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c0ffa30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 -- 172.21.15.203:0/1153950887 >> 172.21.15.203:0/1153950887 conn(0x7fe26c0f70d0 msgr2=0x7fe26c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 -- 172.21.15.203:0/1153950887 shutdown_connections 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.038+0000 7fe27148a700 1 -- 172.21.15.203:0/1153950887 wait complete. 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe27148a700 1 Processor -- start 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe27148a700 1 -- start start 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe27148a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe27148a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10d5c0 con 0x7fe26c0fd650 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe26affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe26affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33048/0 (socket says 172.21.15.203:33048) 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe26affd700 1 -- 172.21.15.203:0/3969923932 learned_addr learned my addr 172.21.15.203:0/3969923932 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe26affd700 1 -- 172.21.15.203:0/3969923932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c005040 con 0x7fe26c0fd650 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.039+0000 7fe26affd700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b900 tx=0x7fe25c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:06.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c00b5c0 con 0x7fe26c0fd650 2024-04-05T02:55:06.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe25c00be70 con 0x7fe26c0fd650 2024-04-05T02:55:06.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c01c590 con 0x7fe26c0fd650 2024-04-05T02:55:06.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c10d7c0 con 0x7fe26c0fd650 2024-04-05T02:55:06.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c10e330 con 0x7fe26c0fd650 2024-04-05T02:55:06.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.040+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe25c01c800 con 0x7fe26c0fd650 2024-04-05T02:55:06.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.041+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c102f60 con 0x7fe26c0fd650 2024-04-05T02:55:06.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.041+0000 7fe2537fe700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 0x7fe254038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:06.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.041+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe25c049500 con 0x7fe26c0fd650 2024-04-05T02:55:06.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.041+0000 7fe26a7fc700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 0x7fe254038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:06.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.041+0000 7fe26a7fc700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 0x7fe254038d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe260009990 tx=0x7fe260006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:06.046 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.046+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe25c011960 con 0x7fe26c0fd650 2024-04-05T02:55:06.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.305+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe26c10ed40 con 0x7fe26c0fd650 2024-04-05T02:55:06.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.306+0000 7fe2537fe700 1 -- 172.21.15.203:0/3969923932 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fe25c011370 con 0x7fe26c0fd650 2024-04-05T02:55:06.306 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:06.306 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:06.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 msgr2=0x7fe254038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 0x7fe254038d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe260009990 tx=0x7fe260006e40 comp rx=0 tx=0).stop 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 msgr2=0x7fe26c10d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b900 tx=0x7fe25c009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 shutdown_connections 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7fe2540368d0 0x7fe254038d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:06.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 --2- 172.21.15.203:0/3969923932 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26c0fd650 0x7fe26c10d080 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:06.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 >> 172.21.15.203:0/3969923932 conn(0x7fe26c0f70d0 msgr2=0x7fe26c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:06.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 shutdown_connections 2024-04-05T02:55:06.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:06.308+0000 7fe27148a700 1 -- 172.21.15.203:0/3969923932 wait complete. 2024-04-05T02:55:06.310 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:07.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:06 smithi153 ceph-mon[29381]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:06 smithi153 ceph-mon[29381]: Failed while placing prometheus.smithi153 on smithi153: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus-smithi153 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: /usr/bin/podman: stderr Error: no such container ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus-smithi153 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus.smithi153 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: /usr/bin/podman: stderr Error: no such container ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus.smithi153 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Deploy daemon prometheus.smithi153 ... 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.43.0 -e NODE_NAME=smithi153 -e CEPH_USE_RANDOM_NONCE=1 quay.io/prometheus/prometheus:v2.43.0 -c %u %g /etc/prometheus 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Trying to pull quay.io/prometheus/prometheus:v2.43.0... 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Getting image source signatures 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:225de5a6f1e724884cc9ebbe5671c3438d0fc76f4ce45c86a1c64a4522e6dcd3 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:98b3fd8aa6c139aa31d964fd60db493191103feb80791867f6fb83145865a356 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:42e880195a22b95f66f21bcc6f52d98e816e1d90341875b0cca30e1e1700b139 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:e6b68b840279de5ed0608798acd288f547245c7ac8f9abfce90ca631cf0cff6c 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:b2eff9ee7deb2deae22e16d39f98be3985d1c2ba4b3064f9b128b6b77ab5dd43 2024-04-05T02:55:07.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:4399114b4c59be45238d2958ccaef1fa50ac18f83d2c1bb585e15a4af273b31f 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:5a6f1da24a94bff679f9ff91bd8dbc37626a90d3ad133e6e2d06708a3d1d1081 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:e5ca35ebcd01b6485ff52a278aa7565453845908eee018aa2d7a8f90cb91fac8 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:aa26eff482ccdf86e483f661c50fb1c6c718c6b788050e103a6b37a4c0b51715 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:63132aee33f8e0415cba8ec6c4994fe92111002083472813de97d3dae27ba54e 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:633e059f1975205781f446279305ec9fa06c1ef627db5a865ce7f9b4834bedd1 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Copying blob sha256:aad4873905acdffa081dc077215138288017c0378f0b291f98155e3e4458a9be 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stat: stderr Error: copying system image from manifest list: reading blob sha256:4399114b4c59be45238d2958ccaef1fa50ac18f83d2c1bb585e15a4af273b31f: fetching blob: received unexpected HTTP status: 502 Bad Gateway 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ERROR: Failed to extract uid/gid for path /etc/prometheus: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.43.0 -e NODE_NAME=smithi153 -e CEPH_USE_RANDOM_NONCE=1 quay.io/prometheus/prometheus:v2.43.0 -c %u %g /etc/prometheus 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:06 smithi153 ceph-mon[29381]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:06 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-05T02:55:07.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:06 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3969923932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:07.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:07.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:08.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:07 smithi153 ceph-mon[29381]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-05T02:55:08.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:07 smithi153 ceph-mon[29381]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-05T02:55:08.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:07 smithi153 ceph-mon[29381]: mgrmap e14: smithi153.npzyqy(active, since 2m) 2024-04-05T02:55:08.167 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:08.214 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 -- 172.21.15.203:0/1388425342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 msgr2=0x7f3a0c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 --2- 172.21.15.203:0/1388425342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c070350 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0048c0 tx=0x7f39fc009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 -- 172.21.15.203:0/1388425342 shutdown_connections 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 --2- 172.21.15.203:0/1388425342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c070350 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 -- 172.21.15.203:0/1388425342 >> 172.21.15.203:0/1388425342 conn(0x7f3a0c0f81c0 msgr2=0x7f3a0c0fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 -- 172.21.15.203:0/1388425342 shutdown_connections 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.600+0000 7f3a12062700 1 -- 172.21.15.203:0/1388425342 wait complete. 2024-04-05T02:55:09.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a12062700 1 Processor -- start 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a12062700 1 -- start start 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a12062700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a12062700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c10f830 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a0b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a0b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33070/0 (socket says 172.21.15.203:33070) 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a0b7fe700 1 -- 172.21.15.203:0/2462197515 learned_addr learned my addr 172.21.15.203:0/2462197515 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a0b7fe700 1 -- 172.21.15.203:0/2462197515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc005040 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a0b7fe700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b900 tx=0x7f39fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc00b5c0 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39fc00be70 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c10fa30 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.601+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc01c590 con 0x7f3a0c071e40 2024-04-05T02:55:09.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c1105a0 con 0x7f3a0c071e40 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f39fc013420 con 0x7f3a0c071e40 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c105180 con 0x7f3a0c071e40 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a08ff9700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 0x7f39f8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f39fc049480 con 0x7f3a0c071e40 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.602+0000 7f3a03fff700 1 -- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 msgr2=0x7f39f8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4288916206 2024-04-05T02:55:09.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.603+0000 7f3a03fff700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 0x7f39f8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T02:55:09.607 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.607+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f39fc011550 con 0x7f3a0c071e40 2024-04-05T02:55:09.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.804+0000 7f3a03fff700 1 -- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 msgr2=0x7f39f8038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4288916206 2024-04-05T02:55:09.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.804+0000 7f3a03fff700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 0x7f39f8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T02:55:09.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.868+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a0c05f5f0 con 0x7f3a0c071e40 2024-04-05T02:55:09.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.868+0000 7f3a08ff9700 1 -- 172.21.15.203:0/2462197515 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f39fc021020 con 0x7f3a0c071e40 2024-04-05T02:55:09.869 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:09.869 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:09.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 msgr2=0x7f39f8038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 0x7f39f8038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 msgr2=0x7f3a0c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00b900 tx=0x7f39fc009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 shutdown_connections 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:6800/4288916206,v1:172.21.15.153:6801/4288916206] conn(0x7f39f80368d0 0x7f39f8038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 --2- 172.21.15.203:0/2462197515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c071e40 0x7f3a0c10f2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 >> 172.21.15.203:0/2462197515 conn(0x7f3a0c0f81c0 msgr2=0x7f3a0c105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 shutdown_connections 2024-04-05T02:55:09.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:09.870+0000 7f3a12062700 1 -- 172.21.15.203:0/2462197515 wait complete. 2024-04-05T02:55:09.872 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:10.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2462197515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:11.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:11.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:11.841 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:11.887 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:13.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1637277311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 msgr2=0x7f0c900fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 --2- 172.21.15.203:0/1637277311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c900fbd60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c840048c0 tx=0x7f0c84009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1637277311 shutdown_connections 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 --2- 172.21.15.203:0/1637277311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c900fbd60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1637277311 >> 172.21.15.203:0/1637277311 conn(0x7f0c900f7090 msgr2=0x7f0c900f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1637277311 shutdown_connections 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1637277311 wait complete. 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 Processor -- start 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- start start 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.401+0000 7f0c96b2f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9010c4c0 con 0x7f0c900fb950 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c948cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c948cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:33086/0 (socket says 172.21.15.203:33086) 2024-04-05T02:55:13.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c948cb700 1 -- 172.21.15.203:0/1266074702 learned_addr learned my addr 172.21.15.203:0/1266074702 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c948cb700 1 -- 172.21.15.203:0/1266074702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84005040 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c948cb700 1 --2- 172.21.15.203:0/1266074702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009680 tx=0x7f0c84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c8400b5c0 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c8400be70 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9010c6c0 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c840135a0 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.402+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9010d230 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.403+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7f0c8401c430 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.403+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c90101e60 con 0x7f0c900fb950 2024-04-05T02:55:13.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.403+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0c84049050 con 0x7f0c900fb950 2024-04-05T02:55:13.408 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.408+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0c8401c7d0 con 0x7f0c900fb950 2024-04-05T02:55:13.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.678+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c9010dc40 con 0x7f0c900fb950 2024-04-05T02:55:13.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.679+0000 7f0c8d7fa700 1 -- 172.21.15.203:0/1266074702 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0c84021070 con 0x7f0c900fb950 2024-04-05T02:55:13.680 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:13.680 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:13.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 msgr2=0x7f0c9010bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 --2- 172.21.15.203:0/1266074702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009680 tx=0x7f0c84009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 shutdown_connections 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 --2- 172.21.15.203:0/1266074702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c900fb950 0x7f0c9010bf80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 >> 172.21.15.203:0/1266074702 conn(0x7f0c900f7090 msgr2=0x7f0c90102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 shutdown_connections 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:13.681+0000 7f0c96b2f700 1 -- 172.21.15.203:0/1266074702 wait complete. 2024-04-05T02:55:13.682 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: Activating manager daemon smithi153.npzyqy 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: mgrmap e15: smithi153.npzyqy(active, starting, since 0.00550009s) 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T02:55:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: Manager daemon smithi153.npzyqy is now available 2024-04-05T02:55:13.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:13.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:13.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1266074702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: mgrmap e16: smithi153.npzyqy(active, since 1.00744s) 2024-04-05T02:55:14.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:15.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:15.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:15.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:15 smithi153 ceph-mon[29381]: [05/Apr/2024:02:55:14] ENGINE Bus STARTING 2024-04-05T02:55:15.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:15 smithi153 ceph-mon[29381]: [05/Apr/2024:02:55:14] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T02:55:15.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:15 smithi153 ceph-mon[29381]: [05/Apr/2024:02:55:14] ENGINE Bus STARTED 2024-04-05T02:55:15.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:15.947 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:16.000 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T02:55:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:17 smithi153 ceph-mon[29381]: mgrmap e17: smithi153.npzyqy(active, since 2s) 2024-04-05T02:55:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:17.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:18.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:55:19.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 -- 172.21.15.203:0/1699585819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 msgr2=0x7f06cc0fdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:19.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1699585819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc0fdea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0048c0 tx=0x7f06bc009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:19.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 -- 172.21.15.203:0/1699585819 shutdown_connections 2024-04-05T02:55:19.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1699585819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc0fdea0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:19.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 -- 172.21.15.203:0/1699585819 >> 172.21.15.203:0/1699585819 conn(0x7f06cc0f74e0 msgr2=0x7f06cc0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:19.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 -- 172.21.15.203:0/1699585819 shutdown_connections 2024-04-05T02:55:19.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.098+0000 7f06d18c8700 1 -- 172.21.15.203:0/1699585819 wait complete. 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06d18c8700 1 Processor -- start 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06d18c8700 1 -- start start 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06d18c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06d18c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc1136a0 con 0x7f06cc0fdad0 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:55676/0 (socket says 172.21.15.203:55676) 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06caffd700 1 -- 172.21.15.203:0/1830501707 learned_addr learned my addr 172.21.15.203:0/1830501707 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:19.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.099+0000 7f06caffd700 1 -- 172.21.15.203:0/1830501707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc005040 con 0x7f06cc0fdad0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06caffd700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00b7f0 tx=0x7f06bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06bc009370 con 0x7f06cc0fdad0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06bc013420 con 0x7f06cc0fdad0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc1138a0 con 0x7f06cc0fdad0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06bc01c590 con 0x7f06cc0fdad0 2024-04-05T02:55:19.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc113d10 con 0x7f06cc0fdad0 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.100+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f06bc026440 con 0x7f06cc0fdad0 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.101+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06cc066180 con 0x7f06cc0fdad0 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.101+0000 7f06b37fe700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 0x7f06b4038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.101+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f06bc0494b0 con 0x7f06cc0fdad0 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.101+0000 7f06ca7fc700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 0x7f06b4038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:19.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.101+0000 7f06ca7fc700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 0x7f06b4038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009990 tx=0x7f06c0006e40 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:19.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.106+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f06bc011780 con 0x7f06cc0fdad0 2024-04-05T02:55:19.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.377+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06cc05f5f0 con 0x7f06cc0fdad0 2024-04-05T02:55:19.378 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.378+0000 7f06b37fe700 1 -- 172.21.15.203:0/1830501707 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f06bc021030 con 0x7f06cc0fdad0 2024-04-05T02:55:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:19.378 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 msgr2=0x7f06b4038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 0x7f06b4038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009990 tx=0x7f06c0006e40 comp rx=0 tx=0).stop 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 msgr2=0x7f06cc113160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00b7f0 tx=0x7f06bc009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 shutdown_connections 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f06b4036970 0x7f06b4038e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 --2- 172.21.15.203:0/1830501707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06cc0fdad0 0x7f06cc113160 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 >> 172.21.15.203:0/1830501707 conn(0x7f06cc0f74e0 msgr2=0x7f06cc0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 shutdown_connections 2024-04-05T02:55:19.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:19.380+0000 7f06d18c8700 1 -- 172.21.15.203:0/1830501707 wait complete. 2024-04-05T02:55:19.382 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:19.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:19.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:55:19.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-05T02:55:19.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:19 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:20 smithi153 ceph-mon[29381]: Deploying daemon crash.smithi203 on smithi203 2024-04-05T02:55:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:20 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1830501707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:21.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:21.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:22.093 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:23.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:22 smithi153 ceph-mon[29381]: Deploying daemon node-exporter.smithi203 on smithi203 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 -- 172.21.15.203:0/2671937192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 msgr2=0x7f69a0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 --2- 172.21.15.203:0/2671937192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a0101d90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f69900048c0 tx=0x7f6990009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 -- 172.21.15.203:0/2671937192 shutdown_connections 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 --2- 172.21.15.203:0/2671937192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a0101d90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 -- 172.21.15.203:0/2671937192 >> 172.21.15.203:0/2671937192 conn(0x7f69a00f74e0 msgr2=0x7f69a00f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 -- 172.21.15.203:0/2671937192 shutdown_connections 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 -- 172.21.15.203:0/2671937192 wait complete. 2024-04-05T02:55:23.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.706+0000 7f69a62b8700 1 Processor -- start 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f69a62b8700 1 -- start start 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f69a62b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f69a62b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a00668c0 con 0x7f69a01019c0 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f699ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f699ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:55694/0 (socket says 172.21.15.203:55694) 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f699ffff700 1 -- 172.21.15.203:0/622768415 learned_addr learned my addr 172.21.15.203:0/622768415 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:23.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f699ffff700 1 -- 172.21.15.203:0/622768415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990005040 con 0x7f69a01019c0 2024-04-05T02:55:23.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.707+0000 7f699ffff700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f699000b820 tx=0x7f6990009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:23.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.708+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69900093c0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.708+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a010e960 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.708+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6990013420 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.708+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f699001c5b0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.708+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a010edd0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.709+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f699001c7f0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.709+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a010efc0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.709+0000 7f699cff9700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 0x7f6988038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.709+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6990049bf0 con 0x7f69a01019c0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.709+0000 7f699f7fe700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 0x7f6988038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:23.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.710+0000 7f699f7fe700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 0x7f6988038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6994009990 tx=0x7f6994006e40 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:23.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:23.714+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6990023300 con 0x7f69a01019c0 2024-04-05T02:55:24.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.019+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69a0100c30 con 0x7f69a01019c0 2024-04-05T02:55:24.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.020+0000 7f699cff9700 1 -- 172.21.15.203:0/622768415 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6990011b60 con 0x7f69a01019c0 2024-04-05T02:55:24.020 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:24.020 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 msgr2=0x7f6988038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 0x7f6988038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6994009990 tx=0x7f6994006e40 comp rx=0 tx=0).stop 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 msgr2=0x7f69a010e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f699000b820 tx=0x7f6990009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 shutdown_connections 2024-04-05T02:55:24.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6988036970 0x7f6988038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:24.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 --2- 172.21.15.203:0/622768415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a01019c0 0x7f69a010e360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:24.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 >> 172.21.15.203:0/622768415 conn(0x7f69a00f74e0 msgr2=0x7f69a00f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:24.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 shutdown_connections 2024-04-05T02:55:24.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:24.022+0000 7f69a62b8700 1 -- 172.21.15.203:0/622768415 wait complete. 2024-04-05T02:55:24.023 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:24 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:24 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/622768415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:25.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:25.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:26.573 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:28.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:27 smithi153 ceph-mon[29381]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T02:55:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:29.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:29.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 -- 172.21.15.203:0/2983953696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 msgr2=0x7f6e4c101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 --2- 172.21.15.203:0/2983953696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c101d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6e440048c0 tx=0x7f6e44009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 -- 172.21.15.203:0/2983953696 shutdown_connections 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 --2- 172.21.15.203:0/2983953696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c101d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 -- 172.21.15.203:0/2983953696 >> 172.21.15.203:0/2983953696 conn(0x7f6e4c0f6c30 msgr2=0x7f6e4c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 -- 172.21.15.203:0/2983953696 shutdown_connections 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.023+0000 7f6e51846700 1 -- 172.21.15.203:0/2983953696 wait complete. 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.024+0000 7f6e51846700 1 Processor -- start 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.024+0000 7f6e51846700 1 -- start start 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.024+0000 7f6e51846700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.024+0000 7f6e51846700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1136c0 con 0x7f6e4c101970 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e4affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:29.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e4affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:52998/0 (socket says 172.21.15.203:52998) 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e4affd700 1 -- 172.21.15.203:0/2841791340 learned_addr learned my addr 172.21.15.203:0/2841791340 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e4affd700 1 -- 172.21.15.203:0/2841791340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44005040 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e4affd700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400b820 tx=0x7f6e44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e440093c0 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c1138c0 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e44013420 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e4401c5b0 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.025+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c113d30 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.026+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f6e4401c7f0 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.026+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c0fd2d0 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.026+0000 7f6e337fe700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 0x7f6e34038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.026+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6e44049a90 con 0x7f6e4c101970 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.027+0000 7f6e4a7fc700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 0x7f6e34038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:29.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.027+0000 7f6e4a7fc700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 0x7f6e34038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000ad90 tx=0x7f6e40009410 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:29.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.032+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6e44023300 con 0x7f6e4c101970 2024-04-05T02:55:29.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.312+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e4c05f5f0 con 0x7f6e4c101970 2024-04-05T02:55:29.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.313+0000 7f6e337fe700 1 -- 172.21.15.203:0/2841791340 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6e44011b60 con 0x7f6e4c101970 2024-04-05T02:55:29.313 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:29.313 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:51:49.024022Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 msgr2=0x7f6e34038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 0x7f6e34038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000ad90 tx=0x7f6e40009410 comp rx=0 tx=0).stop 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 msgr2=0x7f6e4c113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400b820 tx=0x7f6e44009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 shutdown_connections 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e34036970 0x7f6e34038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:29.315 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 --2- 172.21.15.203:0/2841791340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4c101970 0x7f6e4c113180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:29.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 >> 172.21.15.203:0/2841791340 conn(0x7f6e4c0f6c30 msgr2=0x7f6e4c0f85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:29.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 shutdown_connections 2024-04-05T02:55:29.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:29.315+0000 7f6e51846700 1 -- 172.21.15.203:0/2841791340 wait complete. 2024-04-05T02:55:29.316 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-05T02:55:30.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:29 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/2841791340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:30.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T02:55:30.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T02:55:31.334 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:31.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:31 smithi153 ceph-mon[29381]: Deploying daemon mon.smithi203 on smithi203 2024-04-05T02:55:34.074 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 podman[39860]: 2024-04-05 02:55:33.959921161 +0000 UTC m=+0.198349039 image pull a49113eacc58832dc36e13e21ab00e230e7480fa0f05e368736a82d00d143a48 quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030 2024-04-05T02:55:34.074 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 podman[39860]: 2024-04-05 02:55:34.074053348 +0000 UTC m=+0.312481208 container create ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-6fda41f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 -- 172.21.15.203:0/2525698364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 msgr2=0x7f7440101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 --2- 172.21.15.203:0/2525698364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440101dc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74340048c0 tx=0x7f7434009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 -- 172.21.15.203:0/2525698364 shutdown_connections 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 --2- 172.21.15.203:0/2525698364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440101dc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 -- 172.21.15.203:0/2525698364 >> 172.21.15.203:0/2525698364 conn(0x7f74400f6c30 msgr2=0x7f74400f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:34.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 -- 172.21.15.203:0/2525698364 shutdown_connections 2024-04-05T02:55:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.415+0000 7f7446df7700 1 -- 172.21.15.203:0/2525698364 wait complete. 2024-04-05T02:55:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7446df7700 1 Processor -- start 2024-04-05T02:55:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7446df7700 1 -- start start 2024-04-05T02:55:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7446df7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:34.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7446df7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401125b0 con 0x7f74401019f0 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7444b93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7444b93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.203:53040/0 (socket says 172.21.15.203:53040) 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.416+0000 7f7444b93700 1 -- 172.21.15.203:0/3262486800 learned_addr learned my addr 172.21.15.203:0/3262486800 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f7444b93700 1 -- 172.21.15.203:0/3262486800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7434005040 con 0x7f74401019f0 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f7444b93700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f743400b820 tx=0x7f7434009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74340093c0 con 0x7f74401019f0 2024-04-05T02:55:34.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74401127b0 con 0x7f74401019f0 2024-04-05T02:55:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7434013420 con 0x7f74401019f0 2024-04-05T02:55:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743401c5b0 con 0x7f74401019f0 2024-04-05T02:55:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.417+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7440112c20 con 0x7f74401019f0 2024-04-05T02:55:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.418+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f743401c7f0 con 0x7f74401019f0 2024-04-05T02:55:34.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.418+0000 7f743d7fa700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 0x7f7430038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:34.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.418+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74400fd290 con 0x7f74401019f0 2024-04-05T02:55:34.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.418+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7434049b20 con 0x7f74401019f0 2024-04-05T02:55:34.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.418+0000 7f743ffff700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 0x7f7430038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:34.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.419+0000 7f743ffff700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 0x7f7430038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f742c009990 tx=0x7f742c006e40 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:34.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.423+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7434023300 con 0x7f74401019f0 2024-04-05T02:55:34.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.525+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7434023300 con 0x7f74401019f0 2024-04-05T02:55:34.621 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 podman[39860]: 2024-04-05 02:55:34.469688491 +0000 UTC m=+0.708116341 container init ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=quincy-6fda41f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T02:55:34.621 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 podman[39860]: 2024-04-05 02:55:34.472682861 +0000 UTC m=+0.711110720 container start ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-6fda41f, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T02:55:34.621 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable), process ceph-mon, pid 2 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pidfile_write: ignore empty --pid-file 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: load: jerasure load: lrc 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: RocksDB version: 6.15.5 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Compile date Apr 3 2024 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: DB SUMMARY 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: DB Session ID: 1PJI1T30YVERRL33LMCL 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: CURRENT file: CURRENT 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi203/store.db dir, Total Num: 0, files: 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi203/store.db: 000004.log size: 511 ; 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.error_if_exists: 0 2024-04-05T02:55:34.622 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.create_if_missing: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.paranoid_checks: 1 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.env: 0x55d1af052a80 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.fs: Posix File System 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.info_log: 0x55d1b0623c00 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.statistics: (nil) 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.use_fsync: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_log_file_size: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T02:55:34.623 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_fallocate: 1 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.use_direct_reads: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.db_log_dir: 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi203/store.db 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T02:55:34.624 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.write_buffer_manager: 0x55d1b1398990 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.unordered_write: 0 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T02:55:34.625 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T02:55:34.626 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T02:55:34.626 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T02:55:34.626 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.row_cache: None 2024-04-05T02:55:34.626 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.wal_filter: None 2024-04-05T02:55:34.626 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T02:55:34.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T02:55:34.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.preserve_deletes: 0 2024-04-05T02:55:34.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.two_write_queues: 0 2024-04-05T02:55:34.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T02:55:34.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.atomic_flush: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.log_readahead_size: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_background_jobs: 2 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_background_compactions: -1 2024-04-05T02:55:34.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_subcompactions: 1 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_open_files: -1 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_background_flushes: -1 2024-04-05T02:55:34.630 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Compression algorithms supported: 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kZSTD supported: 0 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kXpressCompression supported: 0 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kZlibCompression supported: 1 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kSnappyCompression supported: 1 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kLZ4Compression supported: 1 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: kBZip2Compression supported: 0 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi203/store.db/MANIFEST-000003 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.merge_operator: 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_filter: None 2024-04-05T02:55:34.631 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d1b0678f58) 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: cache_index_and_filter_blocks: 1 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: pin_top_level_index_and_filter: 1 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_type: 0 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: data_block_index_type: 0 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_shortening: 1 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: hash_index_allow_collision: 1 2024-04-05T02:55:34.632 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: checksum: 1 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: no_block_cache: 0 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache: 0x55d1b06ab090 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_name: BinnedLRUCache 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_options: 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: capacity : 536870912 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: num_shard_bits : 4 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: strict_capacity_limit : 0 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: high_pri_pool_ratio: 0.000 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_compressed: (nil) 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: persistent_cache: (nil) 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_size: 4096 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_size_deviation: 10 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_restart_interval: 16 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_block_restart_interval: 1 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: metadata_block_size: 4096 2024-04-05T02:55:34.633 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: partition_filters: 0 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: use_delta_encoding: 1 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: whole_key_filtering: 1 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: verify_compression: 0 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: read_amp_bytes_per_bit: 0 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: format_version: 4 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: enable_index_compression: 1 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_align: 0 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression: NoCompression 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T02:55:34.634 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.num_levels: 7 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T02:55:34.635 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T02:55:34.636 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.arena_block_size: 4194304 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T02:55:34.637 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.table_properties_collectors: 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.inplace_update_support: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.bloom_locality: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.max_successive_merges: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T02:55:34.638 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.ttl: 2592000 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.enable_blob_files: false 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.min_blob_size: 0 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi203/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285734499216, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-04-05T02:55:34.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285734499527, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712285734, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4b8f1a6e-e6dd-47f6-9779-a4d981dc2f51", "db_session_id": "1PJI1T30YVERRL33LMCL"}} 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712285734499846, "job": 1, "event": "recovery_finished"} 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi203/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55d1b063f180 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: DB pointer 0x55d1b0714000 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** DB Stats ** 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T02:55:34.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T02:55:34.641 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.642 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T02:55:34.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203 does not exist in monmap, will attempt to join an existing cluster 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: using public_addr v2:172.21.15.203:0/0 -> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: starting mon.smithi203 rank -1 at public addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] at bind addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi203 fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(???) e0 preinit fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).mds e1 new map 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).mds e1 print_map 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: e1 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: legacy client fscid: -1 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: No filesystems configured 2024-04-05T02:55:34.644 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2747768561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Deploying daemon prometheus.smithi153 on smithi153 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2136882071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1837307297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/3771736596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/614223901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2352699090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/3313920418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/809145483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2689682241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.647 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/4057341467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1550246903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Failed while placing prometheus.smithi153 on smithi153: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus-smithi153 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: /usr/bin/podman: stderr Error: no such container ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus-smithi153 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus.smithi153 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: /usr/bin/podman: stderr Error: no such container ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-prometheus.smithi153 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Deploy daemon prometheus.smithi153 ... 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.43.0 -e NODE_NAME=smithi153 -e CEPH_USE_RANDOM_NONCE=1 quay.io/prometheus/prometheus:v2.43.0 -c %u %g /etc/prometheus 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Trying to pull quay.io/prometheus/prometheus:v2.43.0... 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Getting image source signatures 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:225de5a6f1e724884cc9ebbe5671c3438d0fc76f4ce45c86a1c64a4522e6dcd3 2024-04-05T02:55:34.648 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:98b3fd8aa6c139aa31d964fd60db493191103feb80791867f6fb83145865a356 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:42e880195a22b95f66f21bcc6f52d98e816e1d90341875b0cca30e1e1700b139 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:e6b68b840279de5ed0608798acd288f547245c7ac8f9abfce90ca631cf0cff6c 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:b2eff9ee7deb2deae22e16d39f98be3985d1c2ba4b3064f9b128b6b77ab5dd43 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:4399114b4c59be45238d2958ccaef1fa50ac18f83d2c1bb585e15a4af273b31f 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:5a6f1da24a94bff679f9ff91bd8dbc37626a90d3ad133e6e2d06708a3d1d1081 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:e5ca35ebcd01b6485ff52a278aa7565453845908eee018aa2d7a8f90cb91fac8 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:aa26eff482ccdf86e483f661c50fb1c6c718c6b788050e103a6b37a4c0b51715 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:63132aee33f8e0415cba8ec6c4994fe92111002083472813de97d3dae27ba54e 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:633e059f1975205781f446279305ec9fa06c1ef627db5a865ce7f9b4834bedd1 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Copying blob sha256:aad4873905acdffa081dc077215138288017c0378f0b291f98155e3e4458a9be 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stat: stderr Error: copying system image from manifest list: reading blob sha256:4399114b4c59be45238d2958ccaef1fa50ac18f83d2c1bb585e15a4af273b31f: fetching blob: received unexpected HTTP status: 502 Bad Gateway 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ERROR: Failed to extract uid/gid for path /etc/prometheus: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.43.0 -e NODE_NAME=smithi153 -e CEPH_USE_RANDOM_NONCE=1 quay.io/prometheus/prometheus:v2.43.0 -c %u %g /etc/prometheus 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/3969923932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.649 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14162 172.21.15.153:0/2983087345' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mgrmap e14: smithi153.npzyqy(active, since 2m) 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2462197515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Activating manager daemon smithi153.npzyqy 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mgrmap e15: smithi153.npzyqy(active, starting, since 0.00550009s) 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Manager daemon smithi153.npzyqy is now available 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.650 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1266074702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mgrmap e16: smithi153.npzyqy(active, since 1.00744s) 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.651 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: [05/Apr/2024:02:55:14] ENGINE Bus STARTING 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: [05/Apr/2024:02:55:14] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: [05/Apr/2024:02:55:14] ENGINE Bus STARTED 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mgrmap e17: smithi153.npzyqy(active, since 2s) 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.652 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:55:34.653 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:34.653 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:55:34.653 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T02:55:34.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Deploying daemon crash.smithi203 on smithi203 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1830501707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.655 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Deploying daemon node-exporter.smithi203 on smithi203 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/622768415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/2841791340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:34.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Deploying daemon mon.smithi203 on smithi203 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 ceph-mon[39911]: mon.smithi203@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 bash[39860]: ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 2024-04-05T02:55:34.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:34 smithi203 systemd[1]: Started Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T02:55:34.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:34.715+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f744004bed0 con 0x7f74401019f0 2024-04-05T02:55:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:34 smithi153 ceph-mon[29381]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:34.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:34 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:34.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:34 smithi153 ceph-mon[29381]: Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-04-05T02:55:34.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:34 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T02:55:39.539 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-05T02:55:39.539 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","modified":"2024-04-05T02:55:34.525318Z","created":"2024-04-05T02:51:49.024022Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi203","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-05T02:55:39.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.537+0000 7f743d7fa700 1 -- 172.21.15.203:0/3262486800 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f7434012e40 con 0x7f74401019f0 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 msgr2=0x7f7430038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 0x7f7430038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f742c009990 tx=0x7f742c006e40 comp rx=0 tx=0).stop 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 msgr2=0x7f7440112070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f743400b820 tx=0x7f7434009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 shutdown_connections 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f7430036970 0x7f7430038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:39.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 --2- 172.21.15.203:0/3262486800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74401019f0 0x7f7440112070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:39.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 >> 172.21.15.203:0/3262486800 conn(0x7f74400f6c30 msgr2=0x7f74400f7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:39.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 shutdown_connections 2024-04-05T02:55:39.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:55:39.541+0000 7f7446df7700 1 -- 172.21.15.203:0/3262486800 wait complete. 2024-04-05T02:55:39.542 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: mon.smithi153 calling monitor election 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/3262486800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: mon.smithi203 calling monitor election 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi203=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: fsmap 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: mgrmap e17: smithi153.npzyqy(active, since 26s) 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: overall HEALTH_OK 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz started 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T02:55:39.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:39 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: mon.smithi153 calling monitor election 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3262486800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: mon.smithi203 calling monitor election 2024-04-05T02:55:40.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi203=[v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: fsmap 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: mgrmap e17: smithi153.npzyqy(active, since 26s) 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: overall HEALTH_OK 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz started 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T02:55:40.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3517331358' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T02:55:40.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:39 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:40.223 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-05T02:55:40.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-05T02:55:40.486 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: mgrmap e18: smithi153.npzyqy(active, since 26s), standbys: smithi203.iakodz 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: Deploying daemon prometheus.smithi203 on smithi203 2024-04-05T02:55:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: mgrmap e18: smithi153.npzyqy(active, since 26s), standbys: smithi203.iakodz 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: Deploying daemon prometheus.smithi203 on smithi203 2024-04-05T02:55:41.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:55:42.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2722441734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60102ae0 msgr2=0x7f3d60102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2722441734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60102ae0 0x7f3d60102eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009c60 tx=0x7f3d50004d10 comp rx=0 tx=0).stop 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2722441734 shutdown_connections 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2722441734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60102ae0 0x7f3d60102eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2722441734 >> 172.21.15.153:0/2722441734 conn(0x7f3d600f85e0 msgr2=0x7f3d600fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2722441734 shutdown_connections 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2722441734 wait complete. 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.005+0000 7f3d65e5d700 1 Processor -- start 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d65e5d700 1 -- start start 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d65e5d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d60102ae0 0x7f3d600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d65e5d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:42.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d65e5d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d60119610 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d65e5d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d60119750 con 0x7f3d60102ae0 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42086/0 (socket says 172.21.15.153:42086) 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 -- 172.21.15.153:0/2190909478 learned_addr learned my addr 172.21.15.153:0/2190909478 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5f7fe700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d60102ae0 0x7f3d600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 -- 172.21.15.153:0/2190909478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d60102ae0 msgr2=0x7f3d600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d60102ae0 0x7f3d600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 -- 172.21.15.153:0/2190909478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d50009940 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d5effd700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400b790 tx=0x7f3d54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.006+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d54010870 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.007+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d5400f3e0 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.007+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d54010ea0 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.007+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d60119950 con 0x7f3d60070800 2024-04-05T02:55:42.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.007+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d60119f10 con 0x7f3d60070800 2024-04-05T02:55:42.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.007+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d60101d50 con 0x7f3d60070800 2024-04-05T02:55:42.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.008+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f3d5400f550 con 0x7f3d60070800 2024-04-05T02:55:42.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.008+0000 7f3d47fff700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 0x7f3d4806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:42.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.008+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3d54086610 con 0x7f3d60070800 2024-04-05T02:55:42.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.008+0000 7f3d5f7fe700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 0x7f3d4806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:42.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.009+0000 7f3d5f7fe700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 0x7f3d4806b820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d50015040 tx=0x7f3d50004d10 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:42.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.013+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3d54053a20 con 0x7f3d60070800 2024-04-05T02:55:42.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.199+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3d600fe1e0 con 0x7f3d60070800 2024-04-05T02:55:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.200+0000 7f3d47fff700 1 -- 172.21.15.153:0/2190909478 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f3d54014100 con 0x7f3d60070800 2024-04-05T02:55:42.200 INFO:teuthology.orchestra.run.smithi153.stdout:# minimal ceph.conf for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:55:42.200 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-04-05T02:55:42.200 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T02:55:42.200 INFO:teuthology.orchestra.run.smithi153.stdout: mon_host = [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.201+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 msgr2=0x7f3d4806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.201+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 0x7f3d4806b820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d50015040 tx=0x7f3d50004d10 comp rx=0 tx=0).stop 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 msgr2=0x7f3d60070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400b790 tx=0x7f3d54009250 comp rx=0 tx=0).stop 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 shutdown_connections 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069370 0x7f3d4806b820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d60102ae0 0x7f3d600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 --2- 172.21.15.153:0/2190909478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d60070800 0x7f3d60070c70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 >> 172.21.15.153:0/2190909478 conn(0x7f3d600f85e0 msgr2=0x7f3d600fa0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 shutdown_connections 2024-04-05T02:55:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:42.202+0000 7f3d65e5d700 1 -- 172.21.15.153:0/2190909478 wait complete. 2024-04-05T02:55:42.744 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:42 smithi153 ceph-mon[29381]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:42.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:42 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2190909478' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:42.771 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-05T02:55:42.771 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:55:42.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T02:55:42.801 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:55:42.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:42.870 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:55:42.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T02:55:42.897 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:42 smithi203 ceph-mon[39911]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:42.897 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:42 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2190909478' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:42.901 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:55:42.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T02:55:42.971 INFO:tasks.cephadm:Deploying OSDs... 2024-04-05T02:55:42.971 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:55:42.971 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T02:55:42.988 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T02:55:42.988 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-05 02:54:27.610075861 +0000 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-05 02:47:12.244421610 +0000 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-05 02:47:12.244421610 +0000 2024-04-05T02:55:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-05 02:47:12.244421610 +0000 2024-04-05T02:55:43.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T02:55:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-05T02:55:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-05T02:55:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000198337 s, 2.6 MB/s 2024-04-05T02:55:43.112 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T02:55:43.168 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-05 02:54:27.610075861 +0000 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-05 02:47:12.576416285 +0000 2024-04-05T02:55:43.223 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-05 02:47:12.576416285 +0000 2024-04-05T02:55:43.224 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-05 02:47:12.576416285 +0000 2024-04-05T02:55:43.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T02:55:43.290 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-05T02:55:43.290 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-05T02:55:43.290 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000235726 s, 2.2 MB/s 2024-04-05T02:55:43.291 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T02:55:43.348 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-05T02:55:43.404 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T02:55:43.404 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.404 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-05T02:55:43.404 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.404 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.405 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-05 02:54:27.611075845 +0000 2024-04-05T02:55:43.405 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-05 02:47:12.918410799 +0000 2024-04-05T02:55:43.405 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-05 02:47:12.918410799 +0000 2024-04-05T02:55:43.405 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-05 02:47:12.918410799 +0000 2024-04-05T02:55:43.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T02:55:43.470 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-05T02:55:43.471 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-05T02:55:43.471 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000200717 s, 2.6 MB/s 2024-04-05T02:55:43.472 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T02:55:43.527 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-05T02:55:43.583 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T02:55:43.583 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-05 02:54:27.611075845 +0000 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-05 02:47:13.255405393 +0000 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-05 02:47:13.255405393 +0000 2024-04-05T02:55:43.584 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-05 02:47:13.255405393 +0000 2024-04-05T02:55:43.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T02:55:43.650 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-05T02:55:43.650 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-05T02:55:43.650 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000308481 s, 1.7 MB/s 2024-04-05T02:55:43.651 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T02:55:43.706 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:55:43.707 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T02:55:43.722 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T02:55:43.722 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-05T02:55:43.777 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-05 02:55:32.927432349 +0000 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-05 02:47:19.031992412 +0000 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-05 02:47:19.031992412 +0000 2024-04-05T02:55:43.778 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-05 02:47:19.031992412 +0000 2024-04-05T02:55:43.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T02:55:43.844 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-05T02:55:43.844 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-05T02:55:43.844 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000268031 s, 1.9 MB/s 2024-04-05T02:55:43.845 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T02:55:43.902 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-05T02:55:43.956 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-05 02:55:32.928432324 +0000 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-05 02:47:19.373983883 +0000 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-05 02:47:19.373983883 +0000 2024-04-05T02:55:43.957 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-05 02:47:19.373983883 +0000 2024-04-05T02:55:43.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T02:55:44.022 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-05T02:55:44.022 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-05T02:55:44.023 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000191129 s, 2.7 MB/s 2024-04-05T02:55:44.023 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T02:55:44.079 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-05 02:55:32.928432324 +0000 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-05 02:47:19.712975428 +0000 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-05 02:47:19.712975428 +0000 2024-04-05T02:55:44.135 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-05 02:47:19.712975428 +0000 2024-04-05T02:55:44.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T02:55:44.200 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-05T02:55:44.200 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-05T02:55:44.201 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000219922 s, 2.3 MB/s 2024-04-05T02:55:44.202 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T02:55:44.257 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-05T02:55:44.312 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-05 02:55:32.929432299 +0000 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-05 02:47:20.054966899 +0000 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-05 02:47:20.054966899 +0000 2024-04-05T02:55:44.313 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-05 02:47:20.054966899 +0000 2024-04-05T02:55:44.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T02:55:44.379 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-05T02:55:44.379 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-05T02:55:44.379 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000185819 s, 2.8 MB/s 2024-04-05T02:55:44.380 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T02:55:44.437 INFO:tasks.cephadm:Deploying osd.0 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-05T02:55:44.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T02:55:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:44 smithi203 ceph-mon[39911]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:55:44.819 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:44 smithi153 ceph-mon[29381]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:46.456 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T02:55:46.456 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T02:55:46.456 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-05T02:55:46.456 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-05T02:55:46.456 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115381 s, 909 MB/s 2024-04-05T02:55:46.457 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-05T02:55:46.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:46 smithi153 ceph-mon[29381]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:46 smithi203 ceph-mon[39911]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:47.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-05T02:55:47.284 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:55:48.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:48 smithi153 ceph-mon[29381]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:48 smithi203 ceph-mon[39911]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 -- 172.21.15.153:0/3389176492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6200fbf50 msgr2=0x7fa6200fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 --2- 172.21.15.153:0/3389176492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6200fbf50 0x7fa6200fc360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6080048c0 tx=0x7fa608009e30 comp rx=0 tx=0).stop 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 -- 172.21.15.153:0/3389176492 shutdown_connections 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 --2- 172.21.15.153:0/3389176492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6200fbf50 0x7fa6200fc360 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 -- 172.21.15.153:0/3389176492 >> 172.21.15.153:0/3389176492 conn(0x7fa6200f6840 msgr2=0x7fa6200f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 -- 172.21.15.153:0/3389176492 shutdown_connections 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.816+0000 7fa626780700 1 -- 172.21.15.153:0/3389176492 wait complete. 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 Processor -- start 2024-04-05T02:55:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 -- start start 2024-04-05T02:55:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6200fbf50 0x7fa62010d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62010dc10 con 0x7fa62010d6c0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa626780700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62010dd50 con 0x7fa6200fbf50 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39202/0 (socket says 172.21.15.153:39202) 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 -- 172.21.15.153:0/33793189 learned_addr learned my addr 172.21.15.153:0/33793189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61ffff700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6200fbf50 0x7fa62010d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 -- 172.21.15.153:0/33793189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6200fbf50 msgr2=0x7fa62010d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6200fbf50 0x7fa62010d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.817+0000 7fa61f7fe700 1 -- 172.21.15.153:0/33793189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608005040 con 0x7fa62010d6c0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa61f7fe700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b790 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa610010870 con 0x7fa62010d6c0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa61000f3e0 con 0x7fa62010d6c0 2024-04-05T02:55:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62010ebf0 con 0x7fa62010d6c0 2024-04-05T02:55:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa610010ea0 con 0x7fa62010d6c0 2024-04-05T02:55:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.818+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62010f1b0 con 0x7fa62010d6c0 2024-04-05T02:55:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.819+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62005f5f0 con 0x7fa62010d6c0 2024-04-05T02:55:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.819+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fa61000f550 con 0x7fa62010d6c0 2024-04-05T02:55:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.819+0000 7fa61cff9700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 0x7fa60c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:55:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.820+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa6100866a0 con 0x7fa62010d6c0 2024-04-05T02:55:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.820+0000 7fa61ffff700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 0x7fa60c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:55:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.820+0000 7fa61ffff700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 0x7fa60c06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa60800b810 tx=0x7fa608009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:55:48.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:48.824+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa6100574c0 con 0x7fa62010d6c0 2024-04-05T02:55:49.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:55:49.020+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6200fb2c0 con 0x7fa60c069360 2024-04-05T02:55:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:55:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:55:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:49.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:55:49.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:55:49.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:55:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:50.818 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:55:51.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:51 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2623404248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3638168-6ff3-4751-b73c-032b22097ffa"}]: dispatch 2024-04-05T02:55:51.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:51 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2623404248' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3638168-6ff3-4751-b73c-032b22097ffa"}]': finished 2024-04-05T02:55:51.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:51 smithi153 ceph-mon[29381]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T02:55:51.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:51 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:55:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:51 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2623404248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3638168-6ff3-4751-b73c-032b22097ffa"}]: dispatch 2024-04-05T02:55:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:51 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2623404248' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3638168-6ff3-4751-b73c-032b22097ffa"}]': finished 2024-04-05T02:55:51.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:51 smithi203 ceph-mon[39911]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T02:55:51.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:51 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:55:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:52 smithi203 ceph-mon[39911]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:52 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2558074096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:55:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:52 smithi153 ceph-mon[29381]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:52 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2558074096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:55:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-05T02:55:54.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:55:54.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T02:55:54.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:54.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-05T02:55:54.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:56 smithi203 ceph-mon[39911]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:56.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:56 smithi153 ceph-mon[29381]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: Reconfiguring mgr.smithi153.npzyqy (unknown last config time)... 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: Reconfiguring daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T02:55:58.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:58 smithi203 ceph-mon[39911]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: Reconfiguring mgr.smithi153.npzyqy (unknown last config time)... 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: Reconfiguring daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T02:55:58.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:58 smithi153 ceph-mon[29381]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:55:59.359 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:55:59 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:55:59.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:55:59 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:00.465 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:00.465 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:00.465 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-05T02:56:00.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:56:00.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:00.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-05T02:56:00.466 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:00 smithi153 ceph-mon[29381]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:00.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-05T02:56:00.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:56:00.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:00.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-05T02:56:00.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:00 smithi203 ceph-mon[39911]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:01.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T02:56:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:01 smithi203 ceph-mon[39911]: Deploying daemon osd.0 on smithi153 2024-04-05T02:56:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T02:56:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:01 smithi153 ceph-mon[29381]: Deploying daemon osd.0 on smithi153 2024-04-05T02:56:02.729 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:02 smithi153 ceph-mon[29381]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:02.729 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:02.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:02.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:02 smithi153 ceph-mon[29381]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-04-05T02:56:02.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:02 smithi153 ceph-mon[29381]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-04-05T02:56:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:02 smithi203 ceph-mon[39911]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:02.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:02 smithi203 ceph-mon[39911]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-04-05T02:56:02.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:02 smithi203 ceph-mon[39911]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-04-05T02:56:04.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:04 smithi153 ceph-mon[29381]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:04 smithi203 ceph-mon[39911]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:06.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:06 smithi153 ceph-mon[29381]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:06.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:06.671 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:06 smithi203 ceph-mon[39911]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:08.773 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:08 smithi153 ceph-mon[29381]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:08 smithi203 ceph-mon[39911]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:10.559 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:10 smithi153 ceph-mon[29381]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:10.559 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:10 smithi153 ceph-mon[29381]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T02:56:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:10 smithi203 ceph-mon[39911]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:10 smithi203 ceph-mon[39911]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T02:56:11.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:11 smithi153 ceph-mon[29381]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T02:56:11.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:11 smithi153 ceph-mon[29381]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T02:56:11.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:11 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:11.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:11 smithi153 ceph-mon[29381]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:56:11.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:11 smithi203 ceph-mon[39911]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T02:56:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:11 smithi203 ceph-mon[39911]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T02:56:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:11 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:11 smithi203 ceph-mon[39911]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: osdmap e8: 1 total, 0 up, 1 in 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:12.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: from='osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: osdmap e8: 1 total, 0 up, 1 in 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:12.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:13.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.406+0000 7fa61cff9700 1 -- 172.21.15.153:0/33793189 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa6200fb2c0 con 0x7fa60c069360 2024-04-05T02:56:13.406 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0 on host 'smithi153' 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 msgr2=0x7fa60c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 0x7fa60c06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa60800b810 tx=0x7fa608009e30 comp rx=0 tx=0).stop 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 msgr2=0x7fa62010e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b790 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 shutdown_connections 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa60c069360 0x7fa60c06b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6200fbf50 0x7fa62010d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:13.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 --2- 172.21.15.153:0/33793189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa62010d6c0 0x7fa62010e510 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:13.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 >> 172.21.15.153:0/33793189 conn(0x7fa6200f6840 msgr2=0x7fa6200fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:56:13.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 shutdown_connections 2024-04-05T02:56:13.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:13.408+0000 7fa626780700 1 -- 172.21.15.153:0/33793189 wait complete. 2024-04-05T02:56:13.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:56:13.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:56:13.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-04-05T02:56:13.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] boot 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: osdmap e9: 1 total, 1 up, 1 in 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:13.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: osd.0 [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] boot 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: osdmap e9: 1 total, 1 up, 1 in 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:14.621 DEBUG:teuthology.orchestra.run.smithi153:osd.0> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-05T02:56:14.624 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-05T02:56:14.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T02:56:14.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:14 smithi153 ceph-mon[29381]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:14.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:14 smithi153 ceph-mon[29381]: osdmap e10: 1 total, 1 up, 1 in 2024-04-05T02:56:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:14 smithi203 ceph-mon[39911]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T02:56:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:14 smithi203 ceph-mon[39911]: osdmap e10: 1 total, 1 up, 1 in 2024-04-05T02:56:16.129 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:56:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:16 smithi203 ceph-mon[39911]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:16 smithi153 ceph-mon[29381]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:18.664 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:18 smithi153 ceph-mon[29381]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011781 s, 890 MB/s 2024-04-05T02:56:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-05T02:56:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:18 smithi203 ceph-mon[39911]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:19.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-05T02:56:19.817 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: Reconfiguring crash.smithi203 (monmap changed)... 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: Reconfiguring daemon crash.smithi203 on smithi203 2024-04-05T02:56:20.243 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:20 smithi153 ceph-mon[29381]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:20.394 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: Reconfiguring crash.smithi203 (monmap changed)... 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: Reconfiguring daemon crash.smithi203 on smithi203 2024-04-05T02:56:20.395 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:20 smithi203 ceph-mon[39911]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1366891839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 msgr2=0x7f3d5c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1366891839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c0fd3f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c004940 tx=0x7f3d4c009e30 comp rx=0 tx=0).stop 2024-04-05T02:56:21.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1366891839 shutdown_connections 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1366891839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c0fd3f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1366891839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 0x7f3d5c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1366891839 >> 172.21.15.153:0/1366891839 conn(0x7f3d5c0f7430 msgr2=0x7f3d5c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1366891839 shutdown_connections 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1366891839 wait complete. 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 Processor -- start 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- start start 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 0x7f3d5c066ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c10e500 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.335+0000 7f3d63e4f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c115060 con 0x7f3d5c0fbcd0 2024-04-05T02:56:21.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54600/0 (socket says 172.21.15.153:54600) 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 -- 172.21.15.153:0/1889194094 learned_addr learned my addr 172.21.15.153:0/1889194094 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d61beb700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 0x7f3d5c066ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 -- 172.21.15.153:0/1889194094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 msgr2=0x7f3d5c066ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 0x7f3d5c066ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 -- 172.21.15.153:0/1889194094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c005040 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d613ea700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009150 tx=0x7f3d4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c00e040 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d4c004290 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c1151a0 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d4c01c3f0 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.336+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c115730 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.337+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c04c710 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.337+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f3d4c00bbf0 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.338+0000 7f3d527fc700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 0x7f3d4806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:21.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.338+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f3d4c087660 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.338+0000 7f3d61beb700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 0x7f3d4806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.338+0000 7f3d61beb700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 0x7f3d4806b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009fd0 tx=0x7f3d5800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:56:21.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.342+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3d4c0582a0 con 0x7f3d5c0fcf80 2024-04-05T02:56:21.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:21.540+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d5c1011c0 con 0x7f3d48069210 2024-04-05T02:56:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: Reconfiguring mgr.smithi203.iakodz (monmap changed)... 2024-04-05T02:56:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:56:21.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:56:21.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:21.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:21 smithi153 ceph-mon[29381]: Reconfiguring daemon mgr.smithi203.iakodz on smithi203 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: Reconfiguring mgr.smithi203.iakodz (monmap changed)... 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:21.760 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:21 smithi203 ceph-mon[39911]: Reconfiguring daemon mgr.smithi203.iakodz on smithi203 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T02:56:22.497 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: pgmap v32: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T02:56:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: Reconfiguring mon.smithi203 (monmap changed)... 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: Reconfiguring daemon mon.smithi203 on smithi203 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.719 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:23 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: Reconfiguring mon.smithi203 (monmap changed)... 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: Reconfiguring daemon mon.smithi203 on smithi203 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T02:56:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T02:56:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:23.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:23 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:24.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2211326335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f4fcb24-415c-494b-b034-5ca7e7472e4b"}]: dispatch 2024-04-05T02:56:24.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2211326335' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f4fcb24-415c-494b-b034-5ca7e7472e4b"}]': finished 2024-04-05T02:56:24.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T02:56:24.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:24 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T02:56:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2211326335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f4fcb24-415c-494b-b034-5ca7e7472e4b"}]: dispatch 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2211326335' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f4fcb24-415c-494b-b034-5ca7e7472e4b"}]': finished 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T02:56:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:24 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:25.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:25 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2620924617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:56:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:25 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2620924617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:56:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:26 smithi203 ceph-mon[39911]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:26 smithi153 ceph-mon[29381]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:29.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:28 smithi153 ceph-mon[29381]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:29.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:28 smithi203 ceph-mon[39911]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:31.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:30 smithi153 ceph-mon[29381]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:30 smithi203 ceph-mon[39911]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: Detected new or changed devices on smithi153 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi153 to 13888M 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:32.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi153 2024-04-05T02:56:32.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:32.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi153 to 13888M 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:32.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:33.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:33 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T02:56:33.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:33 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:33.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:33 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T02:56:33.628 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:33 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:34 smithi203 ceph-mon[39911]: Deploying daemon osd.1 on smithi153 2024-04-05T02:56:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:34 smithi203 ceph-mon[39911]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:34 smithi153 ceph-mon[29381]: Deploying daemon osd.1 on smithi153 2024-04-05T02:56:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:34 smithi153 ceph-mon[29381]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:34 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:34.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:34 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:36.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:36 smithi153 ceph-mon[29381]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:36.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:36 smithi203 ceph-mon[39911]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:38.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:38.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:38.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:38 smithi203 ceph-mon[39911]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:38.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:38.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:38.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:38.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:38 smithi153 ceph-mon[29381]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:40.717 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:40.717 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:40.717 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:40.718 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:40 smithi203 ceph-mon[39911]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:40 smithi153 ceph-mon[29381]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:42.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:42 smithi153 ceph-mon[29381]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:42.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:42.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:42 smithi203 ceph-mon[39911]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:42.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:43.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.199+0000 7f3d527fc700 1 -- 172.21.15.153:0/1889194094 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3d5c1011c0 con 0x7f3d48069210 2024-04-05T02:56:43.200 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 msgr2=0x7f3d4806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 0x7f3d4806b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009fd0 tx=0x7f3d5800b040 comp rx=0 tx=0).stop 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 msgr2=0x7f3d5c114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009150 tx=0x7f3d4c009e30 comp rx=0 tx=0).stop 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 shutdown_connections 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3d48069210 0x7f3d4806b6c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d5c0fbcd0 0x7f3d5c066ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 --2- 172.21.15.153:0/1889194094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d5c0fcf80 0x7f3d5c114b20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 >> 172.21.15.153:0/1889194094 conn(0x7f3d5c0f7430 msgr2=0x7f3d5c0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:56:43.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 shutdown_connections 2024-04-05T02:56:43.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:43.202+0000 7f3d63e4f700 1 -- 172.21.15.153:0/1889194094 wait complete. 2024-04-05T02:56:44.368 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-05T02:56:44.370 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-05T02:56:44.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T02:56:44.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:44.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:44.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:44 smithi153 ceph-mon[29381]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T02:56:44.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:44 smithi153 ceph-mon[29381]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:44.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:44 smithi203 ceph-mon[39911]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T02:56:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:44 smithi203 ceph-mon[39911]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:44.865 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:56:45.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-mon[29381]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T02:56:45.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-mon[29381]: osdmap e12: 2 total, 1 up, 2 in 2024-04-05T02:56:45.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:45.452 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-mon[29381]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:56:45.453 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[54573]: 2024-04-05T02:56:45.177+0000 7f3818ff2700 -1 osd.1 0 waiting for initial osdmap 2024-04-05T02:56:45.453 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:56:45 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[54573]: 2024-04-05T02:56:45.183+0000 7f3813ded700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T02:56:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:45 smithi203 ceph-mon[39911]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T02:56:45.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:45 smithi203 ceph-mon[39911]: osdmap e12: 2 total, 1 up, 2 in 2024-04-05T02:56:45.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:45 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:45.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:45 smithi203 ceph-mon[39911]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:56:46.463 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:46 smithi153 ceph-mon[29381]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:56:46.463 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:46 smithi153 ceph-mon[29381]: osdmap e13: 2 total, 1 up, 2 in 2024-04-05T02:56:46.464 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:46 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:46.464 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:46 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:46.464 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:46 smithi153 ceph-mon[29381]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:46.506 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:46 smithi203 ceph-mon[39911]: from='osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:56:46.506 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:46 smithi203 ceph-mon[39911]: osdmap e13: 2 total, 1 up, 2 in 2024-04-05T02:56:46.506 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:46 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:46.506 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:46 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:46.506 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:46 smithi203 ceph-mon[39911]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-05T02:56:47.022 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T02:56:47.022 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T02:56:47.022 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-05T02:56:47.022 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-05T02:56:47.022 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128734 s, 815 MB/s 2024-04-05T02:56:47.023 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-05T02:56:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] boot 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: osdmap e14: 2 total, 2 up, 2 in 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: osd.1 [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] boot 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: osdmap e14: 2 total, 2 up, 2 in 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:47.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-05T02:56:48.277 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:56:48.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:48 smithi203 ceph-mon[39911]: osdmap e15: 2 total, 2 up, 2 in 2024-04-05T02:56:48.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:48 smithi203 ceph-mon[39911]: pgmap v50: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:48.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:48 smithi153 ceph-mon[29381]: osdmap e15: 2 total, 2 up, 2 in 2024-04-05T02:56:48.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:48 smithi153 ceph-mon[29381]: pgmap v50: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:50.758 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:50 smithi153 ceph-mon[29381]: pgmap v51: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:50 smithi203 ceph-mon[39911]: pgmap v51: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- 172.21.15.153:0/1711155172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fdfe0 msgr2=0x7fe2200fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 --2- 172.21.15.153:0/1711155172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fdfe0 0x7fe2200fe450 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe210004940 tx=0x7fe210009e30 comp rx=0 tx=0).stop 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- 172.21.15.153:0/1711155172 shutdown_connections 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 --2- 172.21.15.153:0/1711155172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fdfe0 0x7fe2200fe450 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 --2- 172.21.15.153:0/1711155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fce70 0x7fe2200fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- 172.21.15.153:0/1711155172 >> 172.21.15.153:0/1711155172 conn(0x7fe2200f8550 msgr2=0x7fe2200fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- 172.21.15.153:0/1711155172 shutdown_connections 2024-04-05T02:56:51.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- 172.21.15.153:0/1711155172 wait complete. 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 Processor -- start 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.070+0000 7fe2272c2700 1 -- start start 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe2272c2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe2272c2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fdfe0 0x7fe220070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe2272c2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220070850 con 0x7fe2200fce70 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe2272c2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200709c0 con 0x7fe2200fdfe0 2024-04-05T02:56:51.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32810/0 (socket says 172.21.15.153:32810) 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 -- 172.21.15.153:0/3812900700 learned_addr learned my addr 172.21.15.153:0/3812900700 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22485d700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fdfe0 0x7fe220070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 -- 172.21.15.153:0/3812900700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fdfe0 msgr2=0x7fe220070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fdfe0 0x7fe220070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 -- 172.21.15.153:0/3812900700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210005040 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe22505e700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00b790 tx=0x7fe21c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe21c00f770 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe21c00fda0 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220070ca0 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.071+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe21c00d310 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.072+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220115a40 con 0x7fe2200fce70 2024-04-05T02:56:51.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.072+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22004c710 con 0x7fe2200fce70 2024-04-05T02:56:51.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.073+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fe21c010420 con 0x7fe2200fce70 2024-04-05T02:56:51.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.073+0000 7fe215ffb700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 0x7fe20c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:56:51.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.073+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fe21c0863e0 con 0x7fe2200fce70 2024-04-05T02:56:51.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.073+0000 7fe22485d700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 0x7fe20c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:56:51.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.074+0000 7fe22485d700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 0x7fe20c06b6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe210000c00 tx=0x7fe210009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:56:51.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.077+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe21c056f10 con 0x7fe2200fce70 2024-04-05T02:56:51.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:56:51.273+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe220102300 con 0x7fe20c069210 2024-04-05T02:56:51.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:51 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:56:51.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:51 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:56:51.706 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:51 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:51 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:56:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:51 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:56:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:51 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='client.14318 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: pgmap v52: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi153 to 6944M 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:52.738 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:52.738 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.738 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:52 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='client.14318 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:56:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: pgmap v52: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi153 to 6944M 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:52.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:52 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:56:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3536664234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d951e5ca-f53c-4d05-94c4-f732a3910a78"}]: dispatch 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3536664234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d951e5ca-f53c-4d05-94c4-f732a3910a78"}]': finished 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:54 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1189635490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:56:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3536664234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d951e5ca-f53c-4d05-94c4-f732a3910a78"}]: dispatch 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3536664234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d951e5ca-f53c-4d05-94c4-f732a3910a78"}]': finished 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:54 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1189635490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:56.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:56.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:56.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:56:56.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:56:56.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:56:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:58 smithi153 ceph-mon[29381]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:56:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:56:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:58 smithi203 ceph-mon[39911]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:56:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:56:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:00.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:00 smithi203 ceph-mon[39911]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:00.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:00 smithi153 ceph-mon[29381]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T02:57:01.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T02:57:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:02.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:02 smithi153 ceph-mon[29381]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:02.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:02 smithi153 ceph-mon[29381]: Deploying daemon osd.2 on smithi153 2024-04-05T02:57:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:02 smithi203 ceph-mon[39911]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:02 smithi203 ceph-mon[39911]: Deploying daemon osd.2 on smithi153 2024-04-05T02:57:04.791 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:04 smithi153 ceph-mon[29381]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:04 smithi203 ceph-mon[39911]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:06.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:06 smithi203 ceph-mon[39911]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:06.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:06 smithi153 ceph-mon[29381]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:06.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:06.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:06.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:08.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:08 smithi153 ceph-mon[29381]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:08 smithi203 ceph-mon[39911]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:09.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.528+0000 7fe215ffb700 1 -- 172.21.15.153:0/3812900700 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe220102300 con 0x7fe20c069210 2024-04-05T02:57:09.528 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.530+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 msgr2=0x7fe20c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.530+0000 7fe2272c2700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 0x7fe20c06b6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe210000c00 tx=0x7fe210009e30 comp rx=0 tx=0).stop 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.530+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 msgr2=0x7fe220071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.530+0000 7fe2272c2700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00b790 tx=0x7fe21c009250 comp rx=0 tx=0).stop 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.530+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 shutdown_connections 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe20c069210 0x7fe20c06b6c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2200fce70 0x7fe220071c60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 --2- 172.21.15.153:0/3812900700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2200fdfe0 0x7fe220070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 >> 172.21.15.153:0/3812900700 conn(0x7fe2200f8550 msgr2=0x7fe2200f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 shutdown_connections 2024-04-05T02:57:09.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:57:09.531+0000 7fe2272c2700 1 -- 172.21.15.153:0/3812900700 wait complete. 2024-04-05T02:57:10.188 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-05T02:57:10.190 INFO:tasks.cephadm:Deploying osd.3 on smithi203 with /dev/vg_nvme/lv_4... 2024-04-05T02:57:10.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T02:57:10.454 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:57:10.521 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:10 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T02:57:10.390+0000 7fc6b7f8e540 -1 osd.2 0 log_to_monitors true 2024-04-05T02:57:10.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:10 smithi153 ceph-mon[29381]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:10.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:10 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:10.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:10 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:10.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:10 smithi153 ceph-mon[29381]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T02:57:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:10 smithi203 ceph-mon[39911]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:10 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:10 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:10 smithi203 ceph-mon[39911]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T02:57:11.572 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T02:57:11.548+0000 7fc6ae760700 -1 osd.2 0 waiting for initial osdmap 2024-04-05T02:57:11.572 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T02:57:11.553+0000 7fc6ab55f700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T02:57:11.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-mon[29381]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T02:57:11.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-mon[29381]: osdmap e17: 3 total, 2 up, 3 in 2024-04-05T02:57:11.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:11.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:11 smithi153 ceph-mon[29381]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:57:11.808 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:11 smithi203 ceph-mon[39911]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T02:57:11.808 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:11 smithi203 ceph-mon[39911]: osdmap e17: 3 total, 2 up, 3 in 2024-04-05T02:57:11.808 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:11 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:11.808 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:11 smithi203 ceph-mon[39911]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114134 s, 919 MB/s 2024-04-05T02:57:12.379 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-05T02:57:12.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:12 smithi153 ceph-mon[29381]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:12.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:12 smithi153 ceph-mon[29381]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:57:12.627 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:12 smithi153 ceph-mon[29381]: osdmap e18: 3 total, 2 up, 3 in 2024-04-05T02:57:12.627 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:12.627 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:12 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:12.687 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:12 smithi203 ceph-mon[39911]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:12.687 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:12 smithi203 ceph-mon[39911]: from='osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-05T02:57:12.687 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:12 smithi203 ceph-mon[39911]: osdmap e18: 3 total, 2 up, 3 in 2024-04-05T02:57:12.687 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:12.688 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:12 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:12.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-04-05T02:57:13.250 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:57:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] boot 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: osdmap e19: 3 total, 3 up, 3 in 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:14.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: osd.2 [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] boot 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: osdmap e19: 3 total, 3 up, 3 in 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 -- 172.21.15.203:0/679626955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 msgr2=0x7f58a40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 --2- 172.21.15.203:0/679626955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a40fc0e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58a0004940 tx=0x7f58a0009e30 comp rx=0 tx=0).stop 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 -- 172.21.15.203:0/679626955 shutdown_connections 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 --2- 172.21.15.203:0/679626955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 0x7f58a40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 --2- 172.21.15.203:0/679626955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a40fc0e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 -- 172.21.15.203:0/679626955 >> 172.21.15.203:0/679626955 conn(0x7f58a40f7430 msgr2=0x7f58a40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 -- 172.21.15.203:0/679626955 shutdown_connections 2024-04-05T02:57:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.760+0000 7f58ab610700 1 -- 172.21.15.203:0/679626955 wait complete. 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 Processor -- start 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 -- start start 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 0x7f58a410ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a410d480 con 0x7f58a40fcf80 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58ab610700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a410dcd0 con 0x7f58a40fbcd0 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:34518/0 (socket says 172.21.15.203:34518) 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 -- 172.21.15.203:0/22178543 learned_addr learned my addr 172.21.15.203:0/22178543 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a8bab700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 0x7f58a410ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 -- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 msgr2=0x7f58a410ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 0x7f58a410ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.761+0000 7f58a93ac700 1 -- 172.21.15.203:0/22178543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0005040 con 0x7f58a40fbcd0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f58a93ac700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009fd0 tx=0x7f58a0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a000e040 con 0x7f58a40fbcd0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58a000be00 con 0x7f58a40fbcd0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a410df50 con 0x7f58a40fbcd0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58a00137f0 con 0x7f58a40fbcd0 2024-04-05T02:57:14.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.762+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a410e490 con 0x7f58a40fbcd0 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.763+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a405f5f0 con 0x7f58a40fbcd0 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.763+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f58a0013a60 con 0x7f58a40fbcd0 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.764+0000 7f5899ffb700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 0x7f589006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.764+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f58a008a100 con 0x7f58a40fbcd0 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.764+0000 7f58a8bab700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 0x7f589006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:14.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.764+0000 7f58a8bab700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 0x7f589006b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5894004f40 tx=0x7f589400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:57:14.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.768+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f58a005a890 con 0x7f58a40fbcd0 2024-04-05T02:57:14.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:14.958+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f58a410e880 con 0x7f5890069210 2024-04-05T02:57:14.958 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:14 smithi203 ceph-mon[39911]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:14.958 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:14 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-05T02:57:14.959 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:14 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-05T02:57:14.959 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:14 smithi203 ceph-mon[39911]: osdmap e20: 3 total, 3 up, 3 in 2024-04-05T02:57:14.959 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:14 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T02:57:15.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:14 smithi153 ceph-mon[29381]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-05T02:57:15.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-05T02:57:15.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-05T02:57:15.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:14 smithi153 ceph-mon[29381]: osdmap e20: 3 total, 3 up, 3 in 2024-04-05T02:57:15.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T02:57:15.553 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66202]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T02:57:15.553 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66202]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:15.553 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66202]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:15.553 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66202]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:15.702 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:15.702+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f58a0013d10 con 0x7f58a40fbcd0 2024-04-05T02:57:15.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T02:57:15.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:15.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66208]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:15.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66208]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:15.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T02:57:15.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:15 smithi153 ceph-mon[29381]: osdmap e21: 3 total, 3 up, 3 in 2024-04-05T02:57:15.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:57:15.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:57:15.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:16.013 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T02:57:16.013 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:15 smithi203 ceph-mon[39911]: osdmap e21: 3 total, 3 up, 3 in 2024-04-05T02:57:16.013 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:57:16.013 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:57:16.013 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:16.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T02:57:16.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:16.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:16.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:15 smithi153 sudo[66212]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:16.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T02:57:16.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66279]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T02:57:16.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66279]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:16.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66279]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:16.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66279]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:16.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:16.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66282]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:16.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66282]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: osdmap e22: 3 total, 3 up, 3 in 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: mgrmap e19: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:16.733 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:16.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66285]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T02:57:16.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:16.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66285]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:16.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66285]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:16.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T02:57:16.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:16.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66288]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:16.992 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:16 smithi153 sudo[66288]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: osdmap e22: 3 total, 3 up, 3 in 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: mgrmap e19: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:17.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:17.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:17.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:17.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:16 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 sudo[66291]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T02:57:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 sudo[66291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 sudo[66291]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 sudo[66291]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:17.736 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T02:57:17.736 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:17.737 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44332]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:17.737 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44332]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:18.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44335]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44335]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44335]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 sudo[44335]: pam_unix(sudo:session): session closed for user root 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi153 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi153 to 4629M 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:57:18.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:57:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: Detected new or changed devices on smithi153 2024-04-05T02:57:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi153 to 4629M 2024-04-05T02:57:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T02:57:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T02:57:18.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:57:18.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:57:18.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T02:57:18.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T02:57:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/387475971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]: dispatch 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]: dispatch 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]': finished 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: osdmap e23: 4 total, 3 up, 4 in 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:19.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:18 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1352367678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:57:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T02:57:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T02:57:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/387475971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]: dispatch 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]: dispatch 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1"}]': finished 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: osdmap e23: 4 total, 3 up, 4 in 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:18 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1352367678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:57:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:20 smithi203 ceph-mon[39911]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:20 smithi153 ceph-mon[29381]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:22 smithi203 ceph-mon[39911]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:22 smithi153 ceph-mon[29381]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:24 smithi203 ceph-mon[39911]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-05T02:57:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:24 smithi153 ceph-mon[29381]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-05T02:57:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:25 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T02:57:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:25 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:25 smithi203 ceph-mon[39911]: Deploying daemon osd.3 on smithi203 2024-04-05T02:57:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:25 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T02:57:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:25 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:25 smithi153 ceph-mon[29381]: Deploying daemon osd.3 on smithi203 2024-04-05T02:57:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:26 smithi203 ceph-mon[39911]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-05T02:57:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:26 smithi153 ceph-mon[29381]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-05T02:57:28.918 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:28 smithi203 ceph-mon[39911]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T02:57:28.918 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:28 smithi153 ceph-mon[29381]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T02:57:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:30 smithi203 ceph-mon[39911]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-04-05T02:57:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:30 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:30 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:30 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:30 smithi153 ceph-mon[29381]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-04-05T02:57:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:30 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:30 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:30 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:32.907 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:32 smithi203 ceph-mon[39911]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T02:57:33.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:32 smithi153 ceph-mon[29381]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T02:57:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.569+0000 7f5899ffb700 1 -- 172.21.15.203:0/22178543 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f58a410e880 con 0x7f5890069210 2024-04-05T02:57:33.569 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 3 on host 'smithi203' 2024-04-05T02:57:33.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 msgr2=0x7f589006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:33.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 0x7f589006b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5894004f40 tx=0x7f589400b040 comp rx=0 tx=0).stop 2024-04-05T02:57:33.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 msgr2=0x7f58a410c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009fd0 tx=0x7f58a0009e30 comp rx=0 tx=0).stop 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 shutdown_connections 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5890069210 0x7f589006b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58a40fbcd0 0x7f58a410c940 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 --2- 172.21.15.203:0/22178543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58a40fcf80 0x7f58a410ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 >> 172.21.15.203:0/22178543 conn(0x7f58a40f7430 msgr2=0x7f58a40f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 shutdown_connections 2024-04-05T02:57:33.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:33.571+0000 7f58ab610700 1 -- 172.21.15.203:0/22178543 wait complete. 2024-04-05T02:57:34.162 DEBUG:teuthology.orchestra.run.smithi203:osd.3> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-05T02:57:34.165 INFO:tasks.cephadm:Deploying osd.4 on smithi203 with /dev/vg_nvme/lv_3... 2024-04-05T02:57:34.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T02:57:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:34 smithi203 ceph-mon[39911]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T02:57:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:34 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:34 smithi203 ceph-mon[39911]: from='osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T02:57:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:34 smithi203 ceph-mon[39911]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T02:57:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:34 smithi153 ceph-mon[29381]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T02:57:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:34 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:34 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:34 smithi153 ceph-mon[29381]: from='osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T02:57:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:34 smithi153 ceph-mon[29381]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T02:57:34.872 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: osdmap e24: 4 total, 3 up, 4 in 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:35.889 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:35.890 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[46382]: 2024-04-05T02:57:35.582+0000 7fe910a41700 -1 osd.3 0 waiting for initial osdmap 2024-04-05T02:57:35.890 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 02:57:35 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[46382]: 2024-04-05T02:57:35.587+0000 7fe90c03d700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T02:57:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T02:57:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: osdmap e24: 4 total, 3 up, 4 in 2024-04-05T02:57:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:57:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:57:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:35 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136371 s, 769 MB/s 2024-04-05T02:57:36.730 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-05T02:57:36.730 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:36.730 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:57:36.730 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: osdmap e25: 4 total, 3 up, 4 in 2024-04-05T02:57:36.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:36.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:36.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] boot 2024-04-05T02:57:36.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: osdmap e26: 4 total, 4 up, 4 in 2024-04-05T02:57:36.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-05T02:57:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:57:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: osdmap e25: 4 total, 3 up, 4 in 2024-04-05T02:57:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:37.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:37.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: osd.3 [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] boot 2024-04-05T02:57:37.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: osdmap e26: 4 total, 4 up, 4 in 2024-04-05T02:57:37.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T02:57:37.666 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:37 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:57:37.666 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:37 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:57:37.666 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:37 smithi203 ceph-mon[39911]: osdmap e27: 4 total, 4 up, 4 in 2024-04-05T02:57:37.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-04-05T02:57:38.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:37 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:57:38.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:37 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:57:38.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:37 smithi153 ceph-mon[29381]: osdmap e27: 4 total, 4 up, 4 in 2024-04-05T02:57:38.647 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:57:38.907 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:38 smithi203 ceph-mon[39911]: pgmap v86: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:38.907 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:38 smithi203 ceph-mon[39911]: osdmap e28: 4 total, 4 up, 4 in 2024-04-05T02:57:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:38 smithi153 ceph-mon[29381]: pgmap v86: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:38 smithi153 ceph-mon[29381]: osdmap e28: 4 total, 4 up, 4 in 2024-04-05T02:57:40.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- 172.21.15.203:0/3473443443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fed90 msgr2=0x7efebc0faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 --2- 172.21.15.203:0/3473443443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fed90 0x7efebc0faed0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efea4004940 tx=0x7efea4009e30 comp rx=0 tx=0).stop 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- 172.21.15.203:0/3473443443 shutdown_connections 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 --2- 172.21.15.203:0/3473443443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fed90 0x7efebc0faed0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 --2- 172.21.15.203:0/3473443443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fe440 0x7efebc0fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- 172.21.15.203:0/3473443443 >> 172.21.15.203:0/3473443443 conn(0x7efebc0f6be0 msgr2=0x7efebc0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- 172.21.15.203:0/3473443443 shutdown_connections 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- 172.21.15.203:0/3473443443 wait complete. 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 Processor -- start 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.794+0000 7efec1625700 1 -- start start 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 0x7efebc10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc10e4c0 con 0x7efebc0fed90 2024-04-05T02:57:40.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc10e630 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:33296/0 (socket says 172.21.15.203:33296) 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 -- 172.21.15.203:0/1781118550 learned_addr learned my addr 172.21.15.203:0/1781118550 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efeb27fc700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 0x7efebc10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 -- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 msgr2=0x7efebc10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 0x7efebc10dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 -- 172.21.15.203:0/1781118550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea4005040 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efeb27fc700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 0x7efebc10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efebaffd700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efeac00c830 tx=0x7efeac004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efeac00a760 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc112fe0 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efeac00ad90 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efeac0113a0 con 0x7efebc0fe440 2024-04-05T02:57:40.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.795+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc113570 con 0x7efebc0fe440 2024-04-05T02:57:40.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.796+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc04c710 con 0x7efebc0fe440 2024-04-05T02:57:40.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.797+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efeac012400 con 0x7efebc0fe440 2024-04-05T02:57:40.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.797+0000 7efeb3fff700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 0x7efea806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:57:40.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.797+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7efeac086b50 con 0x7efebc0fe440 2024-04-05T02:57:40.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.798+0000 7efeb27fc700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 0x7efea806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:57:40.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.798+0000 7efeb27fc700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 0x7efea806b750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efea400b3b0 tx=0x7efea4009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:57:40.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.802+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efeac056fb0 con 0x7efebc0fe440 2024-04-05T02:57:40.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:40.996+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7efebc1137f0 con 0x7efea80692a0 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: osdmap e29: 4 total, 4 up, 4 in 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:40.996 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:40.997 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: pgmap v89: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: osdmap e29: 4 total, 4 up, 4 in 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:57:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:41.730 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi203 2024-04-05T02:57:41.730 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi203 to 14912M 2024-04-05T02:57:41.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: osdmap e30: 4 total, 4 up, 4 in 2024-04-05T02:57:41.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:57:41.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:57:41.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:42.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: Detected new or changed devices on smithi203 2024-04-05T02:57:42.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi203 to 14912M 2024-04-05T02:57:42.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: osdmap e30: 4 total, 4 up, 4 in 2024-04-05T02:57:42.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:57:42.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:57:42.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:42.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:42 smithi203 ceph-mon[39911]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:57:42.643 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:42 smithi203 ceph-mon[39911]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:42.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:42 smithi153 ceph-mon[29381]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:57:42.974 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:42 smithi153 ceph-mon[29381]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:43.880 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:43.880 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/398682818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]: dispatch 2024-04-05T02:57:43.881 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]: dispatch 2024-04-05T02:57:43.881 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]': finished 2024-04-05T02:57:43.881 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: osdmap e31: 5 total, 4 up, 5 in 2024-04-05T02:57:43.881 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:43 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:57:44.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:44.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/398682818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]: dispatch 2024-04-05T02:57:44.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]: dispatch 2024-04-05T02:57:44.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2b77994-438d-4c3b-8fb7-44217f7b3b44"}]': finished 2024-04-05T02:57:44.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: osdmap e31: 5 total, 4 up, 5 in 2024-04-05T02:57:44.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:43 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:57:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:44 smithi203 ceph-mon[39911]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-05T02:57:45.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:44 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/484897792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:57:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:44 smithi153 ceph-mon[29381]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-05T02:57:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:44 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/484897792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:57:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:46 smithi203 ceph-mon[39911]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T02:57:47.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:46 smithi153 ceph-mon[29381]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T02:57:48.788 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:48 smithi203 ceph-mon[39911]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-05T02:57:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:48 smithi153 ceph-mon[29381]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-05T02:57:51.007 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:50 smithi203 ceph-mon[39911]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-05T02:57:51.008 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T02:57:51.008 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:50 smithi153 ceph-mon[29381]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-05T02:57:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T02:57:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:57:51.879 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:51 smithi203 ceph-mon[39911]: Deploying daemon osd.4 on smithi203 2024-04-05T02:57:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:51 smithi153 ceph-mon[29381]: Deploying daemon osd.4 on smithi203 2024-04-05T02:57:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:52 smithi203 ceph-mon[39911]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T02:57:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:52 smithi153 ceph-mon[29381]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-05T02:57:54.776 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:54 smithi203 ceph-mon[39911]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:54 smithi153 ceph-mon[29381]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:56.927 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:56 smithi203 ceph-mon[39911]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:56.927 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:56.927 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:56.928 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:56 smithi153 ceph-mon[29381]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:57:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:57:58.932 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:58 smithi203 ceph-mon[39911]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:58.932 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:57:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:58 smithi153 ceph-mon[29381]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:57:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:57:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:57:59.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.260+0000 7efeb3fff700 1 -- 172.21.15.203:0/1781118550 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efebc1137f0 con 0x7efea80692a0 2024-04-05T02:57:59.260 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 4 on host 'smithi203' 2024-04-05T02:57:59.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 msgr2=0x7efea806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:59.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 0x7efea806b750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efea400b3b0 tx=0x7efea4009e30 comp rx=0 tx=0).stop 2024-04-05T02:57:59.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 msgr2=0x7efebc10d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:57:59.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efeac00c830 tx=0x7efeac004470 comp rx=0 tx=0).stop 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 shutdown_connections 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efea80692a0 0x7efea806b750 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efebc0fe440 0x7efebc10d980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 --2- 172.21.15.203:0/1781118550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efebc0fed90 0x7efebc10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 >> 172.21.15.203:0/1781118550 conn(0x7efebc0f6be0 msgr2=0x7efebc0f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 shutdown_connections 2024-04-05T02:57:59.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:57:59.262+0000 7efec1625700 1 -- 172.21.15.203:0/1781118550 wait complete. 2024-04-05T02:57:59.828 DEBUG:teuthology.orchestra.run.smithi203:osd.4> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-05T02:57:59.831 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_2... 2024-04-05T02:57:59.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T02:58:00.509 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:58:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:00 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:00 smithi203 ceph-mon[39911]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:58:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:00 smithi203 ceph-mon[39911]: from='osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T02:58:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:00 smithi203 ceph-mon[39911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T02:58:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:00 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:00 smithi153 ceph-mon[29381]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:58:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:00 smithi153 ceph-mon[29381]: from='osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T02:58:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:00 smithi153 ceph-mon[29381]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: osdmap e32: 5 total, 4 up, 5 in 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:01.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:01.547 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[53454]: 2024-04-05T02:58:01.263+0000 7f45fa9f8700 -1 osd.4 0 waiting for initial osdmap 2024-04-05T02:58:01.547 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 02:58:01 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[53454]: 2024-04-05T02:58:01.272+0000 7f45f47f1700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T02:58:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T02:58:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: osdmap e32: 5 total, 4 up, 5 in 2024-04-05T02:58:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:01.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:01.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:01.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:01.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:01 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:02.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: osdmap e33: 5 total, 4 up, 5 in 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] boot 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: osdmap e34: 5 total, 5 up, 5 in 2024-04-05T02:58:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:58:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: osdmap e33: 5 total, 4 up, 5 in 2024-04-05T02:58:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-05T02:58:02.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: osd.4 [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] boot 2024-04-05T02:58:02.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: osdmap e34: 5 total, 5 up, 5 in 2024-04-05T02:58:02.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T02:58:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:03 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:58:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:03 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:58:03.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:03 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:58:03.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:03 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114554 s, 915 MB/s 2024-04-05T02:58:03.620 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-05T02:58:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:04 smithi203 ceph-mon[39911]: osdmap e35: 5 total, 5 up, 5 in 2024-04-05T02:58:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:04 smithi203 ceph-mon[39911]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:04.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-04-05T02:58:04.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:04 smithi153 ceph-mon[29381]: osdmap e35: 5 total, 5 up, 5 in 2024-04-05T02:58:04.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:04 smithi153 ceph-mon[29381]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:05.146 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:58:06.277 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi203 2024-04-05T02:58:06.277 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi203 to 7456M 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:06.278 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:06.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:06.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:06 smithi203 ceph-mon[39911]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: Detected new or changed devices on smithi203 2024-04-05T02:58:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi203 to 7456M 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:06 smithi153 ceph-mon[29381]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 -- 172.21.15.203:0/3430158701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 msgr2=0x7f38640fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 --2- 172.21.15.203:0/3430158701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fe860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3854004940 tx=0x7f3854009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 -- 172.21.15.203:0/3430158701 shutdown_connections 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 --2- 172.21.15.203:0/3430158701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38640feda0 0x7f38640faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 --2- 172.21.15.203:0/3430158701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fe860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 -- 172.21.15.203:0/3430158701 >> 172.21.15.203:0/3430158701 conn(0x7f38640f6c10 msgr2=0x7f38640f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 -- 172.21.15.203:0/3430158701 shutdown_connections 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.843+0000 7f3869166700 1 -- 172.21.15.203:0/3430158701 wait complete. 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 Processor -- start 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 -- start start 2024-04-05T02:58:06.845 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38640feda0 0x7f38640fc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640fe060 con 0x7f38640feda0 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3869166700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640fc650 con 0x7f38640fe450 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:38232/0 (socket says 172.21.15.203:38232) 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 -- 172.21.15.203:0/295371557 learned_addr learned my addr 172.21.15.203:0/295371557 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 -- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38640feda0 msgr2=0x7f38640fc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38640feda0 0x7f38640fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.844+0000 7f3862d9d700 1 -- 172.21.15.203:0/295371557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854005040 con 0x7f38640fe450 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f3862d9d700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f385400b3b0 tx=0x7f3854009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385400e040 con 0x7f38640fe450 2024-04-05T02:58:06.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f385400bcc0 con 0x7f38640fe450 2024-04-05T02:58:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38640fc8a0 con 0x7f38640fe450 2024-04-05T02:58:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f385401c630 con 0x7f38640fe450 2024-04-05T02:58:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.845+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640fce60 con 0x7f38640fe450 2024-04-05T02:58:06.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.846+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386404c710 con 0x7f38640fe450 2024-04-05T02:58:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.846+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3854013980 con 0x7f38640fe450 2024-04-05T02:58:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.846+0000 7f384b7fe700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 0x7f384c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.847+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f3854087ad0 con 0x7f38640fe450 2024-04-05T02:58:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.847+0000 7f386259c700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 0x7f384c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:06.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.847+0000 7f386259c700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 0x7f384c06b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3858004d00 tx=0x7f3858009ce0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:06.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:06.851+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3854057d00 con 0x7f38640fe450 2024-04-05T02:58:07.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:07.043+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3864102520 con 0x7f384c0692a0 2024-04-05T02:58:07.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:07 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:58:07.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:07 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:58:07.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:07 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:07 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T02:58:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:07 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T02:58:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:07 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:08.342 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:08 smithi203 ceph-mon[39911]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:58:08.342 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:08 smithi203 ceph-mon[39911]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:08 smithi153 ceph-mon[29381]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:58:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:08 smithi153 ceph-mon[29381]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:09.767 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:09 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/3365729047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]: dispatch 2024-04-05T02:58:09.767 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:09 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]: dispatch 2024-04-05T02:58:09.767 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:09 smithi203 ceph-mon[39911]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]': finished 2024-04-05T02:58:09.767 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:09 smithi203 ceph-mon[39911]: osdmap e36: 6 total, 5 up, 6 in 2024-04-05T02:58:09.767 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/3365729047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]: dispatch 2024-04-05T02:58:09.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:09 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]: dispatch 2024-04-05T02:58:09.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:09 smithi153 ceph-mon[29381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708"}]': finished 2024-04-05T02:58:09.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:09 smithi153 ceph-mon[29381]: osdmap e36: 6 total, 5 up, 6 in 2024-04-05T02:58:09.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:10 smithi203 ceph-mon[39911]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:10.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:10 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/1368317192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:58:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:10 smithi153 ceph-mon[29381]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:10 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/1368317192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T02:58:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:12 smithi203 ceph-mon[39911]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:12 smithi153 ceph-mon[29381]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:13.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:14.737 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:14 smithi203 ceph-mon[39911]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:14 smithi153 ceph-mon[29381]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:16 smithi203 ceph-mon[39911]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:16 smithi153 ceph-mon[29381]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T02:58:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:17 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:17 smithi203 ceph-mon[39911]: Deploying daemon osd.5 on smithi203 2024-04-05T02:58:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T02:58:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:17 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:17 smithi153 ceph-mon[29381]: Deploying daemon osd.5 on smithi203 2024-04-05T02:58:18.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:18 smithi203 ceph-mon[39911]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:18 smithi153 ceph-mon[29381]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:20.727 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:20 smithi203 ceph-mon[39911]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:20 smithi153 ceph-mon[29381]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:22.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:22 smithi203 ceph-mon[39911]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:22.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:22.639 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:22.640 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:22 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:22 smithi153 ceph-mon[29381]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:22 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:24.791 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:24 smithi203 ceph-mon[39911]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:24 smithi153 ceph-mon[29381]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:25.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.258+0000 7f384b7fe700 1 -- 172.21.15.203:0/295371557 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3864102520 con 0x7f384c0692a0 2024-04-05T02:58:25.258 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 msgr2=0x7f384c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 0x7f384c06b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3858004d00 tx=0x7f3858009ce0 comp rx=0 tx=0).stop 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 msgr2=0x7f38640fda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f385400b3b0 tx=0x7f3854009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 shutdown_connections 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f384c0692a0 0x7f384c06b750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38640fe450 0x7f38640fda60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 --2- 172.21.15.203:0/295371557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38640feda0 0x7f38640fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 >> 172.21.15.203:0/295371557 conn(0x7f38640f6c10 msgr2=0x7f38640f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:25.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 shutdown_connections 2024-04-05T02:58:25.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:25.260+0000 7f3869166700 1 -- 172.21.15.203:0/295371557 wait complete. 2024-04-05T02:58:26.394 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-05T02:58:26.397 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-05T02:58:26.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-05T02:58:26.535 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:26.535 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:26 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:26.535 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:26 smithi203 ceph-mon[39911]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:26.535 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:26 smithi203 ceph-mon[39911]: from='osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T02:58:26.535 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:26 smithi203 ceph-mon[39911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T02:58:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:26 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:26 smithi153 ceph-mon[29381]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:26 smithi153 ceph-mon[29381]: from='osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T02:58:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:26 smithi153 ceph-mon[29381]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T02:58:26.664 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:27.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-mon[39911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T02:58:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-mon[39911]: osdmap e37: 6 total, 5 up, 6 in 2024-04-05T02:58:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-mon[39911]: from='osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-mon[39911]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:27.547 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[60518]: 2024-04-05T02:58:27.258+0000 7f9a5a0c2700 -1 osd.5 0 waiting for initial osdmap 2024-04-05T02:58:27.547 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 02:58:27 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[60518]: 2024-04-05T02:58:27.266+0000 7f9a54ebd700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T02:58:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:27 smithi153 ceph-mon[29381]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T02:58:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:27 smithi153 ceph-mon[29381]: osdmap e37: 6 total, 5 up, 6 in 2024-04-05T02:58:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:27 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:27 smithi153 ceph-mon[29381]: from='osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:27 smithi153 ceph-mon[29381]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T02:58:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 -- 172.21.15.153:0/2213795449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 msgr2=0x7fa5700ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 --2- 172.21.15.153:0/2213795449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700ff9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa560004940 tx=0x7fa560009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 -- 172.21.15.153:0/2213795449 shutdown_connections 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 --2- 172.21.15.153:0/2213795449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 0x7fa570102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 --2- 172.21.15.153:0/2213795449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700ff9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 -- 172.21.15.153:0/2213795449 >> 172.21.15.153:0/2213795449 conn(0x7fa5700f7430 msgr2=0x7fa5700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 -- 172.21.15.153:0/2213795449 shutdown_connections 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.316+0000 7fa5760dc700 1 -- 172.21.15.153:0/2213795449 wait complete. 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 Processor -- start 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 -- start start 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 0x7fa5700fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700fbe70 con 0x7fa5700662e0 2024-04-05T02:58:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa5760dc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700fbfe0 con 0x7fa5700fff00 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45630/0 (socket says 172.21.15.153:45630) 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 -- 172.21.15.153:0/1782394456 learned_addr learned my addr 172.21.15.153:0/1782394456 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa566ffd700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 0x7fa5700fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 -- 172.21.15.153:0/1782394456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 msgr2=0x7fa5700fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 0x7fa5700fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.317+0000 7fa56f7fe700 1 -- 172.21.15.153:0/1782394456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560005040 con 0x7fa5700662e0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa56f7fe700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b970 tx=0x7fa560009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa56000e040 con 0x7fa5700662e0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa560004290 con 0x7fa5700662e0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5700fc260 con 0x7fa5700662e0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa56001c3f0 con 0x7fa5700662e0 2024-04-05T02:58:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.318+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa570115a60 con 0x7fa5700662e0 2024-04-05T02:58:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.319+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57005f5f0 con 0x7fa5700662e0 2024-04-05T02:58:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.319+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa56000bbf0 con 0x7fa5700662e0 2024-04-05T02:58:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.320+0000 7fa56cff9700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 0x7fa55c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.320+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa560088100 con 0x7fa5700662e0 2024-04-05T02:58:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.320+0000 7fa566ffd700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 0x7fa55c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.320+0000 7fa566ffd700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 0x7fa55c06b760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa5580102e0 tx=0x7fa55800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:28.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.324+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa560054770 con 0x7fa5700662e0 2024-04-05T02:58:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.521+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa57004bed0 con 0x7fa5700662e0 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: osdmap e38: 6 total, 5 up, 6 in 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:28.521 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.521+0000 7fa56cff9700 1 -- 172.21.15.153:0/1782394456 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa57004bed0 con 0x7fa5700662e0 2024-04-05T02:58:28.522 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 msgr2=0x7fa55c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 0x7fa55c06b760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa5580102e0 tx=0x7fa55800b040 comp rx=0 tx=0).stop 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 msgr2=0x7fa5700fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa56000b970 tx=0x7fa560009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 shutdown_connections 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa55c0692b0 0x7fa55c06b760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5700662e0 0x7fa5700fd3f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 --2- 172.21.15.153:0/1782394456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5700fff00 0x7fa5700fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 >> 172.21.15.153:0/1782394456 conn(0x7fa5700f7430 msgr2=0x7fa5700f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 shutdown_connections 2024-04-05T02:58:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:28.524+0000 7fa5760dc700 1 -- 172.21.15.153:0/1782394456 wait complete. 2024-04-05T02:58:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-05T02:58:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: osdmap e38: 6 total, 5 up, 6 in 2024-04-05T02:58:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:28.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:28.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-05T02:58:28.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:28.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:29.111 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1712285908,"num_in_osds":6,"osd_in_since":1712285889,"num_remapped_pgs":0} 2024-04-05T02:58:29.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T02:58:29.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: purged_snaps scrub starts 2024-04-05T02:58:29.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: purged_snaps scrub ok 2024-04-05T02:58:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] boot 2024-04-05T02:58:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: osdmap e39: 6 total, 6 up, 6 in 2024-04-05T02:58:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:29 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1782394456' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T02:58:29.382 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: purged_snaps scrub starts 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: purged_snaps scrub ok 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: osd.5 [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] boot 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: osdmap e39: 6 total, 6 up, 6 in 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T02:58:29.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:29.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:29 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1782394456' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T02:58:30.475 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:30 smithi153 ceph-mon[29381]: osdmap e40: 6 total, 6 up, 6 in 2024-04-05T02:58:30.476 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:30 smithi153 ceph-mon[29381]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:30.493 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:30 smithi203 ceph-mon[39911]: osdmap e40: 6 total, 6 up, 6 in 2024-04-05T02:58:30.493 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:30 smithi203 ceph-mon[39911]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 -- 172.21.15.153:0/1992145942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6941037b0 msgr2=0x7fd6940fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 --2- 172.21.15.153:0/1992145942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6941037b0 0x7fd6940fce20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd688004940 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 -- 172.21.15.153:0/1992145942 shutdown_connections 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 --2- 172.21.15.153:0/1992145942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6941037b0 0x7fd6940fce20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 --2- 172.21.15.153:0/1992145942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd694102e10 0x7fd6941031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 -- 172.21.15.153:0/1992145942 >> 172.21.15.153:0/1992145942 conn(0x7fd6940f8980 msgr2=0x7fd6940fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 -- 172.21.15.153:0/1992145942 shutdown_connections 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.960+0000 7fd69a84f700 1 -- 172.21.15.153:0/1992145942 wait complete. 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 Processor -- start 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 -- start start 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6941037b0 0x7fd6941140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6941147d0 con 0x7fd694102e10 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd69a84f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69410db30 con 0x7fd6941037b0 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd693fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd693fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47414/0 (socket says 172.21.15.153:47414) 2024-04-05T02:58:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.961+0000 7fd693fff700 1 -- 172.21.15.153:0/161492332 learned_addr learned my addr 172.21.15.153:0/161492332 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd6937fe700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6941037b0 0x7fd6941140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd693fff700 1 -- 172.21.15.153:0/161492332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6941037b0 msgr2=0x7fd6941140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd693fff700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6941037b0 0x7fd6941140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd693fff700 1 -- 172.21.15.153:0/161492332 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688005040 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd693fff700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd68400b790 tx=0x7fd684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68400f770 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd68400fda0 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69410de10 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68400d310 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.962+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69410e3a0 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.963+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6940fe560 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.963+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd684010420 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.964+0000 7fd690ff9700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 0x7fd67c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.964+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd684086e70 con 0x7fd694102e10 2024-04-05T02:58:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.964+0000 7fd6937fe700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 0x7fd67c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:30.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.964+0000 7fd6937fe700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 0x7fd67c06b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd69410ee10 tx=0x7fd688009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:30.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:30.968+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd684056f70 con 0x7fd694102e10 2024-04-05T02:58:31.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.161+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd69404bed0 con 0x7fd694102e10 2024-04-05T02:58:31.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.162+0000 7fd690ff9700 1 -- 172.21.15.153:0/161492332 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11108 (secure 0 0 0) 0x7fd684056b00 con 0x7fd694102e10 2024-04-05T02:58:31.162 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:58:31.162 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T02:58:29.264740+0000","last_up_change":"2024-04-05T02:58:28.256825+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6803","nonce":78196267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6805","nonce":78196267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6809","nonce":78196267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6807","nonce":78196267}]},"public_addr":"172.21.15.153:6803/78196267","cluster_addr":"172.21.15.153:6805/78196267","heartbeat_back_addr":"172.21.15.153:6809/78196267","heartbeat_front_addr":"172.21.15.153:6807/78196267","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6811","nonce":4220639279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6813","nonce":4220639279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6817","nonce":4220639279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6815","nonce":4220639279}]},"public_addr":"172.21.15.153:6811/4220639279","cluster_addr":"172.21.15.153:6813/4220639279","heartbeat_back_addr":"172.21.15.153:6817/4220639279","heartbeat_front_addr":"172.21.15.153:6815/4220639279","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6819","nonce":1027037149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6821","nonce":1027037149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6825","nonce":1027037149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6823","nonce":1027037149}]},"public_addr":"172.21.15.153:6819/1027037149","cluster_addr":"172.21.15.153:6821/1027037149","heartbeat_back_addr":"172.21.15.153:6825/1027037149","heartbeat_front_addr":"172.21.15.153:6823/1027037149","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6801","nonce":4064942773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6803","nonce":4064942773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6807","nonce":4064942773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6805","nonce":4064942773}]},"public_addr":"172.21.15.203:6801/4064942773","cluster_addr":"172.21.15.203:6803/4064942773","heartbeat_back_addr":"172.21.15.203:6807/4064942773","heartbeat_front_addr":"172.21.15.203:6805/4064942773","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6809","nonce":519188752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6811","nonce":519188752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6815","nonce":519188752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6813","nonce":519188752}]},"public_addr":"172.21.15.203:6809/519188752","cluster_addr":"172.21.15.203:6811/519188752","heartbeat_back_addr":"172.21.15.203:6815/519188752","heartbeat_front_addr":"172.21.15.203:6813/519188752","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6817","nonce":2086848603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6819","nonce":2086848603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6823","nonce":2086848603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6821","nonce":2086848603}]},"public_addr":"172.21.15.203:6817/2086848603","cluster_addr":"172.21.15.203:6819/2086848603","heartbeat_back_addr":"172.21.15.203:6823/2086848603","heartbeat_front_addr":"172.21.15.203:6821/2086848603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T02:58:31.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 msgr2=0x7fd67c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:31.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 0x7fd67c06b750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd69410ee10 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:31.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 msgr2=0x7fd694113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:31.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd68400b790 tx=0x7fd684009250 comp rx=0 tx=0).stop 2024-04-05T02:58:31.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 shutdown_connections 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fd67c0692a0 0x7fd67c06b750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd694102e10 0x7fd694113bb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 --2- 172.21.15.153:0/161492332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6941037b0 0x7fd6941140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 >> 172.21.15.153:0/161492332 conn(0x7fd6940f8980 msgr2=0x7fd6940f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 shutdown_connections 2024-04-05T02:58:31.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:31.164+0000 7fd69a84f700 1 -- 172.21.15.153:0/161492332 wait complete. 2024-04-05T02:58:31.823 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T02:57:13.688102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-05T02:58:31.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-05T02:58:32.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: Detected new or changed devices on smithi203 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi203 to 4970M 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/161492332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:32.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:32.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:32.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:32 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi203 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T02:58:32.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi203 to 4970M 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/161492332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:58:32.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:32 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- 172.21.15.153:0/4103818471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 msgr2=0x7f92041031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 --2- 172.21.15.153:0/4103818471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f92041031f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9200004940 tx=0x7f9200009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- 172.21.15.153:0/4103818471 shutdown_connections 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 --2- 172.21.15.153:0/4103818471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 0x7f92040fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 --2- 172.21.15.153:0/4103818471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f92041031f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- 172.21.15.153:0/4103818471 >> 172.21.15.153:0/4103818471 conn(0x7f92040f89d0 msgr2=0x7f92040fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- 172.21.15.153:0/4103818471 shutdown_connections 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- 172.21.15.153:0/4103818471 wait complete. 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 Processor -- start 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- start start 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:33.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 0x7f92041140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9204114720 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.474+0000 7f920bd18700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920410db10 con 0x7f92041037c0 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47432/0 (socket says 172.21.15.153:47432) 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 -- 172.21.15.153:0/3598761247 learned_addr learned my addr 172.21.15.153:0/3598761247 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f92092b3700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 0x7f92041140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 -- 172.21.15.153:0/3598761247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 msgr2=0x7f92041140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 0x7f92041140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 -- 172.21.15.153:0/3598761247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200005040 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f9209ab4700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9200009150 tx=0x7f9200009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920000e040 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9200013420 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920001c610 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920410dd90 con 0x7f9204102e20 2024-04-05T02:58:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.475+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920410e350 con 0x7f9204102e20 2024-04-05T02:58:33.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.476+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9204100130 con 0x7f9204102e20 2024-04-05T02:58:33.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.477+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f920000baf0 con 0x7f9204102e20 2024-04-05T02:58:33.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.477+0000 7f91fa7fc700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 0x7f91f006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:33.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.477+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9200087f90 con 0x7f9204102e20 2024-04-05T02:58:33.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.477+0000 7f92092b3700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 0x7f91f006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:33.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.477+0000 7f92092b3700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 0x7f91f006b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f920410ecd0 tx=0x7f91f400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:33.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.481+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9200058130 con 0x7f9204102e20 2024-04-05T02:58:33.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.666+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f920410f1e0 con 0x7f9204102e20 2024-04-05T02:58:33.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.667+0000 7f91fa7fc700 1 -- 172.21.15.153:0/3598761247 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9200057cc0 con 0x7f9204102e20 2024-04-05T02:58:33.667 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 msgr2=0x7f91f006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 0x7f91f006b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f920410ecd0 tx=0x7f91f400b040 comp rx=0 tx=0).stop 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 msgr2=0x7f9204113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9200009150 tx=0x7f9200009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 shutdown_connections 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f91f00692b0 0x7f91f006b760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9204102e20 0x7f9204113b90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 --2- 172.21.15.153:0/3598761247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92041037c0 0x7f92041140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 >> 172.21.15.153:0/3598761247 conn(0x7f92040f89d0 msgr2=0x7f9204107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:33.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 shutdown_connections 2024-04-05T02:58:33.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:33.669+0000 7f920bd18700 1 -- 172.21.15.153:0/3598761247 wait complete. 2024-04-05T02:58:34.338 INFO:tasks.cephadm:Setting up client nodes... 2024-04-05T02:58:34.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-05T02:58:34.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:34 smithi153 ceph-mon[29381]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:34.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:34 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3598761247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T02:58:34.595 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:34 smithi203 ceph-mon[39911]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:34 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3598761247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T02:58:36.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 -- 172.21.15.153:0/3442894010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 msgr2=0x7f44580ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 --2- 172.21.15.153:0/3442894010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f44580ffa70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4448004940 tx=0x7f4448009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 -- 172.21.15.153:0/3442894010 shutdown_connections 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 --2- 172.21.15.153:0/3442894010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 0x7f4458102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 --2- 172.21.15.153:0/3442894010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f44580ffa70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 -- 172.21.15.153:0/3442894010 >> 172.21.15.153:0/3442894010 conn(0x7f44580f7480 msgr2=0x7f44580f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 -- 172.21.15.153:0/3442894010 shutdown_connections 2024-04-05T02:58:36.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.094+0000 7f445d402700 1 -- 172.21.15.153:0/3442894010 wait complete. 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.095+0000 7f445d402700 1 Processor -- start 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.095+0000 7f445d402700 1 -- start start 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.095+0000 7f445d402700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.095+0000 7f445d402700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 0x7f445810def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f445d402700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810e580 con 0x7f4458066ab0 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f445d402700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4458112ed0 con 0x7f44580fffb0 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47456/0 (socket says 172.21.15.153:47456) 2024-04-05T02:58:36.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 -- 172.21.15.153:0/2466314987 learned_addr learned my addr 172.21.15.153:0/2466314987 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f44567fc700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 0x7f445810def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 -- 172.21.15.153:0/2466314987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 msgr2=0x7f445810def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 0x7f445810def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 -- 172.21.15.153:0/2466314987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448005040 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f4456ffd700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f444800b3b0 tx=0x7f4448009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444800e040 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581130d0 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f444800bbf0 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f444801c630 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.096+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4458113640 con 0x7f4458066ab0 2024-04-05T02:58:36.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.097+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445804c710 con 0x7f4458066ab0 2024-04-05T02:58:36.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.098+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f444800bd60 con 0x7f4458066ab0 2024-04-05T02:58:36.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.098+0000 7f443f7fe700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 0x7f444006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:36.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.098+0000 7f44567fc700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 0x7f444006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:36.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.098+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4448087b50 con 0x7f4458066ab0 2024-04-05T02:58:36.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.098+0000 7f44567fc700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 0x7f444006b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f444c009bf0 tx=0x7f444c0091a0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:36.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.102+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4448057bf0 con 0x7f4458066ab0 2024-04-05T02:58:36.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.366+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4458105980 con 0x7f4458066ab0 2024-04-05T02:58:36.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.369+0000 7f443f7fe700 1 -- 172.21.15.153:0/2466314987 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4448057780 con 0x7f4458066ab0 2024-04-05T02:58:36.369 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-04-05T02:58:36.369 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDcaA9mL/zmFRAAOPQpYUwnHawlooKEApVFVg== 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 msgr2=0x7f444006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 0x7f444006b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f444c009bf0 tx=0x7f444c0091a0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 msgr2=0x7f445810d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f444800b3b0 tx=0x7f4448009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 shutdown_connections 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f44400692a0 0x7f444006b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4458066ab0 0x7f445810d9b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 --2- 172.21.15.153:0/2466314987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44580fffb0 0x7f445810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.371+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 >> 172.21.15.153:0/2466314987 conn(0x7f44580f7480 msgr2=0x7f44580f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.372+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 shutdown_connections 2024-04-05T02:58:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:36.372+0000 7f445d402700 1 -- 172.21.15.153:0/2466314987 wait complete. 2024-04-05T02:58:36.690 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:36 smithi153 ceph-mon[29381]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:36.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:36 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2466314987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:36.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:36 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2466314987' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T02:58:36.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:36 smithi203 ceph-mon[39911]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:36 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2466314987' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:36 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2466314987' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T02:58:36.958 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T02:58:36.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-05T02:58:36.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-05T02:58:37.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-05T02:58:37.268 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi203/config 2024-04-05T02:58:38.606 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:38 smithi203 ceph-mon[39911]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:38 smithi153 ceph-mon[29381]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:38.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 -- 172.21.15.203:0/1981651326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fce80 msgr2=0x7f84500fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:38.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 --2- 172.21.15.203:0/1981651326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fce80 0x7f84500fd290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8440009ce0 tx=0x7f8440004d10 comp rx=0 tx=0).stop 2024-04-05T02:58:38.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 -- 172.21.15.203:0/1981651326 shutdown_connections 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 --2- 172.21.15.203:0/1981651326 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fdff0 0x7f84500fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 --2- 172.21.15.203:0/1981651326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fce80 0x7f84500fd290 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 -- 172.21.15.203:0/1981651326 >> 172.21.15.203:0/1981651326 conn(0x7f84500f8540 msgr2=0x7f84500fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 -- 172.21.15.203:0/1981651326 shutdown_connections 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.863+0000 7f8456155700 1 -- 172.21.15.203:0/1981651326 wait complete. 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 Processor -- start 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 -- start start 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 0x7f8450071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450072260 con 0x7f84500fce80 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f8456155700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450070850 con 0x7f84500fdff0 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f844effd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.864+0000 7f844effd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:59342/0 (socket says 172.21.15.203:59342) 2024-04-05T02:58:38.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844effd700 1 -- 172.21.15.203:0/749012848 learned_addr learned my addr 172.21.15.203:0/749012848 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844f7fe700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 0x7f8450071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844effd700 1 -- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 msgr2=0x7f8450071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844effd700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 0x7f8450071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844effd700 1 -- 172.21.15.203:0/749012848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440009940 con 0x7f84500fdff0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844f7fe700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 0x7f8450071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f844effd700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8444002a50 tx=0x7f84440046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8444007cc0 con 0x7f84500fdff0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f844400f040 con 0x7f84500fdff0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8444009640 con 0x7f84500fdff0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8450070b00 con 0x7f84500fdff0 2024-04-05T02:58:38.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.865+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84500710c0 con 0x7f84500fdff0 2024-04-05T02:58:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.866+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845004c710 con 0x7f84500fdff0 2024-04-05T02:58:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.867+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f84440097a0 con 0x7f84500fdff0 2024-04-05T02:58:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.867+0000 7f8437fff700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 0x7f843806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.867+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8444086b70 con 0x7f84500fdff0 2024-04-05T02:58:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.867+0000 7f844f7fe700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 0x7f843806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:38.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.868+0000 7f844f7fe700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 0x7f843806b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f84400095a0 tx=0x7f8440004d10 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:38.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:38.871+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8444056c70 con 0x7f84500fdff0 2024-04-05T02:58:39.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.129+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f84501058e0 con 0x7f84500fdff0 2024-04-05T02:58:39.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.135+0000 7f8437fff700 1 -- 172.21.15.203:0/749012848 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8444056800 con 0x7f84500fdff0 2024-04-05T02:58:39.135 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-04-05T02:58:39.135 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDfaA9mymLaBxAAKKi9gz/kzNuOEUw2fIkteg== 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 msgr2=0x7f843806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 0x7f843806b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f84400095a0 tx=0x7f8440004d10 comp rx=0 tx=0).stop 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 msgr2=0x7f8450070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8444002a50 tx=0x7f84440046b0 comp rx=0 tx=0).stop 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 shutdown_connections 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f84380692a0 0x7f843806b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84500fce80 0x7f8450071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 --2- 172.21.15.203:0/749012848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84500fdff0 0x7f8450070310 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 >> 172.21.15.203:0/749012848 conn(0x7f84500f8540 msgr2=0x7f84500f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:39.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 shutdown_connections 2024-04-05T02:58:39.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T02:58:39.137+0000 7f8456155700 1 -- 172.21.15.203:0/749012848 wait complete. 2024-04-05T02:58:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:39 smithi203 ceph-mon[39911]: from='client.? 172.21.15.203:0/749012848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:39 smithi203 ceph-mon[39911]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:39 smithi203 ceph-mon[39911]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T02:58:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:39 smithi153 ceph-mon[29381]: from='client.? 172.21.15.203:0/749012848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:39 smithi153 ceph-mon[29381]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T02:58:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:39 smithi153 ceph-mon[29381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T02:58:39.927 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T02:58:39.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-05T02:58:39.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-05T02:58:39.970 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-05T02:58:39.970 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-05T02:58:39.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-05T02:58:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:40.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:40 smithi153 ceph-mon[29381]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:40 smithi203 ceph-mon[39911]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.774+0000 7fed22233700 1 -- 172.21.15.153:0/1878309731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070c10 msgr2=0x7fed1c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.774+0000 7fed22233700 1 --2- 172.21.15.153:0/1878309731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070c10 0x7fed1c109170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fed04004970 tx=0x7fed04009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- 172.21.15.153:0/1878309731 shutdown_connections 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 --2- 172.21.15.153:0/1878309731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070c10 0x7fed1c109170 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 --2- 172.21.15.153:0/1878309731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070300 0x7fed1c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:41.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- 172.21.15.153:0/1878309731 >> 172.21.15.153:0/1878309731 conn(0x7fed1c0f89b0 msgr2=0x7fed1c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- 172.21.15.153:0/1878309731 shutdown_connections 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- 172.21.15.153:0/1878309731 wait complete. 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 Processor -- start 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- start start 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070c10 0x7fed1c1160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c116740 con 0x7fed1c070300 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.775+0000 7fed22233700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c10fd20 con 0x7fed1c070c10 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55602/0 (socket says 172.21.15.153:55602) 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 -- 172.21.15.153:0/2544948379 learned_addr learned my addr 172.21.15.153:0/2544948379 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed13fff700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070c10 0x7fed1c1160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 -- 172.21.15.153:0/2544948379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070c10 msgr2=0x7fed1c1160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:41.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070c10 0x7fed1c1160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 -- 172.21.15.153:0/2544948379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04005040 con 0x7fed1c070300 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed1b7fe700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fed0c002a50 tx=0x7fed0c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed0c007cc0 con 0x7fed1c070300 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed0c00f040 con 0x7fed1c070300 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c110000 con 0x7fed1c070300 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed0c00db70 con 0x7fed1c070300 2024-04-05T02:58:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.776+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c110510 con 0x7fed1c070300 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.777+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c071a40 con 0x7fed1c070300 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.778+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fed0c01c400 con 0x7fed1c070300 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.778+0000 7fed18ff9700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 0x7fed0806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.778+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fed0c086550 con 0x7fed1c070300 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.778+0000 7fed13fff700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 0x7fed0806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.778+0000 7fed13fff700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 0x7fed0806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed04004f50 tx=0x7fed04009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:41.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:41.782+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fed0c056650 con 0x7fed1c070300 2024-04-05T02:58:42.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.018+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fed1c04c710 con 0x7fed1c070300 2024-04-05T02:58:42.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.021+0000 7fed18ff9700 1 -- 172.21.15.153:0/2544948379 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167173 (secure 0 0 0) 0x7fed0c0561e0 con 0x7fed1c070300 2024-04-05T02:58:42.025 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 msgr2=0x7fed0806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 0x7fed0806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed04004f50 tx=0x7fed04009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 msgr2=0x7fed1c115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fed0c002a50 tx=0x7fed0c0046b0 comp rx=0 tx=0).stop 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 shutdown_connections 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fed080692a0 0x7fed0806b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:42.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c070300 0x7fed1c115bb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:42.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 --2- 172.21.15.153:0/2544948379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1c070c10 0x7fed1c1160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:42.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 >> 172.21.15.153:0/2544948379 conn(0x7fed1c0f89b0 msgr2=0x7fed1c0f9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:42.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 shutdown_connections 2024-04-05T02:58:42.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:42.024+0000 7fed22233700 1 -- 172.21.15.153:0/2544948379 wait complete. 2024-04-05T02:58:42.655 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:42 smithi153 ceph-mon[29381]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:42.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:42 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2544948379' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T02:58:42.679 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":19,"active_gid":14254,"active_name":"smithi153.npzyqy","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":663556540},{"type":"v1","addr":"172.21.15.153:6801","nonce":663556540}]},"active_addr":"172.21.15.153:6801/663556540","active_change":"2024-04-05T02:55:13.379708+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14272,"name":"smithi203.iakodz","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.153:8443/","prometheus":"http://172.21.15.153:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":220940347}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3393095164}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1792434239}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":971996676}]}]} 2024-04-05T02:58:42.683 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-05T02:58:42.683 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-05T02:58:42.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T02:58:42.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:42 smithi203 ceph-mon[39911]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:42.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:42 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2544948379' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T02:58:42.938 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:43.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:43 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:43 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:44.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 -- 172.21.15.153:0/4153628469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 msgr2=0x7f2234102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 --2- 172.21.15.153:0/4153628469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f2234102140 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2224004940 tx=0x7f2224009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 -- 172.21.15.153:0/4153628469 shutdown_connections 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 --2- 172.21.15.153:0/4153628469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 0x7f22340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 --2- 172.21.15.153:0/4153628469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f2234102140 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 -- 172.21.15.153:0/4153628469 >> 172.21.15.153:0/4153628469 conn(0x7f2234072bc0 msgr2=0x7f2234072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 -- 172.21.15.153:0/4153628469 shutdown_connections 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 -- 172.21.15.153:0/4153628469 wait complete. 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.497+0000 7f223a37d700 1 Processor -- start 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f223a37d700 1 -- start start 2024-04-05T02:58:44.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f223a37d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f223a37d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 0x7f223410f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f223a37d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223410f700 con 0x7f2234101d70 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f223a37d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223410f840 con 0x7f2234102710 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55634/0 (socket says 172.21.15.153:55634) 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 -- 172.21.15.153:0/303637364 learned_addr learned my addr 172.21.15.153:0/303637364 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f22337fe700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 0x7f223410f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 -- 172.21.15.153:0/303637364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 msgr2=0x7f223410f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 0x7f223410f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.498+0000 7f2233fff700 1 -- 172.21.15.153:0/303637364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224005040 con 0x7f2234101d70 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f2233fff700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f222400b8b0 tx=0x7f2224009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222400e040 con 0x7f2234101d70 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2234117b30 con 0x7f2234101d70 2024-04-05T02:58:44.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2224004290 con 0x7f2234101d70 2024-04-05T02:58:44.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22341180c0 con 0x7f2234101d70 2024-04-05T02:58:44.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.499+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222401c3f0 con 0x7f2234101d70 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.500+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340ff080 con 0x7f2234101d70 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.500+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2224013420 con 0x7f2234101d70 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.501+0000 7f2230ff9700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 0x7f221c06fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.501+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2224087f50 con 0x7f2234101d70 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.501+0000 7f22337fe700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 0x7f221c06fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:44.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.501+0000 7f22337fe700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 0x7f221c06fb60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2234066020 tx=0x7f2228009040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:44.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.505+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f22240ba190 con 0x7f2234101d70 2024-04-05T02:58:44.547 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:44 smithi153 ceph-mon[29381]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.701+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f223404bed0 con 0x7f2234101d70 2024-04-05T02:58:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.701+0000 7f2230ff9700 1 -- 172.21.15.153:0/303637364 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11108 (secure 0 0 0) 0x7f2224054710 con 0x7f2234101d70 2024-04-05T02:58:44.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:58:44.703 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T02:58:29.264740+0000","last_up_change":"2024-04-05T02:58:28.256825+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6803","nonce":78196267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6805","nonce":78196267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6809","nonce":78196267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6807","nonce":78196267}]},"public_addr":"172.21.15.153:6803/78196267","cluster_addr":"172.21.15.153:6805/78196267","heartbeat_back_addr":"172.21.15.153:6809/78196267","heartbeat_front_addr":"172.21.15.153:6807/78196267","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6811","nonce":4220639279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6813","nonce":4220639279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6817","nonce":4220639279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6815","nonce":4220639279}]},"public_addr":"172.21.15.153:6811/4220639279","cluster_addr":"172.21.15.153:6813/4220639279","heartbeat_back_addr":"172.21.15.153:6817/4220639279","heartbeat_front_addr":"172.21.15.153:6815/4220639279","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6819","nonce":1027037149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6821","nonce":1027037149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6825","nonce":1027037149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6823","nonce":1027037149}]},"public_addr":"172.21.15.153:6819/1027037149","cluster_addr":"172.21.15.153:6821/1027037149","heartbeat_back_addr":"172.21.15.153:6825/1027037149","heartbeat_front_addr":"172.21.15.153:6823/1027037149","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6801","nonce":4064942773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6803","nonce":4064942773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6807","nonce":4064942773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6805","nonce":4064942773}]},"public_addr":"172.21.15.203:6801/4064942773","cluster_addr":"172.21.15.203:6803/4064942773","heartbeat_back_addr":"172.21.15.203:6807/4064942773","heartbeat_front_addr":"172.21.15.203:6805/4064942773","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6809","nonce":519188752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6811","nonce":519188752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6815","nonce":519188752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6813","nonce":519188752}]},"public_addr":"172.21.15.203:6809/519188752","cluster_addr":"172.21.15.203:6811/519188752","heartbeat_back_addr":"172.21.15.203:6815/519188752","heartbeat_front_addr":"172.21.15.203:6813/519188752","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6817","nonce":2086848603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6819","nonce":2086848603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6823","nonce":2086848603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6821","nonce":2086848603}]},"public_addr":"172.21.15.203:6817/2086848603","cluster_addr":"172.21.15.203:6819/2086848603","heartbeat_back_addr":"172.21.15.203:6823/2086848603","heartbeat_front_addr":"172.21.15.203:6821/2086848603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T02:58:44.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 msgr2=0x7f221c06fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 0x7f221c06fb60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2234066020 tx=0x7f2228009040 comp rx=0 tx=0).stop 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 msgr2=0x7f223410ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f222400b8b0 tx=0x7f2224009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 shutdown_connections 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f221c06d6b0 0x7f221c06fb60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2234101d70 0x7f223410ec80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 --2- 172.21.15.153:0/303637364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2234102710 0x7f223410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 >> 172.21.15.153:0/303637364 conn(0x7f2234072bc0 msgr2=0x7f22340fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 shutdown_connections 2024-04-05T02:58:44.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:44.705+0000 7f223a37d700 1 -- 172.21.15.153:0/303637364 wait complete. 2024-04-05T02:58:44.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:44 smithi203 ceph-mon[39911]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:45.352 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-05T02:58:45.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T02:58:45.617 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:45.738 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:45 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/303637364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:45 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/303637364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:46.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:46 smithi153 ceph-mon[29381]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:46 smithi203 ceph-mon[39911]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 -- 172.21.15.153:0/3678192615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 msgr2=0x7f12241031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 --2- 172.21.15.153:0/3678192615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f12241031b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1214004940 tx=0x7f1214009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 -- 172.21.15.153:0/3678192615 shutdown_connections 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 --2- 172.21.15.153:0/3678192615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12240fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 --2- 172.21.15.153:0/3678192615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f12241031b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 -- 172.21.15.153:0/3678192615 >> 172.21.15.153:0/3678192615 conn(0x7f12240f8970 msgr2=0x7f12240fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 -- 172.21.15.153:0/3678192615 shutdown_connections 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.335+0000 7f122963c700 1 -- 172.21.15.153:0/3678192615 wait complete. 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 Processor -- start 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 -- start start 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12241162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1224116900 con 0x7f1224102de0 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f122963c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122410fd20 con 0x7f1224103780 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f1222ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.336+0000 7f1222ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55638/0 (socket says 172.21.15.153:55638) 2024-04-05T02:58:47.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f1222ffd700 1 -- 172.21.15.153:0/1852809492 learned_addr learned my addr 172.21.15.153:0/1852809492 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f12227fc700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12241162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f1222ffd700 1 -- 172.21.15.153:0/1852809492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 msgr2=0x7f12241162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f1222ffd700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12241162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f1222ffd700 1 -- 172.21.15.153:0/1852809492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214005040 con 0x7f1224102de0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f12227fc700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12241162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f1222ffd700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1214009150 tx=0x7f1214009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121400e040 con 0x7f1224102de0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1214013420 con 0x7f1224102de0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121401c630 con 0x7f1224102de0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122410ffa0 con 0x7f1224102de0 2024-04-05T02:58:47.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.337+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1224110560 con 0x7f1224102de0 2024-04-05T02:58:47.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.338+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1224106a80 con 0x7f1224102de0 2024-04-05T02:58:47.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.339+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1214013590 con 0x7f1224102de0 2024-04-05T02:58:47.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.339+0000 7f120b7fe700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 0x7f120c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:47.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.339+0000 7f12227fc700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 0x7f120c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:47.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.339+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1214087fe0 con 0x7f1224102de0 2024-04-05T02:58:47.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.340+0000 7f12227fc700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 0x7f120c06b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1224111610 tx=0x7f1218009040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:47.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.344+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1214058180 con 0x7f1224102de0 2024-04-05T02:58:47.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.531+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12240fe530 con 0x7f1224102de0 2024-04-05T02:58:47.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.532+0000 7f120b7fe700 1 -- 172.21.15.153:0/1852809492 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11108 (secure 0 0 0) 0x7f1214057d10 con 0x7f1224102de0 2024-04-05T02:58:47.532 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:58:47.532 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T02:58:29.264740+0000","last_up_change":"2024-04-05T02:58:28.256825+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6803","nonce":78196267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6805","nonce":78196267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6809","nonce":78196267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6807","nonce":78196267}]},"public_addr":"172.21.15.153:6803/78196267","cluster_addr":"172.21.15.153:6805/78196267","heartbeat_back_addr":"172.21.15.153:6809/78196267","heartbeat_front_addr":"172.21.15.153:6807/78196267","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6811","nonce":4220639279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6813","nonce":4220639279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6817","nonce":4220639279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6815","nonce":4220639279}]},"public_addr":"172.21.15.153:6811/4220639279","cluster_addr":"172.21.15.153:6813/4220639279","heartbeat_back_addr":"172.21.15.153:6817/4220639279","heartbeat_front_addr":"172.21.15.153:6815/4220639279","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6819","nonce":1027037149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6821","nonce":1027037149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6825","nonce":1027037149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6823","nonce":1027037149}]},"public_addr":"172.21.15.153:6819/1027037149","cluster_addr":"172.21.15.153:6821/1027037149","heartbeat_back_addr":"172.21.15.153:6825/1027037149","heartbeat_front_addr":"172.21.15.153:6823/1027037149","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6801","nonce":4064942773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6803","nonce":4064942773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6807","nonce":4064942773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6805","nonce":4064942773}]},"public_addr":"172.21.15.203:6801/4064942773","cluster_addr":"172.21.15.203:6803/4064942773","heartbeat_back_addr":"172.21.15.203:6807/4064942773","heartbeat_front_addr":"172.21.15.203:6805/4064942773","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6809","nonce":519188752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6811","nonce":519188752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6815","nonce":519188752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6813","nonce":519188752}]},"public_addr":"172.21.15.203:6809/519188752","cluster_addr":"172.21.15.203:6811/519188752","heartbeat_back_addr":"172.21.15.203:6815/519188752","heartbeat_front_addr":"172.21.15.203:6813/519188752","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6817","nonce":2086848603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6819","nonce":2086848603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6823","nonce":2086848603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6821","nonce":2086848603}]},"public_addr":"172.21.15.203:6817/2086848603","cluster_addr":"172.21.15.203:6819/2086848603","heartbeat_back_addr":"172.21.15.203:6823/2086848603","heartbeat_front_addr":"172.21.15.203:6821/2086848603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.534+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 msgr2=0x7f120c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.534+0000 7f122963c700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 0x7f120c06b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1224111610 tx=0x7f1218009040 comp rx=0 tx=0).stop 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.534+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 msgr2=0x7f1224115d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1214009150 tx=0x7f1214009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 shutdown_connections 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f120c0692b0 0x7f120c06b760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1224102de0 0x7f1224115d70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 --2- 172.21.15.153:0/1852809492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224103780 0x7f12241162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 >> 172.21.15.153:0/1852809492 conn(0x7f12240f8970 msgr2=0x7f12241073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 shutdown_connections 2024-04-05T02:58:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:47.535+0000 7f122963c700 1 -- 172.21.15.153:0/1852809492 wait complete. 2024-04-05T02:58:48.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-05T02:58:48.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-05T02:58:48.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-05T02:58:48.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-05T02:58:48.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-05T02:58:48.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-05T02:58:48.497 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:48 smithi153 ceph-mon[29381]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:48.498 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:48 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1852809492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:48.531 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:48 smithi203 ceph-mon[39911]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:48 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1852809492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T02:58:49.890 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:49.890 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:49.892 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:49.892 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:50.652 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:50 smithi153 ceph-mon[29381]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:50 smithi203 ceph-mon[39911]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:52.793 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:52 smithi153 ceph-mon[29381]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:52 smithi203 ceph-mon[39911]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:54 smithi203 ceph-mon[39911]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:54 smithi153 ceph-mon[29381]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 -- 172.21.15.153:0/240058383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070c10 msgr2=0x7f2504109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 --2- 172.21.15.153:0/240058383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070c10 0x7f2504109180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24f4004970 tx=0x7f24f4009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 -- 172.21.15.153:0/240058383 shutdown_connections 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 --2- 172.21.15.153:0/240058383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070c10 0x7f2504109180 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 --2- 172.21.15.153:0/240058383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070300 0x7f25040706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 -- 172.21.15.153:0/240058383 >> 172.21.15.153:0/240058383 conn(0x7f25040f89a0 msgr2=0x7f25040fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 -- 172.21.15.153:0/240058383 shutdown_connections 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.405+0000 7f250b291700 1 -- 172.21.15.153:0/240058383 wait complete. 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 Processor -- start 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 -- start start 2024-04-05T02:58:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070c10 0x7f2504116340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504116990 con 0x7f2504070300 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250b291700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250410fd60 con 0x7f2504070c10 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60014/0 (socket says 172.21.15.153:60014) 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 -- 172.21.15.153:0/2692642089 learned_addr learned my addr 172.21.15.153:0/2692642089 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250882c700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070c10 0x7f2504116340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 -- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070c10 msgr2=0x7f2504116340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070c10 0x7f2504116340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f4005040 con 0x7f2504070300 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f250902d700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24fc001d80 tx=0x7f24fc009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24fc007d10 con 0x7f2504070300 2024-04-05T02:58:55.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.406+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24fc007e70 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.407+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2504110040 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.407+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24fc014640 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.407+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2504110550 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.408+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2504110a10 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.408+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f24fc00e960 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f24f9ffb700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 0x7f24f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f24fc086c30 con 0x7f2504070300 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f250882c700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 0x7f24f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f24f9ffb700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 0x7f24f0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24f0071a30 con 0x7f24f006ef70 2024-04-05T02:58:55.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24fc08c290 con 0x7f2504070300 2024-04-05T02:58:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f250882c700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 0x7f24f006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f24f4004f50 tx=0x7f24f4009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.409+0000 7f250982e700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 0x7f24f0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.410+0000 7f250982e700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 0x7f24f0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:55.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.410+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== osd.3 v2:172.21.15.203:6800/4064942773 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f24f0071a30 con 0x7f24f006ef70 2024-04-05T02:58:55.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.425+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 --> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2504110b80 con 0x7f24f006ef70 2024-04-05T02:58:55.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f24f9ffb700 1 -- 172.21.15.153:0/2692642089 <== osd.3 v2:172.21.15.203:6800/4064942773 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2504110b80 con 0x7f24f006ef70 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 msgr2=0x7f24f0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 0x7f24f0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 msgr2=0x7f24f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 0x7f24f006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f24f4004f50 tx=0x7f24f4009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 msgr2=0x7f2504115e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f24fc001d80 tx=0x7f24fc009ec0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 shutdown_connections 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f24f00692a0 0x7f24f006b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2504070300 0x7f2504115e00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:6800/4064942773,v1:172.21.15.203:6801/4064942773] conn(0x7f24f006ef70 0x7f24f0071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 --2- 172.21.15.153:0/2692642089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2504070c10 0x7f2504116340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:55.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 >> 172.21.15.153:0/2692642089 conn(0x7f25040f89a0 msgr2=0x7f25040f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:55.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 shutdown_connections 2024-04-05T02:58:55.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:55.426+0000 7f250b291700 1 -- 172.21.15.153:0/2692642089 wait complete. 2024-04-05T02:58:56.365 INFO:teuthology.orchestra.run.smithi153.stdout:111669149714 2024-04-05T02:58:56.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-05T02:58:56.625 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:56 smithi153 ceph-mon[29381]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:56 smithi203 ceph-mon[39911]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:56.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.955+0000 7f9847d39700 1 -- 172.21.15.153:0/1355785254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98400fb3b0 msgr2=0x7f98400fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:56.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.955+0000 7f9847d39700 1 --2- 172.21.15.153:0/1355785254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98400fb3b0 0x7f98400fb820 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9830004940 tx=0x7f9830009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:56.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- 172.21.15.153:0/1355785254 shutdown_connections 2024-04-05T02:58:56.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 --2- 172.21.15.153:0/1355785254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98400fb3b0 0x7f98400fb820 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 --2- 172.21.15.153:0/1355785254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9840102f70 0x7f9840103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- 172.21.15.153:0/1355785254 >> 172.21.15.153:0/1355785254 conn(0x7f9840072310 msgr2=0x7f9840072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- 172.21.15.153:0/1355785254 shutdown_connections 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- 172.21.15.153:0/1355785254 wait complete. 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 Processor -- start 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- start start 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98400fb3b0 0x7f98400fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400fe4e0 con 0x7f9840102f70 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.956+0000 7f9847d39700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400fd010 con 0x7f98400fb3b0 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60056/0 (socket says 172.21.15.153:60056) 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 -- 172.21.15.153:0/1220555592 learned_addr learned my addr 172.21.15.153:0/1220555592 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f9845ad5700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98400fb3b0 0x7f98400fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:56.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 -- 172.21.15.153:0/1220555592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98400fb3b0 msgr2=0x7f98400fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98400fb3b0 0x7f98400fc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830005040 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98452d4700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f983000b3b0 tx=0x7f9830009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f983000e040 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98400fd290 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9830013420 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f983001c630 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.957+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984011a0b0 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.958+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f984005e6c0 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9830013590 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 0x7f982c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9830088aa0 con 0x7f9840102f70 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f9845ad5700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 0x7f982c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 0x7f982c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:56.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f982c071a30 con 0x7f982c06ef70 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9830088e80 con 0x7f9840102f70 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f98462d6700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 0x7f982c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.959+0000 7f9845ad5700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 0x7f982c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f983c0041d0 tx=0x7f983c0043c0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.960+0000 7f98462d6700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 0x7f982c071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.960+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== osd.1 v2:172.21.15.153:6810/4220639279 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f982c071a30 con 0x7f982c06ef70 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 --> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f984005eb40 con 0x7f982c06ef70 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f98367fc700 1 -- 172.21.15.153:0/1220555592 <== osd.1 v2:172.21.15.153:6810/4220639279 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f984005eb40 con 0x7f982c06ef70 2024-04-05T02:58:56.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 msgr2=0x7f982c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 0x7f982c071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 msgr2=0x7f982c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 0x7f982c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f983c0041d0 tx=0x7f983c0043c0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 msgr2=0x7f98400fcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f983000b3b0 tx=0x7f9830009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 shutdown_connections 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f982c0692a0 0x7f982c06b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:6810/4220639279,v1:172.21.15.153:6811/4220639279] conn(0x7f982c06ef70 0x7f982c071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98400fb3b0 0x7f98400fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 --2- 172.21.15.153:0/1220555592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9840102f70 0x7f98400fcad0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 >> 172.21.15.153:0/1220555592 conn(0x7f9840072310 msgr2=0x7f98400f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 shutdown_connections 2024-04-05T02:58:56.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.975+0000 7f9847d39700 1 -- 172.21.15.153:0/1220555592 wait complete. 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- 172.21.15.153:0/304890443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc1026b0 msgr2=0x7f85bc0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 --2- 172.21.15.153:0/304890443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc1026b0 0x7f85bc0fbd20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f85b0004940 tx=0x7f85b0009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- 172.21.15.153:0/304890443 shutdown_connections 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 --2- 172.21.15.153:0/304890443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc1026b0 0x7f85bc0fbd20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 --2- 172.21.15.153:0/304890443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc101d10 0x7f85bc1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- 172.21.15.153:0/304890443 >> 172.21.15.153:0/304890443 conn(0x7f85bc0f7860 msgr2=0x7f85bc0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- 172.21.15.153:0/304890443 shutdown_connections 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- 172.21.15.153:0/304890443 wait complete. 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 Processor -- start 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.862+0000 7f85c1495700 1 -- start start 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc1026b0 0x7f85bc10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc067100 con 0x7f85bc101d10 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10d340 con 0x7f85bc1026b0 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60042/0 (socket says 172.21.15.153:60042) 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 -- 172.21.15.153:0/1372617448 learned_addr learned my addr 172.21.15.153:0/1372617448 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:57.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85ba7fc700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc1026b0 0x7f85bc10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 -- 172.21.15.153:0/1372617448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc1026b0 msgr2=0x7f85bc10ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc1026b0 0x7f85bc10ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0005040 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85baffd700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85ac002a50 tx=0x7f85ac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac007cc0 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85ac00f040 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc10d620 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac00da30 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.863+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc11a170 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.864+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85bc0f91f0 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f85ac00db90 con 0x7f85bc101d10 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 0x7f85a406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:57.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f85ac086c00 con 0x7f85bc101d10 2024-04-05T02:58:57.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85ba7fc700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 0x7f85a406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:57.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 0x7f85a4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:57.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85a4071a30 con 0x7f85a406ef70 2024-04-05T02:58:57.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.865+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85ac08c200 con 0x7f85bc101d10 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.866+0000 7f85bb7fe700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 0x7f85a4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.866+0000 7f85ba7fc700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 0x7f85a406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85bc066b10 tx=0x7f85b0009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.866+0000 7f85bb7fe700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 0x7f85a4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.867+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== osd.0 v2:172.21.15.153:6802/78196267 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f85a4071a30 con 0x7f85a406ef70 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.882+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 --> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f85bc0fd460 con 0x7f85a406ef70 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.882+0000 7f85a37fe700 1 -- 172.21.15.153:0/1372617448 <== osd.0 v2:172.21.15.153:6802/78196267 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f85bc0fd460 con 0x7f85a406ef70 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.882+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 msgr2=0x7f85a4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.882+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 0x7f85a4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 msgr2=0x7f85a406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 0x7f85a406b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f85bc066b10 tx=0x7f85b0009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 msgr2=0x7f85bc10c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85ac002a50 tx=0x7f85ac0046b0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 shutdown_connections 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6802/78196267,v1:172.21.15.153:6803/78196267] conn(0x7f85a406ef70 0x7f85a4071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f85a40692a0 0x7f85a406b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85bc101d10 0x7f85bc10c8c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 --2- 172.21.15.153:0/1372617448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85bc1026b0 0x7f85bc10ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 >> 172.21.15.153:0/1372617448 conn(0x7f85bc0f7860 msgr2=0x7f85bc0f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 shutdown_connections 2024-04-05T02:58:57.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:56.883+0000 7f85c1495700 1 -- 172.21.15.153:0/1372617448 wait complete. 2024-04-05T02:58:57.890 INFO:teuthology.orchestra.run.smithi153.stdout:60129542172 2024-04-05T02:58:57.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-05T02:58:58.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 -- 172.21.15.153:0/3533763033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 msgr2=0x7f99cc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 --2- 172.21.15.153:0/3533763033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc0fbd80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f99c0004940 tx=0x7f99c0009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 -- 172.21.15.153:0/3533763033 shutdown_connections 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 --2- 172.21.15.153:0/3533763033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc0fbd80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 --2- 172.21.15.153:0/3533763033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 0x7f99cc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 -- 172.21.15.153:0/3533763033 >> 172.21.15.153:0/3533763033 conn(0x7f99cc072bc0 msgr2=0x7f99cc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 -- 172.21.15.153:0/3533763033 shutdown_connections 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 -- 172.21.15.153:0/3533763033 wait complete. 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.136+0000 7f99d211d700 1 Processor -- start 2024-04-05T02:58:58.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99d211d700 1 -- start start 2024-04-05T02:58:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99d211d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 0x7f99cc113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99d211d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99d211d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc1146f0 con 0x7f99cc102710 2024-04-05T02:58:58.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99d211d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc10dae0 con 0x7f99cc101d70 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60084/0 (socket says 172.21.15.153:60084) 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 -- 172.21.15.153:0/45478726 learned_addr learned my addr 172.21.15.153:0/45478726 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99cb7fe700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 0x7f99cc113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 -- 172.21.15.153:0/45478726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 msgr2=0x7f99cc113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 0x7f99cc113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.137+0000 7f99caffd700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c0005040 con 0x7f99cc102710 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99caffd700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009100 tx=0x7f99c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c000e040 con 0x7f99cc102710 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc10dd60 con 0x7f99cc102710 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99c0004290 con 0x7f99cc102710 2024-04-05T02:58:58.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99c001c3f0 con 0x7f99cc102710 2024-04-05T02:58:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.138+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc10e320 con 0x7f99cc102710 2024-04-05T02:58:58.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.139+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99cc10e800 con 0x7f99cc102710 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.139+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f99c0013690 con 0x7f99cc102710 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99b3fff700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 0x7f99b406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f99c0088a10 con 0x7f99cc102710 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99cb7fe700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 0x7f99b406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99b3fff700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 0x7f99b4071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99b4071a40 con 0x7f99b406ef80 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99c0088df0 con 0x7f99cc102710 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.140+0000 7f99cbfff700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 0x7f99b4071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.141+0000 7f99cb7fe700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 0x7f99b406b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0041d0 tx=0x7f99bc0043c0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.141+0000 7f99cbfff700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 0x7f99b4071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.142+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== osd.2 v2:172.21.15.153:6818/1027037149 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f99b4071a40 con 0x7f99b406ef80 2024-04-05T02:58:58.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.157+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 --> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f99cc100e00 con 0x7f99b406ef80 2024-04-05T02:58:58.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.157+0000 7f99b3fff700 1 -- 172.21.15.153:0/45478726 <== osd.2 v2:172.21.15.153:6818/1027037149 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f99cc100e00 con 0x7f99b406ef80 2024-04-05T02:58:58.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.157+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 msgr2=0x7f99b4071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 0x7f99b4071390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 msgr2=0x7f99b406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 0x7f99b406b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0041d0 tx=0x7f99bc0043c0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 msgr2=0x7f99cc1140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009100 tx=0x7f99c0009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:58.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 shutdown_connections 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6818/1027037149,v1:172.21.15.153:6819/1027037149] conn(0x7f99b406ef80 0x7f99b4071390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f99b40692b0 0x7f99b406b760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99cc101d70 0x7f99cc113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 --2- 172.21.15.153:0/45478726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99cc102710 0x7f99cc1140a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 >> 172.21.15.153:0/45478726 conn(0x7f99cc072bc0 msgr2=0x7f99cc0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 shutdown_connections 2024-04-05T02:58:58.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.158+0000 7f99d211d700 1 -- 172.21.15.153:0/45478726 wait complete. 2024-04-05T02:58:58.684 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:58 smithi153 ceph-mon[29381]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:58.684 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:58:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:58 smithi203 ceph-mon[39911]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:58:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:58:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:58:58.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 -- 172.21.15.153:0/1815400732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c0702e0 msgr2=0x7f4f0c0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 --2- 172.21.15.153:0/1815400732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c0706b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4f00004940 tx=0x7f4f00009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 -- 172.21.15.153:0/1815400732 shutdown_connections 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 --2- 172.21.15.153:0/1815400732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 --2- 172.21.15.153:0/1815400732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c0706b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 -- 172.21.15.153:0/1815400732 >> 172.21.15.153:0/1815400732 conn(0x7f4f0c0f8980 msgr2=0x7f4f0c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 -- 172.21.15.153:0/1815400732 shutdown_connections 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.803+0000 7f4f11687700 1 -- 172.21.15.153:0/1815400732 wait complete. 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 Processor -- start 2024-04-05T02:58:58.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 -- start start 2024-04-05T02:58:58.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c1169f0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f11687700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c10fd30 con 0x7f4f0c0702e0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f0a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f0a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55392/0 (socket says 172.21.15.153:55392) 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f0a7fc700 1 -- 172.21.15.153:0/3086974803 learned_addr learned my addr 172.21.15.153:0/3086974803 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.804+0000 7f4f0affd700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f0a7fc700 1 -- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c0702e0 msgr2=0x7f4f0c115dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f0a7fc700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c115dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f0a7fc700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f00005040 con 0x7f4f0c070bf0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f0a7fc700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00b790 tx=0x7f4efc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4efc00f6f0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4efc00fd20 con 0x7f4f0c070bf0 2024-04-05T02:58:58.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4efc0103a0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c10ffb0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.805+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c1104a0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.806+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f0c110a30 con 0x7f4f0c070bf0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4efc011400 con 0x7f4f0c070bf0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 0x7f4ef406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4efc087600 con 0x7f4f0c070bf0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4f0affd700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 0x7f4ef406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 0x7f4ef4071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ef4071a40 con 0x7f4ef406ef80 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.807+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4efc0879e0 con 0x7f4f0c070bf0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.808+0000 7f4f0affd700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 0x7f4ef406b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000b3b0 tx=0x7f4f00009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.808+0000 7f4f0b7fe700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 0x7f4ef4071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.808+0000 7f4f0b7fe700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 0x7f4ef4071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:58:58.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.809+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== osd.4 v2:172.21.15.203:6808/519188752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4ef4071a40 con 0x7f4ef406ef80 2024-04-05T02:58:58.820 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:58:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.825+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 --> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f0c0fa730 con 0x7f4ef406ef80 2024-04-05T02:58:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4ef37fe700 1 -- 172.21.15.153:0/3086974803 <== osd.4 v2:172.21.15.203:6808/519188752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4f0c0fa730 con 0x7f4ef406ef80 2024-04-05T02:58:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 msgr2=0x7f4ef4071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 0x7f4ef4071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 msgr2=0x7f4ef406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 0x7f4ef406b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000b3b0 tx=0x7f4f00009e30 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 msgr2=0x7f4f0c116310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00b790 tx=0x7f4efc009250 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 shutdown_connections 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4ef40692b0 0x7f4ef406b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:6808/519188752,v1:172.21.15.203:6809/519188752] conn(0x7f4ef406ef80 0x7f4ef4071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f0c0702e0 0x7f4f0c115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 --2- 172.21.15.153:0/3086974803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f0c070bf0 0x7f4f0c116310 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 >> 172.21.15.153:0/3086974803 conn(0x7f4f0c0f8980 msgr2=0x7f4f0c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 shutdown_connections 2024-04-05T02:58:58.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:58:58.826+0000 7f4f11687700 1 -- 172.21.15.153:0/3086974803 wait complete. 2024-04-05T02:58:59.480 INFO:teuthology.orchestra.run.smithi153.stdout:146028888078 2024-04-05T02:58:59.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-05T02:58:59.895 INFO:teuthology.orchestra.run.smithi153.stdout:81604378647 2024-04-05T02:58:59.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-05T02:59:00.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:00 smithi203 ceph-mon[39911]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:00 smithi153 ceph-mon[29381]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:00.935 INFO:teuthology.orchestra.run.smithi153.stdout:38654705700 2024-04-05T02:59:00.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/3400169475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070300 msgr2=0x7f4e080706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/3400169475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070300 0x7f4e080706d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4e04004970 tx=0x7f4e04009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/3400169475 shutdown_connections 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/3400169475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070c10 0x7f4e08109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/3400169475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070300 0x7f4e080706d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/3400169475 >> 172.21.15.153:0/3400169475 conn(0x7f4e080f8980 msgr2=0x7f4e080fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:01.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/3400169475 shutdown_connections 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/3400169475 wait complete. 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 Processor -- start 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.150+0000 7f4e0f2f4700 1 -- start start 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0f2f4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070300 0x7f4e08115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0f2f4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0f2f4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e08116970 con 0x7f4e08070c10 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0f2f4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0810fd40 con 0x7f4e08070300 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55402/0 (socket says 172.21.15.153:55402) 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 -- 172.21.15.153:0/876766066 learned_addr learned my addr 172.21.15.153:0/876766066 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:01.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0d090700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070300 0x7f4e08115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 -- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070300 msgr2=0x7f4e08115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070300 0x7f4e08115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04005040 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0c88f700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4df800c8c0 tx=0x7f4df80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df800a980 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4df800ce60 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e08110020 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.151+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df80115e0 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.152+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e08110560 con 0x7f4e08070c10 2024-04-05T02:59:01.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.152+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e08110a20 con 0x7f4e08070c10 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.153+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4df8012400 con 0x7f4e08070c10 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4dfdffb700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 0x7f4df406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4e0d090700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 0x7f4df406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4df8087ba0 con 0x7f4e08070c10 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4dfdffb700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 0x7f4df4071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4df4071a40 con 0x7f4df406ef80 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4df8087ee0 con 0x7f4e08070c10 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4e0d090700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 0x7f4df406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009fd0 tx=0x7f4e04009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.154+0000 7f4e0d891700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 0x7f4df4071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.155+0000 7f4e0d891700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 0x7f4df4071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.156+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== osd.5 v2:172.21.15.203:6816/2086848603 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4df4071a40 con 0x7f4df406ef80 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.170+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 --> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e08110be0 con 0x7f4df406ef80 2024-04-05T02:59:01.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.171+0000 7f4dfdffb700 1 -- 172.21.15.153:0/876766066 <== osd.5 v2:172.21.15.203:6816/2086848603 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4e08110be0 con 0x7f4df406ef80 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.171+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 msgr2=0x7f4df4071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.171+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 0x7f4df4071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.171+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 msgr2=0x7f4df406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.171+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 0x7f4df406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009fd0 tx=0x7f4e04009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 msgr2=0x7f4e08116320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4df800c8c0 tx=0x7f4df80044c0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 shutdown_connections 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4df40692b0 0x7f4df406b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e08070300 0x7f4e08115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.203:6816/2086848603,v1:172.21.15.203:6817/2086848603] conn(0x7f4df406ef80 0x7f4df4071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 --2- 172.21.15.153:0/876766066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e08070c10 0x7f4e08116320 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:01.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 >> 172.21.15.153:0/876766066 conn(0x7f4e080f8980 msgr2=0x7f4e080f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:01.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 shutdown_connections 2024-04-05T02:59:01.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:01.172+0000 7f4e0f2f4700 1 -- 172.21.15.153:0/876766066 wait complete. 2024-04-05T02:59:01.716 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:02.295 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:02 smithi203 ceph-mon[39911]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:02 smithi153 ceph-mon[29381]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 -- 172.21.15.153:0/950561250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618102710 msgr2=0x7f26180fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 --2- 172.21.15.153:0/950561250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618102710 0x7f26180fbd80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2604004940 tx=0x7f2604009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 -- 172.21.15.153:0/950561250 shutdown_connections 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 --2- 172.21.15.153:0/950561250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618102710 0x7f26180fbd80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 --2- 172.21.15.153:0/950561250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618101d70 0x7f2618102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 -- 172.21.15.153:0/950561250 >> 172.21.15.153:0/950561250 conn(0x7f2618072bc0 msgr2=0x7f2618072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:03.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 -- 172.21.15.153:0/950561250 shutdown_connections 2024-04-05T02:59:03.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.332+0000 7f261ecd3700 1 -- 172.21.15.153:0/950561250 wait complete. 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 Processor -- start 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 -- start start 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618102710 0x7f26181140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2618114730 con 0x7f2618101d70 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ecd3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261810db40 con 0x7f2618102710 2024-04-05T02:59:03.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55424/0 (socket says 172.21.15.153:55424) 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 -- 172.21.15.153:0/83175477 learned_addr learned my addr 172.21.15.153:0/83175477 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f2617fff700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618102710 0x7f26181140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 -- 172.21.15.153:0/83175477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618102710 msgr2=0x7f26181140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618102710 0x7f26181140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.333+0000 7f261ca6f700 1 -- 172.21.15.153:0/83175477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2604005040 con 0x7f2618101d70 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f261ca6f700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f260c002a50 tx=0x7f260c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f260c007cc0 con 0x7f2618101d70 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f260c00f040 con 0x7f2618101d70 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261810de20 con 0x7f2618101d70 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f260c00db70 con 0x7f2618101d70 2024-04-05T02:59:03.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.334+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261810e3e0 con 0x7f2618101d70 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.335+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26180fd4c0 con 0x7f2618101d70 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.335+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f260c00dd10 con 0x7f2618101d70 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.336+0000 7f26157fa700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 0x7f260806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.336+0000 7f2617fff700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 0x7f260806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.336+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f260c0865c0 con 0x7f2618101d70 2024-04-05T02:59:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.336+0000 7f2617fff700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 0x7f260806b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f260400b3b0 tx=0x7f2604009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:03.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.340+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f260c052bb0 con 0x7f2618101d70 2024-04-05T02:59:03.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.536+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f261805f5f0 con 0x7f2618101d70 2024-04-05T02:59:03.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.536+0000 7f26157fa700 1 -- 172.21.15.153:0/83175477 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f260c0561d0 con 0x7f2618101d70 2024-04-05T02:59:03.536 INFO:teuthology.orchestra.run.smithi153.stdout:60129542174 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 msgr2=0x7f260806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 0x7f260806b760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f260400b3b0 tx=0x7f2604009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 msgr2=0x7f2618113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f260c002a50 tx=0x7f260c0046b0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 shutdown_connections 2024-04-05T02:59:03.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f26080692b0 0x7f260806b760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2618101d70 0x7f2618113ba0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 --2- 172.21.15.153:0/83175477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2618102710 0x7f26181140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:03.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 >> 172.21.15.153:0/83175477 conn(0x7f2618072bc0 msgr2=0x7f26180659d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:03.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 shutdown_connections 2024-04-05T02:59:03.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:03.538+0000 7f261ecd3700 1 -- 172.21.15.153:0/83175477 wait complete. 2024-04-05T02:59:04.336 INFO:teuthology.orchestra.run.smithi153.stdout:167503724554 2024-04-05T02:59:04.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-05T02:59:04.363 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:04.396 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:04.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:04 smithi153 ceph-mon[29381]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:04.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:04 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/83175477' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T02:59:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:04 smithi203 ceph-mon[39911]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:04 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/83175477' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T02:59:06.796 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:06 smithi153 ceph-mon[29381]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:06 smithi203 ceph-mon[39911]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:06.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- 172.21.15.153:0/3381843300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 msgr2=0x7f58701020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:06.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 --2- 172.21.15.153:0/3381843300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701020f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5860004940 tx=0x7f5860009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:06.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- 172.21.15.153:0/3381843300 shutdown_connections 2024-04-05T02:59:06.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 --2- 172.21.15.153:0/3381843300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 0x7f58700fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 --2- 172.21.15.153:0/3381843300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701020f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- 172.21.15.153:0/3381843300 >> 172.21.15.153:0/3381843300 conn(0x7f58700f7870 msgr2=0x7f58700f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- 172.21.15.153:0/3381843300 shutdown_connections 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- 172.21.15.153:0/3381843300 wait complete. 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 Processor -- start 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.796+0000 7f5874de2700 1 -- start start 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 0x7f5870113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5870114580 con 0x7f5870101d20 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587010db80 con 0x7f58701026c0 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55438/0 (socket says 172.21.15.153:55438) 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 -- 172.21.15.153:0/91908976 learned_addr learned my addr 172.21.15.153:0/91908976 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:06.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586dd9b700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 0x7f5870113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 -- 172.21.15.153:0/91908976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 msgr2=0x7f5870113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 0x7f5870113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 -- 172.21.15.153:0/91908976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5860005040 con 0x7f5870101d20 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f586e59c700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f586000b3b0 tx=0x7f5860009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f586000e040 con 0x7f5870101d20 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5860013420 con 0x7f5870101d20 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f586001c630 con 0x7f5870101d20 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587010de00 con 0x7f5870101d20 2024-04-05T02:59:06.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.797+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587010e3c0 con 0x7f5870101d20 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.798+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58700fd470 con 0x7f5870101d20 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.799+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5860013590 con 0x7f5870101d20 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.800+0000 7f585effd700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 0x7f585806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.800+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5860088040 con 0x7f5870101d20 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.800+0000 7f586dd9b700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 0x7f585806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:06.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.800+0000 7f586dd9b700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 0x7f585806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f587010f220 tx=0x7f58640043c0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:06.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.804+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5860058140 con 0x7f5870101d20 2024-04-05T02:59:06.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.995+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f587010efd0 con 0x7f5870101d20 2024-04-05T02:59:06.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.995+0000 7f585effd700 1 -- 172.21.15.153:0/91908976 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5860057cd0 con 0x7f5870101d20 2024-04-05T02:59:06.995 INFO:teuthology.orchestra.run.smithi153.stdout:81604378649 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 msgr2=0x7f585806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 0x7f585806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f587010f220 tx=0x7f58640043c0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 msgr2=0x7f58701139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f586000b3b0 tx=0x7f5860009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 shutdown_connections 2024-04-05T02:59:06.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f58580692a0 0x7f585806b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5870101d20 0x7f58701139f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 --2- 172.21.15.153:0/91908976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58701026c0 0x7f5870113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:06.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 >> 172.21.15.153:0/91908976 conn(0x7f58700f7870 msgr2=0x7f58700f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:06.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 shutdown_connections 2024-04-05T02:59:06.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:06.997+0000 7f5874de2700 1 -- 172.21.15.153:0/91908976 wait complete. 2024-04-05T02:59:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:07 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/91908976' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T02:59:07.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:07 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/91908976' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.810+0000 7f8319c06700 1 -- 172.21.15.153:0/2421332724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 msgr2=0x7f83140fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.810+0000 7f8319c06700 1 --2- 172.21.15.153:0/2421332724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f83140fe250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8304004940 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- 172.21.15.153:0/2421332724 shutdown_connections 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 --2- 172.21.15.153:0/2421332724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83140fe820 0x7f8314106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 --2- 172.21.15.153:0/2421332724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f83140fe250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:07.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- 172.21.15.153:0/2421332724 >> 172.21.15.153:0/2421332724 conn(0x7f83140f7870 msgr2=0x7f83140f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- 172.21.15.153:0/2421332724 shutdown_connections 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- 172.21.15.153:0/2421332724 wait complete. 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 Processor -- start 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- start start 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83140fe820 0x7f8314113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314114530 con 0x7f83140fde80 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.811+0000 7f8319c06700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831410daa0 con 0x7f83140fe820 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55458/0 (socket says 172.21.15.153:55458) 2024-04-05T02:59:07.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 -- 172.21.15.153:0/3485066824 learned_addr learned my addr 172.21.15.153:0/3485066824 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 -- 172.21.15.153:0/3485066824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83140fe820 msgr2=0x7f8314113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83140fe820 0x7f8314113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 -- 172.21.15.153:0/3485066824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304005040 con 0x7f83140fde80 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f83137fe700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8304009380 tx=0x7f8304009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f830400e040 con 0x7f83140fde80 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8304009af0 con 0x7f83140fde80 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831410dd80 con 0x7f83140fde80 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f830401c630 con 0x7f83140fde80 2024-04-05T02:59:07.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.812+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831410e2c0 con 0x7f83140fde80 2024-04-05T02:59:07.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.813+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314066af0 con 0x7f83140fde80 2024-04-05T02:59:07.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.814+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f830400bbc0 con 0x7f83140fde80 2024-04-05T02:59:07.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.814+0000 7f82fbfff700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 0x7f82fc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:07.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.814+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8304088110 con 0x7f83140fde80 2024-04-05T02:59:07.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.814+0000 7f8312ffd700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 0x7f82fc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:07.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.815+0000 7f8312ffd700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 0x7f82fc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8308006fd0 tx=0x7f8308006db0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:07.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:07.819+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8304058210 con 0x7f83140fde80 2024-04-05T02:59:08.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.014+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8314109f10 con 0x7f83140fde80 2024-04-05T02:59:08.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.014+0000 7f82fbfff700 1 -- 172.21.15.153:0/3485066824 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8304057da0 con 0x7f83140fde80 2024-04-05T02:59:08.015 INFO:teuthology.orchestra.run.smithi153.stdout:146028888080 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 msgr2=0x7f82fc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 0x7f82fc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8308006fd0 tx=0x7f8308006db0 comp rx=0 tx=0).stop 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 msgr2=0x7f8314113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8304009380 tx=0x7f8304009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 shutdown_connections 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f82fc0692a0 0x7f82fc06b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83140fde80 0x7f8314113910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 --2- 172.21.15.153:0/3485066824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83140fe820 0x7f8314113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.016+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 >> 172.21.15.153:0/3485066824 conn(0x7f83140f7870 msgr2=0x7f83140f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.017+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 shutdown_connections 2024-04-05T02:59:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:08.017+0000 7f8319c06700 1 -- 172.21.15.153:0/3485066824 wait complete. 2024-04-05T02:59:08.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542172 got 60129542174 for osd.1 2024-04-05T02:59:08.474 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:08 smithi153 ceph-mon[29381]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:08 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3485066824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T02:59:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:08 smithi203 ceph-mon[39911]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:08 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3485066824' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T02:59:09.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378649 for osd.2 2024-04-05T02:59:09.466 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- 172.21.15.153:0/1110579026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5088102f70 msgr2=0x7f5088103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 --2- 172.21.15.153:0/1110579026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5088102f70 0x7f5088103340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5084004940 tx=0x7f5084009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- 172.21.15.153:0/1110579026 shutdown_connections 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 --2- 172.21.15.153:0/1110579026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50880fb3b0 0x7f50880fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 --2- 172.21.15.153:0/1110579026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5088102f70 0x7f5088103340 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- 172.21.15.153:0/1110579026 >> 172.21.15.153:0/1110579026 conn(0x7f5088072310 msgr2=0x7f5088072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- 172.21.15.153:0/1110579026 shutdown_connections 2024-04-05T02:59:09.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- 172.21.15.153:0/1110579026 wait complete. 2024-04-05T02:59:09.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 Processor -- start 2024-04-05T02:59:09.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.884+0000 7f508fc4a700 1 -- start start 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088102f70 0x7f5088113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5088114570 con 0x7f50880fb3b0 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508810db30 con 0x7f5088102f70 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33564/0 (socket says 172.21.15.153:33564) 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 -- 172.21.15.153:0/542380367 learned_addr learned my addr 172.21.15.153:0/542380367 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d1e5700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088102f70 0x7f5088113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 -- 172.21.15.153:0/542380367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088102f70 msgr2=0x7f5088113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088102f70 0x7f5088113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:09.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 -- 172.21.15.153:0/542380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084005040 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508d9e6700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5084009150 tx=0x7f5084009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508400e040 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508810ddb0 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5084013420 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508401c630 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.885+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508810e2f0 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.886+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50881053b0 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.887+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5084013590 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.887+0000 7f507e7fc700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 0x7f507406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.887+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f50840880d0 con 0x7f50880fb3b0 2024-04-05T02:59:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.887+0000 7f508d1e5700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 0x7f507406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:09.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.887+0000 7f508d1e5700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 0x7f507406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5078006fd0 tx=0x7f5078008040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:09.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:09.891+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f50840547f0 con 0x7f50880fb3b0 2024-04-05T02:59:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.084+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f508810a080 con 0x7f50880fb3b0 2024-04-05T02:59:10.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.085+0000 7f507e7fc700 1 -- 172.21.15.153:0/542380367 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5084057e10 con 0x7f50880fb3b0 2024-04-05T02:59:10.085 INFO:teuthology.orchestra.run.smithi153.stdout:38654705703 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 msgr2=0x7f507406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 0x7f507406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5078006fd0 tx=0x7f5078008040 comp rx=0 tx=0).stop 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 msgr2=0x7f50881139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5084009150 tx=0x7f5084009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 shutdown_connections 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f50740692a0 0x7f507406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50880fb3b0 0x7f50881139f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 --2- 172.21.15.153:0/542380367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5088102f70 0x7f5088113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 >> 172.21.15.153:0/542380367 conn(0x7f5088072310 msgr2=0x7f50880f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:10.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 shutdown_connections 2024-04-05T02:59:10.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:10.087+0000 7f508fc4a700 1 -- 172.21.15.153:0/542380367 wait complete. 2024-04-05T02:59:10.285 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888080 for osd.4 2024-04-05T02:59:10.285 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:10.789 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:10 smithi153 ceph-mon[29381]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:10.790 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:10 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/542380367' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T02:59:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:10 smithi203 ceph-mon[39911]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:10 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/542380367' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T02:59:10.816 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705703 for osd.0 2024-04-05T02:59:10.816 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:10.995 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:11.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 -- 172.21.15.153:0/2489595091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 msgr2=0x7f5f5c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:11.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2489595091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c1020e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c004940 tx=0x7f5f4c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:11.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 -- 172.21.15.153:0/2489595091 shutdown_connections 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2489595091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 0x7f5f5c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2489595091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c1020e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 -- 172.21.15.153:0/2489595091 >> 172.21.15.153:0/2489595091 conn(0x7f5f5c0f7860 msgr2=0x7f5f5c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 -- 172.21.15.153:0/2489595091 shutdown_connections 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.017+0000 7f5f619f8700 1 -- 172.21.15.153:0/2489595091 wait complete. 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 Processor -- start 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 -- start start 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 0x7f5f5c10f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c10f7a0 con 0x7f5f5c101d10 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f619f8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c10f910 con 0x7f5f5c1026b0 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33576/0 (socket says 172.21.15.153:33576) 2024-04-05T02:59:11.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 -- 172.21.15.153:0/2822168992 learned_addr learned my addr 172.21.15.153:0/2822168992 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5a7fc700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 0x7f5f5c10f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 -- 172.21.15.153:0/2822168992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 msgr2=0x7f5f5c10f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 0x7f5f5c10f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 -- 172.21.15.153:0/2822168992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c005040 con 0x7f5f5c101d10 2024-04-05T02:59:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.018+0000 7f5f5affd700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00b3b0 tx=0x7f5f4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.019+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f4c00e040 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.019+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c117bd0 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.019+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c1180e0 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.019+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f4c00bbc0 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.019+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f4c025400 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.020+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f480052b0 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.021+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5f4c013420 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.021+0000 7f5f437fe700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 0x7f5f4406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.021+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5f4c087ff0 con 0x7f5f5c101d10 2024-04-05T02:59:11.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.021+0000 7f5f5a7fc700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 0x7f5f4406fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:11.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.022+0000 7f5f5a7fc700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 0x7f5f4406fb60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c066890 tx=0x7f5f54009040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:11.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.025+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f4c0545e0 con 0x7f5f5c101d10 2024-04-05T02:59:11.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.221+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5f48005560 con 0x7f5f5c101d10 2024-04-05T02:59:11.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.221+0000 7f5f437fe700 1 -- 172.21.15.153:0/2822168992 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5f4c057c00 con 0x7f5f5c101d10 2024-04-05T02:59:11.221 INFO:teuthology.orchestra.run.smithi153.stdout:111669149717 2024-04-05T02:59:11.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 msgr2=0x7f5f4406fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:11.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 0x7f5f4406fb60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c066890 tx=0x7f5f54009040 comp rx=0 tx=0).stop 2024-04-05T02:59:11.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 msgr2=0x7f5f5c10ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00b3b0 tx=0x7f5f4c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 shutdown_connections 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f4406d6b0 0x7f5f4406fb60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f5c101d10 0x7f5f5c10ed20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 --2- 172.21.15.153:0/2822168992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c1026b0 0x7f5f5c10f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.223+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 >> 172.21.15.153:0/2822168992 conn(0x7f5f5c0f7860 msgr2=0x7f5f5c0f88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.224+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 shutdown_connections 2024-04-05T02:59:11.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:11.224+0000 7f5f619f8700 1 -- 172.21.15.153:0/2822168992 wait complete. 2024-04-05T02:59:11.764 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:11 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2822168992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T02:59:11.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:11 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2822168992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T02:59:12.792 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:12 smithi153 ceph-mon[29381]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:12 smithi203 ceph-mon[39911]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:12.817 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149714 got 111669149717 for osd.3 2024-04-05T02:59:12.818 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:13.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 -- 172.21.15.153:0/2009130775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948101d70 msgr2=0x7f5948102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 --2- 172.21.15.153:0/2009130775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948101d70 0x7f5948102140 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5944004940 tx=0x7f5944009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 -- 172.21.15.153:0/2009130775 shutdown_connections 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 --2- 172.21.15.153:0/2009130775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948102710 0x7f59480fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 --2- 172.21.15.153:0/2009130775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948101d70 0x7f5948102140 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 -- 172.21.15.153:0/2009130775 >> 172.21.15.153:0/2009130775 conn(0x7f5948072bc0 msgr2=0x7f5948072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 -- 172.21.15.153:0/2009130775 shutdown_connections 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.307+0000 7f594ff2c700 1 -- 172.21.15.153:0/2009130775 wait complete. 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 Processor -- start 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 -- start start 2024-04-05T02:59:13.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948101d70 0x7f594810ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594810f700 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594ff2c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594810f840 con 0x7f5948101d70 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33606/0 (socket says 172.21.15.153:33606) 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 -- 172.21.15.153:0/90623548 learned_addr learned my addr 172.21.15.153:0/90623548 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594dcc8700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948101d70 0x7f594810ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 -- 172.21.15.153:0/90623548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948101d70 msgr2=0x7f594810ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948101d70 0x7f594810ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.308+0000 7f594d4c7700 1 -- 172.21.15.153:0/90623548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5944005040 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f594d4c7700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f593800c890 tx=0x7f59380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f593800a700 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5948117b30 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f593800ad30 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59380113a0 con 0x7f5948102710 2024-04-05T02:59:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.309+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59481180c0 con 0x7f5948102710 2024-04-05T02:59:13.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.310+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59480ff080 con 0x7f5948102710 2024-04-05T02:59:13.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.310+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5938012400 con 0x7f5948102710 2024-04-05T02:59:13.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.311+0000 7f593e7fc700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 0x7f593406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:13.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.311+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5938086f20 con 0x7f5948102710 2024-04-05T02:59:13.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.311+0000 7f594dcc8700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 0x7f593406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:13.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.311+0000 7f594dcc8700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 0x7f593406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5944004f90 tx=0x7f5944009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:13.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.315+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5938053590 con 0x7f5948102710 2024-04-05T02:59:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.526+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5948108e70 con 0x7f5948102710 2024-04-05T02:59:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.526+0000 7f593e7fc700 1 -- 172.21.15.153:0/90623548 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5938051e60 con 0x7f5948102710 2024-04-05T02:59:13.527 INFO:teuthology.orchestra.run.smithi153.stdout:167503724556 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.528+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 msgr2=0x7f593406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 0x7f593406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5944004f90 tx=0x7f5944009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 msgr2=0x7f594810f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f593800c890 tx=0x7f59380044c0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 shutdown_connections 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f59340692a0 0x7f593406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5948101d70 0x7f594810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 --2- 172.21.15.153:0/90623548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5948102710 0x7f594810f1c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 >> 172.21.15.153:0/90623548 conn(0x7f5948072bc0 msgr2=0x7f59480fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 shutdown_connections 2024-04-05T02:59:13.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:13.529+0000 7f594ff2c700 1 -- 172.21.15.153:0/90623548 wait complete. 2024-04-05T02:59:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:13 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/90623548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T02:59:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:13 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/90623548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T02:59:14.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-04-05T02:59:14.226 DEBUG:teuthology.parallel:result is None 2024-04-05T02:59:14.227 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-05T02:59:14.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T02:59:14.497 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:14 smithi203 ceph-mon[39911]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:14 smithi153 ceph-mon[29381]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1450485906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c18101db0 msgr2=0x7f3c18102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 --2- 172.21.15.153:0/1450485906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c18101db0 0x7f3c18102180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3c08004940 tx=0x7f3c08009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1450485906 shutdown_connections 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 --2- 172.21.15.153:0/1450485906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c180fc4c0 0x7f3c180fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 --2- 172.21.15.153:0/1450485906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c18101db0 0x7f3c18102180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1450485906 >> 172.21.15.153:0/1450485906 conn(0x7f3c18072310 msgr2=0x7f3c18072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1450485906 shutdown_connections 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1450485906 wait complete. 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 Processor -- start 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.106+0000 7f3c1cdfd700 1 -- start start 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 0x7f3c1810f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1810f5e0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1810f720 con 0x7f3c18101db0 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33628/0 (socket says 172.21.15.153:33628) 2024-04-05T02:59:16.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 -- 172.21.15.153:0/1640699345 learned_addr learned my addr 172.21.15.153:0/1640699345 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c0dd9b700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 0x7f3c1810f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 -- 172.21.15.153:0/1640699345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 msgr2=0x7f3c1810f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 0x7f3c1810f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 -- 172.21.15.153:0/1640699345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08005040 con 0x7f3c180fc4c0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c0dd9b700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 0x7f3c1810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1659c700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800b8b0 tx=0x7f3c08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0800e040 con 0x7f3c180fc4c0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c08004290 con 0x7f3c180fc4c0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1640699345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1810f9a0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c0801c3f0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.107+0000 7f3c1cdfd700 1 -- 172.21.15.153:0/1640699345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c18117ef0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.108+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc0052b0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.109+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3c080137b0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.109+0000 7f3c0f7fe700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 0x7f3c0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.109+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3c08088130 con 0x7f3c180fc4c0 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.110+0000 7f3c0dd9b700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 0x7f3c0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:16.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.110+0000 7f3c0dd9b700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 0x7f3c0406b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006fd0 tx=0x7f3c00008040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:16.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.114+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3c080547a0 con 0x7f3c180fc4c0 2024-04-05T02:59:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.294+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3bfc000bc0 con 0x7f3c040692a0 2024-04-05T02:59:16.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.296+0000 7f3c0f7fe700 1 -- 172.21.15.153:0/1640699345 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28000 (secure 0 0 0) 0x7f3bfc000bc0 con 0x7f3c040692a0 2024-04-05T02:59:16.296 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:16.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 msgr2=0x7f3c0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 0x7f3c0406b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006fd0 tx=0x7f3c00008040 comp rx=0 tx=0).stop 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 msgr2=0x7f3c1810eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800b8b0 tx=0x7f3c08009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 shutdown_connections 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f3c040692a0 0x7f3c0406b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c180fc4c0 0x7f3c1810eb60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 --2- 172.21.15.153:0/1640699345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c18101db0 0x7f3c1810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 >> 172.21.15.153:0/1640699345 conn(0x7f3c18072310 msgr2=0x7f3c180fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 shutdown_connections 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:16.298+0000 7f3c0cd99700 1 -- 172.21.15.153:0/1640699345 wait complete. 2024-04-05T02:59:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-04-05T02:59:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:16 smithi203 ceph-mon[39911]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:16 smithi153 ceph-mon[29381]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:16.897 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-05T02:59:15.426810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788024,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682888,"statfs":{"total":575970213888,"available":574139277312,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002268"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-05T02:58:28.262531+0000","last_change":"2024-04-05T02:58:02.258246+0000","last_active":"2024-04-05T02:58:28.262531+0000","last_peered":"2024-04-05T02:58:28.262531+0000","last_clean":"2024-04-05T02:58:28.262531+0000","last_became_active":"2024-04-05T02:58:02.257878+0000","last_became_peered":"2024-04-05T02:58:02.257878+0000","last_unstale":"2024-04-05T02:58:28.262531+0000","last_undegraded":"2024-04-05T02:58:28.262531+0000","last_fullsized":"2024-04-05T02:58:28.262531+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T02:57:13.688501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T02:57:13.688501+0000","last_clean_scrub_stamp":"2024-04-05T02:57:13.688501+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T10:57:38.583846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.503}]},{"osd":2,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38600000000000001}]},{"osd":4,"last update":"Fri Apr 5 02:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.25700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.314},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.37}]},{"osd":3,"last update":"Fri Apr 5 02:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Apr 5 02:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32300000000000001},"min":{"1min":0.253,"5min":0.189,"15min":0.189},"max":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.375},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.32700000000000001},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.52800000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.30399999999999999}]},{"osd":2,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Apr 5 02:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.621},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Apr 5 02:59:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.33100000000000002,"15min":0.32500000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.33200000000000002,"15min":0.32600000000000001},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.22600000000000001}]},{"osd":2,"last update":"Fri Apr 5 02:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.248},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.40699999999999997}]},{"osd":4,"last update":"Fri Apr 5 02:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":17.373999999999999,"5min":17.373999999999999,"15min":17.373999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T02:59:16.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T02:59:17.151 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:17 smithi153 ceph-mon[29381]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T02:59:17.949 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:17 smithi203 ceph-mon[39911]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T02:59:18.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 -- 172.21.15.153:0/1858812704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3801026c0 msgr2=0x7fc3800fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 --2- 172.21.15.153:0/1858812704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3801026c0 0x7fc3800fbd30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc374004940 tx=0x7fc374009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 -- 172.21.15.153:0/1858812704 shutdown_connections 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 --2- 172.21.15.153:0/1858812704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3801026c0 0x7fc3800fbd30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 --2- 172.21.15.153:0/1858812704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc380101d20 0x7fc3801020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 -- 172.21.15.153:0/1858812704 >> 172.21.15.153:0/1858812704 conn(0x7fc3800f7870 msgr2=0x7fc3800f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 -- 172.21.15.153:0/1858812704 shutdown_connections 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.711+0000 7fc384edf700 1 -- 172.21.15.153:0/1858812704 wait complete. 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 Processor -- start 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 -- start start 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3801026c0 0x7fc38010ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:18.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc380067190 con 0x7fc380101d20 2024-04-05T02:59:18.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc384edf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38010d3b0 con 0x7fc3801026c0 2024-04-05T02:59:18.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:18.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:18 smithi153 ceph-mon[29381]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:18.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc37e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43752/0 (socket says 172.21.15.153:43752) 2024-04-05T02:59:18.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc37e59c700 1 -- 172.21.15.153:0/1271399768 learned_addr learned my addr 172.21.15.153:0/1271399768 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:18.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.712+0000 7fc37e59c700 1 -- 172.21.15.153:0/1271399768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3801026c0 msgr2=0x7fc38010ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:18.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc37e59c700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3801026c0 0x7fc38010ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc37e59c700 1 -- 172.21.15.153:0/1271399768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374005040 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc37e59c700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc37000b790 tx=0x7fc370009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37000f770 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc37000fda0 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37000d310 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38010d690 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.713+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38011a230 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.714+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3800fd470 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.715+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc370010420 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.715+0000 7fc36effd700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 0x7fc36806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.715+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc370086d60 con 0x7fc380101d20 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.715+0000 7fc37dd9b700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 0x7fc36806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.716+0000 7fc37dd9b700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 0x7fc36806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc37400b3b0 tx=0x7fc374009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:18.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.720+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc370056f00 con 0x7fc380101d20 2024-04-05T02:59:18.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.902+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3800f8fc0 con 0x7fc3680692a0 2024-04-05T02:59:18.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.903+0000 7fc36effd700 1 -- 172.21.15.153:0/1271399768 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28012 (secure 0 0 0) 0x7fc3800f8fc0 con 0x7fc3680692a0 2024-04-05T02:59:18.903 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 msgr2=0x7fc36806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 0x7fc36806b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc37400b3b0 tx=0x7fc374009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 msgr2=0x7fc38010c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc37000b790 tx=0x7fc370009250 comp rx=0 tx=0).stop 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 shutdown_connections 2024-04-05T02:59:18.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc3680692a0 0x7fc36806b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc380101d20 0x7fc38010c930 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 --2- 172.21.15.153:0/1271399768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3801026c0 0x7fc38010ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 >> 172.21.15.153:0/1271399768 conn(0x7fc3800f7870 msgr2=0x7fc3800f88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 shutdown_connections 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:18.905+0000 7fc384edf700 1 -- 172.21.15.153:0/1271399768 wait complete. 2024-04-05T02:59:18.906 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-04-05T02:59:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:18 smithi203 ceph-mon[39911]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:19.569 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-04-05T02:59:17.427255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788024,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682888,"statfs":{"total":575970213888,"available":574139277312,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002374"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-05T02:58:28.262531+0000","last_change":"2024-04-05T02:58:02.258246+0000","last_active":"2024-04-05T02:58:28.262531+0000","last_peered":"2024-04-05T02:58:28.262531+0000","last_clean":"2024-04-05T02:58:28.262531+0000","last_became_active":"2024-04-05T02:58:02.257878+0000","last_became_peered":"2024-04-05T02:58:02.257878+0000","last_unstale":"2024-04-05T02:58:28.262531+0000","last_undegraded":"2024-04-05T02:58:28.262531+0000","last_fullsized":"2024-04-05T02:58:28.262531+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T02:57:13.688501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T02:57:13.688501+0000","last_clean_scrub_stamp":"2024-04-05T02:57:13.688501+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T10:57:38.583846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.499},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39200000000000002}]},{"osd":1,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.44400000000000001}]},{"osd":3,"last update":"Fri Apr 5 02:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46999999999999997}]},{"osd":2,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.5},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.52700000000000002}]},{"osd":4,"last update":"Fri Apr 5 02:59:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.39600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.314},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.37}]},{"osd":3,"last update":"Fri Apr 5 02:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Apr 5 02:59:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 02:58:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32300000000000001},"min":{"1min":0.253,"5min":0.189,"15min":0.189},"max":{"1min":0.51100000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.375},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.32700000000000001},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.52800000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.30399999999999999}]},{"osd":2,"last update":"Fri Apr 5 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Apr 5 02:58:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.621},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Apr 5 02:59:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 02:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.33100000000000002,"15min":0.32500000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.33200000000000002,"15min":0.32600000000000001},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.22600000000000001}]},{"osd":2,"last update":"Fri Apr 5 02:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.248},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Fri Apr 5 02:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.40699999999999997}]},{"osd":4,"last update":"Fri Apr 5 02:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":17.373999999999999,"5min":17.373999999999999,"15min":17.373999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T02:59:19.570 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-05T02:59:19.570 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-05T02:59:19.570 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-05T02:59:19.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-05T02:59:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:20 smithi153 ceph-mon[29381]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T02:59:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:20 smithi153 ceph-mon[29381]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:20 smithi203 ceph-mon[39911]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T02:59:21.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:20 smithi203 ceph-mon[39911]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:21.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 -- 172.21.15.153:0/3906378613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fc5b0 msgr2=0x7f6e2c100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 --2- 172.21.15.153:0/3906378613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c100bd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c004940 tx=0x7f6e1c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 -- 172.21.15.153:0/3906378613 shutdown_connections 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 --2- 172.21.15.153:0/3906378613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c100bd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 --2- 172.21.15.153:0/3906378613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c0fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 -- 172.21.15.153:0/3906378613 >> 172.21.15.153:0/3906378613 conn(0x7f6e2c0f7870 msgr2=0x7f6e2c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 -- 172.21.15.153:0/3906378613 shutdown_connections 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 -- 172.21.15.153:0/3906378613 wait complete. 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.344+0000 7f6e33299700 1 Processor -- start 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e33299700 1 -- start start 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e33299700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e33299700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c11a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:21.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e33299700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c11ac60 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e33299700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c1141b0 con 0x7f6e2c0fc5b0 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43766/0 (socket says 172.21.15.153:43766) 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 -- 172.21.15.153:0/386533033 learned_addr learned my addr 172.21.15.153:0/386533033 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e30834700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c11a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 -- 172.21.15.153:0/386533033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fc5b0 msgr2=0x7f6e2c11a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c11a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 -- 172.21.15.153:0/386533033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c005040 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e31035700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800b790 tx=0x7f6e28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e2800f6f0 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.345+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e2800fd20 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.346+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c114430 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.346+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e280103a0 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.346+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c114970 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.347+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c066b70 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.347+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6e28011400 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.347+0000 7f6e21ffb700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 0x7f6e1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:21.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.347+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6e28086c30 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.347+0000 7f6e30834700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 0x7f6e1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:21.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.348+0000 7f6e30834700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 0x7f6e1806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c1158e0 tx=0x7f6e1c009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:21.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.352+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6e28056d30 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.581+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6e2c115750 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.581+0000 7f6e21ffb700 1 -- 172.21.15.153:0/386533033 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6e280568c0 con 0x7f6e2c0fbc10 2024-04-05T02:59:21.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:21.581 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 msgr2=0x7f6e1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 0x7f6e1806b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c1158e0 tx=0x7f6e1c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 msgr2=0x7f6e2c11a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800b790 tx=0x7f6e28009250 comp rx=0 tx=0).stop 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 shutdown_connections 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6e180692a0 0x7f6e1806b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e2c0fbc10 0x7f6e2c11a040 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 --2- 172.21.15.153:0/386533033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e2c0fc5b0 0x7f6e2c11a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 >> 172.21.15.153:0/386533033 conn(0x7f6e2c0f7870 msgr2=0x7f6e2c0f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 shutdown_connections 2024-04-05T02:59:21.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:21.583+0000 7f6e33299700 1 -- 172.21.15.153:0/386533033 wait complete. 2024-04-05T02:59:22.186 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-05T02:59:22.186 INFO:tasks.cephadm:Setup complete, yielding 2024-04-05T02:59:22.186 INFO:teuthology.run_tasks:Running task print... 2024-04-05T02:59:22.198 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-05T02:59:22.198 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:22.212 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:22.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-05T02:59:22.467 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:22 smithi153 ceph-mon[29381]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:22 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/386533033' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T02:59:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:22 smithi203 ceph-mon[39911]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:22 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/386533033' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 -- 172.21.15.153:0/1921297442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 msgr2=0x7fdc540fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/1921297442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc540fc0e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdc50004940 tx=0x7fdc50009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 -- 172.21.15.153:0/1921297442 shutdown_connections 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/1921297442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 0x7fdc540fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/1921297442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc540fc0e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 -- 172.21.15.153:0/1921297442 >> 172.21.15.153:0/1921297442 conn(0x7fdc540f7430 msgr2=0x7fdc540f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 -- 172.21.15.153:0/1921297442 shutdown_connections 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.015+0000 7fdc5ba09700 1 -- 172.21.15.153:0/1921297442 wait complete. 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 Processor -- start 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 -- start start 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 0x7fdc5410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5410e470 con 0x7fdc540fbcd0 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc5ba09700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5410e5e0 con 0x7fdc540fcf80 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc597a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc597a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43776/0 (socket says 172.21.15.153:43776) 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc597a5700 1 -- 172.21.15.153:0/774862218 learned_addr learned my addr 172.21.15.153:0/774862218 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.016+0000 7fdc58fa4700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 0x7fdc5410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc597a5700 1 -- 172.21.15.153:0/774862218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 msgr2=0x7fdc5410de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:24.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc597a5700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 0x7fdc5410de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc597a5700 1 -- 172.21.15.153:0/774862218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50005040 con 0x7fdc540fbcd0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc597a5700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b3b0 tx=0x7fdc50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5000e040 con 0x7fdc540fbcd0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc5000bbf0 con 0x7fdc540fbcd0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54112fe0 con 0x7fdc540fbcd0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5001c630 con 0x7fdc540fbcd0 2024-04-05T02:59:24.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.017+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc54113510 con 0x7fdc540fbcd0 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.018+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc54101b10 con 0x7fdc540fbcd0 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.019+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdc5000bd60 con 0x7fdc540fbcd0 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.019+0000 7fdc49ffb700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 0x7fdc4006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.019+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc50087b00 con 0x7fdc540fbcd0 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.019+0000 7fdc58fa4700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 0x7fdc4006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:24.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.019+0000 7fdc58fa4700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 0x7fdc4006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc44004f40 tx=0x7fdc4400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:24.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.023+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdc50057c00 con 0x7fdc540fbcd0 2024-04-05T02:59:24.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.212+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fdc5404c710 con 0x7fdc540fbcd0 2024-04-05T02:59:24.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.217+0000 7fdc49ffb700 1 -- 172.21.15.153:0/774862218 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fdc50057790 con 0x7fdc540fbcd0 2024-04-05T02:59:24.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 msgr2=0x7fdc4006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:24.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 0x7fdc4006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc44004f40 tx=0x7fdc4400b040 comp rx=0 tx=0).stop 2024-04-05T02:59:24.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 msgr2=0x7fdc5410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:24.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdc5000b3b0 tx=0x7fdc50009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 shutdown_connections 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fdc400692a0 0x7fdc4006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc540fbcd0 0x7fdc5410d930 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 --2- 172.21.15.153:0/774862218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc540fcf80 0x7fdc5410de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 >> 172.21.15.153:0/774862218 conn(0x7fdc540f7430 msgr2=0x7fdc540668e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 shutdown_connections 2024-04-05T02:59:24.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:24.219+0000 7fdc5ba09700 1 -- 172.21.15.153:0/774862218 wait complete. 2024-04-05T02:59:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:24 smithi153 ceph-mon[29381]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:24 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/774862218' entity='client.admin' 2024-04-05T02:59:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:24 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:24.872 INFO:teuthology.run_tasks:Running task print... 2024-04-05T02:59:24.882 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-05T02:59:24.882 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:24.894 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:24.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-05T02:59:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:24 smithi203 ceph-mon[39911]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:25.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:24 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/774862218' entity='client.admin' 2024-04-05T02:59:25.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:24 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:25.148 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:25 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:25 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:59:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:25 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:25 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:25 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:59:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:25 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- 172.21.15.153:0/2489080370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 msgr2=0x7f1c480fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 --2- 172.21.15.153:0/2489080370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c480fc0e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1c44004940 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- 172.21.15.153:0/2489080370 shutdown_connections 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 --2- 172.21.15.153:0/2489080370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 0x7f1c480fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 --2- 172.21.15.153:0/2489080370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c480fc0e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- 172.21.15.153:0/2489080370 >> 172.21.15.153:0/2489080370 conn(0x7f1c480f7450 msgr2=0x7f1c480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- 172.21.15.153:0/2489080370 shutdown_connections 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- 172.21.15.153:0/2489080370 wait complete. 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 Processor -- start 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.725+0000 7f1c50423700 1 -- start start 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c50423700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c50423700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 0x7f1c4810dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c50423700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4810e3e0 con 0x7f1c480fbcd0 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c50423700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4810e550 con 0x7f1c480fcf80 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43800/0 (socket says 172.21.15.153:43800) 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 -- 172.21.15.153:0/879503185 learned_addr learned my addr 172.21.15.153:0/879503185 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4d9be700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 0x7f1c4810dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 -- 172.21.15.153:0/879503185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 msgr2=0x7f1c4810dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:26.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 0x7f1c4810dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 -- 172.21.15.153:0/879503185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c44005040 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c4e1bf700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009150 tx=0x7f1c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c4400e040 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c44004290 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c48112f50 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.726+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c4401c630 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.727+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48113480 con 0x7f1c480fbcd0 2024-04-05T02:59:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.727+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48101b10 con 0x7f1c480fbcd0 2024-04-05T02:59:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.728+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1c4400bbf0 con 0x7f1c480fbcd0 2024-04-05T02:59:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.728+0000 7f1c3effd700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 0x7f1c3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.728+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1c44087b10 con 0x7f1c480fbcd0 2024-04-05T02:59:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.728+0000 7f1c4d9be700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 0x7f1c3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:26.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.729+0000 7f1c4d9be700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 0x7f1c3406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009fd0 tx=0x7f1c3800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.733+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1c44057c10 con 0x7f1c480fbcd0 2024-04-05T02:59:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:26 smithi153 ceph-mon[29381]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.933+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c4802b420 con 0x7f1c340692a0 2024-04-05T02:59:26.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.934+0000 7f1c3effd700 1 -- 172.21.15.153:0/879503185 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1c4802b420 con 0x7f1c340692a0 2024-04-05T02:59:26.934 INFO:teuthology.orchestra.run.smithi153.stdout:Backend: cephadm 2024-04-05T02:59:26.934 INFO:teuthology.orchestra.run.smithi153.stdout:Available: Yes 2024-04-05T02:59:26.934 INFO:teuthology.orchestra.run.smithi153.stdout:Paused: No 2024-04-05T02:59:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 msgr2=0x7f1c3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 0x7f1c3406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009fd0 tx=0x7f1c3800b040 comp rx=0 tx=0).stop 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 msgr2=0x7f1c4810d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009150 tx=0x7f1c44009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 shutdown_connections 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1c340692a0 0x7f1c3406b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c480fbcd0 0x7f1c4810d8a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 --2- 172.21.15.153:0/879503185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c480fcf80 0x7f1c4810dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 >> 172.21.15.153:0/879503185 conn(0x7f1c480f7450 msgr2=0x7f1c48066760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 shutdown_connections 2024-04-05T02:59:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:26.936+0000 7f1c50423700 1 -- 172.21.15.153:0/879503185 wait complete. 2024-04-05T02:59:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:26 smithi203 ceph-mon[39911]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:27.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-05T02:59:27.939 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:29.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:28 smithi153 ceph-mon[29381]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:29.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:28 smithi153 ceph-mon[29381]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:29.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:28 smithi203 ceph-mon[39911]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:28 smithi203 ceph-mon[39911]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 -- 172.21.15.153:0/268737917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 msgr2=0x7f7484102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 --2- 172.21.15.153:0/268737917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f7484102320 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7478004940 tx=0x7f7478009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 -- 172.21.15.153:0/268737917 shutdown_connections 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 --2- 172.21.15.153:0/268737917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f7484102320 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 --2- 172.21.15.153:0/268737917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 -- 172.21.15.153:0/268737917 >> 172.21.15.153:0/268737917 conn(0x7f74840f7430 msgr2=0x7f74840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 -- 172.21.15.153:0/268737917 shutdown_connections 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.453+0000 7f7488dec700 1 -- 172.21.15.153:0/268737917 wait complete. 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 Processor -- start 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 -- start start 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840fd830 con 0x7f74840fff00 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7488dec700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840fbe00 con 0x7f7484066f70 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7481d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35050/0 (socket says 172.21.15.153:35050) 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f7481d9b700 1 -- 172.21.15.153:0/3827530091 learned_addr learned my addr 172.21.15.153:0/3827530091 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.454+0000 7f748259c700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7481d9b700 1 -- 172.21.15.153:0/3827530091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 msgr2=0x7f74840fd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7481d9b700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840fd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7481d9b700 1 -- 172.21.15.153:0/3827530091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478005040 con 0x7f74840fff00 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f748259c700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7481d9b700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7478009150 tx=0x7f7478009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747800e040 con 0x7f74840fff00 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f747800bbf0 con 0x7f74840fff00 2024-04-05T02:59:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74840fc080 con 0x7f74840fff00 2024-04-05T02:59:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747801c630 con 0x7f74840fff00 2024-04-05T02:59:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.455+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74840fc640 con 0x7f74840fff00 2024-04-05T02:59:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.456+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748404c710 con 0x7f74840fff00 2024-04-05T02:59:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.456+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f747800bd60 con 0x7f74840fff00 2024-04-05T02:59:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.457+0000 7f7472ffd700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.457+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7478088110 con 0x7f74840fff00 2024-04-05T02:59:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.457+0000 7f748259c700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.457+0000 7f748259c700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 0x7f746c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7484066cb0 tx=0x7f7474008040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:29.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.461+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f74780582b0 con 0x7f74840fff00 2024-04-05T02:59:29.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.655+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7484101b70 con 0x7f746c0692a0 2024-04-05T02:59:29.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.663+0000 7f7472ffd700 1 -- 172.21.15.153:0/3827530091 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2700 (secure 0 0 0) 0x7f7484101b70 con 0x7f746c0692a0 2024-04-05T02:59:29.663 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T02:59:29.663 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (3m) 2m ago 5m 24.5M - 0.25.0 c8568f914cd2 db2172977a37 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (6m) 2m ago 6m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (4m) 61s ago 4m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (3m) 2m ago 5m 78.1M - 9.4.7 954c08fa6188 0fafa83f96b9 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (7m) 2m ago 7m 467M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (3m) 61s ago 3m 434M - 17.2.7-805-g6fda41ff a49113eacc58 3ebddafb5706 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (7m) 2m ago 7m 43.2M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (3m) 61s ago 3m 41.6M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 2m ago 5m 15.6M - 1.5.0 0da6a335fe13 61d64f55ec69 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (4m) 61s ago 4m 15.7M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 2m ago 3m 36.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 2m ago 2m 35.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 2m ago 2m 11.4M 4629M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (2m) 61s ago 119s 38.1M 4970M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (94s) 61s ago 94s 36.7M 4970M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (68s) 61s ago 67s 11.0M 4970M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T02:59:29.664 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (3m) 61s ago 3m 50.2M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.665+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 msgr2=0x7f746c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.665+0000 7f7488dec700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 0x7f746c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7484066cb0 tx=0x7f7474008040 comp rx=0 tx=0).stop 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.665+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 msgr2=0x7f74840fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.665+0000 7f7488dec700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7478009150 tx=0x7f7478009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 shutdown_connections 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f746c0692a0 0x7f746c06b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7484066f70 0x7f74840fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 --2- 172.21.15.153:0/3827530091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74840fff00 0x7f74840fb8c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 >> 172.21.15.153:0/3827530091 conn(0x7f74840f7430 msgr2=0x7f74840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 shutdown_connections 2024-04-05T02:59:29.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:29.666+0000 7f7488dec700 1 -- 172.21.15.153:0/3827530091 wait complete. 2024-04-05T02:59:30.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-05T02:59:30.570 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:30.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:30 smithi153 ceph-mon[29381]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:30.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:30 smithi153 ceph-mon[29381]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:30 smithi203 ceph-mon[39911]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:30 smithi203 ceph-mon[39911]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:32.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 -- 172.21.15.153:0/2194454661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fcf80 msgr2=0x7f8a9c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/2194454661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c0fd3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8a90004940 tx=0x7f8a90009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 -- 172.21.15.153:0/2194454661 shutdown_connections 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/2194454661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c0fd3f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/2194454661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 -- 172.21.15.153:0/2194454661 >> 172.21.15.153:0/2194454661 conn(0x7f8a9c0f7430 msgr2=0x7f8a9c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 -- 172.21.15.153:0/2194454661 shutdown_connections 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.107+0000 7f8aa11bf700 1 -- 172.21.15.153:0/2194454661 wait complete. 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 Processor -- start 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 -- start start 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:32.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c10e510 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8aa11bf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c112eb0 con 0x7f8a9c0fcf80 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8a9ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8a9ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35070/0 (socket says 172.21.15.153:35070) 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.108+0000 7f8a9ad9d700 1 -- 172.21.15.153:0/3508269921 learned_addr learned my addr 172.21.15.153:0/3508269921 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9a59c700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9ad9d700 1 -- 172.21.15.153:0/3508269921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 msgr2=0x7f8a9c10de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9ad9d700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c10de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9ad9d700 1 -- 172.21.15.153:0/3508269921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90005040 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9a59c700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a9ad9d700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00b790 tx=0x7f8a8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a8c00f770 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a8c00fda0 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c113110 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a8c00d310 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.109+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c113680 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.110+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c04c710 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.111+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8a8c010420 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.111+0000 7f8a837fe700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 0x7f8a8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.111+0000 7f8a9a59c700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 0x7f8a8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.111+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a8c0875f0 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.112+0000 7f8a9a59c700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 0x7f8a8406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000b970 tx=0x7f8a90009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:32.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.115+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8a8c0576d0 con 0x7f8a9c0fbcd0 2024-04-05T02:59:32.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.311+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a9c113960 con 0x7f8a840692a0 2024-04-05T02:59:32.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.316+0000 7f8a837fe700 1 -- 172.21.15.153:0/3508269921 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f8a9c113960 con 0x7f8a840692a0 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:crash 2/2 2m ago 6m * 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:mgr 2/2 2m ago 6m count:2 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:mon 2/2 2m ago 6m smithi153:172.21.15.153=smithi153;smithi203:172.21.15.203=smithi203;count:2 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:osd 6 2m ago - 2024-04-05T02:59:32.316 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus ?:9095 1/1 64s ago 6m count:1 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 msgr2=0x7f8a8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 0x7f8a8406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000b970 tx=0x7f8a90009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 msgr2=0x7f8a9c10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00b790 tx=0x7f8a8c009250 comp rx=0 tx=0).stop 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 shutdown_connections 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8a840692a0 0x7f8a8406b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c0fbcd0 0x7f8a9c10d940 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 --2- 172.21.15.153:0/3508269921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a9c0fcf80 0x7f8a9c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 >> 172.21.15.153:0/3508269921 conn(0x7f8a9c0f7430 msgr2=0x7f8a9c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 shutdown_connections 2024-04-05T02:59:32.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:32.318+0000 7f8aa11bf700 1 -- 172.21.15.153:0/3508269921 wait complete. 2024-04-05T02:59:32.893 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:32 smithi153 ceph-mon[29381]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:32.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-05T02:59:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:32 smithi203 ceph-mon[39911]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:33.151 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:33.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:33 smithi153 ceph-mon[29381]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:33 smithi203 ceph-mon[39911]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.698+0000 7fa2dfa72700 1 -- 172.21.15.153:0/2020419493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fdfe0 msgr2=0x7fa2d80fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.698+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/2020419493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d80fe450 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8004940 tx=0x7fa2c8009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- 172.21.15.153:0/2020419493 shutdown_connections 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/2020419493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d80fe450 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/2020419493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fce70 0x7fa2d80fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- 172.21.15.153:0/2020419493 >> 172.21.15.153:0/2020419493 conn(0x7fa2d80f8550 msgr2=0x7fa2d80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- 172.21.15.153:0/2020419493 shutdown_connections 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- 172.21.15.153:0/2020419493 wait complete. 2024-04-05T02:59:34.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 Processor -- start 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- start start 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d810e500 con 0x7fa2d80fce70 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dfa72700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8112ea0 con 0x7fa2d80fdfe0 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dd80e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dd80e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35082/0 (socket says 172.21.15.153:35082) 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.699+0000 7fa2dd80e700 1 -- 172.21.15.153:0/3004149605 learned_addr learned my addr 172.21.15.153:0/3004149605 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dd00d700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dd80e700 1 -- 172.21.15.153:0/3004149605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fdfe0 msgr2=0x7fa2d810de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dd80e700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dd80e700 1 -- 172.21.15.153:0/3004149605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8005040 con 0x7fa2d80fce70 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dd80e700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400b790 tx=0x7fa2d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d400f770 con 0x7fa2d80fce70 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2d400fda0 con 0x7fa2d80fce70 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d400d310 con 0x7fa2d80fce70 2024-04-05T02:59:34.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d8113100 con 0x7fa2d80fce70 2024-04-05T02:59:34.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.700+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d8113670 con 0x7fa2d80fce70 2024-04-05T02:59:34.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.701+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d804c710 con 0x7fa2d80fce70 2024-04-05T02:59:34.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.702+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa2d4010420 con 0x7fa2d80fce70 2024-04-05T02:59:34.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.702+0000 7fa2ce7fc700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 0x7fa2c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:34.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.702+0000 7fa2dd00d700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 0x7fa2c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:34.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.702+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa2d4086d60 con 0x7fa2d80fce70 2024-04-05T02:59:34.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.702+0000 7fa2dd00d700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 0x7fa2c406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:34.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.706+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa2d4056f00 con 0x7fa2d80fce70 2024-04-05T02:59:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.898+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2d8102300 con 0x7fa2c40692a0 2024-04-05T02:59:34.898 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:34 smithi153 ceph-mon[29381]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:34.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.900+0000 7fa2ce7fc700 1 -- 172.21.15.153:0/3004149605 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fa2d8102300 con 0x7fa2c40692a0 2024-04-05T02:59:34.899 INFO:teuthology.orchestra.run.smithi153.stdout:HOST ADDR LABELS STATUS 2024-04-05T02:59:34.900 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 172.21.15.153 2024-04-05T02:59:34.900 INFO:teuthology.orchestra.run.smithi153.stdout:smithi203 172.21.15.203 2024-04-05T02:59:34.900 INFO:teuthology.orchestra.run.smithi153.stdout:2 hosts in cluster 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 msgr2=0x7fa2c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 0x7fa2c406b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 msgr2=0x7fa2d810d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400b790 tx=0x7fa2d4009250 comp rx=0 tx=0).stop 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 shutdown_connections 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa2c40692a0 0x7fa2c406b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2d80fce70 0x7fa2d810d930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 --2- 172.21.15.153:0/3004149605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2d80fdfe0 0x7fa2d810de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 >> 172.21.15.153:0/3004149605 conn(0x7fa2d80f8550 msgr2=0x7fa2d80f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 shutdown_connections 2024-04-05T02:59:34.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:34.902+0000 7fa2dfa72700 1 -- 172.21.15.153:0/3004149605 wait complete. 2024-04-05T02:59:34.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:34 smithi203 ceph-mon[39911]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:35.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-05T02:59:35.690 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:36.809 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:36 smithi153 ceph-mon[29381]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:36.809 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:36 smithi153 ceph-mon[29381]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:36 smithi203 ceph-mon[39911]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:37.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:36 smithi203 ceph-mon[39911]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:37.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.278+0000 7f4fd0eee700 1 -- 172.21.15.153:0/3468315528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 msgr2=0x7f4fcc0fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:37.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.278+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/3468315528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc0fe450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0004940 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:37.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.278+0000 7f4fd0eee700 1 -- 172.21.15.153:0/3468315528 shutdown_connections 2024-04-05T02:59:37.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.278+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/3468315528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc0fe450 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.278+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/3468315528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- 172.21.15.153:0/3468315528 >> 172.21.15.153:0/3468315528 conn(0x7f4fcc0f8550 msgr2=0x7f4fcc0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- 172.21.15.153:0/3468315528 shutdown_connections 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- 172.21.15.153:0/3468315528 wait complete. 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 Processor -- start 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- start start 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc070850 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fd0eee700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0709c0 con 0x7f4fcc0fce70 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fc9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fc9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35094/0 (socket says 172.21.15.153:35094) 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.279+0000 7f4fc9d9b700 1 -- 172.21.15.153:0/1496405794 learned_addr learned my addr 172.21.15.153:0/1496405794 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:37.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fca59c700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fc9d9b700 1 -- 172.21.15.153:0/1496405794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 msgr2=0x7f4fcc071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fc9d9b700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fc9d9b700 1 -- 172.21.15.153:0/1496405794 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0005040 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fca59c700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fc9d9b700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000b3b0 tx=0x7f4fc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc000e040 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fc000bcc0 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc070c40 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc001c630 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.280+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc115a20 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.281+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc04c710 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.282+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4fc0013980 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.282+0000 7f4fbaffd700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 0x7f4fb406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.282+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4fc0087c80 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.282+0000 7f4fca59c700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 0x7f4fb406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:37.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.282+0000 7f4fca59c700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 0x7f4fb406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006fd0 tx=0x7f4fbc008040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:37.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.286+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4fc0057d00 con 0x7f4fcc0fdfe0 2024-04-05T02:59:37.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.480+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fcc102280 con 0x7f4fb40692a0 2024-04-05T02:59:37.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.482+0000 7f4fbaffd700 1 -- 172.21.15.153:0/1496405794 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f4fcc102280 con 0x7f4fb40692a0 2024-04-05T02:59:37.482 INFO:teuthology.orchestra.run.smithi153.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-05T02:59:37.482 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-05T02:59:37.483 INFO:teuthology.orchestra.run.smithi153.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 66s ago Has a FileSystem, LVM detected 2024-04-05T02:59:37.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.484+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 msgr2=0x7f4fb406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.484+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 0x7f4fb406b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc006fd0 tx=0x7f4fbc008040 comp rx=0 tx=0).stop 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 msgr2=0x7f4fcc070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000b3b0 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 shutdown_connections 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fb40692a0 0x7f4fb406b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc0fce70 0x7f4fcc071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 --2- 172.21.15.153:0/1496405794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc0fdfe0 0x7f4fcc070310 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 >> 172.21.15.153:0/1496405794 conn(0x7f4fcc0f8550 msgr2=0x7f4fcc0f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 shutdown_connections 2024-04-05T02:59:37.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:37.485+0000 7f4fd0eee700 1 -- 172.21.15.153:0/1496405794 wait complete. 2024-04-05T02:59:38.107 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:38.117 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:38.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-05T02:59:38.375 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:38.819 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:38 smithi153 ceph-mon[29381]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:38.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:38 smithi153 ceph-mon[29381]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:38 smithi203 ceph-mon[39911]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:38 smithi203 ceph-mon[39911]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 -- 172.21.15.153:0/2878174170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 msgr2=0x7fe40c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 --2- 172.21.15.153:0/2878174170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c070350 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009ce0 tx=0x7fe3fc004d10 comp rx=0 tx=0).stop 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 -- 172.21.15.153:0/2878174170 shutdown_connections 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 --2- 172.21.15.153:0/2878174170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe40c070890 0x7fe40c070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 --2- 172.21.15.153:0/2878174170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c070350 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 -- 172.21.15.153:0/2878174170 >> 172.21.15.153:0/2878174170 conn(0x7fe40c0f8580 msgr2=0x7fe40c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 -- 172.21.15.153:0/2878174170 shutdown_connections 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.935+0000 7fe411c99700 1 -- 172.21.15.153:0/2878174170 wait complete. 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 Processor -- start 2024-04-05T02:59:39.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 -- start start 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe40c070890 0x7fe40c10fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c111110 con 0x7fe40c071e40 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.936+0000 7fe411c99700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c111280 con 0x7fe40c070890 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57458/0 (socket says 172.21.15.153:57458) 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 -- 172.21.15.153:0/1122361730 learned_addr learned my addr 172.21.15.153:0/1122361730 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 -- 172.21.15.153:0/1122361730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe40c070890 msgr2=0x7fe40c10fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe40c070890 0x7fe40c10fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 -- 172.21.15.153:0/1122361730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009940 con 0x7fe40c071e40 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe40affd700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c830 tx=0x7fe3f4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3f400a760 con 0x7fe40c071e40 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c111500 con 0x7fe40c071e40 2024-04-05T02:59:39.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3f400ad90 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3f40113a0 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.937+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c111ac0 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.938+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c05f5f0 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.938+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe3f4012400 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.939+0000 7fe403fff700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 0x7fe3f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.939+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe3f4086e90 con 0x7fe40c071e40 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.939+0000 7fe40b7fe700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 0x7fe3f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.939+0000 7fe40b7fe700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 0x7fe3f806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009f90 tx=0x7fe3fc004d10 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:39.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:39.943+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe3f4053500 con 0x7fe40c071e40 2024-04-05T02:59:40.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:40.157+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe40c111d40 con 0x7fe3f80692a0 2024-04-05T02:59:41.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:40 smithi203 ceph-mon[39911]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:41.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:40 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-05T02:59:41.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:40 smithi153 ceph-mon[29381]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:41.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:40 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-05T02:59:41.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.676+0000 7fe403fff700 1 -- 172.21.15.153:0/1122361730 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe40c111d40 con 0x7fe3f80692a0 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 msgr2=0x7fe3f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 0x7fe3f806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009f90 tx=0x7fe3fc004d10 comp rx=0 tx=0).stop 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 msgr2=0x7fe40c1103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400c830 tx=0x7fe3f4004470 comp rx=0 tx=0).stop 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 shutdown_connections 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe3f80692a0 0x7fe3f806b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:41.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe40c070890 0x7fe40c10fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:41.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 --2- 172.21.15.153:0/1122361730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe40c071e40 0x7fe40c1103a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:41.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 >> 172.21.15.153:0/1122361730 conn(0x7fe40c0f8580 msgr2=0x7fe40c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:41.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 shutdown_connections 2024-04-05T02:59:41.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:41.680+0000 7fe411c99700 1 -- 172.21.15.153:0/1122361730 wait complete. 2024-04-05T02:59:41.890 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[29377]: 2024-04-05T02:59:41.654+0000 7f53190c7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: osdmap e41: 6 total, 6 up, 6 in 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: pgmap v163: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-05T02:59:42.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: osdmap e42: 6 total, 6 up, 6 in 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: osdmap e43: 6 total, 6 up, 6 in 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: fsmap cephfs:0 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: Saving service mds.cephfs spec with placement count:4 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:42.273 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:41 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:42.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: osdmap e41: 6 total, 6 up, 6 in 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: pgmap v163: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: osdmap e42: 6 total, 6 up, 6 in 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: osdmap e43: 6 total, 6 up, 6 in 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: fsmap cephfs:0 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: Saving service mds.cephfs spec with placement count:4 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:42.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:41 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:42.585 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: Deploying daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T02:59:42.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:42 smithi153 ceph-mon[29381]: osdmap e44: 6 total, 6 up, 6 in 2024-04-05T02:59:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: Deploying daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T02:59:43.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:42 smithi203 ceph-mon[39911]: osdmap e44: 6 total, 6 up, 6 in 2024-04-05T02:59:44.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:44 smithi203 ceph-mon[39911]: pgmap v167: 65 pgs: 9 creating+peering, 37 unknown, 19 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:44.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:44.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:44 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:44.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:44 smithi203 ceph-mon[39911]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T02:59:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:44 smithi153 ceph-mon[29381]: pgmap v167: 65 pgs: 9 creating+peering, 37 unknown, 19 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:44.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:44 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:44.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:44 smithi153 ceph-mon[29381]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T02:59:45.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 -- 172.21.15.153:0/822606309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 msgr2=0x7f9eb00fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 --2- 172.21.15.153:0/822606309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb00fd250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0004940 tx=0x7f9ea0009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 -- 172.21.15.153:0/822606309 shutdown_connections 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 --2- 172.21.15.153:0/822606309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 0x7f9eb00fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 --2- 172.21.15.153:0/822606309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb00fd250 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 -- 172.21.15.153:0/822606309 >> 172.21.15.153:0/822606309 conn(0x7f9eb00f8510 msgr2=0x7f9eb00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 -- 172.21.15.153:0/822606309 shutdown_connections 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 -- 172.21.15.153:0/822606309 wait complete. 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.137+0000 7f9eb605b700 1 Processor -- start 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eb605b700 1 -- start start 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eb605b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:45.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eb605b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 0x7f9eb010bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eb605b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb010c340 con 0x7f9eb00fce40 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eb605b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb010cb90 con 0x7f9eb00fe040 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57470/0 (socket says 172.21.15.153:57470) 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 -- 172.21.15.153:0/2853453604 learned_addr learned my addr 172.21.15.153:0/2853453604 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:45.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaeffd700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 0x7f9eb010bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 -- 172.21.15.153:0/2853453604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 msgr2=0x7f9eb010bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 0x7f9eb010bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.138+0000 7f9eaf7fe700 1 -- 172.21.15.153:0/2853453604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0005040 con 0x7f9eb00fce40 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9eaf7fe700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea000e030 con 0x7f9eb00fce40 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ea0013de0 con 0x7f9eb00fce40 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb010ce10 con 0x7f9eb00fce40 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea001ce50 con 0x7f9eb00fce40 2024-04-05T02:59:45.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.139+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb010d350 con 0x7f9eb00fce40 2024-04-05T02:59:45.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.140+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb01019d0 con 0x7f9eb00fce40 2024-04-05T02:59:45.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.141+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9ea001c400 con 0x7f9eb00fce40 2024-04-05T02:59:45.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.141+0000 7f9e97fff700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 0x7f9e9806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:45.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.141+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9ea0088b30 con 0x7f9eb00fce40 2024-04-05T02:59:45.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.141+0000 7f9eaeffd700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 0x7f9e9806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:45.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.142+0000 7f9eaeffd700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 0x7f9e9806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009b80 tx=0x7f9ea400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:45.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.146+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9ea0054e60 con 0x7f9eb00fce40 2024-04-05T02:59:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.396+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9eb01059d0 con 0x7f9eb00fce40 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.397+0000 7f9e97fff700 1 -- 172.21.15.153:0/2853453604 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f9ea0058480 con 0x7f9eb00fce40 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:e2 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 2 2024-04-05T02:59:45.399 INFO:teuthology.orchestra.run.smithi153.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T02:59:41.656649+0000 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 0 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 1 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:in 2024-04-05T02:59:45.400 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 0 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:45.401 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 msgr2=0x7f9e9806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 0x7f9e9806b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009b80 tx=0x7f9ea400b040 comp rx=0 tx=0).stop 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 msgr2=0x7f9eb010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000c00 tx=0x7f9ea0009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 shutdown_connections 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9e980692a0 0x7f9e9806b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb00fce40 0x7f9eb010b7b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 --2- 172.21.15.153:0/2853453604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb00fe040 0x7f9eb010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:45.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 >> 172.21.15.153:0/2853453604 conn(0x7f9eb00f8510 msgr2=0x7f9eb00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:45.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 shutdown_connections 2024-04-05T02:59:45.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:45.400+0000 7f9eb605b700 1 -- 172.21.15.153:0/2853453604 wait complete. 2024-04-05T02:59:45.403 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 2 2024-04-05T02:59:45.716 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:45 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2853453604' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T02:59:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:45 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2853453604' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T02:59:45.976 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:45.986 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:45.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-05T02:59:46.488 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:46.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:46 smithi203 ceph-mon[39911]: pgmap v168: 65 pgs: 9 creating+peering, 6 unknown, 50 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:46.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:46 smithi153 ceph-mon[29381]: pgmap v168: 65 pgs: 9 creating+peering, 6 unknown, 50 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-05T02:59:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6826/2246788375,v1:172.21.15.153:6827/2246788375] up:boot 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: daemon mds.cephfs.smithi153.vicaky assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: fsmap cephfs:0 1 up:standby 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:creating} 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: daemon mds.cephfs.smithi153.vicaky is now active in filesystem cephfs as rank 0 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:47.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:47.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:47.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:47 smithi203 ceph-mon[39911]: Deploying daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T02:59:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6826/2246788375,v1:172.21.15.153:6827/2246788375] up:boot 2024-04-05T02:59:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: daemon mds.cephfs.smithi153.vicaky assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-05T02:59:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T02:59:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T02:59:47.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: fsmap cephfs:0 1 up:standby 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:creating} 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: daemon mds.cephfs.smithi153.vicaky is now active in filesystem cephfs as rank 0 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:47.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:47 smithi153 ceph-mon[29381]: Deploying daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T02:59:48.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- 172.21.15.153:0/3859135049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 msgr2=0x7f9a54102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/3859135049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a54102360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c004940 tx=0x7f9a3c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- 172.21.15.153:0/3859135049 shutdown_connections 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/3859135049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a54102360 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/3859135049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 0x7f9a540ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- 172.21.15.153:0/3859135049 >> 172.21.15.153:0/3859135049 conn(0x7f9a540f7430 msgr2=0x7f9a540f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- 172.21.15.153:0/3859135049 shutdown_connections 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- 172.21.15.153:0/3859135049 wait complete. 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 Processor -- start 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.093+0000 7f9a58e9c700 1 -- start start 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a58e9c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 0x7f9a5410d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a58e9c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a58e9c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5410e490 con 0x7f9a540fff80 2024-04-05T02:59:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a58e9c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5410e600 con 0x7f9a54066ef0 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57508/0 (socket says 172.21.15.153:57508) 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 -- 172.21.15.153:0/153263886 learned_addr learned my addr 172.21.15.153:0/153263886 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a5259c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 0x7f9a5410d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 -- 172.21.15.153:0/153263886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 msgr2=0x7f9a5410d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 0x7f9a5410d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 -- 172.21.15.153:0/153263886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c005040 con 0x7f9a540fff80 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a49d9b700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00b3b0 tx=0x7f9a3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a3c00e040 con 0x7f9a540fff80 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a3c00bbf0 con 0x7f9a540fff80 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a54113000 con 0x7f9a540fff80 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.094+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a3c01c630 con 0x7f9a540fff80 2024-04-05T02:59:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.095+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a541135c0 con 0x7f9a540fff80 2024-04-05T02:59:48.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.095+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5404c710 con 0x7f9a540fff80 2024-04-05T02:59:48.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.096+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9a3c00bd60 con 0x7f9a540fff80 2024-04-05T02:59:48.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.097+0000 7f9a4b7fe700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 0x7f9a4006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:48.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.097+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9a3c087e40 con 0x7f9a540fff80 2024-04-05T02:59:48.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.097+0000 7f9a5259c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 0x7f9a4006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:48.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.097+0000 7f9a5259c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 0x7f9a4006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9a54066c30 tx=0x7f9a4400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:48.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.101+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9a3c057c00 con 0x7f9a540fff80 2024-04-05T02:59:48.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.332+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f9a54113a00 con 0x7f9a540fff80 2024-04-05T02:59:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.540+0000 7f9a4b7fe700 1 -- 172.21.15.153:0/153263886 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9a3c057790 con 0x7f9a540fff80 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 msgr2=0x7f9a4006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 0x7f9a4006b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9a54066c30 tx=0x7f9a4400b040 comp rx=0 tx=0).stop 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 msgr2=0x7f9a5410de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00b3b0 tx=0x7f9a3c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 shutdown_connections 2024-04-05T02:59:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f9a400692a0 0x7f9a4006b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a54066ef0 0x7f9a5410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 --2- 172.21.15.153:0/153263886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a540fff80 0x7f9a5410de90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 >> 172.21.15.153:0/153263886 conn(0x7f9a540f7430 msgr2=0x7f9a540f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 shutdown_connections 2024-04-05T02:59:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:48.542+0000 7f9a58e9c700 1 -- 172.21.15.153:0/153263886 wait complete. 2024-04-05T02:59:48.683 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:48 smithi153 ceph-mon[29381]: pgmap v169: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-04-05T02:59:48.683 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:48 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6826/2246788375,v1:172.21.15.153:6827/2246788375] up:active 2024-04-05T02:59:48.683 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:48 smithi153 ceph-mon[29381]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 2024-04-05T02:59:48.683 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:48 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/153263886' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-05T02:59:48.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:48 smithi203 ceph-mon[39911]: pgmap v169: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-04-05T02:59:48.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:48 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6826/2246788375,v1:172.21.15.153:6827/2246788375] up:active 2024-04-05T02:59:48.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:48 smithi203 ceph-mon[39911]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 2024-04-05T02:59:48.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:48 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/153263886' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-05T02:59:49.218 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:49.228 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:49.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/153263886' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:49.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:49 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/153263886' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:49.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:49 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:50.136 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: Deploying daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T02:59:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: pgmap v170: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-05T02:59:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:boot 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: daemon mds.cephfs.smithi203.xpteyc assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 1 up:standby 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:creating} 2024-04-05T02:59:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:50 smithi203 ceph-mon[39911]: daemon mds.cephfs.smithi203.xpteyc is now active in filesystem cephfs as rank 1 2024-04-05T02:59:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: Deploying daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T02:59:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: pgmap v170: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:boot 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: daemon mds.cephfs.smithi203.xpteyc assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: fsmap cephfs:1 {0=cephfs.smithi153.vicaky=up:active} 1 up:standby 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:creating} 2024-04-05T02:59:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:50 smithi153 ceph-mon[29381]: daemon mds.cephfs.smithi203.xpteyc is now active in filesystem cephfs as rank 1 2024-04-05T02:59:51.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:51 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:active 2024-04-05T02:59:51.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:51 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T02:59:52.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:51 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:active 2024-04-05T02:59:52.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:51 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T02:59:52.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 -- 172.21.15.153:0/1984148513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b8066240 msgr2=0x7f98b80ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 --2- 172.21.15.153:0/1984148513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b8066240 0x7f98b80ffa60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f98a8004940 tx=0x7f98a8009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 -- 172.21.15.153:0/1984148513 shutdown_connections 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 --2- 172.21.15.153:0/1984148513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80fffa0 0x7f98b8102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 --2- 172.21.15.153:0/1984148513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b8066240 0x7f98b80ffa60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 -- 172.21.15.153:0/1984148513 >> 172.21.15.153:0/1984148513 conn(0x7f98b80f7430 msgr2=0x7f98b80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 -- 172.21.15.153:0/1984148513 shutdown_connections 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.271+0000 7f98be858700 1 -- 172.21.15.153:0/1984148513 wait complete. 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 Processor -- start 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 -- start start 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 0x7f98b810dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b810e6a0 con 0x7f98b80fffa0 2024-04-05T02:59:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98be858700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b810ee40 con 0x7f98b8066240 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41678/0 (socket says 172.21.15.153:41678) 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 -- 172.21.15.153:0/4015121331 learned_addr learned my addr 172.21.15.153:0/4015121331 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b7fff700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 0x7f98b810dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 -- 172.21.15.153:0/4015121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 msgr2=0x7f98b810dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 0x7f98b810dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b77fe700 1 -- 172.21.15.153:0/4015121331 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a8005040 con 0x7f98b80fffa0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.272+0000 7f98b7fff700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 0x7f98b810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98b77fe700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b790 tx=0x7f98ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98ac00f770 con 0x7f98b80fffa0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98ac00fda0 con 0x7f98b80fffa0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b810f120 con 0x7f98b80fffa0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98ac00d310 con 0x7f98b80fffa0 2024-04-05T02:59:52.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.273+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b810f6e0 con 0x7f98b80fffa0 2024-04-05T02:59:52.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.274+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b805f5f0 con 0x7f98b80fffa0 2024-04-05T02:59:52.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.274+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f98ac010420 con 0x7f98b80fffa0 2024-04-05T02:59:52.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.275+0000 7f98b4ff9700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 0x7f98a006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:52.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.275+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f98ac0871b0 con 0x7f98b80fffa0 2024-04-05T02:59:52.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.275+0000 7f98b7fff700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 0x7f98a006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:52.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.275+0000 7f98b7fff700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 0x7f98a006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98a800b970 tx=0x7f98a8009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:52.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.279+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f98ac056f70 con 0x7f98b80fffa0 2024-04-05T02:59:52.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.512+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f98b804bed0 con 0x7f98b80fffa0 2024-04-05T02:59:52.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.557+0000 7f98b4ff9700 1 -- 172.21.15.153:0/4015121331 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7f98b804bed0 con 0x7f98b80fffa0 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 msgr2=0x7f98a006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 0x7f98a006b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98a800b970 tx=0x7f98a8009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 msgr2=0x7f98b810e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b790 tx=0x7f98ac009250 comp rx=0 tx=0).stop 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 shutdown_connections 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f98a00692a0 0x7f98a006b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b8066240 0x7f98b810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 --2- 172.21.15.153:0/4015121331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b80fffa0 0x7f98b810e010 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 >> 172.21.15.153:0/4015121331 conn(0x7f98b80f7430 msgr2=0x7f98b80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 shutdown_connections 2024-04-05T02:59:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:52.560+0000 7f98be858700 1 -- 172.21.15.153:0/4015121331 wait complete. 2024-04-05T02:59:52.689 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:52 smithi153 ceph-mon[29381]: pgmap v171: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-05T02:59:52.690 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:52 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/4015121331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T02:59:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:52 smithi203 ceph-mon[39911]: pgmap v171: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-05T02:59:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:52 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/4015121331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T02:59:53.413 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:53.426 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:53.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-05T02:59:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:53.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/4015121331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:53.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:53 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/4015121331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T02:59:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:53 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: Deploying daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: pgmap v172: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:boot 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby 2024-04-05T02:59:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T02:59:54.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:54 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: Deploying daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: pgmap v172: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 6 op/s 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:boot 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T02:59:54.990 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:54 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 2024-04-05T02:59:55.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- 172.21.15.153:0/2464842647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 msgr2=0x7f87ac0fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:55.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 --2- 172.21.15.153:0/2464842647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac0fc100 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f879c004940 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:55.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- 172.21.15.153:0/2464842647 shutdown_connections 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 --2- 172.21.15.153:0/2464842647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 0x7f87ac0fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 --2- 172.21.15.153:0/2464842647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac0fc100 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- 172.21.15.153:0/2464842647 >> 172.21.15.153:0/2464842647 conn(0x7f87ac0f7450 msgr2=0x7f87ac0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- 172.21.15.153:0/2464842647 shutdown_connections 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- 172.21.15.153:0/2464842647 wait complete. 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 Processor -- start 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- start start 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 0x7f87ac10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac10e600 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.250+0000 7f87b35b9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac10edb0 con 0x7f87ac0fcfa0 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41708/0 (socket says 172.21.15.153:41708) 2024-04-05T02:59:55.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 -- 172.21.15.153:0/337705322 learned_addr learned my addr 172.21.15.153:0/337705322 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b0b54700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 0x7f87ac10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 -- 172.21.15.153:0/337705322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 msgr2=0x7f87ac10e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 0x7f87ac10e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 -- 172.21.15.153:0/337705322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c005040 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b1355700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f879c009150 tx=0x7f879c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c00e040 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f879c00bbf0 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ac10f030 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c01c630 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.251+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ac10f570 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.253+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ac101b30 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.253+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f879c00bd60 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.253+0000 7f87a1ffb700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 0x7f879806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T02:59:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.253+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f879c0884f0 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.254+0000 7f87b0b54700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 0x7f879806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T02:59:55.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.254+0000 7f87b0b54700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 0x7f879806b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87a8004f40 tx=0x7f87a800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T02:59:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.258+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f879c058360 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.494+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f87ac05f5f0 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.570+0000 7f87a1ffb700 1 -- 172.21.15.153:0/337705322 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v13) v1 ==== 133+0+0 (secure 0 0 0) 0x7f879c013e60 con 0x7f87ac0fbcf0 2024-04-05T02:59:55.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.572+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 msgr2=0x7f879806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.572+0000 7f87b35b9700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 0x7f879806b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87a8004f40 tx=0x7f87a800b040 comp rx=0 tx=0).stop 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.572+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 msgr2=0x7f87ac10dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.572+0000 7f87b35b9700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f879c009150 tx=0x7f879c009e30 comp rx=0 tx=0).stop 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 shutdown_connections 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f87980692b0 0x7f879806b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87ac0fbcf0 0x7f87ac10dac0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 --2- 172.21.15.153:0/337705322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87ac0fcfa0 0x7f87ac10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 >> 172.21.15.153:0/337705322 conn(0x7f87ac0f7450 msgr2=0x7f87ac066350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 shutdown_connections 2024-04-05T02:59:55.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T02:59:55.573+0000 7f87b35b9700 1 -- 172.21.15.153:0/337705322 wait complete. 2024-04-05T02:59:55.574 INFO:teuthology.orchestra.run.smithi153.stderr:inline data disabled 2024-04-05T02:59:56.199 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T02:59:56.208 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T02:59:56.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T02:59:56.560 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: pgmap v173: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/337705322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] up:boot 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/337705322' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T02:59:56.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:56 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: pgmap v173: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/337705322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] up:boot 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/337705322' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-05T02:59:56.690 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T02:59:56.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T02:59:56.691 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:56 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T02:59:57.127 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T02:59:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: pgmap v174: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-05T02:59:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:standby-replay 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: Dropping low affinity active daemon mds.cephfs.smithi153.vicaky in favor of higher affinity standby. 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: Replacing daemon mds.cephfs.smithi153.vicaky as rank 0 with standby daemon mds.cephfs.smithi153.fmpmvd 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: osdmap e45: 6 total, 6 up, 6 in 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:replay,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: pgmap v174: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:standby-replay 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: Dropping low affinity active daemon mds.cephfs.smithi153.vicaky in favor of higher affinity standby. 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: Replacing daemon mds.cephfs.smithi153.vicaky as rank 0 with standby daemon mds.cephfs.smithi153.fmpmvd 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: osdmap e45: 6 total, 6 up, 6 in 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:replay,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] up:boot 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:active 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:resolve 2024-04-05T02:59:59.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:resolve,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T02:59:59.614 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T02:59:59.614 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 02:59:59 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] up:boot 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] up:active 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:resolve 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:resolve,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T02:59:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 02:59:59 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.033+0000 7f31c8133700 1 -- 172.21.15.153:0/63723137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c0066b10 msgr2=0x7f31c00ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.033+0000 7f31c8133700 1 --2- 172.21.15.153:0/63723137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c0066b10 0x7f31c00ffab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f31bc004940 tx=0x7f31bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 -- 172.21.15.153:0/63723137 shutdown_connections 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 --2- 172.21.15.153:0/63723137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c00ffff0 0x7f31c01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 --2- 172.21.15.153:0/63723137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c0066b10 0x7f31c00ffab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 -- 172.21.15.153:0/63723137 >> 172.21.15.153:0/63723137 conn(0x7f31c00f74e0 msgr2=0x7f31c00f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 -- 172.21.15.153:0/63723137 shutdown_connections 2024-04-05T03:00:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.034+0000 7f31c8133700 1 -- 172.21.15.153:0/63723137 wait complete. 2024-04-05T03:00:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 Processor -- start 2024-04-05T03:00:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 -- start start 2024-04-05T03:00:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0066b10 0x7f31c010da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c010e5c0 con 0x7f31c00ffff0 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c8133700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c010ee10 con 0x7f31c0066b10 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c56ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c56ce700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47134/0 (socket says 172.21.15.153:47134) 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c56ce700 1 -- 172.21.15.153:0/2046450129 learned_addr learned my addr 172.21.15.153:0/2046450129 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.035+0000 7f31c5ecf700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0066b10 0x7f31c010da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c56ce700 1 -- 172.21.15.153:0/2046450129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0066b10 msgr2=0x7f31c010da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c56ce700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0066b10 0x7f31c010da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c56ce700 1 -- 172.21.15.153:0/2046450129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc005040 con 0x7f31c00ffff0 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c56ce700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c8c0 tx=0x7f31b00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b000a760 con 0x7f31c00ffff0 2024-04-05T03:00:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c010f0f0 con 0x7f31c00ffff0 2024-04-05T03:00:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31b000ad90 con 0x7f31c00ffff0 2024-04-05T03:00:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b001b400 con 0x7f31c00ffff0 2024-04-05T03:00:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.036+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c010f630 con 0x7f31c00ffff0 2024-04-05T03:00:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.037+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c005f5f0 con 0x7f31c00ffff0 2024-04-05T03:00:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.038+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f31b0012400 con 0x7f31c00ffff0 2024-04-05T03:00:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.038+0000 7f31b67fc700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 0x7f31ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.038+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f31b0087230 con 0x7f31c00ffff0 2024-04-05T03:00:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.038+0000 7f31c5ecf700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 0x7f31ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.039+0000 7f31c5ecf700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 0x7f31ac06b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31bc004890 tx=0x7f31bc009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:00.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.047+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f31b0056fa0 con 0x7f31c00ffff0 2024-04-05T03:00:00.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.300+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f31c004bed0 con 0x7f31c00ffff0 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.300+0000 7f31b67fc700 1 -- 172.21.15.153:0/2046450129 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1852 (secure 0 0 0) 0x7f31b0056b30 con 0x7f31c00ffff0 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stdout:e18 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:00:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 18 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T02:59:59.442228+0000 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:00:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:00.305 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:00:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:00.305 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:14518} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:00:00.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 msgr2=0x7f31ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:00.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 0x7f31ac06b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f31bc004890 tx=0x7f31bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 msgr2=0x7f31c010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c8c0 tx=0x7f31b00044c0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 shutdown_connections 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f31ac0692a0 0x7f31ac06b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0066b10 0x7f31c010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 --2- 172.21.15.153:0/2046450129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31c00ffff0 0x7f31c010dfc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.302+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 >> 172.21.15.153:0/2046450129 conn(0x7f31c00f74e0 msgr2=0x7f31c00f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.303+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 shutdown_connections 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:00.303+0000 7f31c8133700 1 -- 172.21.15.153:0/2046450129 wait complete. 2024-04-05T03:00:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 18 2024-04-05T03:00:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: pgmap v176: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-05T03:00:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:reconnect 2024-04-05T03:00:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] up:standby-replay 2024-04-05T03:00:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:reconnect,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-04-05T03:00:00.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:00:00.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: fs cephfs is degraded 2024-04-05T03:00:00.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:00 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2046450129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: pgmap v176: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:reconnect 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] up:standby-replay 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:reconnect,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: Health detail: HEALTH_WARN 1 filesystem is degraded 2024-04-05T03:00:00.770 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:00:00.771 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: fs cephfs is degraded 2024-04-05T03:00:00.771 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:00 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2046450129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:00:01.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-05T03:00:01.546 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:01 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:rejoin 2024-04-05T03:00:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:01 smithi153 ceph-mon[29381]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:rejoin,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:01 smithi153 ceph-mon[29381]: daemon mds.cephfs.smithi153.fmpmvd is now active in filesystem cephfs as rank 0 2024-04-05T03:00:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:01 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:rejoin 2024-04-05T03:00:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:01 smithi203 ceph-mon[39911]: fsmap cephfs:2/2 {0=cephfs.smithi153.fmpmvd=up:rejoin,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:01 smithi203 ceph-mon[39911]: daemon mds.cephfs.smithi153.fmpmvd is now active in filesystem cephfs as rank 0 2024-04-05T03:00:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: pgmap v177: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-05T03:00:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T03:00:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T03:00:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:active 2024-04-05T03:00:02.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:00:02.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:02 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: pgmap v177: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: mds.? [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] up:active 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T03:00:02.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:02.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:02 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 -- 172.21.15.153:0/87319137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fbcd0 msgr2=0x7f20ac0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/87319137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac0fc0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f20a0004940 tx=0x7f20a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 -- 172.21.15.153:0/87319137 shutdown_connections 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/87319137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fcf80 0x7f20ac0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/87319137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac0fc0e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 -- 172.21.15.153:0/87319137 >> 172.21.15.153:0/87319137 conn(0x7f20ac0f7450 msgr2=0x7f20ac0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 -- 172.21.15.153:0/87319137 shutdown_connections 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.070+0000 7f20b2cd2700 1 -- 172.21.15.153:0/87319137 wait complete. 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 Processor -- start 2024-04-05T03:00:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 -- start start 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac10c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac10d380 con 0x7f20ac0fcf80 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.071+0000 7f20b2cd2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac10d4f0 con 0x7f20ac0fbcd0 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47152/0 (socket says 172.21.15.153:47152) 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 -- 172.21.15.153:0/242762965 learned_addr learned my addr 172.21.15.153:0/242762965 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20b0a6e700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac10c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 -- 172.21.15.153:0/242762965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fbcd0 msgr2=0x7f20ac10c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac10c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 -- 172.21.15.153:0/242762965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0005040 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20abfff700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2098002a50 tx=0x7f20980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2098007cc0 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f209800f040 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac111f50 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f209800da70 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.072+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac1124e0 con 0x7f20ac0fcf80 2024-04-05T03:00:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.073+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac04c710 con 0x7f20ac0fcf80 2024-04-05T03:00:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.074+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f209800dc10 con 0x7f20ac0fcf80 2024-04-05T03:00:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.074+0000 7f20a97fa700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 0x7f209c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.074+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2098086830 con 0x7f20ac0fcf80 2024-04-05T03:00:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.074+0000 7f20b0a6e700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 0x7f209c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.075+0000 7f20b0a6e700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 0x7f209c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b970 tx=0x7f20a0009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.079+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f209804d190 con 0x7f20ac0fcf80 2024-04-05T03:00:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.322+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20ac1047e0 con 0x7f20ac0fcf80 2024-04-05T03:00:03.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.322+0000 7f20a97fa700 1 -- 172.21.15.153:0/242762965 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4728 (secure 0 0 0) 0x7f2098056520 con 0x7f20ac0fcf80 2024-04-05T03:00:03.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.324+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 msgr2=0x7f209c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:03.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.324+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 0x7f209c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b970 tx=0x7f20a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.324+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 msgr2=0x7f20ac10cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.324+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2098002a50 tx=0x7f20980046b0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 shutdown_connections 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f209c0692a0 0x7f209c06b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ac0fbcd0 0x7f20ac10c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 --2- 172.21.15.153:0/242762965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20ac0fcf80 0x7f20ac10cd80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 >> 172.21.15.153:0/242762965 conn(0x7f20ac0f7450 msgr2=0x7f20ac0f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 shutdown_connections 2024-04-05T03:00:03.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:03.325+0000 7f20b2cd2700 1 -- 172.21.15.153:0/242762965 wait complete. 2024-04-05T03:00:03.326 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 21 2024-04-05T03:00:03.334 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:00:03.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:03 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/242762965' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:00:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:03 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/242762965' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:00:04.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:04 smithi153 ceph-mon[29381]: pgmap v178: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 614 B/s wr, 7 op/s 2024-04-05T03:00:04.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-05T03:00:04.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:04 smithi203 ceph-mon[39911]: pgmap v178: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 614 B/s wr, 7 op/s 2024-04-05T03:00:05.216 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:05.661 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:05 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:05.661 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:05 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:05.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:05 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:05.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:05 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:06 smithi203 ceph-mon[39911]: pgmap v179: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-05T03:00:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:06 smithi153 ceph-mon[29381]: pgmap v179: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 -- 172.21.15.153:0/2266940407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 msgr2=0x7fa65c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 --2- 172.21.15.153:0/2266940407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0ffa40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa650004940 tx=0x7fa650009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 -- 172.21.15.153:0/2266940407 shutdown_connections 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 --2- 172.21.15.153:0/2266940407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 0x7fa65c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 --2- 172.21.15.153:0/2266940407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0ffa40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 -- 172.21.15.153:0/2266940407 >> 172.21.15.153:0/2266940407 conn(0x7fa65c0f7430 msgr2=0x7fa65c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 -- 172.21.15.153:0/2266940407 shutdown_connections 2024-04-05T03:00:07.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 -- 172.21.15.153:0/2266940407 wait complete. 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.853+0000 7fa662c89700 1 Processor -- start 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa662c89700 1 -- start start 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa662c89700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa662c89700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 0x7fa65c0fb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa662c89700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0fbef0 con 0x7fa65c066260 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa662c89700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0fc060 con 0x7fa65c0fff80 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47180/0 (socket says 172.21.15.153:47180) 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 -- 172.21.15.153:0/203029183 learned_addr learned my addr 172.21.15.153:0/203029183 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa65bfff700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 0x7fa65c0fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 -- 172.21.15.153:0/203029183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 msgr2=0x7fa65c0fb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 0x7fa65c0fb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 -- 172.21.15.153:0/203029183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650005040 con 0x7fa65c066260 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.854+0000 7fa660a25700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b3b0 tx=0x7fa650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:07.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.855+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa65000e040 con 0x7fa65c066260 2024-04-05T03:00:07.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.855+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa65000bbf0 con 0x7fa65c066260 2024-04-05T03:00:07.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.855+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c0fc2e0 con 0x7fa65c066260 2024-04-05T03:00:07.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.855+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa65001c630 con 0x7fa65c066260 2024-04-05T03:00:07.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.855+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c115aa0 con 0x7fa65c066260 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.856+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c05f5f0 con 0x7fa65c066260 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.856+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa65000bd60 con 0x7fa65c066260 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.856+0000 7fa6597fa700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 0x7fa64c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.857+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa650087e80 con 0x7fa65c066260 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.857+0000 7fa65bfff700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 0x7fa64c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:07.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.857+0000 7fa65bfff700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 0x7fa64c06b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6480102e0 tx=0x7fa64800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:07.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:07.861+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa650054160 con 0x7fa65c066260 2024-04-05T03:00:08.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.130+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fa65c04bed0 con 0x7fa65c066260 2024-04-05T03:00:08.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.130+0000 7fa6597fa700 1 -- 172.21.15.153:0/203029183 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v21) v1 ==== 78+0+99 (secure 0 0 0) 0x7fa65c04bed0 con 0x7fa65c066260 2024-04-05T03:00:08.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.133+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 msgr2=0x7fa64c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.133+0000 7fa662c89700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 0x7fa64c06b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6480102e0 tx=0x7fa64800b040 comp rx=0 tx=0).stop 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.133+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 msgr2=0x7fa65c0fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.133+0000 7fa662c89700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b3b0 tx=0x7fa650009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 shutdown_connections 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa64c0692a0 0x7fa64c06b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa65c066260 0x7fa65c0fd470 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 --2- 172.21.15.153:0/203029183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa65c0fff80 0x7fa65c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 >> 172.21.15.153:0/203029183 conn(0x7fa65c0f7430 msgr2=0x7fa65c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 shutdown_connections 2024-04-05T03:00:08.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.134+0000 7fa662c89700 1 -- 172.21.15.153:0/203029183 wait complete. 2024-04-05T03:00:08.146 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:00:08.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:08 smithi153 ceph-mon[29381]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T03:00:08.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:08 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/203029183' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-05T03:00:08.742 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-05T03:00:08.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T03:00:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:08 smithi203 ceph-mon[39911]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T03:00:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:08 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/203029183' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/3379719807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 msgr2=0x7f1c080fac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/3379719807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c080fac50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00049c0 tx=0x7f1bf000aeb0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/3379719807 shutdown_connections 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/3379719807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c080fbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/3379719807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c080fac50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/3379719807 >> 172.21.15.153:0/3379719807 conn(0x7f1c080f6000 msgr2=0x7f1c080f8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/3379719807 shutdown_connections 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.875+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/3379719807 wait complete. 2024-04-05T03:00:08.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 Processor -- start 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 -- start start 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c0810d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0810d8d0 con 0x7f1c080fa840 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0cc6e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08112240 con 0x7f1c080fbb10 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35790/0 (socket says 172.21.15.153:35790) 2024-04-05T03:00:08.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.876+0000 7f1c0659c640 1 -- 172.21.15.153:0/2975364311 learned_addr learned my addr 172.21.15.153:0/2975364311 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c05d9b640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c0810d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0659c640 1 -- 172.21.15.153:0/2975364311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 msgr2=0x7f1c0810d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0659c640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c0810d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0659c640 1 -- 172.21.15.153:0/2975364311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0005040 con 0x7f1c080fa840 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c05d9b640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c0810d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0659c640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0032040 tx=0x7f1bf000aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf000a2b0 con 0x7f1c080fa840 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c08112380 con 0x7f1c080fa840 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bf000a410 con 0x7f1c080fa840 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf003a640 con 0x7f1c080fa840 2024-04-05T03:00:08.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.877+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c081128f0 con 0x7f1c080fa840 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.878+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0804bcc0 con 0x7f1c080fa840 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.879+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1bf00449e0 con 0x7f1c080fa840 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.879+0000 7f1bfeffd640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 0x7f1bf406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.879+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1bf00a9c00 con 0x7f1c080fa840 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.879+0000 7f1c05d9b640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 0x7f1bf406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:08.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.879+0000 7f1c05d9b640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 0x7f1bf406b6c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c0810e1b0 tx=0x7f1c0000b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:08.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:08.884+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1bf00af020 con 0x7f1c080fa840 2024-04-05T03:00:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.073+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c08112b10 con 0x7f1c080fa840 2024-04-05T03:00:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.074+0000 7f1bfeffd640 1 -- 172.21.15.153:0/2975364311 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+14425 (secure 0 0 0) 0x7f1bf0075e60 con 0x7f1c080fa840 2024-04-05T03:00:09.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:09.075 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":45,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T02:59:57.435468+0000","last_up_change":"2024-04-05T02:58:28.256825+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T02:59:40.392548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T02:59:40.884499+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6803","nonce":78196267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6805","nonce":78196267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6809","nonce":78196267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":78196267},{"type":"v1","addr":"172.21.15.153:6807","nonce":78196267}]},"public_addr":"172.21.15.153:6803/78196267","cluster_addr":"172.21.15.153:6805/78196267","heartbeat_back_addr":"172.21.15.153:6809/78196267","heartbeat_front_addr":"172.21.15.153:6807/78196267","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6811","nonce":4220639279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6813","nonce":4220639279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6817","nonce":4220639279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4220639279},{"type":"v1","addr":"172.21.15.153:6815","nonce":4220639279}]},"public_addr":"172.21.15.153:6811/4220639279","cluster_addr":"172.21.15.153:6813/4220639279","heartbeat_back_addr":"172.21.15.153:6817/4220639279","heartbeat_front_addr":"172.21.15.153:6815/4220639279","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6819","nonce":1027037149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6821","nonce":1027037149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6825","nonce":1027037149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1027037149},{"type":"v1","addr":"172.21.15.153:6823","nonce":1027037149}]},"public_addr":"172.21.15.153:6819/1027037149","cluster_addr":"172.21.15.153:6821/1027037149","heartbeat_back_addr":"172.21.15.153:6825/1027037149","heartbeat_front_addr":"172.21.15.153:6823/1027037149","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6801","nonce":4064942773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6803","nonce":4064942773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6807","nonce":4064942773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":4064942773},{"type":"v1","addr":"172.21.15.203:6805","nonce":4064942773}]},"public_addr":"172.21.15.203:6801/4064942773","cluster_addr":"172.21.15.203:6803/4064942773","heartbeat_back_addr":"172.21.15.203:6807/4064942773","heartbeat_front_addr":"172.21.15.203:6805/4064942773","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6809","nonce":519188752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6811","nonce":519188752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6815","nonce":519188752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":519188752},{"type":"v1","addr":"172.21.15.203:6813","nonce":519188752}]},"public_addr":"172.21.15.203:6809/519188752","cluster_addr":"172.21.15.203:6811/519188752","heartbeat_back_addr":"172.21.15.203:6815/519188752","heartbeat_front_addr":"172.21.15.203:6813/519188752","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6817","nonce":2086848603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6819","nonce":2086848603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6823","nonce":2086848603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2086848603},{"type":"v1","addr":"172.21.15.203:6821","nonce":2086848603}]},"public_addr":"172.21.15.203:6817/2086848603","cluster_addr":"172.21.15.203:6819/2086848603","heartbeat_back_addr":"172.21.15.203:6823/2086848603","heartbeat_front_addr":"172.21.15.203:6821/2086848603","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6827/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6826/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T03:00:09.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 msgr2=0x7f1bf406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 0x7f1bf406b6c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c0810e1b0 tx=0x7f1c0000b040 comp rx=0 tx=0).stop 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 msgr2=0x7f1c0810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0032040 tx=0x7f1bf000aeb0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 shutdown_connections 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f1bf4069210 0x7f1bf406b6c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c080fbb10 0x7f1c0810d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 --2- 172.21.15.153:0/2975364311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c080fa840 0x7f1c0810cd10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.076+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 >> 172.21.15.153:0/2975364311 conn(0x7f1c080f6000 msgr2=0x7f1c080f7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.077+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 shutdown_connections 2024-04-05T03:00:09.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.077+0000 7f1c0cc6e640 1 -- 172.21.15.153:0/2975364311 wait complete. 2024-04-05T03:00:09.087 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T02:57:13.688102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T02:59:40.392548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T02:59:40.884499+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-05T03:00:09.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 -- 172.21.15.153:0/1303414389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fc360 msgr2=0x7f604c0fc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 --2- 172.21.15.153:0/1303414389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fc360 0x7f604c0fc7d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f603c0048d0 tx=0x7f603c009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 -- 172.21.15.153:0/1303414389 shutdown_connections 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 --2- 172.21.15.153:0/1303414389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fc360 0x7f604c0fc7d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 --2- 172.21.15.153:0/1303414389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f604c0fb090 0x7f604c0fb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 -- 172.21.15.153:0/1303414389 >> 172.21.15.153:0/1303414389 conn(0x7f604c0f6870 msgr2=0x7f604c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 -- 172.21.15.153:0/1303414389 shutdown_connections 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.189+0000 7f605359d640 1 -- 172.21.15.153:0/1303414389 wait complete. 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 Processor -- start 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 -- start start 2024-04-05T03:00:09.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f604c0fc360 0x7f604c10c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c10c700 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c111100 con 0x7f604c0fc360 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35804/0 (socket says 172.21.15.153:35804) 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 -- 172.21.15.153:0/3957332903 learned_addr learned my addr 172.21.15.153:0/3957332903 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 -- 172.21.15.153:0/3957332903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f604c0fc360 msgr2=0x7f604c10c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f604c0fc360 0x7f604c10c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 -- 172.21.15.153:0/3957332903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c005040 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6051339640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f604800c830 tx=0x7f6048004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f604800a850 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f604800ae80 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6048012ea0 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.190+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c1112a0 con 0x7f604c0fb090 2024-04-05T03:00:09.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.191+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c111790 con 0x7f604c0fb090 2024-04-05T03:00:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.192+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c103be0 con 0x7f604c0fb090 2024-04-05T03:00:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.192+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6048012400 con 0x7f604c0fb090 2024-04-05T03:00:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.192+0000 7f6041ffb640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 0x7f603806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.192+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6048087450 con 0x7f604c0fb090 2024-04-05T03:00:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.192+0000 7f6050b38640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 0x7f603806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.193+0000 7f6050b38640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 0x7f603806b6c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f603c009040 tx=0x7f603c009da0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.197+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6048053730 con 0x7f604c0fb090 2024-04-05T03:00:09.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.363+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f604c02a980 con 0x7f604c0fb090 2024-04-05T03:00:09.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.363+0000 7f6041ffb640 1 -- 172.21.15.153:0/3957332903 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60480126b0 con 0x7f604c0fb090 2024-04-05T03:00:09.363 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-04-05T03:00:09.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 msgr2=0x7f603806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 0x7f603806b6c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f603c009040 tx=0x7f603c009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 msgr2=0x7f604c10bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f604800c830 tx=0x7f6048004430 comp rx=0 tx=0).stop 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 shutdown_connections 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f6038069210 0x7f603806b6c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f604c0fc360 0x7f604c10c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 --2- 172.21.15.153:0/3957332903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f604c0fb090 0x7f604c10bb70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 >> 172.21.15.153:0/3957332903 conn(0x7f604c0f6870 msgr2=0x7f604c0ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 shutdown_connections 2024-04-05T03:00:09.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.365+0000 7f605359d640 1 -- 172.21.15.153:0/3957332903 wait complete. 2024-04-05T03:00:09.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T03:00:09.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 -- 172.21.15.153:0/3079098943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 msgr2=0x7f02880fb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 --2- 172.21.15.153:0/3079098943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f02880fb4a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f02780048d0 tx=0x7f0278009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 -- 172.21.15.153:0/3079098943 shutdown_connections 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 --2- 172.21.15.153:0/3079098943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 0x7f02880fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 --2- 172.21.15.153:0/3079098943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f02880fb4a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 -- 172.21.15.153:0/3079098943 >> 172.21.15.153:0/3079098943 conn(0x7f02880f6870 msgr2=0x7f02880f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 -- 172.21.15.153:0/3079098943 shutdown_connections 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 -- 172.21.15.153:0/3079098943 wait complete. 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.477+0000 7f028d2ae640 1 Processor -- start 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 -- start start 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 0x7f0288111ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288112020 con 0x7f02880fb090 2024-04-05T03:00:09.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288112160 con 0x7f02880fc360 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35828/0 (socket says 172.21.15.153:35828) 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 -- 172.21.15.153:0/1731335886 learned_addr learned my addr 172.21.15.153:0/1731335886 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f027e7fc640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 0x7f0288111ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 -- 172.21.15.153:0/1731335886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 msgr2=0x7f0288111ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 0x7f0288111ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 -- 172.21.15.153:0/1731335886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278005040 con 0x7f02880fb090 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f0286ffd640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0278007da0 tx=0x7f0278009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02780094c0 con 0x7f02880fb090 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0278009620 con 0x7f02880fb090 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.478+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0288112360 con 0x7f02880fb090 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.479+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f027803b620 con 0x7f02880fb090 2024-04-05T03:00:09.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.479+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0288112880 con 0x7f02880fb090 2024-04-05T03:00:09.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.480+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0288105590 con 0x7f02880fb090 2024-04-05T03:00:09.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.480+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f027802f070 con 0x7f02880fb090 2024-04-05T03:00:09.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.480+0000 7f027ffff640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 0x7f027406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.480+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f02780a9810 con 0x7f02880fb090 2024-04-05T03:00:09.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.480+0000 7f027e7fc640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 0x7f027406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.481+0000 7f027e7fc640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 0x7f027406b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0270007fd0 tx=0x7f0270009e90 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.485+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0278075a70 con 0x7f02880fb090 2024-04-05T03:00:09.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.652+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0288103c00 con 0x7f02880fb090 2024-04-05T03:00:09.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.652+0000 7f027ffff640 1 -- 172.21.15.153:0/1731335886 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0278079090 con 0x7f02880fb090 2024-04-05T03:00:09.652 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:00:09.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 msgr2=0x7f027406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 0x7f027406b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0270007fd0 tx=0x7f0270009e90 comp rx=0 tx=0).stop 2024-04-05T03:00:09.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 msgr2=0x7f0288064dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0278007da0 tx=0x7f0278009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 shutdown_connections 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f0274069210 0x7f027406b6c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02880fc360 0x7f0288111ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 --2- 172.21.15.153:0/1731335886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02880fb090 0x7f0288064dd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 >> 172.21.15.153:0/1731335886 conn(0x7f02880f6870 msgr2=0x7f02880ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 shutdown_connections 2024-04-05T03:00:09.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.654+0000 7f028d2ae640 1 -- 172.21.15.153:0/1731335886 wait complete. 2024-04-05T03:00:09.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T03:00:09.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 -- 172.21.15.153:0/2978452426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2870064fb0 msgr2=0x7f28700fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 --2- 172.21.15.153:0/2978452426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2870064fb0 0x7f28700fee20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f28600048d0 tx=0x7f2860009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 -- 172.21.15.153:0/2978452426 shutdown_connections 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 --2- 172.21.15.153:0/2978452426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28700ff360 0x7f2870101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 --2- 172.21.15.153:0/2978452426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2870064fb0 0x7f28700fee20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.765+0000 7f2876cae640 1 -- 172.21.15.153:0/2978452426 >> 172.21.15.153:0/2978452426 conn(0x7f28700f6870 msgr2=0x7f28700f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 -- 172.21.15.153:0/2978452426 shutdown_connections 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 -- 172.21.15.153:0/2978452426 wait complete. 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 Processor -- start 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 -- start start 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2870064fb0 0x7f287010cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287010d8c0 con 0x7f28700ff360 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f2876cae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870112280 con 0x7f2870064fb0 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f286ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f286ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35854/0 (socket says 172.21.15.153:35854) 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.766+0000 7f286ffff640 1 -- 172.21.15.153:0/1028854517 learned_addr learned my addr 172.21.15.153:0/1028854517 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:09.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f2874a4a640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2870064fb0 0x7f287010cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286ffff640 1 -- 172.21.15.153:0/1028854517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2870064fb0 msgr2=0x7f287010cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286ffff640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2870064fb0 0x7f287010cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286ffff640 1 -- 172.21.15.153:0/1028854517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860005040 con 0x7f28700ff360 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286ffff640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f286400c800 tx=0x7f2864004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286400a820 con 0x7f28700ff360 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2870112420 con 0x7f28700ff360 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f286400ae50 con 0x7f28700ff360 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28640115e0 con 0x7f28700ff360 2024-04-05T03:00:09.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.767+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28701128f0 con 0x7f28700ff360 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.768+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287004bcc0 con 0x7f28700ff360 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.769+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2864012400 con 0x7f28700ff360 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.769+0000 7f286d7fa640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 0x7f285806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.769+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2864087380 con 0x7f28700ff360 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.769+0000 7f2874a4a640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 0x7f285806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:09.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.769+0000 7f2874a4a640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 0x7f285806b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f287010e1f0 tx=0x7f2860009da0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:09.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.773+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2864053660 con 0x7f28700ff360 2024-04-05T03:00:09.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.937+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2870104d10 con 0x7f28700ff360 2024-04-05T03:00:09.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.938+0000 7f286d7fa640 1 -- 172.21.15.153:0/1028854517 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v45) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2864056c80 con 0x7f28700ff360 2024-04-05T03:00:09.938 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:00:09.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.939+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 msgr2=0x7f285806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.939+0000 7f2876cae640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 0x7f285806b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f287010e1f0 tx=0x7f2860009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.939+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 msgr2=0x7f287010d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.939+0000 7f2876cae640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f286400c800 tx=0x7f2864004430 comp rx=0 tx=0).stop 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 shutdown_connections 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2858069210 0x7f285806b6c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28700ff360 0x7f287010d270 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 --2- 172.21.15.153:0/1028854517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2870064fb0 0x7f287010cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 >> 172.21.15.153:0/1028854517 conn(0x7f28700f6870 msgr2=0x7f28700f6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 shutdown_connections 2024-04-05T03:00:09.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:09.940+0000 7f2876cae640 1 -- 172.21.15.153:0/1028854517 wait complete. 2024-04-05T03:00:09.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T03:00:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.052+0000 7f4bf146e640 1 -- 172.21.15.153:0/1007279379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec065da0 msgr2=0x7f4bec0fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.052+0000 7f4bf146e640 1 --2- 172.21.15.153:0/1007279379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec065da0 0x7f4bec0fee20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4be00048d0 tx=0x7f4be0009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.052+0000 7f4bf146e640 1 -- 172.21.15.153:0/1007279379 shutdown_connections 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 --2- 172.21.15.153:0/1007279379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec0ff360 0x7f4bec101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 --2- 172.21.15.153:0/1007279379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec065da0 0x7f4bec0fee20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- 172.21.15.153:0/1007279379 >> 172.21.15.153:0/1007279379 conn(0x7f4bec0f6890 msgr2=0x7f4bec0f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- 172.21.15.153:0/1007279379 shutdown_connections 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- 172.21.15.153:0/1007279379 wait complete. 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 Processor -- start 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- start start 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec065da0 0x7f4bec0fc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec0fcca0 con 0x7f4bec0ff360 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.053+0000 7f4bf146e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec0fb260 con 0x7f4bec065da0 2024-04-05T03:00:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35880/0 (socket says 172.21.15.153:35880) 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 -- 172.21.15.153:0/2477454747 learned_addr learned my addr 172.21.15.153:0/2477454747 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4beaffd640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec065da0 0x7f4bec0fc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 -- 172.21.15.153:0/2477454747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec065da0 msgr2=0x7f4bec0fc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec065da0 0x7f4bec0fc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 -- 172.21.15.153:0/2477454747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be0005040 con 0x7f4bec0ff360 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bea7fc640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00eb70 tx=0x7f4bdc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bdc007c30 con 0x7f4bec0ff360 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bdc00f040 con 0x7f4bec0ff360 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec0fb4e0 con 0x7f4bec0ff360 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bdc0095b0 con 0x7f4bec0ff360 2024-04-05T03:00:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.054+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec0fba00 con 0x7f4bec0ff360 2024-04-05T03:00:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.055+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec04bcc0 con 0x7f4bec0ff360 2024-04-05T03:00:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.056+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4bdc009730 con 0x7f4bec0ff360 2024-04-05T03:00:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.056+0000 7f4bd37fe640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 0x7f4bd406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.056+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4bdc086ec0 con 0x7f4bec0ff360 2024-04-05T03:00:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.056+0000 7f4beaffd640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 0x7f4bd406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.057+0000 7f4beaffd640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 0x7f4bd406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4be0030930 tx=0x7f4be0009da0 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.061+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4bdc0531d0 con 0x7f4bec0ff360 2024-04-05T03:00:10.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.264+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bec104d30 con 0x7f4bec0ff360 2024-04-05T03:00:10.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.264+0000 7f4bd37fe640 1 -- 172.21.15.153:0/2477454747 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4728 (secure 0 0 0) 0x7f4bdc0567f0 con 0x7f4bec0ff360 2024-04-05T03:00:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:00:10.264 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:00:01.459558+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14518":{"gid":14518,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/1440749410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1440749410},{"type":"v1","addr":"172.21.15.153:6827","nonce":1440749410}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.203:6827/215598905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":215598905},{"type":"v1","addr":"172.21.15.203:6827","nonce":215598905}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-05T03:00:10.265 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 21 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.266+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 msgr2=0x7f4bd406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.266+0000 7f4bf146e640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 0x7f4bd406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4be0030930 tx=0x7f4be0009da0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.266+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 msgr2=0x7f4bec0fad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.266+0000 7f4bf146e640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00eb70 tx=0x7f4bdc00b040 comp rx=0 tx=0).stop 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 shutdown_connections 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4bd4069210 0x7f4bd406b6c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bec0ff360 0x7f4bec0fad20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 --2- 172.21.15.153:0/2477454747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bec065da0 0x7f4bec0fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 >> 172.21.15.153:0/2477454747 conn(0x7f4bec0f6890 msgr2=0x7f4bec0f6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 shutdown_connections 2024-04-05T03:00:10.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:10.267+0000 7f4bf146e640 1 -- 172.21.15.153:0/2477454747 wait complete. 2024-04-05T03:00:10.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:00:10.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:00:10.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2975364311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:00:10.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3957332903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-05T03:00:10.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1731335886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:09 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1028854517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.278 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 21, 'max_mds': 2, 'flags': 50} 2024-04-05T03:00:10.278 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-05T03:00:10.304 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-05T03:00:10.305 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-05T03:00:10.305 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-05T03:00:10.305 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-05T03:00:10.305 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:10.305 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-05T03:00:10.305 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-05T03:00:10.305 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:10.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:10.306 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-04-05T03:00:10.306 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.306 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.306 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:00:10.306 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2975364311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3957332903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1731335886' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.307 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:09 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1028854517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:00:10.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:10.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-04-05T03:00:10.391 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-04-05T03:00:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:00:10.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:10.393 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-04-05T03:00:10.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:10.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link delete ceph-brx 2024-04-05T03:00:10.489 INFO:teuthology.orchestra.run.smithi203.stderr:Cannot find device "ceph-brx" 2024-04-05T03:00:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:00:10.491 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-05T03:00:10.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-05T03:00:10.750 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:11.238 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:10 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2477454747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:00:11.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:10 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2477454747' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:00:12.153 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:11 smithi153 ceph-mon[29381]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-05T03:00:12.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:11 smithi203 ceph-mon[39911]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 -- 172.21.15.153:0/2991886087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea601026c0 msgr2=0x7fea600fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 --2- 172.21.15.153:0/2991886087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea601026c0 0x7fea600fbd30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fea54004940 tx=0x7fea54009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 -- 172.21.15.153:0/2991886087 shutdown_connections 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 --2- 172.21.15.153:0/2991886087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea601026c0 0x7fea600fbd30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 --2- 172.21.15.153:0/2991886087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea60101d20 0x7fea601020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 -- 172.21.15.153:0/2991886087 >> 172.21.15.153:0/2991886087 conn(0x7fea600f7870 msgr2=0x7fea600f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 -- 172.21.15.153:0/2991886087 shutdown_connections 2024-04-05T03:00:12.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 -- 172.21.15.153:0/2991886087 wait complete. 2024-04-05T03:00:12.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.386+0000 7fea663a4700 1 Processor -- start 2024-04-05T03:00:12.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea663a4700 1 -- start start 2024-04-05T03:00:12.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea663a4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:12.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea663a4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea601026c0 0x7fea6010f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:12.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea663a4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6010f5e0 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea663a4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6010f720 con 0x7fea601026c0 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35892/0 (socket says 172.21.15.153:35892) 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 -- 172.21.15.153:0/2635524156 learned_addr learned my addr 172.21.15.153:0/2635524156 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5f7fe700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea601026c0 0x7fea6010f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 -- 172.21.15.153:0/2635524156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea601026c0 msgr2=0x7fea6010f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea601026c0 0x7fea6010f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.387+0000 7fea5ffff700 1 -- 172.21.15.153:0/2635524156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea54005040 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea5ffff700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fea50002a50 tx=0x7fea500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea50007cc0 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea60117bd0 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea5000f040 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea5000da30 con 0x7fea60101d20 2024-04-05T03:00:12.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.388+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea60117f20 con 0x7fea60101d20 2024-04-05T03:00:12.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.389+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea600fd470 con 0x7fea60101d20 2024-04-05T03:00:12.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.389+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fea5000db90 con 0x7fea60101d20 2024-04-05T03:00:12.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.389+0000 7fea5cff9700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 0x7fea4806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:12.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.390+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fea50086f70 con 0x7fea60101d20 2024-04-05T03:00:12.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.390+0000 7fea5f7fe700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 0x7fea4806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:12.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.390+0000 7fea5f7fe700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 0x7fea4806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea54004890 tx=0x7fea54009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:12.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.394+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fea500531d0 con 0x7fea60101d20 2024-04-05T03:00:12.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.624+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fea60066860 con 0x7fea60101d20 2024-04-05T03:00:12.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.624+0000 7fea5cff9700 1 -- 172.21.15.153:0/2635524156 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+83 (secure 0 0 0) 0x7fea500567f0 con 0x7fea60101d20 2024-04-05T03:00:12.624 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-05T03:00:12.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 msgr2=0x7fea4806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:12.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 0x7fea4806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea54004890 tx=0x7fea54009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:12.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 msgr2=0x7fea6010eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:12.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fea50002a50 tx=0x7fea500046b0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 shutdown_connections 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fea480692a0 0x7fea4806b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea60101d20 0x7fea6010eb60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 --2- 172.21.15.153:0/2635524156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea601026c0 0x7fea6010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.626+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 >> 172.21.15.153:0/2635524156 conn(0x7fea600f7870 msgr2=0x7fea600f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.627+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 shutdown_connections 2024-04-05T03:00:12.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:12.627+0000 7fea663a4700 1 -- 172.21.15.153:0/2635524156 wait complete. 2024-04-05T03:00:13.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:12 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2635524156' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T03:00:13.248 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-05T03:00:13.249 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-05T03:00:13.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:13.249 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-04-05T03:00:13.265 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-05T03:00:13.265 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-05T03:00:13.265 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-04-05T03:00:13.265 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-05T03:00:13.266 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42134sec preferred_lft 42134sec 2024-04-05T03:00:13.267 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-04-05T03:00:13.267 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:13.267 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add name ceph-brx type bridge 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr flush dev ceph-brx 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set ceph-brx up 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-05T03:00:13.267 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-05T03:00:13.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:12 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2635524156' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T03:00:13.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:13.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:13.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:13.482 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-05T03:00:13.511 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-04-05T03:00:13.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:13.512 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-04-05T03:00:13.583 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-04-05T03:00:13.583 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-05T03:00:13.583 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-05T03:00:13.583 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-05T03:00:13.583 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-05T03:00:13.584 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-05T03:00:13.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:13.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:13.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:13.917 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-04-05T03:00:13.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:13.932 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-04-05T03:00:13.987 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:13.987 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-05T03:00:13.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T03:00:13.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-05T03:00:13.987 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-05T03:00:14.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:14.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:14.113 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-05T03:00:14.113 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-05T03:00:14.114 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-05T03:00:14.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:14.255 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:14.255 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-04-05T03:00:14.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.0 up 2024-04-05T03:00:14.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.0 master ceph-brx 2024-04-05T03:00:14.255 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-04-05T03:00:14.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:14.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:14.338 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-05T03:00:14.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T03:00:14.338 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:14.354 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-05T03:00:14.355 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T03:00:14.355 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:14.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe fuse 2024-04-05T03:00:14.480 DEBUG:teuthology.orchestra.run.smithi153:> cat /proc/self/mounts | awk '{print $2}' 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/proc 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/sys 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/dev 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/security 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/shm 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/pts 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/run 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/cgroup 2024-04-05T03:00:14.537 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/pstore 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/bpf 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/ 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/selinux 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/hugepages 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/mqueue 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/debug 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/tracing 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/fuse/connections 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-sysctl.service 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/config 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-05T03:00:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/nfs/rpc_pipefs 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/1000 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/ceph 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/ba73080405ea5fb9e23fc030972fffb03f72136df3bf05eaeca983045a40204f/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/c8acffac32ce2b58c92837be72fe7f7c5107a7bfc3b74e453985ea94f7756ba2/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/0 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/82fb2897f771471e5badf3acc99d275a8abd4cba3da4e211086d3c0ef5f7b6d5/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/2d083276a1791e6b7081ee65aaa0cc525fb64803de78a9e1363f5ab4f8d3e844/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/4a6742ee1f064cc5db5e7a3d611362506d49ede0b3886358abdbb81a1d79fdcf/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/f73634a5eed0726ff5700ae71cdf132b4caa2e68a202a187b71210babe135529/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/aa2309aa9a5ad9886c263d68d0d23d538ea05a9766b96f9c18564b6f20263461/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/f8c410357544d2d962c1b259e0361508771864679623df204b63be1223fe8997/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/4bba2cab9ebd0d5169f000de8d6b18c0ca9a822140f9b19b7a4095c5c77d1a20/merged 2024-04-05T03:00:14.539 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/97cf93917eb17a46c8aa98245691913c70b8b09a466048fb624c8014bf0f3185/merged 2024-04-05T03:00:14.540 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/fc81a720510cad109c6bdf0a1a1137495b6d4b2a359098dd3ce81a6b85bf05a6/merged 2024-04-05T03:00:14.540 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns 2024-04-05T03:00:14.540 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T03:00:14.540 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T03:00:14.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:14.540 DEBUG:teuthology.orchestra.run.smithi153:> ls /sys/fs/fuse/connections 2024-04-05T03:00:14.593 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-05T03:00:14.593 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-05T03:00:14.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo modprobe fuse 2024-04-05T03:00:14.661 DEBUG:teuthology.orchestra.run.smithi153:> cat /proc/self/mounts | awk '{print $2}' 2024-04-05T03:00:14.717 INFO:teuthology.orchestra.run.smithi153.stdout:/proc 2024-04-05T03:00:14.717 INFO:teuthology.orchestra.run.smithi153.stdout:/sys 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/dev 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/security 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/shm 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/pts 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/run 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/cgroup 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/pstore 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/bpf 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/ 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/selinux 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/hugepages 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/dev/mqueue 2024-04-05T03:00:14.718 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/debug 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/tracing 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/fs/fuse/connections 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-sysctl.service 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/sys/kernel/config 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/nfs/rpc_pipefs 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/1000 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/ceph 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/ba73080405ea5fb9e23fc030972fffb03f72136df3bf05eaeca983045a40204f/merged 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/c8acffac32ce2b58c92837be72fe7f7c5107a7bfc3b74e453985ea94f7756ba2/merged 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/run/user/0 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/82fb2897f771471e5badf3acc99d275a8abd4cba3da4e211086d3c0ef5f7b6d5/merged 2024-04-05T03:00:14.719 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/2d083276a1791e6b7081ee65aaa0cc525fb64803de78a9e1363f5ab4f8d3e844/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/4a6742ee1f064cc5db5e7a3d611362506d49ede0b3886358abdbb81a1d79fdcf/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/f73634a5eed0726ff5700ae71cdf132b4caa2e68a202a187b71210babe135529/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/aa2309aa9a5ad9886c263d68d0d23d538ea05a9766b96f9c18564b6f20263461/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/f8c410357544d2d962c1b259e0361508771864679623df204b63be1223fe8997/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/4bba2cab9ebd0d5169f000de8d6b18c0ca9a822140f9b19b7a4095c5c77d1a20/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/97cf93917eb17a46c8aa98245691913c70b8b09a466048fb624c8014bf0f3185/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/var/lib/containers/storage/overlay/fc81a720510cad109c6bdf0a1a1137495b6d4b2a359098dd3ce81a6b85bf05a6/merged 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run.smithi153.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T03:00:14.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:14.720 DEBUG:teuthology.orchestra.run.smithi153:> ls /sys/fs/fuse/connections 2024-04-05T03:00:14.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi153.stderr:2024-04-05T03:00:14.734+0000 7f833de37300 -1 init, newargv = 0x55a5540db870 newargc=15 2024-04-05T03:00:14.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi153.stderr:ceph-fuse[75258]: starting ceph client 2024-04-05T03:00:14.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi153.stderr:ceph-fuse[75258]: starting fuse 2024-04-05T03:00:14.749 INFO:teuthology.orchestra.run.smithi153.stdout:126 2024-04-05T03:00:14.749 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [126] 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo stdin-killer -- python3 -c ' 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> import glob 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> import re 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> import os 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> import subprocess 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-05T03:00:14.749 DEBUG:teuthology.orchestra.run.smithi153:> def _find_admin_socket(client_name): 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> files = glob.glob(asok_path) 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> # Given a non-glob path, it better be there 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> if "*" not in asok_path: 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> assert(len(files) == 1) 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> return files[0] 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> for f in files: 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> contents = proc_f.read() 2024-04-05T03:00:14.750 DEBUG:teuthology.orchestra.run.smithi153:> if mountpoint in contents: 2024-04-05T03:00:14.751 DEBUG:teuthology.orchestra.run.smithi153:> return f 2024-04-05T03:00:14.751 DEBUG:teuthology.orchestra.run.smithi153:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-05T03:00:14.751 DEBUG:teuthology.orchestra.run.smithi153:> 2024-04-05T03:00:14.751 DEBUG:teuthology.orchestra.run.smithi153:> print(_find_admin_socket("client.0")) 2024-04-05T03:00:14.751 DEBUG:teuthology.orchestra.run.smithi153:> ' 2024-04-05T03:00:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:14 smithi203 ceph-mon[39911]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T03:00:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:14 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:14 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:14.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:14 smithi153 ceph-mon[29381]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T03:00:14.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:00:14.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:14 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:14.870 INFO:teuthology.orchestra.run.smithi153.stdout:/var/run/ceph/ceph-client.0.75258.asok 2024-04-05T03:00:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:14.880 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75258.asok 2024-04-05T03:00:14.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:14.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75258.asok status 2024-04-05T03:00:15.012 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "metadata": { 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_sha1": "93c790e135cc94d629f832b908e3f6c70360bfd5", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.7-815-g93c790e1 (93c790e135cc94d629f832b908e3f6c70360bfd5) quincy (stable)", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "entity_id": "0", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "hostname": "smithi153", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "pid": "75258", 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "root": "/" 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "dentry_count": 0, 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "dentry_pinned_count": 0, 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "id": 14558, 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "inst": { 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "name": { 2024-04-05T03:00:15.013 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "client", 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 14558 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": { 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "192.168.0.1:0", 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 761609335 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": { 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "192.168.0.1:0", 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 761609335 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:00:15.014 INFO:teuthology.orchestra.run.smithi153.stdout: "inst_str": "client.14558 192.168.0.1:0/761609335", 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "addr_str": "192.168.0.1:0/761609335", 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "inode_count": 1, 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_epoch": 21, 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_epoch": 45, 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_epoch_barrier": 0, 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "blocklisted": false, 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout: "fs_name": "cephfs" 2024-04-05T03:00:15.015 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:00:15.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-05T03:00:15.275 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:16.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:16 smithi153 ceph-mon[29381]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-05T03:00:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:16 smithi203 ceph-mon[39911]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 -- 172.21.15.153:0/2717967643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fd1d0 msgr2=0x7f27f40fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/2717967643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fd1d0 0x7f27f40fd5a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f27e4004940 tx=0x7f27e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 -- 172.21.15.153:0/2717967643 shutdown_connections 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/2717967643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fe9d0 0x7f27f40fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/2717967643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fd1d0 0x7f27f40fd5a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 -- 172.21.15.153:0/2717967643 >> 172.21.15.153:0/2717967643 conn(0x7f27f40f89b0 msgr2=0x7f27f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 -- 172.21.15.153:0/2717967643 shutdown_connections 2024-04-05T03:00:16.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 -- 172.21.15.153:0/2717967643 wait complete. 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.801+0000 7f27fa1fb700 1 Processor -- start 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27fa1fb700 1 -- start start 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27fa1fb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fd1d0 0x7f27f41138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27fa1fb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27fa1fb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4114450 con 0x7f27f40fe9d0 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27fa1fb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410da50 con 0x7f27f40fd1d0 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35908/0 (socket says 172.21.15.153:35908) 2024-04-05T03:00:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 -- 172.21.15.153:0/3301597584 learned_addr learned my addr 172.21.15.153:0/3301597584 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27f37fe700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fd1d0 0x7f27f41138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 -- 172.21.15.153:0/3301597584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fd1d0 msgr2=0x7f27f41138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fd1d0 0x7f27f41138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.802+0000 7f27eaffd700 1 -- 172.21.15.153:0/3301597584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e4005040 con 0x7f27f40fe9d0 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27eaffd700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f27dc002a80 tx=0x7f27dc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27dc007cc0 con 0x7f27f40fe9d0 2024-04-05T03:00:16.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27dc00f040 con 0x7f27f40fe9d0 2024-04-05T03:00:16.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27dc00dac0 con 0x7f27f40fe9d0 2024-04-05T03:00:16.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f410dd30 con 0x7f27f40fe9d0 2024-04-05T03:00:16.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.803+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f410e2f0 con 0x7f27f40fe9d0 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.804+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40fe570 con 0x7f27f40fe9d0 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.804+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f27dc00dc20 con 0x7f27f40fe9d0 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.804+0000 7f27f0ff9700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 0x7f27e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.805+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f27dc086e00 con 0x7f27f40fe9d0 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.805+0000 7f27f37fe700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 0x7f27e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:16.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.805+0000 7f27f37fe700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 0x7f27e006b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f27e4004f90 tx=0x7f27e4009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:16.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:16.809+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f27dc056ad0 con 0x7f27f40fe9d0 2024-04-05T03:00:17.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.039+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f27f410ef10 con 0x7f27f40fe9d0 2024-04-05T03:00:17.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.039+0000 7f27f0ff9700 1 -- 172.21.15.153:0/3301597584 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+83 (secure 0 0 0) 0x7f27dc056660 con 0x7f27f40fe9d0 2024-04-05T03:00:17.040 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-05T03:00:17.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 msgr2=0x7f27e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 0x7f27e006b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f27e4004f90 tx=0x7f27e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 msgr2=0x7f27f4113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f27dc002a80 tx=0x7f27dc0046b0 comp rx=0 tx=0).stop 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 shutdown_connections 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e00692a0 0x7f27e006b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f40fd1d0 0x7f27f41138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 --2- 172.21.15.153:0/3301597584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f40fe9d0 0x7f27f4113e00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.041+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 >> 172.21.15.153:0/3301597584 conn(0x7f27f40f89b0 msgr2=0x7f27f40f9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.042+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 shutdown_connections 2024-04-05T03:00:17.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:17.042+0000 7f27fa1fb700 1 -- 172.21.15.153:0/3301597584 wait complete. 2024-04-05T03:00:17.689 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:17 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3301597584' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T03:00:17.714 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-05T03:00:17.715 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-05T03:00:17.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:17.715 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2024-04-05T03:00:17.734 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42133sec preferred_lft 42133sec 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2024-04-05T03:00:17.735 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2024-04-05T03:00:17.736 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add name ceph-brx type bridge 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr flush dev ceph-brx 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set ceph-brx up 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-05T03:00:17.736 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-04-05T03:00:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:17 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3301597584' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T03:00:17.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:17.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:17.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:17.953 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-05T03:00:18.022 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-04-05T03:00:18.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:18.024 DEBUG:teuthology.orchestra.run.smithi203:> route 2024-04-05T03:00:18.095 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2024-04-05T03:00:18.096 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-05T03:00:18.096 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-05T03:00:18.096 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-05T03:00:18.096 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-05T03:00:18.096 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-04-05T03:00:18.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:18.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:18.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:18.397 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2024-04-05T03:00:18.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:18.412 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list-id 2024-04-05T03:00:18.468 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:18.468 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-04-05T03:00:18.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T03:00:18.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-05T03:00:18.468 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-04-05T03:00:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:18.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:18.591 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-05T03:00:18.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-05T03:00:18.592 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-04-05T03:00:18.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:18.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:18 smithi203 ceph-mon[39911]: pgmap v185: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-05T03:00:18.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:18.733 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-05T03:00:18.733 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2024-04-05T03:00:18.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.0 up 2024-04-05T03:00:18.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set dev brx.0 master ceph-brx 2024-04-05T03:00:18.733 DEBUG:teuthology.orchestra.run.smithi203:> ') 2024-04-05T03:00:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-05T03:00:18.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:18.815 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-05T03:00:18.815 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T03:00:18.815 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:18 smithi153 ceph-mon[29381]: pgmap v185: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-05T03:00:18.841 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-05T03:00:18.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T03:00:18.842 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:18.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-04-05T03:00:18.965 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-04-05T03:00:19.022 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-05T03:00:19.023 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/0 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/e198afea19dc27d39aee81aeaeba5bf15266a5f90e414168d1d098728c4db24e/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/42636ea68e59cb7d46d47ffe0518391e596d4e80904f2530344f859f5ee147fb/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/a58770cf63c59faf654c5092353c9e786d56002a19c0987e7f3668ef24938056/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/a1215fcfa1a3f357a00818ef589f3f40b8d0efeea64b7e03b07b298015f26c6a/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/e71b9387a5868fb7883c8de698b7b4b581cf5e3154d6a8631d43b65e98392e75/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/05b63df97ea1c94df7ed623ff7baaca7d4881b67c9e89f57d1302f4a5d0e8431/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/38cdc7e7cb2c7573bde6f3e4cebc1791eb29acb08f176e64b5c8a46cc4c8c623/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/3716c2cda95d032c8fa718a28b801f6c19048c0419b3036d747239d236c327a4/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/9893355cc46d0d00f2343e9258996c7c35be9d3401c7d185495ed4e328692b48/merged 2024-04-05T03:00:19.024 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/b951835cb58cd191faa764acfaf8e862f11cb22b299985702c4f882a69afbf3a/merged 2024-04-05T03:00:19.025 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-04-05T03:00:19.025 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T03:00:19.025 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T03:00:19.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.025 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-04-05T03:00:19.077 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-05T03:00:19.077 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-05T03:00:19.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2024-04-05T03:00:19.146 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2024-04-05T03:00:19.202 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:19.203 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/0 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay 2024-04-05T03:00:19.204 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/e198afea19dc27d39aee81aeaeba5bf15266a5f90e414168d1d098728c4db24e/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/42636ea68e59cb7d46d47ffe0518391e596d4e80904f2530344f859f5ee147fb/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/a58770cf63c59faf654c5092353c9e786d56002a19c0987e7f3668ef24938056/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/a1215fcfa1a3f357a00818ef589f3f40b8d0efeea64b7e03b07b298015f26c6a/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/e71b9387a5868fb7883c8de698b7b4b581cf5e3154d6a8631d43b65e98392e75/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/05b63df97ea1c94df7ed623ff7baaca7d4881b67c9e89f57d1302f4a5d0e8431/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/38cdc7e7cb2c7573bde6f3e4cebc1791eb29acb08f176e64b5c8a46cc4c8c623/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/3716c2cda95d032c8fa718a28b801f6c19048c0419b3036d747239d236c327a4/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/9893355cc46d0d00f2343e9258996c7c35be9d3401c7d185495ed4e328692b48/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/containers/storage/overlay/b951835cb58cd191faa764acfaf8e862f11cb22b299985702c4f882a69afbf3a/merged 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T03:00:19.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.206 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2024-04-05T03:00:19.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:2024-04-05T03:00:19.223+0000 7f4cf42cf300 -1 init, newargv = 0x55a7c7ca39a0 newargc=15 2024-04-05T03:00:19.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[67067]: starting ceph client 2024-04-05T03:00:19.229 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[67067]: starting fuse 2024-04-05T03:00:19.238 INFO:teuthology.orchestra.run.smithi203.stdout:126 2024-04-05T03:00:19.238 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [126] 2024-04-05T03:00:19.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2024-04-05T03:00:19.238 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2024-04-05T03:00:19.238 DEBUG:teuthology.orchestra.run.smithi203:> import re 2024-04-05T03:00:19.238 DEBUG:teuthology.orchestra.run.smithi203:> import os 2024-04-05T03:00:19.238 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2024-04-05T03:00:19.239 DEBUG:teuthology.orchestra.run.smithi203:> 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> return f 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.1")) 2024-04-05T03:00:19.240 DEBUG:teuthology.orchestra.run.smithi203:> ' 2024-04-05T03:00:19.357 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.1.67067.asok 2024-04-05T03:00:19.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-05T03:00:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-05T03:00:19.368 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.67067.asok 2024-04-05T03:00:19.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.67067.asok status 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "93c790e135cc94d629f832b908e3f6c70360bfd5", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 17.2.7-815-g93c790e1 (93c790e135cc94d629f832b908e3f6c70360bfd5) quincy (stable)", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "1", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "67067", 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2024-04-05T03:00:19.502 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 24311, 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 24311 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2642975444 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2024-04-05T03:00:19.503 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2642975444 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.24311 192.168.0.1:0/2642975444", 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.1:0/2642975444", 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 21, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 45, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2024-04-05T03:00:19.504 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-05T03:00:19.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.510 DEBUG:teuthology.orchestra.run.smithi153:> stat --file-system '--printf=%T 2024-04-05T03:00:19.511 DEBUG:teuthology.orchestra.run.smithi153:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:19.531 INFO:teuthology.orchestra.run.smithi153.stdout:fuseblk 2024-04-05T03:00:19.531 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:19.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:19.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.609 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2024-04-05T03:00:19.609 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:19.625 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2024-04-05T03:00:19.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:19.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:00:19.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:20 smithi203 ceph-mon[39911]: pgmap v186: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T03:00:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:20 smithi153 ceph-mon[29381]: pgmap v186: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T03:00:22.720 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:22 smithi203 ceph-mon[39911]: pgmap v187: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-05T03:00:22.721 INFO:teuthology.run_tasks:Running task print... 2024-04-05T03:00:22.733 INFO:teuthology.task.print:**** done client 2024-04-05T03:00:22.733 INFO:teuthology.run_tasks:Running task parallel... 2024-04-05T03:00:22.748 INFO:teuthology.task.parallel:starting parallel... 2024-04-05T03:00:22.748 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-05T03:00:22.749 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T03:00:22.749 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:00:22.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-05T03:00:22.750 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-05T03:00:22.750 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-05T03:00:22.753 INFO:tasks.workunit:Pulling workunits from ref 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-05T03:00:22.754 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-05T03:00:22.754 INFO:tasks.workunit:timeout=3h 2024-04-05T03:00:22.754 INFO:tasks.workunit:cleanup=True 2024-04-05T03:00:22.754 DEBUG:teuthology.orchestra.run.smithi153:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:22.775 INFO:teuthology.orchestra.run.smithi153.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:22.775 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-05T03:00:22.775 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 7eh/126d Inode: 1 Links: 2 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-05 02:59:46.447617084 +0000 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-05 03:00:19.606936062 +0000 2024-04-05T03:00:22.776 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-05T03:00:22.776 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-05T03:00:22.776 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-05T03:00:22.784 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:22 smithi153 ceph-mon[29381]: pgmap v187: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-05T03:00:22.820 DEBUG:teuthology.orchestra.run.smithi203:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-05T03:00:22.841 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-05T03:00:22.842 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-05 03:00:22.815141530 +0000 2024-04-05T03:00:22.842 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-05 03:00:22.815141530 +0000 2024-04-05T03:00:22.842 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-04-05T03:00:22.842 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-05T03:00:22.842 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-05T03:00:22.920 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-05T03:00:22.921 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-05T03:00:22.937 INFO:tasks.workunit.client.0.smithi153.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-05T03:00:22.978 INFO:tasks.workunit.client.1.smithi203.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-05T03:00:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:24.520 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:24 smithi153 ceph-mon[29381]: pgmap v188: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 -- 172.21.15.153:0/2762121128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fdff0 msgr2=0x7f64bc0fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 --2- 172.21.15.153:0/2762121128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fdff0 0x7f64bc0fe460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f64ac004940 tx=0x7f64ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 -- 172.21.15.153:0/2762121128 shutdown_connections 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 --2- 172.21.15.153:0/2762121128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fdff0 0x7f64bc0fe460 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 --2- 172.21.15.153:0/2762121128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fce80 0x7f64bc0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 -- 172.21.15.153:0/2762121128 >> 172.21.15.153:0/2762121128 conn(0x7f64bc0f8580 msgr2=0x7f64bc0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 -- 172.21.15.153:0/2762121128 shutdown_connections 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.738+0000 7f64c37b5700 1 -- 172.21.15.153:0/2762121128 wait complete. 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 Processor -- start 2024-04-05T03:00:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 -- start start 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 0x7f64bc070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc0708c0 con 0x7f64bc0fce80 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c37b5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc070a30 con 0x7f64bc0fdff0 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38952/0 (socket says 172.21.15.153:38952) 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 -- 172.21.15.153:0/1807595679 learned_addr learned my addr 172.21.15.153:0/1807595679 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c0d50700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 0x7f64bc070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 -- 172.21.15.153:0/1807595679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 msgr2=0x7f64bc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 0x7f64bc070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 -- 172.21.15.153:0/1807595679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac005040 con 0x7f64bc0fce80 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c0d50700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 0x7f64bc070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.739+0000 7f64c1551700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64b8001d80 tx=0x7f64b8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.740+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64b8007d10 con 0x7f64bc0fce80 2024-04-05T03:00:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.740+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64b8007e70 con 0x7f64bc0fce80 2024-04-05T03:00:24.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.740+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64b8014640 con 0x7f64bc0fce80 2024-04-05T03:00:24.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.740+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc070d10 con 0x7f64bc0fce80 2024-04-05T03:00:24.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.740+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc115a80 con 0x7f64bc0fce80 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.741+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc05f5f0 con 0x7f64bc0fce80 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.741+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f64b800e960 con 0x7f64bc0fce80 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.741+0000 7f64b1ffb700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 0x7f64a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.742+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f64b8086fc0 con 0x7f64bc0fce80 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.742+0000 7f64c0d50700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 0x7f64a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.742+0000 7f64c0d50700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 0x7f64a806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00b970 tx=0x7f64ac009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:24.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.746+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f64b80532d0 con 0x7f64bc0fce80 2024-04-05T03:00:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:24 smithi203 ceph-mon[39911]: pgmap v188: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T03:00:24.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.936+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f64bc04bed0 con 0x7f64bc0fce80 2024-04-05T03:00:24.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.936+0000 7f64b1ffb700 1 -- 172.21.15.153:0/1807595679 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f64b80568f0 con 0x7f64bc0fce80 2024-04-05T03:00:24.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.938+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 msgr2=0x7f64a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.938+0000 7f64c37b5700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 0x7f64a806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00b970 tx=0x7f64ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.938+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 msgr2=0x7f64bc071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f64b8001d80 tx=0x7f64b8009ec0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 shutdown_connections 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f64a80692a0 0x7f64a806b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64bc0fce80 0x7f64bc071e40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 --2- 172.21.15.153:0/1807595679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64bc0fdff0 0x7f64bc070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 >> 172.21.15.153:0/1807595679 conn(0x7f64bc0f8580 msgr2=0x7f64bc0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 shutdown_connections 2024-04-05T03:00:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:24.939+0000 7f64c37b5700 1 -- 172.21.15.153:0/1807595679 wait complete. 2024-04-05T03:00:25.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-05T03:00:25.805 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:26 smithi153 ceph-mon[29381]: pgmap v189: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:26 smithi203 ceph-mon[39911]: pgmap v189: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 -- 172.21.15.153:0/415544692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 msgr2=0x7f676c0fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 --2- 172.21.15.153:0/415544692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c0fc0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6760004940 tx=0x7f6760009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 -- 172.21.15.153:0/415544692 shutdown_connections 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 --2- 172.21.15.153:0/415544692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 0x7f676c0fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 --2- 172.21.15.153:0/415544692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c0fc0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 -- 172.21.15.153:0/415544692 >> 172.21.15.153:0/415544692 conn(0x7f676c0f7430 msgr2=0x7f676c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 -- 172.21.15.153:0/415544692 shutdown_connections 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.373+0000 7f6773192700 1 -- 172.21.15.153:0/415544692 wait complete. 2024-04-05T03:00:27.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 Processor -- start 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 -- start start 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 0x7f676c10dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c10e630 con 0x7f676c0fbcd0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6773192700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c10edd0 con 0x7f676c0fcf80 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6770f2e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6770f2e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38964/0 (socket says 172.21.15.153:38964) 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6770f2e700 1 -- 172.21.15.153:0/1681964172 learned_addr learned my addr 172.21.15.153:0/1681964172 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f6770f2e700 1 -- 172.21.15.153:0/1681964172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 msgr2=0x7f676c10dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.374+0000 7f676bfff700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 0x7f676c10dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f6770f2e700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 0x7f676c10dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f6770f2e700 1 -- 172.21.15.153:0/1681964172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760005040 con 0x7f676c0fbcd0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f6770f2e700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f676000b3b0 tx=0x7f6760009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676000e040 con 0x7f676c0fbcd0 2024-04-05T03:00:27.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c10f050 con 0x7f676c0fbcd0 2024-04-05T03:00:27.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f676000bcc0 con 0x7f676c0fbcd0 2024-04-05T03:00:27.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f676001c630 con 0x7f676c0fbcd0 2024-04-05T03:00:27.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.375+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c10f610 con 0x7f676c0fbcd0 2024-04-05T03:00:27.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.376+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c101b10 con 0x7f676c0fbcd0 2024-04-05T03:00:27.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.376+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6760013980 con 0x7f676c0fbcd0 2024-04-05T03:00:27.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.377+0000 7f67697fa700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 0x7f675c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:27.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.377+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6760088070 con 0x7f676c0fbcd0 2024-04-05T03:00:27.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.377+0000 7f676bfff700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 0x7f675c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:27.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.377+0000 7f676bfff700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 0x7f675c06b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6758006fd0 tx=0x7f6758008040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:27.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.381+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6760057d00 con 0x7f676c0fbcd0 2024-04-05T03:00:27.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.567+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f676c04bed0 con 0x7f676c0fbcd0 2024-04-05T03:00:27.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.567+0000 7f67697fa700 1 -- 172.21.15.153:0/1681964172 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f6760057890 con 0x7f676c0fbcd0 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 msgr2=0x7f675c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 0x7f675c06b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6758006fd0 tx=0x7f6758008040 comp rx=0 tx=0).stop 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 msgr2=0x7f676c10da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f676000b3b0 tx=0x7f6760009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 shutdown_connections 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f675c0692a0 0x7f675c06b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f676c0fbcd0 0x7f676c10da60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 --2- 172.21.15.153:0/1681964172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f676c0fcf80 0x7f676c10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:27.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 >> 172.21.15.153:0/1681964172 conn(0x7f676c0f7430 msgr2=0x7f676c065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:27.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 shutdown_connections 2024-04-05T03:00:27.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:27.570+0000 7f6773192700 1 -- 172.21.15.153:0/1681964172 wait complete. 2024-04-05T03:00:28.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-05T03:00:28.418 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:28 smithi203 ceph-mon[39911]: pgmap v190: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T03:00:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:28.803 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:28 smithi153 ceph-mon[29381]: pgmap v190: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T03:00:28.803 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 -- 172.21.15.153:0/1587287254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fbcd0 msgr2=0x7f41440fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 --2- 172.21.15.153:0/1587287254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fbcd0 0x7f41440fc0e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4140004940 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 -- 172.21.15.153:0/1587287254 shutdown_connections 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 --2- 172.21.15.153:0/1587287254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fcf80 0x7f41440fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 --2- 172.21.15.153:0/1587287254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fbcd0 0x7f41440fc0e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 -- 172.21.15.153:0/1587287254 >> 172.21.15.153:0/1587287254 conn(0x7f41440f7430 msgr2=0x7f41440f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 -- 172.21.15.153:0/1587287254 shutdown_connections 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.970+0000 7f414c6a0700 1 -- 172.21.15.153:0/1587287254 wait complete. 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 Processor -- start 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 -- start start 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fbcd0 0x7f4144110740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144066350 con 0x7f41440fcf80 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414c6a0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144066490 con 0x7f41440fbcd0 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49702/0 (socket says 172.21.15.153:49702) 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 -- 172.21.15.153:0/3142904393 learned_addr learned my addr 172.21.15.153:0/3142904393 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f414a43c700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fbcd0 0x7f4144110740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 -- 172.21.15.153:0/3142904393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fbcd0 msgr2=0x7f4144110740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fbcd0 0x7f4144110740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 -- 172.21.15.153:0/3142904393 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005040 con 0x7f41440fcf80 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.971+0000 7f4149c3b700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f413400c830 tx=0x7f4134004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41340106f0 con 0x7f41440fcf80 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4134010d20 con 0x7f41440fcf80 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4144113230 con 0x7f41440fcf80 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41340113a0 con 0x7f41440fcf80 2024-04-05T03:00:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41441137f0 con 0x7f41440fcf80 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.972+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414405f5f0 con 0x7f41440fcf80 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.973+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4134012400 con 0x7f41440fcf80 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.973+0000 7f413affd700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 0x7f413006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.973+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4134087180 con 0x7f41440fcf80 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.973+0000 7f414a43c700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 0x7f413006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:29.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.974+0000 7f414a43c700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 0x7f413006b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4140004890 tx=0x7f4140009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:29.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:29.978+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4134053460 con 0x7f41440fcf80 2024-04-05T03:00:30.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.171+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f41441047c0 con 0x7f41440fcf80 2024-04-05T03:00:30.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.171+0000 7f413affd700 1 -- 172.21.15.153:0/3142904393 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4134056a80 con 0x7f41440fcf80 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 msgr2=0x7f413006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 0x7f413006b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4140004890 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 msgr2=0x7f4144112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f413400c830 tx=0x7f4134004470 comp rx=0 tx=0).stop 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 shutdown_connections 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f41300692a0 0x7f413006b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:30.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41440fbcd0 0x7f4144110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:30.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 --2- 172.21.15.153:0/3142904393 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f41440fcf80 0x7f4144112c90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:30.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 >> 172.21.15.153:0/3142904393 conn(0x7f41440f7430 msgr2=0x7f41440f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:30.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 shutdown_connections 2024-04-05T03:00:30.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:30.173+0000 7f414c6a0700 1 -- 172.21.15.153:0/3142904393 wait complete. 2024-04-05T03:00:30.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:30 smithi153 ceph-mon[29381]: pgmap v191: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-05T03:00:30.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-05T03:00:30.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:30 smithi203 ceph-mon[39911]: pgmap v191: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-05T03:00:31.062 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:00:32.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.695+0000 7f27feb2d700 1 -- 172.21.15.153:0/903083619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 msgr2=0x7f27f80fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:32.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.695+0000 7f27feb2d700 1 --2- 172.21.15.153:0/903083619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f80fb880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f27ec004940 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:00:32.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 -- 172.21.15.153:0/903083619 shutdown_connections 2024-04-05T03:00:32.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 --2- 172.21.15.153:0/903083619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 0x7f27f80fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:32.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 --2- 172.21.15.153:0/903083619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f80fb880 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 -- 172.21.15.153:0/903083619 >> 172.21.15.153:0/903083619 conn(0x7f27f80f6bd0 msgr2=0x7f27f80f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 -- 172.21.15.153:0/903083619 shutdown_connections 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 -- 172.21.15.153:0/903083619 wait complete. 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 Processor -- start 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.696+0000 7f27feb2d700 1 -- start start 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 0x7f27f810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f810e460 con 0x7f27f80fb470 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f810e5d0 con 0x7f27f80fc720 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49714/0 (socket says 172.21.15.153:49714) 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 -- 172.21.15.153:0/833379673 learned_addr learned my addr 172.21.15.153:0/833379673 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27f7fff700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 0x7f27f810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 -- 172.21.15.153:0/833379673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 msgr2=0x7f27f810de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:00:32.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 0x7f27f810de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 -- 172.21.15.153:0/833379673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec005040 con 0x7f27f80fb470 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27fc8c9700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b3b0 tx=0x7f27ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27ec00e040 con 0x7f27f80fb470 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27ec00bbf0 con 0x7f27f80fb470 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f8112fd0 con 0x7f27f80fb470 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27ec01c630 con 0x7f27f80fb470 2024-04-05T03:00:32.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.697+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f8113500 con 0x7f27f80fb470 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.698+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f804c710 con 0x7f27f80fb470 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.699+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f27ec00bd60 con 0x7f27f80fb470 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.699+0000 7f27f57fa700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 0x7f27e806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.699+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f27ec087e90 con 0x7f27f80fb470 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.699+0000 7f27f7fff700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 0x7f27e806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:00:32.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.700+0000 7f27f7fff700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 0x7f27e806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f27e4007fd0 tx=0x7f27e4009040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:00:32.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.704+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f27ec057c00 con 0x7f27f80fb470 2024-04-05T03:00:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:32 smithi203 ceph-mon[39911]: pgmap v192: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-05T03:00:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:32 smithi153 ceph-mon[29381]: pgmap v192: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-05T03:00:32.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:00:32.907+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f80f8480 con 0x7f27e80692a0 2024-04-05T03:00:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:34 smithi203 ceph-mon[39911]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:00:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:34 smithi203 ceph-mon[39911]: pgmap v193: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-05T03:00:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:34 smithi153 ceph-mon[29381]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:00:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:34 smithi153 ceph-mon[29381]: pgmap v193: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-05T03:00:36.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:36 smithi203 ceph-mon[39911]: pgmap v194: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T03:00:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:36 smithi153 ceph-mon[29381]: pgmap v194: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T03:00:38.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:38 smithi203 ceph-mon[39911]: pgmap v195: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-05T03:00:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:38 smithi153 ceph-mon[29381]: pgmap v195: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-04-05T03:00:40.774 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:40 smithi153 ceph-mon[29381]: pgmap v196: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T03:00:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:40 smithi203 ceph-mon[39911]: pgmap v196: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T03:00:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:42 smithi203 ceph-mon[39911]: pgmap v197: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-05T03:00:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:42 smithi153 ceph-mon[29381]: pgmap v197: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-05T03:00:43.735 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:43 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:43 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:44.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:44 smithi203 ceph-mon[39911]: pgmap v198: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-05T03:00:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:44 smithi153 ceph-mon[29381]: pgmap v198: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-05T03:00:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:46 smithi203 ceph-mon[39911]: pgmap v199: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:46 smithi153 ceph-mon[29381]: pgmap v199: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:48 smithi203 ceph-mon[39911]: pgmap v200: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:48 smithi153 ceph-mon[29381]: pgmap v200: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:00:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:50 smithi203 ceph-mon[39911]: pgmap v201: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:00:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:50 smithi153 ceph-mon[29381]: pgmap v201: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:00:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:52 smithi203 ceph-mon[39911]: pgmap v202: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:00:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:52 smithi153 ceph-mon[29381]: pgmap v202: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:00:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:54 smithi203 ceph-mon[39911]: pgmap v203: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:00:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:54 smithi153 ceph-mon[29381]: pgmap v203: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:00:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:56 smithi203 ceph-mon[39911]: pgmap v204: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:00:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:56 smithi153 ceph-mon[29381]: pgmap v204: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:00:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:58 smithi203 ceph-mon[39911]: pgmap v205: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:00:58.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:00:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:00:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:58 smithi153 ceph-mon[29381]: pgmap v205: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:00:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:00:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:00.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:00 smithi203 ceph-mon[39911]: pgmap v206: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:00 smithi153 ceph-mon[29381]: pgmap v206: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:02 smithi203 ceph-mon[39911]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:02 smithi153 ceph-mon[29381]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:04 smithi203 ceph-mon[39911]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:04 smithi153 ceph-mon[29381]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:06 smithi203 ceph-mon[39911]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:06 smithi153 ceph-mon[29381]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:08 smithi203 ceph-mon[39911]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:08 smithi153 ceph-mon[29381]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:09 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:09 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:10 smithi203 ceph-mon[39911]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:10 smithi153 ceph-mon[29381]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:12 smithi203 ceph-mon[39911]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:12 smithi153 ceph-mon[29381]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:13.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:13.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:01:13.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:13.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:01:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:14.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.749+0000 7f27f57fa700 1 -- 172.21.15.153:0/833379673 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f27f80f8480 con 0x7f27e80692a0 2024-04-05T03:01:14.750 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 msgr2=0x7f27e806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 0x7f27e806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f27e4007fd0 tx=0x7f27e4009040 comp rx=0 tx=0).stop 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 msgr2=0x7f27f810d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00b3b0 tx=0x7f27ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 shutdown_connections 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f27e80692a0 0x7f27e806b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80fb470 0x7f27f810d920 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 --2- 172.21.15.153:0/833379673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27f80fc720 0x7f27f810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 >> 172.21.15.153:0/833379673 conn(0x7f27f80f6bd0 msgr2=0x7f27f80f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 shutdown_connections 2024-04-05T03:01:14.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:14.752+0000 7f27feb2d700 1 -- 172.21.15.153:0/833379673 wait complete. 2024-04-05T03:01:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:14 smithi203 ceph-mon[39911]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:14 smithi153 ceph-mon[29381]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:15.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-05T03:01:15.733 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:01:15.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:15 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:01:16.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:15 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:16.958 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:16 smithi153 ceph-mon[29381]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:01:16.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:16 smithi153 ceph-mon[29381]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:17.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:16 smithi203 ceph-mon[39911]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:01:17.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:16 smithi203 ceph-mon[39911]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:18.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1077780922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 msgr2=0x7f4fd00fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:18.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1077780922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00fc0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0004940 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:18.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1077780922 shutdown_connections 2024-04-05T03:01:18.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1077780922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 0x7f4fd00fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:18.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1077780922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00fc0e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1077780922 >> 172.21.15.153:0/1077780922 conn(0x7f4fd00f7430 msgr2=0x7f4fd00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1077780922 shutdown_connections 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.746+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1077780922 wait complete. 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 Processor -- start 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 -- start start 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:18.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 0x7f4fd00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd0111610 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fd59c0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd0111780 con 0x7f4fd00fcf80 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.747+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35244/0 (socket says 172.21.15.153:35244) 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 -- 172.21.15.153:0/1147256044 learned_addr learned my addr 172.21.15.153:0/1147256044 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fc7fff700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 0x7f4fd00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 -- 172.21.15.153:0/1147256044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 msgr2=0x7f4fd00fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 0x7f4fd00fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 -- 172.21.15.153:0/1147256044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0005040 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fceffd700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009150 tx=0x7f4fc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc000e040 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fc0004290 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.748+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd00fe8c0 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.749+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc001c630 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.749+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd00fed70 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.750+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd005f5f0 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.751+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4fc000bbf0 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.751+0000 7f4fc77fe700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 0x7f4fbc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:18.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.751+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4fc0087dc0 con 0x7f4fd00fbcd0 2024-04-05T03:01:18.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.751+0000 7f4fc7fff700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 0x7f4fbc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:18.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.752+0000 7f4fc7fff700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 0x7f4fbc06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009fd0 tx=0x7f4fb800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:18.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:18.760+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4fc0054140 con 0x7f4fd00fbcd0 2024-04-05T03:01:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:18 smithi153 ceph-mon[29381]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:19.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:18 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.155+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fd01047c0 con 0x7f4fbc0692a0 2024-04-05T03:01:19.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.157+0000 7f4fc77fe700 1 -- 172.21.15.153:0/1147256044 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4fd01047c0 con 0x7f4fbc0692a0 2024-04-05T03:01:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.159+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 msgr2=0x7f4fbc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.159+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 0x7f4fbc06b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009fd0 tx=0x7f4fb800b040 comp rx=0 tx=0).stop 2024-04-05T03:01:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.159+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 msgr2=0x7f4fd00ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.159+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009150 tx=0x7f4fc0009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 shutdown_connections 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f4fbc0692a0 0x7f4fbc06b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fd00fbcd0 0x7f4fd00ffdf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 --2- 172.21.15.153:0/1147256044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd00fcf80 0x7f4fd00fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 >> 172.21.15.153:0/1147256044 conn(0x7f4fd00f7430 msgr2=0x7f4fd00f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 shutdown_connections 2024-04-05T03:01:19.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.160+0000 7f4fd59c0700 1 -- 172.21.15.153:0/1147256044 wait complete. 2024-04-05T03:01:19.175 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:01:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 -- 172.21.15.153:0/1054008245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c0fd9c0 msgr2=0x7f5b0c0ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 --2- 172.21.15.153:0/1054008245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c0fd9c0 0x7f5b0c0ffda0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5afc004940 tx=0x7f5afc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 -- 172.21.15.153:0/1054008245 shutdown_connections 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 --2- 172.21.15.153:0/1054008245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 --2- 172.21.15.153:0/1054008245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c0fd9c0 0x7f5b0c0ffda0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 -- 172.21.15.153:0/1054008245 >> 172.21.15.153:0/1054008245 conn(0x7f5b0c0f7480 msgr2=0x7f5b0c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 -- 172.21.15.153:0/1054008245 shutdown_connections 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 -- 172.21.15.153:0/1054008245 wait complete. 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.273+0000 7f5b11b38700 1 Processor -- start 2024-04-05T03:01:19.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b11b38700 1 -- start start 2024-04-05T03:01:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b11b38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b11b38700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c06e9c0 0x7f5b0c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b11b38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c0fd7c0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b11b38700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c06ffb0 con 0x7f5b0c06e9c0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35266/0 (socket says 172.21.15.153:35266) 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 -- 172.21.15.153:0/2202920052 learned_addr learned my addr 172.21.15.153:0/2202920052 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0affd700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c06e9c0 0x7f5b0c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 -- 172.21.15.153:0/2202920052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c06e9c0 msgr2=0x7f5b0c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c06e9c0 0x7f5b0c06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.274+0000 7f5b0b7fe700 1 -- 172.21.15.153:0/2202920052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc005040 con 0x7f5b0c1002e0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5b0b7fe700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5afc013870 tx=0x7f5afc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc00e030 con 0x7f5b0c1002e0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5afc013de0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc00b410 con 0x7f5b0c1002e0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c11e1e0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.275+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c11e640 con 0x7f5b0c1002e0 2024-04-05T03:01:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.276+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c05f5f0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.277+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5afc01c400 con 0x7f5b0c1002e0 2024-04-05T03:01:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.277+0000 7f5af3fff700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 0x7f5af406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.277+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5afc0886f0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.278+0000 7f5b0affd700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 0x7f5af406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.278+0000 7f5b0affd700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 0x7f5af406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c06f730 tx=0x7f5b0000b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.286+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5afc0549d0 con 0x7f5b0c1002e0 2024-04-05T03:01:19.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:18 smithi203 ceph-mon[39911]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:19.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:18 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:19.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:18 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:01:19.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:18 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:19.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.501+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b0c0f8f70 con 0x7f5af40692a0 2024-04-05T03:01:19.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.502+0000 7f5af3fff700 1 -- 172.21.15.153:0/2202920052 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f5b0c0f8f70 con 0x7f5af40692a0 2024-04-05T03:01:19.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.504+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 msgr2=0x7f5af406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.504+0000 7f5b11b38700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 0x7f5af406b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c06f730 tx=0x7f5b0000b040 comp rx=0 tx=0).stop 2024-04-05T03:01:19.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.504+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 msgr2=0x7f5b0c06e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.504+0000 7f5b11b38700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5afc013870 tx=0x7f5afc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 shutdown_connections 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5af40692a0 0x7f5af406b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0c1002e0 0x7f5b0c06e480 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 --2- 172.21.15.153:0/2202920052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b0c06e9c0 0x7f5b0c06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 >> 172.21.15.153:0/2202920052 conn(0x7f5b0c0f7480 msgr2=0x7f5b0c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 shutdown_connections 2024-04-05T03:01:19.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.505+0000 7f5b11b38700 1 -- 172.21.15.153:0/2202920052 wait complete. 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 -- 172.21.15.153:0/3114816397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fce40 msgr2=0x7f69dc0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 --2- 172.21.15.153:0/3114816397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fce40 0x7f69dc0fd250 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f69d0004a90 tx=0x7f69d0009dd0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 -- 172.21.15.153:0/3114816397 shutdown_connections 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 --2- 172.21.15.153:0/3114816397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fe040 0x7f69dc0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 --2- 172.21.15.153:0/3114816397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fce40 0x7f69dc0fd250 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 -- 172.21.15.153:0/3114816397 >> 172.21.15.153:0/3114816397 conn(0x7f69dc0f8570 msgr2=0x7f69dc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 -- 172.21.15.153:0/3114816397 shutdown_connections 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.613+0000 7f69e30b7700 1 -- 172.21.15.153:0/3114816397 wait complete. 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 Processor -- start 2024-04-05T03:01:19.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 -- start start 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fce40 0x7f69dc06e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc06f210 con 0x7f69dc0fe040 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.614+0000 7f69e30b7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc06fa60 con 0x7f69dc0fce40 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69e0e53700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fce40 0x7f69dc06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35282/0 (socket says 172.21.15.153:35282) 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 -- 172.21.15.153:0/4127246651 learned_addr learned my addr 172.21.15.153:0/4127246651 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 -- 172.21.15.153:0/4127246651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fce40 msgr2=0x7f69dc06e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fce40 0x7f69dc06e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 -- 172.21.15.153:0/4127246651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0004650 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.615+0000 7f69dbfff700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f69c800eb40 tx=0x7f69c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.616+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c8007cc0 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.616+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc06fce0 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.616+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69c800f040 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.616+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc070220 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.616+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c800c990 con 0x7f69dc0fe040 2024-04-05T03:01:19.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.617+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc05f5f0 con 0x7f69dc0fe040 2024-04-05T03:01:19.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.618+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f69c800cc00 con 0x7f69dc0fe040 2024-04-05T03:01:19.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.619+0000 7f69d97fa700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 0x7f69cc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.619+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f69c8086d50 con 0x7f69dc0fe040 2024-04-05T03:01:19.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.619+0000 7f69e0e53700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 0x7f69cc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.620+0000 7f69e0e53700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 0x7f69cc06b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f69d00049b0 tx=0x7f69d0016040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.627+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f69c80530d0 con 0x7f69dc0fe040 2024-04-05T03:01:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.832+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f69dc070490 con 0x7f69cc0692b0 2024-04-05T03:01:19.849 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:01:19.849 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 70s ago 7m 26.6M - 0.25.0 c8568f914cd2 db2172977a37 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 70s ago 8m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (5m) 75s ago 5m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 70s ago 6m 88.5M - 9.4.7 954c08fa6188 0fafa83f96b9 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (87s) 70s ago 87s 14.5M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (94s) 70s ago 93s 11.1M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (84s) 75s ago 84s 13.2M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (91s) 75s ago 90s 16.6M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (9m) 70s ago 9m 479M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (5m) 75s ago 5m 435M - 17.2.7-805-g6fda41ff a49113eacc58 3ebddafb5706 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 70s ago 9m 43.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (5m) 75s ago 5m 38.2M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 70s ago 7m 15.8M - 1.5.0 0da6a335fe13 61d64f55ec69 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (5m) 75s ago 5m 16.1M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T03:01:19.850 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 70s ago 5m 46.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 70s ago 4m 46.5M 4629M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 70s ago 4m 44.9M 4629M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (3m) 75s ago 3m 43.7M 4970M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (3m) 75s ago 3m 44.2M 4970M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (2m) 75s ago 2m 42.1M 4970M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (5m) 75s ago 5m 56.7M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T03:01:19.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.849+0000 7f69d97fa700 1 -- 172.21.15.153:0/4127246651 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f69dc070490 con 0x7f69cc0692b0 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 msgr2=0x7f69cc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 0x7f69cc06b760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f69d00049b0 tx=0x7f69d0016040 comp rx=0 tx=0).stop 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 msgr2=0x7f69dc06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f69c800eb40 tx=0x7f69c800b040 comp rx=0 tx=0).stop 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 shutdown_connections 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f69cc0692b0 0x7f69cc06b760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69dc0fce40 0x7f69dc06e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 --2- 172.21.15.153:0/4127246651 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69dc0fe040 0x7f69dc06ebc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 >> 172.21.15.153:0/4127246651 conn(0x7f69dc0f8570 msgr2=0x7f69dc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 shutdown_connections 2024-04-05T03:01:19.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.852+0000 7f69e30b7700 1 -- 172.21.15.153:0/4127246651 wait complete. 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.961+0000 7fa804512700 1 -- 172.21.15.153:0/3752749174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc102360 msgr2=0x7fa7fc104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.961+0000 7fa804512700 1 --2- 172.21.15.153:0/3752749174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc102360 0x7fa7fc104740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec004940 tx=0x7fa7ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.961+0000 7fa804512700 1 -- 172.21.15.153:0/3752749174 shutdown_connections 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.961+0000 7fa804512700 1 --2- 172.21.15.153:0/3752749174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc102360 0x7fa7fc104740 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.961+0000 7fa804512700 1 --2- 172.21.15.153:0/3752749174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc066280 0x7fa7fc101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.962+0000 7fa804512700 1 -- 172.21.15.153:0/3752749174 >> 172.21.15.153:0/3752749174 conn(0x7fa7fc0f7430 msgr2=0x7fa7fc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.962+0000 7fa804512700 1 -- 172.21.15.153:0/3752749174 shutdown_connections 2024-04-05T03:01:19.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.962+0000 7fa804512700 1 -- 172.21.15.153:0/3752749174 wait complete. 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.962+0000 7fa804512700 1 Processor -- start 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.962+0000 7fa804512700 1 -- start start 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa804512700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa804512700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 0x7fa7fc0fe300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa804512700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0fe950 con 0x7fa7fc066280 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa804512700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc0ff150 con 0x7fa7fc102360 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35292/0 (socket says 172.21.15.153:35292) 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 -- 172.21.15.153:0/3634771205 learned_addr learned my addr 172.21.15.153:0/3634771205 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa801aad700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 0x7fa7fc0fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 -- 172.21.15.153:0/3634771205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 msgr2=0x7fa7fc0fe300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 0x7fa7fc0fe300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 -- 172.21.15.153:0/3634771205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec005040 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa801aad700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 0x7fa7fc0fe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.963+0000 7fa8022ae700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800eb40 tx=0x7fa7f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.964+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f8007cc0 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.964+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7f800f040 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.964+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc0ff3d0 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.964+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f800cb30 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.964+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc0ff910 con 0x7fa7fc066280 2024-04-05T03:01:19.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.965+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc05f5f0 con 0x7fa7fc066280 2024-04-05T03:01:19.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.965+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa7f800cc90 con 0x7fa7fc066280 2024-04-05T03:01:19.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.966+0000 7fa7f2ffd700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 0x7fa7e806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:19.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.966+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa7f8086b10 con 0x7fa7fc066280 2024-04-05T03:01:19.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.966+0000 7fa801aad700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 0x7fa7e806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:19.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.967+0000 7fa801aad700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 0x7fa7e806b820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec013870 tx=0x7fa7ec009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:19.974+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa7f8052e90 con 0x7fa7fc066280 2024-04-05T03:01:20.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:19 smithi153 ceph-mon[29381]: Upgrade: Target is version 19.0.0-2013-g0028525c (unknown) 2024-04-05T03:01:20.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:19 smithi153 ceph-mon[29381]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca'] 2024-04-05T03:01:20.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:19 smithi153 ceph-mon[29381]: Upgrade: Need to upgrade myself (mgr.smithi153.npzyqy) 2024-04-05T03:01:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.255+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa7fc0ffb80 con 0x7fa7fc066280 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.256+0000 7fa7f2ffd700 1 -- 172.21.15.153:0/3634771205 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fa7f8019290 con 0x7fa7fc066280 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:01:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 14 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:01:20.257 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.258+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 msgr2=0x7fa7e806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.258+0000 7fa804512700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 0x7fa7e806b820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec013870 tx=0x7fa7ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.258+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 msgr2=0x7fa7fc0fddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.258+0000 7fa804512700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800eb40 tx=0x7fa7f800b040 comp rx=0 tx=0).stop 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 shutdown_connections 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fa7e8069370 0x7fa7e806b820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7fc066280 0x7fa7fc0fddc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 --2- 172.21.15.153:0/3634771205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7fc102360 0x7fa7fc0fe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 >> 172.21.15.153:0/3634771205 conn(0x7fa7fc0f7430 msgr2=0x7fa7fc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 shutdown_connections 2024-04-05T03:01:20.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.259+0000 7fa804512700 1 -- 172.21.15.153:0/3634771205 wait complete. 2024-04-05T03:01:20.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:19 smithi203 ceph-mon[39911]: Upgrade: Target is version 19.0.0-2013-g0028525c (unknown) 2024-04-05T03:01:20.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:19 smithi203 ceph-mon[39911]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca'] 2024-04-05T03:01:20.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:19 smithi203 ceph-mon[39911]: Upgrade: Need to upgrade myself (mgr.smithi153.npzyqy) 2024-04-05T03:01:20.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3313314112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 msgr2=0x7f8b400fd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3313314112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b400fd860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8b30004970 tx=0x7f8b30009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3313314112 shutdown_connections 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3313314112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b401046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3313314112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b400fd860 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3313314112 >> 172.21.15.153:0/3313314112 conn(0x7f8b400f6be0 msgr2=0x7f8b400f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3313314112 shutdown_connections 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.369+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3313314112 wait complete. 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 Processor -- start 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 -- start start 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b40110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40110610 con 0x7f8b400fd450 2024-04-05T03:01:20.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b44a2d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40110780 con 0x7f8b400fdda0 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b3e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b3e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35298/0 (socket says 172.21.15.153:35298) 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b3e59c700 1 -- 172.21.15.153:0/3005669606 learned_addr learned my addr 172.21.15.153:0/3005669606 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.370+0000 7f8b3dd9b700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b40110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b3e59c700 1 -- 172.21.15.153:0/3005669606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 msgr2=0x7f8b40110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b3e59c700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b40110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b3e59c700 1 -- 172.21.15.153:0/3005669606 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b30005040 con 0x7f8b400fd450 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b3dd9b700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b40110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:01:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b3e59c700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005010 tx=0x7f8b30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3000e040 con 0x7f8b400fd450 2024-04-05T03:01:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b40115180 con 0x7f8b400fd450 2024-04-05T03:01:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b3000b580 con 0x7f8b400fd450 2024-04-05T03:01:20.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.371+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b3001c630 con 0x7f8b400fd450 2024-04-05T03:01:20.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.372+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b40115740 con 0x7f8b400fd450 2024-04-05T03:01:20.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.373+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4004c710 con 0x7f8b400fd450 2024-04-05T03:01:20.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.374+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8b30013980 con 0x7f8b400fd450 2024-04-05T03:01:20.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.374+0000 7f8b2effd700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 0x7f8b2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:20.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.374+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8b30087d70 con 0x7f8b400fd450 2024-04-05T03:01:20.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.374+0000 7f8b3dd9b700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 0x7f8b2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:20.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.375+0000 7f8b3dd9b700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 0x7f8b2806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009b80 tx=0x7f8b3400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:20.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.379+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8b30057b90 con 0x7f8b400fd450 2024-04-05T03:01:20.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.590+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b400f8720 con 0x7f8b280692a0 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.592+0000 7f8b2effd700 1 -- 172.21.15.153:0/3005669606 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f8b400f8720 con 0x7f8b280692a0 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca image on host smithi203", 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:01:20.592 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:01:20.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 msgr2=0x7f8b2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 0x7f8b2806b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009b80 tx=0x7f8b3400b040 comp rx=0 tx=0).stop 2024-04-05T03:01:20.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 msgr2=0x7f8b4010fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:20.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8b30005010 tx=0x7f8b30009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 shutdown_connections 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f8b280692a0 0x7f8b2806b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b400fd450 0x7f8b4010fad0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 --2- 172.21.15.153:0/3005669606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b400fdda0 0x7f8b40110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.594+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 >> 172.21.15.153:0/3005669606 conn(0x7f8b400f6be0 msgr2=0x7f8b400f7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.595+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 shutdown_connections 2024-04-05T03:01:20.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:20.595+0000 7f8b44a2d700 1 -- 172.21.15.153:0/3005669606 wait complete. 2024-04-05T03:01:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca on smithi203 2024-04-05T03:01:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:20 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3634771205' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:01:21.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca on smithi203 2024-04-05T03:01:21.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:21.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:21.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:20 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3634771205' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:01:22.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:21 smithi153 ceph-mon[29381]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:22.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:21 smithi203 ceph-mon[39911]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:22 smithi203 ceph-mon[39911]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:22 smithi153 ceph-mon[29381]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:25.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:24 smithi203 ceph-mon[39911]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:24 smithi153 ceph-mon[29381]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:27.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:26 smithi203 ceph-mon[39911]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:27.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:26 smithi153 ceph-mon[29381]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:28 smithi153 ceph-mon[29381]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:28 smithi203 ceph-mon[39911]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:29.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:31.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:30 smithi203 ceph-mon[39911]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:31.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:30 smithi153 ceph-mon[29381]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:33.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:32 smithi203 ceph-mon[39911]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:33.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:32 smithi153 ceph-mon[29381]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:35.098 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:34 smithi203 ceph-mon[39911]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:35.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:34 smithi153 ceph-mon[29381]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:37.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:36 smithi203 ceph-mon[39911]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:37.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:36 smithi153 ceph-mon[29381]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:Note: switching to '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'. 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:state without impacting any branches by switching back to a branch. 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-05T03:01:38.023 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: git switch -c 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr:Or undo this operation with: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: git switch - 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-04-05T03:01:38.024 INFO:tasks.workunit.client.0.smithi153.stderr:HEAD is now at 9f192350ba5 Merge pull request #56691 from afreen23/wip-65156-squid 2024-04-05T03:01:38.027 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-05T03:01:38.053 INFO:tasks.workunit.client.0.smithi153.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-05T03:01:38.055 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T03:01:38.056 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-05T03:01:38.136 INFO:tasks.workunit.client.1.smithi203.stderr:Updating files: 92% (11554/12461) Updating files: 93% (11589/12461) Updating files: 94% (11714/12461) Updating files: 95% (11838/12461) Updating files: 96% (11963/12461) Updating files: 97% (12088/12461) Updating files: 98% (12212/12461) Updating files: 99% (12337/12461) Updating files: 100% (12461/12461) Updating files: 100% (12461/12461), done. 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:Note: switching to '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'. 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:state without impacting any branches by switching back to a branch. 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-05T03:01:38.446 INFO:tasks.workunit.client.1.smithi203.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: git switch -c 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr:Or undo this operation with: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: git switch - 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr: 2024-04-05T03:01:38.447 INFO:tasks.workunit.client.1.smithi203.stderr:HEAD is now at 9f192350ba5 Merge pull request #56691 from afreen23/wip-65156-squid 2024-04-05T03:01:38.450 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-05T03:01:38.519 INFO:tasks.workunit.client.1.smithi203.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-05T03:01:38.521 INFO:tasks.workunit.client.1.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-05T03:01:38.521 INFO:tasks.workunit.client.1.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-05T03:01:39.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:38 smithi153 ceph-mon[29381]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:39.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:38 smithi203 ceph-mon[39911]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:01:39.486 INFO:tasks.workunit.client.1.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-05T03:01:39.565 INFO:tasks.workunit.client.1.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-05T03:01:39.605 INFO:tasks.workunit.client.1.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-05T03:01:39.607 INFO:tasks.workunit.client.1.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-05T03:01:39.607 INFO:tasks.workunit.client.1.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-05T03:01:39.648 INFO:tasks.workunit.client.1.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-05T03:01:39.652 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-05T03:01:39.652 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-05T03:01:39.707 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-05T03:01:39.709 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-05T03:01:39.709 DEBUG:teuthology.orchestra.run.smithi203:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-05T03:01:39.774 INFO:tasks.workunit.client.1.smithi203.stderr:+ mkdir -p fsstress 2024-04-05T03:01:39.777 INFO:tasks.workunit.client.1.smithi203.stderr:+ pushd fsstress 2024-04-05T03:01:39.779 INFO:tasks.workunit.client.1.smithi203.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T03:01:39.779 INFO:tasks.workunit.client.1.smithi203.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-05T03:01:40.300 INFO:tasks.workunit.client.1.smithi203.stderr:+ tar xzf ltp-full.tgz 2024-04-05T03:01:41.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:40 smithi203 ceph-mon[39911]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:41.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:40 smithi153 ceph-mon[29381]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:01:42.622 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-05T03:01:43.123 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:42 smithi153 ceph-mon[29381]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:43.274 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-05T03:01:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:42 smithi203 ceph-mon[39911]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:01:43.330 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T03:01:43.332 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T03:01:43.332 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-05T03:01:43.392 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T03:01:43.396 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T03:01:43.396 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-05T03:01:43.453 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-05T03:01:43.454 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-05T03:01:43.454 DEBUG:teuthology.orchestra.run.smithi153:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-05T03:01:43.517 INFO:tasks.workunit.client.0.smithi153.stderr:+ mkdir -p fsstress 2024-04-05T03:01:43.519 INFO:tasks.workunit.client.0.smithi153.stderr:+ pushd fsstress 2024-04-05T03:01:43.520 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T03:01:43.520 INFO:tasks.workunit.client.0.smithi153.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-05T03:01:44.288 INFO:tasks.workunit.client.0.smithi153.stderr:+ tar xzf ltp-full.tgz 2024-04-05T03:01:44.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:43 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:43 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:45.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:44 smithi203 ceph-mon[39911]: pgmap v228: 65 pgs: 65 active+clean; 8.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s rd, 719 KiB/s wr, 26 op/s 2024-04-05T03:01:45.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:44 smithi153 ceph-mon[29381]: pgmap v228: 65 pgs: 65 active+clean; 8.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s rd, 719 KiB/s wr, 26 op/s 2024-04-05T03:01:47.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:46 smithi203 ceph-mon[39911]: pgmap v229: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 80 op/s 2024-04-05T03:01:47.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:46 smithi153 ceph-mon[29381]: pgmap v229: 65 pgs: 65 active+clean; 15 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 80 op/s 2024-04-05T03:01:48.891 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:48 smithi203 ceph-mon[39911]: pgmap v230: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 2.3 MiB/s wr, 166 op/s 2024-04-05T03:01:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:48 smithi153 ceph-mon[29381]: pgmap v230: 65 pgs: 65 active+clean; 27 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 2.3 MiB/s wr, 166 op/s 2024-04-05T03:01:50.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.730+0000 7f300bb44700 1 -- 172.21.15.153:0/2664105674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fcf90 msgr2=0x7f30040fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.730+0000 7f300bb44700 1 --2- 172.21.15.153:0/2664105674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fcf90 0x7f30040fd400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4004940 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.730+0000 7f300bb44700 1 -- 172.21.15.153:0/2664105674 shutdown_connections 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.730+0000 7f300bb44700 1 --2- 172.21.15.153:0/2664105674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fcf90 0x7f30040fd400 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 --2- 172.21.15.153:0/2664105674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fbce0 0x7f30040fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- 172.21.15.153:0/2664105674 >> 172.21.15.153:0/2664105674 conn(0x7f30040f7440 msgr2=0x7f30040f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- 172.21.15.153:0/2664105674 shutdown_connections 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- 172.21.15.153:0/2664105674 wait complete. 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 Processor -- start 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- start start 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fbce0 0x7f3004110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:50.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004066c40 con 0x7f30040fbce0 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.731+0000 7f300bb44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004066d80 con 0x7f30040fcf90 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:40014/0 (socket says 172.21.15.153:40014) 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 -- 172.21.15.153:0/3930552910 learned_addr learned my addr 172.21.15.153:0/3930552910 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:50.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30098e0700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fbce0 0x7f3004110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:50.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 -- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fbce0 msgr2=0x7f3004110630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:50.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fbce0 0x7f3004110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:50.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 -- 172.21.15.153:0/3930552910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4005040 con 0x7f30040fcf90 2024-04-05T03:01:50.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f30090df700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4013870 tx=0x7f2ff4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:50.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff400e070 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300410c410 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ff401c400 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.732+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff400b5f0 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.733+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300410c8f0 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.734+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300405f5f0 con 0x7f30040fcf90 2024-04-05T03:01:50.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.735+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2ff401ca50 con 0x7f30040fcf90 2024-04-05T03:01:50.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.735+0000 7f2ffa7fc700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 0x7f2ff006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:50.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.735+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2ff40893d0 con 0x7f30040fcf90 2024-04-05T03:01:50.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.736+0000 7f30098e0700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 0x7f2ff006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:50.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.736+0000 7f30098e0700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 0x7f2ff006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3000004600 tx=0x7f300000b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:50.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:50.739+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2ff4055750 con 0x7f30040fcf90 2024-04-05T03:01:51.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.044+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3004101260 con 0x7f2ff00692b0 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: pgmap v231: 65 pgs: 65 active+clean; 40 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: Upgrade: Updating mgr.smithi203.iakodz 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:51.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:50 smithi153 ceph-mon[29381]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:01:51.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.046+0000 7f2ffa7fc700 1 -- 172.21.15.153:0/3930552910 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f3004101260 con 0x7f2ff00692b0 2024-04-05T03:01:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: pgmap v231: 65 pgs: 65 active+clean; 40 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-04-05T03:01:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: Upgrade: Updating mgr.smithi203.iakodz 2024-04-05T03:01:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:01:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:01:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:01:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:01:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:50 smithi203 ceph-mon[39911]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:01:51.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 msgr2=0x7f2ff006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 0x7f2ff006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3000004600 tx=0x7f300000b040 comp rx=0 tx=0).stop 2024-04-05T03:01:51.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 msgr2=0x7f3004112b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4013870 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 shutdown_connections 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f2ff00692b0 0x7f2ff006b760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30040fbce0 0x7f3004110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.049+0000 7f300bb44700 1 --2- 172.21.15.153:0/3930552910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30040fcf90 0x7f3004112b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.050+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 >> 172.21.15.153:0/3930552910 conn(0x7f30040f7440 msgr2=0x7f30040f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.050+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 shutdown_connections 2024-04-05T03:01:51.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.050+0000 7f300bb44700 1 -- 172.21.15.153:0/3930552910 wait complete. 2024-04-05T03:01:51.064 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.192+0000 7fe6073d0700 1 -- 172.21.15.153:0/1400232338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 msgr2=0x7fe6000ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.192+0000 7fe6073d0700 1 --2- 172.21.15.153:0/1400232338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000ff9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc004940 tx=0x7fe5fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 -- 172.21.15.153:0/1400232338 shutdown_connections 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 --2- 172.21.15.153:0/1400232338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe600102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 --2- 172.21.15.153:0/1400232338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000ff9c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 -- 172.21.15.153:0/1400232338 >> 172.21.15.153:0/1400232338 conn(0x7fe6000f7430 msgr2=0x7fe6000f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 -- 172.21.15.153:0/1400232338 shutdown_connections 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 -- 172.21.15.153:0/1400232338 wait complete. 2024-04-05T03:01:51.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 Processor -- start 2024-04-05T03:01:51.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.193+0000 7fe6073d0700 1 -- start start 2024-04-05T03:01:51.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe6073d0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe6073d0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe6000fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe6073d0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000fbe50 con 0x7fe6000662e0 2024-04-05T03:01:51.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe6073d0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000fbf90 con 0x7fe6000fff00 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60516c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60516c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53754/0 (socket says 172.21.15.153:53754) 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60516c700 1 -- 172.21.15.153:0/547263356 learned_addr learned my addr 172.21.15.153:0/547263356 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60496b700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe6000fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60496b700 1 -- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 msgr2=0x7fe6000fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60496b700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60496b700 1 -- 172.21.15.153:0/547263356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc005040 con 0x7fe6000fff00 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60516c700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe60496b700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe6000fb8c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009fd0 tx=0x7fe5f000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f0004e90 con 0x7fe6000fff00 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5f000f040 con 0x7fe6000fff00 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6000fc210 con 0x7fe6000fff00 2024-04-05T03:01:51.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.194+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f000ca50 con 0x7fe6000fff00 2024-04-05T03:01:51.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.195+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe600115a30 con 0x7fe6000fff00 2024-04-05T03:01:51.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.196+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60005f5f0 con 0x7fe6000fff00 2024-04-05T03:01:51.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.196+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe5f0007cc0 con 0x7fe6000fff00 2024-04-05T03:01:51.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.197+0000 7fe5f5ffb700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.197+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe5f0086c90 con 0x7fe6000fff00 2024-04-05T03:01:51.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.197+0000 7fe60516c700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.198+0000 7fe60516c700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 0x7fe5ec06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b0d0 tx=0x7fe5fc009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:51.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.202+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe5f004e190 con 0x7fe6000fff00 2024-04-05T03:01:51.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.512+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6000f8f70 con 0x7fe5ec0692a0 2024-04-05T03:01:51.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.513+0000 7fe5f5ffb700 1 -- 172.21.15.153:0/547263356 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe6000f8f70 con 0x7fe5ec0692a0 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 msgr2=0x7fe5ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 0x7fe5ec06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b0d0 tx=0x7fe5fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 msgr2=0x7fe6000fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe6000fb8c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009fd0 tx=0x7fe5f000b040 comp rx=0 tx=0).stop 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 shutdown_connections 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6000662e0 0x7fe6000fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 --2- 172.21.15.153:0/547263356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6000fff00 0x7fe6000fb8c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 >> 172.21.15.153:0/547263356 conn(0x7fe6000f7430 msgr2=0x7fe6000f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:51.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 shutdown_connections 2024-04-05T03:01:51.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.516+0000 7fe6073d0700 1 -- 172.21.15.153:0/547263356 wait complete. 2024-04-05T03:01:51.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 -- 172.21.15.153:0/2177576750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 msgr2=0x7f65ec104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 --2- 172.21.15.153:0/2177576750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec104760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f65dc004940 tx=0x7f65dc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:51.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 -- 172.21.15.153:0/2177576750 shutdown_connections 2024-04-05T03:01:51.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 --2- 172.21.15.153:0/2177576750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec104760 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 --2- 172.21.15.153:0/2177576750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 0x7f65ec101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 -- 172.21.15.153:0/2177576750 >> 172.21.15.153:0/2177576750 conn(0x7f65ec0f7430 msgr2=0x7f65ec0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:51.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 -- 172.21.15.153:0/2177576750 shutdown_connections 2024-04-05T03:01:51.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.660+0000 7f65f3c91700 1 -- 172.21.15.153:0/2177576750 wait complete. 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 Processor -- start 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 -- start start 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 0x7f65ec10fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec1107d0 con 0x7f65ec102380 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f3c91700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec110fd0 con 0x7f65ec066260 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f122c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f122c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53774/0 (socket says 172.21.15.153:53774) 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f122c700 1 -- 172.21.15.153:0/1119584924 learned_addr learned my addr 172.21.15.153:0/1119584924 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.661+0000 7f65f1a2d700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 0x7f65ec10fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f122c700 1 -- 172.21.15.153:0/1119584924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 msgr2=0x7f65ec10fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f122c700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 0x7f65ec10fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f122c700 1 -- 172.21.15.153:0/1119584924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc005040 con 0x7f65ec102380 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f122c700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0043d0 tx=0x7f65dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65dc00e070 con 0x7f65ec102380 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ec111250 con 0x7f65ec102380 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65dc01c400 con 0x7f65ec102380 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65dc0125e0 con 0x7f65ec102380 2024-04-05T03:01:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.662+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ec111730 con 0x7f65ec102380 2024-04-05T03:01:51.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.664+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ec05f5f0 con 0x7f65ec102380 2024-04-05T03:01:51.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.665+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f65dc01ca50 con 0x7f65ec102380 2024-04-05T03:01:51.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.665+0000 7f65e27fc700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 0x7f65d806b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:51.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.665+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f65dc088860 con 0x7f65ec102380 2024-04-05T03:01:51.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.666+0000 7f65f1a2d700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 0x7f65d806b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.666+0000 7f65f1a2d700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 0x7f65d806b7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009b80 tx=0x7f65e800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.672+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f65dc058530 con 0x7f65ec102380 2024-04-05T03:01:51.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.949+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f65ec111a10 con 0x7f65d80692f0 2024-04-05T03:01:51.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.967+0000 7f65e27fc700 1 -- 172.21.15.153:0/1119584924 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f65ec111a10 con 0x7f65d80692f0 2024-04-05T03:01:51.967 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 102s ago 7m 26.6M - 0.25.0 c8568f914cd2 db2172977a37 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 102s ago 8m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (6m) 107s ago 6m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 102s ago 7m 88.5M - 9.4.7 954c08fa6188 0fafa83f96b9 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (119s) 102s ago 119s 14.5M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (2m) 102s ago 2m 11.1M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (117s) 107s ago 116s 13.2M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (2m) 107s ago 2m 16.6M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (9m) 102s ago 9m 479M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (6m) 107s ago 6m 435M - 17.2.7-805-g6fda41ff a49113eacc58 3ebddafb5706 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 102s ago 10m 43.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (6m) 107s ago 6m 38.2M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 102s ago 7m 15.8M - 1.5.0 0da6a335fe13 61d64f55ec69 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (6m) 107s ago 6m 16.1M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T03:01:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 102s ago 5m 46.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 102s ago 5m 46.5M 4629M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 102s ago 4m 44.9M 4629M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (4m) 107s ago 4m 43.7M 4970M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (3m) 107s ago 3m 44.2M 4970M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (3m) 107s ago 3m 42.1M 4970M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:01:51.969 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (6m) 107s ago 6m 56.7M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 msgr2=0x7f65d806b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 0x7f65d806b7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009b80 tx=0x7f65e800b040 comp rx=0 tx=0).stop 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 msgr2=0x7f65ec1101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0043d0 tx=0x7f65dc009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 shutdown_connections 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f65d80692f0 0x7f65d806b7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65ec066260 0x7f65ec10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 --2- 172.21.15.153:0/1119584924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ec102380 0x7f65ec1101d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:51.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 >> 172.21.15.153:0/1119584924 conn(0x7f65ec0f7430 msgr2=0x7f65ec0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:51.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 shutdown_connections 2024-04-05T03:01:51.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:51.970+0000 7f65f3c91700 1 -- 172.21.15.153:0/1119584924 wait complete. 2024-04-05T03:01:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 -- 172.21.15.153:0/945561731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 msgr2=0x7efd380fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/945561731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd380fc0e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efd2c004940 tx=0x7efd2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 -- 172.21.15.153:0/945561731 shutdown_connections 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/945561731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd380fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/945561731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd380fc0e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 -- 172.21.15.153:0/945561731 >> 172.21.15.153:0/945561731 conn(0x7efd380f7430 msgr2=0x7efd380f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 -- 172.21.15.153:0/945561731 shutdown_connections 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.104+0000 7efd3f1ee700 1 -- 172.21.15.153:0/945561731 wait complete. 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 Processor -- start 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 -- start start 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd38112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380663b0 con 0x7efd380fbcd0 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3f1ee700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380664f0 con 0x7efd380fcf80 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3cf8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3cf8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53802/0 (socket says 172.21.15.153:53802) 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.105+0000 7efd3cf8a700 1 -- 172.21.15.153:0/4072354202 learned_addr learned my addr 172.21.15.153:0/4072354202 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd37fff700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd38112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3cf8a700 1 -- 172.21.15.153:0/4072354202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 msgr2=0x7efd38112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3cf8a700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd38112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3cf8a700 1 -- 172.21.15.153:0/4072354202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c005040 con 0x7efd380fbcd0 2024-04-05T03:01:52.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd37fff700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd38112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3cf8a700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd2c000c00 tx=0x7efd2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd2c021070 con 0x7efd380fbcd0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd38113120 con 0x7efd380fbcd0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.106+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd38113600 con 0x7efd380fbcd0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.107+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd2c01c400 con 0x7efd380fbcd0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.107+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd2c00e070 con 0x7efd380fbcd0 2024-04-05T03:01:52.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.108+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1c0052e0 con 0x7efd380fbcd0 2024-04-05T03:01:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.109+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efd2c01ca50 con 0x7efd380fbcd0 2024-04-05T03:01:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.109+0000 7efd357fa700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 0x7efd2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.109+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7efd2c0882f0 con 0x7efd380fbcd0 2024-04-05T03:01:52.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.109+0000 7efd37fff700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 0x7efd2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.110+0000 7efd37fff700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 0x7efd2806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd24009b80 tx=0x7efd2400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:52.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.116+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efd2c054670 con 0x7efd380fbcd0 2024-04-05T03:01:52.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.545+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efd1c005da0 con 0x7efd380fbcd0 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.546+0000 7efd357fa700 1 -- 172.21.15.153:0/4072354202 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7efd2c057c90 con 0x7efd380fbcd0 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:52.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 14 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:01:52.547 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 msgr2=0x7efd2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 0x7efd2806b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd24009b80 tx=0x7efd2400b040 comp rx=0 tx=0).stop 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 msgr2=0x7efd38110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd2c000c00 tx=0x7efd2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 shutdown_connections 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7efd280692a0 0x7efd2806b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd380fbcd0 0x7efd38110630 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 --2- 172.21.15.153:0/4072354202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd380fcf80 0x7efd38112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 >> 172.21.15.153:0/4072354202 conn(0x7efd380f7430 msgr2=0x7efd380f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:52.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 shutdown_connections 2024-04-05T03:01:52.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.549+0000 7efd3f1ee700 1 -- 172.21.15.153:0/4072354202 wait complete. 2024-04-05T03:01:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1974792104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fbcd0 msgr2=0x7f5f640fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1974792104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fbcd0 0x7f5f640fc0e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5f60004940 tx=0x7f5f60009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1974792104 shutdown_connections 2024-04-05T03:01:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1974792104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fcf80 0x7f5f640fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1974792104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fbcd0 0x7f5f640fc0e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1974792104 >> 172.21.15.153:0/1974792104 conn(0x7f5f640f7430 msgr2=0x7f5f640f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1974792104 shutdown_connections 2024-04-05T03:01:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.693+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1974792104 wait complete. 2024-04-05T03:01:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.694+0000 7f5f6c77a700 1 Processor -- start 2024-04-05T03:01:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.694+0000 7f5f6c77a700 1 -- start start 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.694+0000 7f5f6c77a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.694+0000 7f5f6c77a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 0x7f5f6410ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6c77a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6410d440 con 0x7f5f640fcf80 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6c77a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6410dc90 con 0x7f5f640fbcd0 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:40092/0 (socket says 172.21.15.153:40092) 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 -- 172.21.15.153:0/1073563426 learned_addr learned my addr 172.21.15.153:0/1073563426 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f69d15700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 0x7f5f6410ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 -- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 msgr2=0x7f5f6410ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 0x7f5f6410ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 -- 172.21.15.153:0/1073563426 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f60005040 con 0x7f5f640fbcd0 2024-04-05T03:01:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f69d15700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 0x7f5f6410ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:01:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.695+0000 7f5f6a516700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000b3b0 tx=0x7f5f60009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.696+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f6000e040 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.696+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6410df10 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.696+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6410e450 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.696+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f6000bcc0 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.696+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f6000e040 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.697+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6405f5f0 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.697+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5f6001c3f0 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.698+0000 7f5f5affd700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 0x7f5f5006da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.698+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5f60087c60 con 0x7f5f640fbcd0 2024-04-05T03:01:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.698+0000 7f5f69d15700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 0x7f5f5006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:01:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.698+0000 7f5f69d15700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 0x7f5f5006da20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009fd0 tx=0x7f5f5400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:01:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:52.703+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5f60057c30 con 0x7f5f640fbcd0 2024-04-05T03:01:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:52 smithi153 ceph-mon[29381]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:52 smithi153 ceph-mon[29381]: pgmap v232: 65 pgs: 65 active+clean; 40 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-04-05T03:01:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:52 smithi153 ceph-mon[29381]: from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:52 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/4072354202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:01:53.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.041+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f6410e7c0 con 0x7f5f5006b570 2024-04-05T03:01:53.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.042+0000 7f5f5affd700 1 -- 172.21.15.153:0/1073563426 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f5f6410e7c0 con 0x7f5f5006b570 2024-04-05T03:01:53.042 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:01:53.042 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:01:53.042 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:01:53.042 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:01:53.043 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:01:53.043 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-04-05T03:01:53.043 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T03:01:53.043 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:01:53.043 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:01:53.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 msgr2=0x7f5f5006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 0x7f5f5006da20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009fd0 tx=0x7f5f5400b040 comp rx=0 tx=0).stop 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 msgr2=0x7f5f6410c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000b3b0 tx=0x7f5f60009e30 comp rx=0 tx=0).stop 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 shutdown_connections 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f5f5006b570 0x7f5f5006da20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f640fbcd0 0x7f5f6410c900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 --2- 172.21.15.153:0/1073563426 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f640fcf80 0x7f5f6410ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.045+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 >> 172.21.15.153:0/1073563426 conn(0x7f5f640f7430 msgr2=0x7f5f640f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.046+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 shutdown_connections 2024-04-05T03:01:53.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:01:53.046+0000 7f5f6c77a700 1 -- 172.21.15.153:0/1073563426 wait complete. 2024-04-05T03:01:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:52 smithi203 ceph-mon[39911]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:52 smithi203 ceph-mon[39911]: pgmap v232: 65 pgs: 65 active+clean; 40 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 3.5 MiB/s wr, 274 op/s 2024-04-05T03:01:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:52 smithi203 ceph-mon[39911]: from='client.24345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:52 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/4072354202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:01:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:53 smithi203 ceph-mon[39911]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:53 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:01:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:53 smithi153 ceph-mon[29381]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:53 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:01:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:54 smithi203 ceph-mon[39911]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:54 smithi203 ceph-mon[39911]: pgmap v233: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-04-05T03:01:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:54 smithi153 ceph-mon[29381]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:01:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:54 smithi153 ceph-mon[29381]: pgmap v233: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 355 op/s 2024-04-05T03:01:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:56 smithi203 ceph-mon[39911]: pgmap v234: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-05T03:01:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:56 smithi153 ceph-mon[29381]: pgmap v234: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-05T03:01:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:58 smithi203 ceph-mon[39911]: pgmap v235: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 428 op/s 2024-04-05T03:01:59.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:01:58 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:01:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:58 smithi153 ceph-mon[29381]: pgmap v235: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 428 op/s 2024-04-05T03:01:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:01:58 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:00 smithi203 ceph-mon[39911]: pgmap v236: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-05T03:02:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:00 smithi153 ceph-mon[29381]: pgmap v236: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-05T03:02:03.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:02 smithi203 ceph-mon[39911]: pgmap v237: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 328 op/s 2024-04-05T03:02:03.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:02 smithi153 ceph-mon[29381]: pgmap v237: 65 pgs: 65 active+clean; 79 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 328 op/s 2024-04-05T03:02:04.933 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:04 smithi203 ceph-mon[39911]: pgmap v238: 65 pgs: 65 active+clean; 91 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 430 op/s 2024-04-05T03:02:05.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:04 smithi153 ceph-mon[29381]: pgmap v238: 65 pgs: 65 active+clean; 91 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 430 op/s 2024-04-05T03:02:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:06 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:06 smithi203 ceph-mon[39911]: pgmap v239: 65 pgs: 65 active+clean; 101 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 480 op/s 2024-04-05T03:02:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:06 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:06 smithi153 ceph-mon[29381]: pgmap v239: 65 pgs: 65 active+clean; 101 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 480 op/s 2024-04-05T03:02:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:08 smithi203 ceph-mon[39911]: pgmap v240: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 491 op/s 2024-04-05T03:02:09.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:08 smithi153 ceph-mon[29381]: pgmap v240: 65 pgs: 65 active+clean; 111 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.8 MiB/s wr, 491 op/s 2024-04-05T03:02:10.957 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:10 smithi203 ceph-mon[39911]: pgmap v241: 65 pgs: 65 active+clean; 115 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 530 op/s 2024-04-05T03:02:11.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:10 smithi153 ceph-mon[29381]: pgmap v241: 65 pgs: 65 active+clean; 115 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 530 op/s 2024-04-05T03:02:13.033 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:12 smithi203 ceph-mon[39911]: pgmap v242: 65 pgs: 65 active+clean; 115 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 434 op/s 2024-04-05T03:02:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:12 smithi153 ceph-mon[29381]: pgmap v242: 65 pgs: 65 active+clean; 115 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 434 op/s 2024-04-05T03:02:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:02:13.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:02:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:02:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:02:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:02:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:02:13.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:13 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:02:13.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:02:13.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:13 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:14 smithi203 ceph-mon[39911]: pgmap v243: 65 pgs: 65 active+clean; 124 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 501 op/s 2024-04-05T03:02:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:14 smithi203 ceph-mon[39911]: mgrmap e20: smithi153.npzyqy(active, since 7m), standbys: smithi203.iakodz 2024-04-05T03:02:15.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:14 smithi153 ceph-mon[29381]: pgmap v243: 65 pgs: 65 active+clean; 124 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 501 op/s 2024-04-05T03:02:15.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:14 smithi153 ceph-mon[29381]: mgrmap e20: smithi153.npzyqy(active, since 7m), standbys: smithi203.iakodz 2024-04-05T03:02:17.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:16 smithi203 ceph-mon[39911]: pgmap v244: 65 pgs: 65 active+clean; 138 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 491 op/s 2024-04-05T03:02:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:16 smithi153 ceph-mon[29381]: pgmap v244: 65 pgs: 65 active+clean; 138 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.2 MiB/s wr, 491 op/s 2024-04-05T03:02:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:18 smithi203 ceph-mon[39911]: pgmap v245: 65 pgs: 65 active+clean; 142 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 440 op/s 2024-04-05T03:02:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:18 smithi153 ceph-mon[29381]: pgmap v245: 65 pgs: 65 active+clean; 142 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.7 MiB/s wr, 440 op/s 2024-04-05T03:02:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:20 smithi153 ceph-mon[29381]: pgmap v246: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 451 op/s 2024-04-05T03:02:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:20 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.697 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.697 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.697 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:20 smithi203 ceph-mon[39911]: pgmap v246: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.8 MiB/s wr, 451 op/s 2024-04-05T03:02:20.697 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:20.698 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:20 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:22.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:22 smithi203 ceph-mon[39911]: pgmap v247: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 349 op/s 2024-04-05T03:02:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:22 smithi153 ceph-mon[29381]: pgmap v247: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 349 op/s 2024-04-05T03:02:23.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 -- 172.21.15.153:0/3619313778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fed50 msgr2=0x7f71580fae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 --2- 172.21.15.153:0/3619313778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fed50 0x7f71580fae90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f714c004940 tx=0x7f714c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 -- 172.21.15.153:0/3619313778 shutdown_connections 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 --2- 172.21.15.153:0/3619313778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fed50 0x7f71580fae90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 --2- 172.21.15.153:0/3619313778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fe400 0x7f71580fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 -- 172.21.15.153:0/3619313778 >> 172.21.15.153:0/3619313778 conn(0x7f71580f6ba0 msgr2=0x7f71580f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.185+0000 7f715cae8700 1 -- 172.21.15.153:0/3619313778 shutdown_connections 2024-04-05T03:02:23.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.186+0000 7f715cae8700 1 -- 172.21.15.153:0/3619313778 wait complete. 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.186+0000 7f715cae8700 1 Processor -- start 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.186+0000 7f715cae8700 1 -- start start 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715cae8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715cae8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fed50 0x7f71580fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715cae8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580fc670 con 0x7f71580fe400 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715cae8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71580fc7e0 con 0x7f71580fed50 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40126/0 (socket says 172.21.15.153:40126) 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 -- 172.21.15.153:0/3805638355 learned_addr learned my addr 172.21.15.153:0/3805638355 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f7155d9b700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fed50 0x7f71580fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 -- 172.21.15.153:0/3805638355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fed50 msgr2=0x7f71580fc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fed50 0x7f71580fc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 -- 172.21.15.153:0/3805638355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c005040 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.187+0000 7f715659c700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f714800eb40 tx=0x7f714800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.188+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7148007cc0 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.188+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f714800f040 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.188+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71580fca60 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.188+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71580fcee0 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.188+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714800ca10 con 0x7f71580fe400 2024-04-05T03:02:23.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.190+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715805f5f0 con 0x7f71580fe400 2024-04-05T03:02:23.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.191+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) v1 ==== 89099+0+0 (secure 0 0 0) 0x7f714800cb70 con 0x7f71580fe400 2024-04-05T03:02:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.191+0000 7f7146ffd700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 0x7f714006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.192+0000 7f7155d9b700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 0x7f714006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.192+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7148089890 con 0x7f71580fe400 2024-04-05T03:02:23.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.192+0000 7f7155d9b700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 0x7f714006d8d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f714c009b30 tx=0x7f714c009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:23.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.196+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7148055c20 con 0x7f71580fe400 2024-04-05T03:02:23.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.443+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7158102450 con 0x7f714006b420 2024-04-05T03:02:23.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.444+0000 7f7146ffd700 1 -- 172.21.15.153:0/3805638355 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f7158102450 con 0x7f714006b420 2024-04-05T03:02:23.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 msgr2=0x7f714006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 0x7f714006d8d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f714c009b30 tx=0x7f714c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 msgr2=0x7f71580fdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f714800eb40 tx=0x7f714800b040 comp rx=0 tx=0).stop 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 shutdown_connections 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f714006b420 0x7f714006d8d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71580fe400 0x7f71580fdbf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.448+0000 7f715cae8700 1 --2- 172.21.15.153:0/3805638355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71580fed50 0x7f71580fc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.449+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 >> 172.21.15.153:0/3805638355 conn(0x7f71580f6ba0 msgr2=0x7f71580f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:23.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.449+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 shutdown_connections 2024-04-05T03:02:23.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.449+0000 7f715cae8700 1 -- 172.21.15.153:0/3805638355 wait complete. 2024-04-05T03:02:23.464 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 -- 172.21.15.153:0/3920808078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 msgr2=0x7fc0740fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 --2- 172.21.15.153:0/3920808078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0740fd400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc068004940 tx=0x7fc068009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 -- 172.21.15.153:0/3920808078 shutdown_connections 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 --2- 172.21.15.153:0/3920808078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0740fd400 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 --2- 172.21.15.153:0/3920808078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc0740fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 -- 172.21.15.153:0/3920808078 >> 172.21.15.153:0/3920808078 conn(0x7fc0740f7480 msgr2=0x7fc0740f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 -- 172.21.15.153:0/3920808078 shutdown_connections 2024-04-05T03:02:23.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.600+0000 7fc079826700 1 -- 172.21.15.153:0/3920808078 wait complete. 2024-04-05T03:02:23.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.601+0000 7fc079826700 1 Processor -- start 2024-04-05T03:02:23.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.601+0000 7fc079826700 1 -- start start 2024-04-05T03:02:23.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.601+0000 7fc079826700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.601+0000 7fc079826700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0741144f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc079826700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074114af0 con 0x7fc0740fcf90 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc079826700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074115340 con 0x7fc0740fbce0 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc0727fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0741144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:48930/0 (socket says 172.21.15.153:48930) 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 -- 172.21.15.153:0/3667900229 learned_addr learned my addr 172.21.15.153:0/3667900229 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 -- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 msgr2=0x7fc0741144f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0741144f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 -- 172.21.15.153:0/3667900229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068005040 con 0x7fc0740fbce0 2024-04-05T03:02:23.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc0727fc700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0741144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.602+0000 7fc072ffd700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc06400eb40 tx=0x7fc06400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.603+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc064007cc0 con 0x7fc0740fbce0 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.603+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc06400f040 con 0x7fc0740fbce0 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.603+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc074115540 con 0x7fc0740fbce0 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.603+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06400ca10 con 0x7fc0740fbce0 2024-04-05T03:02:23.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.603+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc074115a80 con 0x7fc0740fbce0 2024-04-05T03:02:23.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.604+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc074101920 con 0x7fc0740fbce0 2024-04-05T03:02:23.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.605+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) v1 ==== 89099+0+0 (secure 0 0 0) 0x7fc06400cb70 con 0x7fc0740fbce0 2024-04-05T03:02:23.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.605+0000 7fc05b7fe700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 0x7fc05c06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:23.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.605+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc064089900 con 0x7fc0740fbce0 2024-04-05T03:02:23.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.606+0000 7fc0727fc700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 0x7fc05c06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:23.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.606+0000 7fc0727fc700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 0x7fc05c06d8d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc068009b30 tx=0x7fc068009e30 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.612+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc064059720 con 0x7fc0740fbce0 2024-04-05T03:02:23.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.927+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc074115cf0 con 0x7fc05c06b420 2024-04-05T03:02:23.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.928+0000 7fc05b7fe700 1 -- 172.21.15.153:0/3667900229 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fc074115cf0 con 0x7fc05c06b420 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 msgr2=0x7fc05c06d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 0x7fc05c06d8d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc068009b30 tx=0x7fc068009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 msgr2=0x7fc074113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc06400eb40 tx=0x7fc06400b040 comp rx=0 tx=0).stop 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 shutdown_connections 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc05c06b420 0x7fc05c06d8d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0740fbce0 0x7fc074113fb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 --2- 172.21.15.153:0/3667900229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0740fcf90 0x7fc0741144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:23.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 >> 172.21.15.153:0/3667900229 conn(0x7fc0740f7480 msgr2=0x7fc0740663e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:23.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 shutdown_connections 2024-04-05T03:02:23.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:23.931+0000 7fc079826700 1 -- 172.21.15.153:0/3667900229 wait complete. 2024-04-05T03:02:24.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 -- 172.21.15.153:0/1571377222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86900fcf80 msgr2=0x7f86900fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 --2- 172.21.15.153:0/1571377222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86900fcf80 0x7f86900fd3f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f867c004940 tx=0x7f867c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 -- 172.21.15.153:0/1571377222 shutdown_connections 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 --2- 172.21.15.153:0/1571377222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86900fcf80 0x7f86900fd3f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 --2- 172.21.15.153:0/1571377222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f86900fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 -- 172.21.15.153:0/1571377222 >> 172.21.15.153:0/1571377222 conn(0x7f86900f7430 msgr2=0x7f86900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 -- 172.21.15.153:0/1571377222 shutdown_connections 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.086+0000 7f8696f42700 1 -- 172.21.15.153:0/1571377222 wait complete. 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 Processor -- start 2024-04-05T03:02:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 -- start start 2024-04-05T03:02:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f869010b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869010c100 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.087+0000 7f8696f42700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869010c240 con 0x7f86900fbcd0 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40154/0 (socket says 172.21.15.153:40154) 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 -- 172.21.15.153:0/1209447839 learned_addr learned my addr 172.21.15.153:0/1209447839 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 -- 172.21.15.153:0/1209447839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 msgr2=0x7f869010b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f8694cde700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f869010b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f869010b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 -- 172.21.15.153:0/1209447839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c005040 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f8694cde700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f869010b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.088+0000 7f868ffff700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f867c004890 tx=0x7f867c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.089+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867c021070 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.089+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f867c01c8d0 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.089+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867c00e070 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.089+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869010d090 con 0x7f869010bc40 2024-04-05T03:02:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.089+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869010d5a0 con 0x7f869010bc40 2024-04-05T03:02:24.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.091+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869005f5f0 con 0x7f869010bc40 2024-04-05T03:02:24.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.093+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) v1 ==== 89099+0+0 (secure 0 0 0) 0x7f867c01ca40 con 0x7f869010bc40 2024-04-05T03:02:24.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.095+0000 7f868d7fa700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 0x7f868006fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.095+0000 7f8694cde700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 0x7f868006fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.095+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f867c08c110 con 0x7f869010bc40 2024-04-05T03:02:24.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.096+0000 7f8694cde700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 0x7f868006fbb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8684009fd0 tx=0x7f868400b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:24.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.100+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f867c05be60 con 0x7f869010bc40 2024-04-05T03:02:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.406+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f86901012f0 con 0x7f868006d700 2024-04-05T03:02:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.422+0000 7f868d7fa700 1 -- 172.21.15.153:0/1209447839 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f86901012f0 con 0x7f868006d700 2024-04-05T03:02:24.422 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:02:24.422 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 2m ago 8m 26.6M - 0.25.0 c8568f914cd2 db2172977a37 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (9m) 2m ago 9m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (7m) 5s ago 7m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 2m ago 7m 88.5M - 9.4.7 954c08fa6188 0fafa83f96b9 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (2m) 2m ago 2m 14.5M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (2m) 2m ago 2m 11.1M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (2m) 5s ago 2m 44.8M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (2m) 5s ago 2m 70.6M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (10m) 2m ago 10m 479M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (23s) 5s ago 6m 82.5M - 19.0.0-2013-g0028525c d8c35eaf3c25 f4c2fc4aeae0 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (10m) 2m ago 10m 43.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (6m) 5s ago 6m 39.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 2m ago 8m 15.8M - 1.5.0 0da6a335fe13 61d64f55ec69 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (6m) 5s ago 6m 16.0M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 2m ago 6m 46.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:02:24.423 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 2m ago 5m 46.5M 4629M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:02:24.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 2m ago 5m 44.9M 4629M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:02:24.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (4m) 5s ago 4m 94.2M 4970M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:02:24.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (4m) 5s ago 4m 93.6M 4970M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:02:24.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (4m) 5s ago 4m 86.5M 4970M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:02:24.424 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (6m) 5s ago 6m 58.9M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T03:02:24.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 msgr2=0x7f868006fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 0x7f868006fbb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8684009fd0 tx=0x7f868400b040 comp rx=0 tx=0).stop 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 msgr2=0x7f869010ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f867c004890 tx=0x7f867c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 shutdown_connections 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7f868006d700 0x7f868006fbb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86900fbcd0 0x7f869010b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 --2- 172.21.15.153:0/1209447839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f869010bc40 0x7f869010ca90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.426+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 >> 172.21.15.153:0/1209447839 conn(0x7f86900f7430 msgr2=0x7f86900f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.427+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 shutdown_connections 2024-04-05T03:02:24.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.427+0000 7f8696f42700 1 -- 172.21.15.153:0/1209447839 wait complete. 2024-04-05T03:02:24.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 -- 172.21.15.153:0/3356563504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 msgr2=0x7ff514103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/3356563504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514103830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff504004940 tx=0x7ff504009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:24.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 -- 172.21.15.153:0/3356563504 shutdown_connections 2024-04-05T03:02:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/3356563504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 0x7ff514104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/3356563504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514103830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.575+0000 7ff51a4ff700 1 -- 172.21.15.153:0/3356563504 >> 172.21.15.153:0/3356563504 conn(0x7ff5140725a0 msgr2=0x7ff5140749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.576+0000 7ff51a4ff700 1 -- 172.21.15.153:0/3356563504 shutdown_connections 2024-04-05T03:02:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.576+0000 7ff51a4ff700 1 -- 172.21.15.153:0/3356563504 wait complete. 2024-04-05T03:02:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.576+0000 7ff51a4ff700 1 Processor -- start 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff51a4ff700 1 -- start start 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff51a4ff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff51a4ff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 0x7ff5141143f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff51a4ff700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5141149f0 con 0x7ff514103420 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff51a4ff700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff514114b60 con 0x7ff5141046d0 2024-04-05T03:02:24.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff513fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff513fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40168/0 (socket says 172.21.15.153:40168) 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff513fff700 1 -- 172.21.15.153:0/1280654358 learned_addr learned my addr 172.21.15.153:0/1280654358 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.577+0000 7ff5137fe700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 0x7ff5141143f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff513fff700 1 -- 172.21.15.153:0/1280654358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 msgr2=0x7ff5141143f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff513fff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 0x7ff5141143f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff513fff700 1 -- 172.21.15.153:0/1280654358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504005040 con 0x7ff514103420 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff513fff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff504000c00 tx=0x7ff504009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff50400e070 con 0x7ff514103420 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff504013d80 con 0x7ff514103420 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff504014670 con 0x7ff514103420 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff514076950 con 0x7ff514103420 2024-04-05T03:02:24.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.578+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff514076e90 con 0x7ff514103420 2024-04-05T03:02:24.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.580+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5000052e0 con 0x7ff514103420 2024-04-05T03:02:24.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.581+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 20) v1 ==== 89099+0+0 (secure 0 0 0) 0x7ff50401c400 con 0x7ff514103420 2024-04-05T03:02:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.581+0000 7ff510ff9700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 0x7ff4fc06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.581+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff50408b2a0 con 0x7ff514103420 2024-04-05T03:02:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.582+0000 7ff5137fe700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 0x7ff4fc06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.583+0000 7ff5137fe700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 0x7ff4fc06d8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff508009b80 tx=0x7ff50800b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:24.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.586+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff50405b010 con 0x7ff514103420 2024-04-05T03:02:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:24 smithi203 ceph-mon[39911]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:24 smithi203 ceph-mon[39911]: pgmap v248: 65 pgs: 65 active+clean; 166 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-05T03:02:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:24 smithi153 ceph-mon[29381]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:24 smithi153 ceph-mon[29381]: pgmap v248: 65 pgs: 65 active+clean; 166 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-05T03:02:24.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.980+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff500005da0 con 0x7ff514103420 2024-04-05T03:02:24.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.980+0000 7ff510ff9700 1 -- 172.21.15.153:0/1280654358 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7ff50405aba0 con 0x7ff514103420 2024-04-05T03:02:24.980 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 13, 2024-04-05T03:02:24.981 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:02:24.982 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:02:24.982 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.983+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 msgr2=0x7ff4fc06d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.983+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 0x7ff4fc06d8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff508009b80 tx=0x7ff50800b040 comp rx=0 tx=0).stop 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.983+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 msgr2=0x7ff514113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff504000c00 tx=0x7ff504009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 shutdown_connections 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7ff4fc06b420 0x7ff4fc06d8d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff514103420 0x7ff514113eb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 --2- 172.21.15.153:0/1280654358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5141046d0 0x7ff5141143f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 >> 172.21.15.153:0/1280654358 conn(0x7ff5140725a0 msgr2=0x7ff514073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:24.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 shutdown_connections 2024-04-05T03:02:24.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:24.984+0000 7ff51a4ff700 1 -- 172.21.15.153:0/1280654358 wait complete. 2024-04-05T03:02:25.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 -- 172.21.15.153:0/1961352558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fcf90 msgr2=0x7fc0ec0fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/1961352558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec0fd400 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004940 tx=0x7fc0d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 -- 172.21.15.153:0/1961352558 shutdown_connections 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/1961352558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec0fd400 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/1961352558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec0fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.126+0000 7fc0f0f44700 1 -- 172.21.15.153:0/1961352558 >> 172.21.15.153:0/1961352558 conn(0x7fc0ec0f7440 msgr2=0x7fc0ec0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 -- 172.21.15.153:0/1961352558 shutdown_connections 2024-04-05T03:02:25.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 -- 172.21.15.153:0/1961352558 wait complete. 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 Processor -- start 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 -- start start 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec110860 con 0x7fc0ec0fbce0 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.127+0000 7fc0f0f44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec110fa0 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0ea59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40186/0 (socket says 172.21.15.153:40186) 2024-04-05T03:02:25.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0e3fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0ea59c700 1 -- 172.21.15.153:0/4229660985 learned_addr learned my addr 172.21.15.153:0/4229660985 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0e3fff700 1 -- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 msgr2=0x7fc0ec10fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0e3fff700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0e3fff700 1 -- 172.21.15.153:0/4229660985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4005040 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0ea59c700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.128+0000 7fc0e3fff700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec110260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400b970 tx=0x7fc0d4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.129+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0d400e040 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.129+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0d4004290 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.129+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0d401c3f0 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.129+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec111220 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.129+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec1117e0 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.130+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec05f5f0 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.131+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) v1 ==== 89099+0+0 (secure 0 0 0) 0x7fc0d400bbf0 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.131+0000 7fc0e2ffd700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 0x7fc0d806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:25.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.131+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc0d408af10 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.131+0000 7fc0ea59c700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 0x7fc0d806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:25.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.132+0000 7fc0ea59c700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 0x7fc0d806d8e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0041d0 tx=0x7fc0dc00b040 comp rx=0 tx=0).ready entity=mgr.14254 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.138+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc0d405ac80 con 0x7fc0ec0fcf90 2024-04-05T03:02:25.156 INFO:tasks.workunit.client.1.smithi203.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-05T03:02:25.162 INFO:tasks.workunit.client.1.smithi203.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T03:02:25.163 INFO:tasks.workunit.client.1.smithi203.stderr:+ make 2024-04-05T03:02:25.309 INFO:tasks.workunit.client.1.smithi203.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-05T03:02:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.414+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 --> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0ec111c20 con 0x7fc0d806b430 2024-04-05T03:02:25.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.415+0000 7fc0e2ffd700 1 -- 172.21.15.153:0/4229660985 <== mgr.14254 v2:172.21.15.153:6800/663556540 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fc0ec111c20 con 0x7fc0d806b430 2024-04-05T03:02:25.414 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:02:25.415 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:02:25.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 msgr2=0x7fc0d806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 0x7fc0d806d8e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0041d0 tx=0x7fc0dc00b040 comp rx=0 tx=0).stop 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 msgr2=0x7fc0ec110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec110260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400b970 tx=0x7fc0d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 shutdown_connections 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:6800/663556540,v1:172.21.15.153:6801/663556540] conn(0x7fc0d806b430 0x7fc0d806d8e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0ec0fbce0 0x7fc0ec10fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 --2- 172.21.15.153:0/4229660985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0ec0fcf90 0x7fc0ec110260 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 >> 172.21.15.153:0/4229660985 conn(0x7fc0ec0f7440 msgr2=0x7fc0ec0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 shutdown_connections 2024-04-05T03:02:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:25.417+0000 7fc0f0f44700 1 -- 172.21.15.153:0/4229660985 wait complete. 2024-04-05T03:02:25.741 INFO:tasks.workunit.client.1.smithi203.stderr:++ readlink -f fsstress 2024-04-05T03:02:25.744 INFO:tasks.workunit.client.1.smithi203.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-05T03:02:25.744 INFO:tasks.workunit.client.1.smithi203.stderr:+ popd 2024-04-05T03:02:25.745 INFO:tasks.workunit.client.1.smithi203.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T03:02:25.745 INFO:tasks.workunit.client.1.smithi203.stderr:+ popd 2024-04-05T03:02:25.746 INFO:tasks.workunit.client.1.smithi203.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-05T03:02:25.746 INFO:tasks.workunit.client.1.smithi203.stderr:++ mktemp -d -p . 2024-04-05T03:02:25.748 INFO:tasks.workunit.client.1.smithi203.stderr:+ T=./tmp.ZzKx9mB6wg 2024-04-05T03:02:25.749 INFO:tasks.workunit.client.1.smithi203.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.ZzKx9mB6wg -l 1 -n 1000 -p 10 -v 2024-04-05T03:02:25.752 INFO:tasks.workunit.client.1.smithi203.stdout:seed = 1711543941 2024-04-05T03:02:25.755 INFO:tasks.workunit.client.1.smithi203.stdout:0/0: dread - no filename 2024-04-05T03:02:25.756 INFO:tasks.workunit.client.1.smithi203.stdout:0/1: chown . 14926 1 2024-04-05T03:02:25.756 INFO:tasks.workunit.client.1.smithi203.stdout:0/2: rename - no filename 2024-04-05T03:02:25.756 INFO:tasks.workunit.client.1.smithi203.stdout:0/3: dwrite - no filename 2024-04-05T03:02:25.756 INFO:tasks.workunit.client.1.smithi203.stdout:1/0: dread - no filename 2024-04-05T03:02:25.758 INFO:tasks.workunit.client.1.smithi203.stdout:2/0: truncate - no filename 2024-04-05T03:02:25.758 INFO:tasks.workunit.client.1.smithi203.stdout:2/1: write - no filename 2024-04-05T03:02:25.758 INFO:tasks.workunit.client.1.smithi203.stdout:0/4: mknod c0 0 2024-04-05T03:02:25.758 INFO:tasks.workunit.client.1.smithi203.stdout:0/5: truncate - no filename 2024-04-05T03:02:25.758 INFO:tasks.workunit.client.1.smithi203.stdout:0/6: dread - no filename 2024-04-05T03:02:25.759 INFO:tasks.workunit.client.1.smithi203.stdout:3/0: write - no filename 2024-04-05T03:02:25.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/1: dread - no filename 2024-04-05T03:02:25.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/2: dread - no filename 2024-04-05T03:02:25.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/3: read - no filename 2024-04-05T03:02:25.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/4: fsync - no filename 2024-04-05T03:02:25.760 INFO:tasks.workunit.client.1.smithi203.stdout:1/1: creat f0 x:0 0 0 2024-04-05T03:02:25.761 INFO:tasks.workunit.client.1.smithi203.stdout:4/0: rename - no filename 2024-04-05T03:02:25.761 INFO:tasks.workunit.client.1.smithi203.stdout:2/2: creat f0 x:0 0 0 2024-04-05T03:02:25.764 INFO:tasks.workunit.client.1.smithi203.stdout:2/3: symlink l1 0 2024-04-05T03:02:25.766 INFO:tasks.workunit.client.1.smithi203.stdout:4/1: creat f0 x:0 0 0 2024-04-05T03:02:25.767 INFO:tasks.workunit.client.1.smithi203.stdout:2/4: symlink l2 0 2024-04-05T03:02:25.767 INFO:tasks.workunit.client.1.smithi203.stdout:6/0: unlink - no file 2024-04-05T03:02:25.767 INFO:tasks.workunit.client.1.smithi203.stdout:6/1: write - no filename 2024-04-05T03:02:25.767 INFO:tasks.workunit.client.1.smithi203.stdout:6/2: rmdir - no directory 2024-04-05T03:02:25.767 INFO:tasks.workunit.client.1.smithi203.stdout:6/3: write - no filename 2024-04-05T03:02:25.768 INFO:tasks.workunit.client.1.smithi203.stdout:6/4: chown . 104698 1 2024-04-05T03:02:25.768 INFO:tasks.workunit.client.1.smithi203.stdout:6/5: chown . 198540454 1 2024-04-05T03:02:25.769 INFO:tasks.workunit.client.1.smithi203.stdout:2/5: chown l2 129358 1 2024-04-05T03:02:25.771 INFO:tasks.workunit.client.1.smithi203.stdout:5/0: creat f0 x:0 0 0 2024-04-05T03:02:25.771 INFO:tasks.workunit.client.1.smithi203.stdout:7/0: link - no file 2024-04-05T03:02:25.771 INFO:tasks.workunit.client.1.smithi203.stdout:4/2: creat f1 x:0 0 0 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/0: dread - no filename 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/1: fsync - no filename 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/2: rmdir - no directory 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/3: write - no filename 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/4: unlink - no file 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:1/2: dwrite f0 [0,4194304] 0 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/5: dread - no filename 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:8/6: dwrite - no filename 2024-04-05T03:02:25.775 INFO:tasks.workunit.client.1.smithi203.stdout:7/1: creat f0 x:0 0 0 2024-04-05T03:02:25.777 INFO:tasks.workunit.client.1.smithi203.stdout:4/3: rename f1 to f2 0 2024-04-05T03:02:25.778 INFO:tasks.workunit.client.1.smithi203.stdout:2/6: link f0 f3 0 2024-04-05T03:02:25.778 INFO:tasks.workunit.client.1.smithi203.stdout:9/0: readlink - no filename 2024-04-05T03:02:25.780 INFO:tasks.workunit.client.1.smithi203.stdout:1/3: mknod c1 0 2024-04-05T03:02:25.783 INFO:tasks.workunit.client.1.smithi203.stdout:7/2: creat f1 x:0 0 0 2024-04-05T03:02:25.784 INFO:tasks.workunit.client.1.smithi203.stdout:2/7: symlink l4 0 2024-04-05T03:02:25.787 INFO:tasks.workunit.client.1.smithi203.stdout:8/7: creat f0 x:0 0 0 2024-04-05T03:02:25.789 INFO:tasks.workunit.client.1.smithi203.stdout:5/1: dwrite f0 [0,4194304] 0 2024-04-05T03:02:25.789 INFO:tasks.workunit.client.1.smithi203.stdout:7/3: creat f2 x:0 0 0 2024-04-05T03:02:25.790 INFO:tasks.workunit.client.1.smithi203.stdout:2/8: mkdir d5 0 2024-04-05T03:02:25.791 INFO:tasks.workunit.client.1.smithi203.stdout:2/9: chown f0 206 1 2024-04-05T03:02:25.793 INFO:tasks.workunit.client.1.smithi203.stdout:1/4: link f0 f2 0 2024-04-05T03:02:25.794 INFO:tasks.workunit.client.1.smithi203.stdout:4/4: dwrite f0 [0,4194304] 0 2024-04-05T03:02:25.794 INFO:tasks.workunit.client.1.smithi203.stdout:4/5: readlink - no filename 2024-04-05T03:02:25.795 INFO:tasks.workunit.client.1.smithi203.stdout:5/2: rename f0 to f1 0 2024-04-05T03:02:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:25 smithi203 ceph-mon[39911]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:25 smithi203 ceph-mon[39911]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:25 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1280654358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:02:25.801 INFO:tasks.workunit.client.1.smithi203.stdout:2/10: getdents d5 0 2024-04-05T03:02:25.801 INFO:tasks.workunit.client.1.smithi203.stdout:5/3: creat f2 x:0 0 0 2024-04-05T03:02:25.804 INFO:tasks.workunit.client.1.smithi203.stdout:1/5: dread f2 [0,4194304] 0 2024-04-05T03:02:25.804 INFO:tasks.workunit.client.1.smithi203.stdout:2/11: unlink l1 0 2024-04-05T03:02:25.805 INFO:tasks.workunit.client.1.smithi203.stdout:2/12: rename d5 to d5/d6 22 2024-04-05T03:02:25.806 INFO:tasks.workunit.client.1.smithi203.stdout:5/4: mkdir d3 0 2024-04-05T03:02:25.808 INFO:tasks.workunit.client.1.smithi203.stdout:2/13: symlink d5/l7 0 2024-04-05T03:02:25.810 INFO:tasks.workunit.client.1.smithi203.stdout:5/5: mkdir d3/d4 0 2024-04-05T03:02:25.811 INFO:tasks.workunit.client.1.smithi203.stdout:1/6: dread f2 [0,4194304] 0 2024-04-05T03:02:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:25 smithi153 ceph-mon[29381]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:25 smithi153 ceph-mon[29381]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:25 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1280654358' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:02:25.862 INFO:tasks.workunit.client.1.smithi203.stdout:2/14: fsync f0 0 2024-04-05T03:02:25.863 INFO:tasks.workunit.client.1.smithi203.stdout:2/15: chown l4 6 1 2024-04-05T03:02:25.863 INFO:tasks.workunit.client.1.smithi203.stdout:7/4: fsync f2 0 2024-04-05T03:02:25.863 INFO:tasks.workunit.client.1.smithi203.stdout:2/16: chown d5/l7 520532 1 2024-04-05T03:02:25.866 INFO:tasks.workunit.client.1.smithi203.stdout:2/17: creat d5/f8 x:0 0 0 2024-04-05T03:02:25.868 INFO:tasks.workunit.client.1.smithi203.stdout:7/5: rename f1 to f3 0 2024-04-05T03:02:25.872 INFO:tasks.workunit.client.1.smithi203.stdout:7/6: write f3 [976455,117199] 0 2024-04-05T03:02:25.872 INFO:tasks.workunit.client.1.smithi203.stdout:4/6: truncate f0 3346703 0 2024-04-05T03:02:25.874 INFO:tasks.workunit.client.1.smithi203.stdout:4/7: fdatasync f0 0 2024-04-05T03:02:25.876 INFO:tasks.workunit.client.1.smithi203.stdout:5/6: getdents d3 0 2024-04-05T03:02:25.876 INFO:tasks.workunit.client.1.smithi203.stdout:4/8: write f2 [207435,87574] 0 2024-04-05T03:02:25.877 INFO:tasks.workunit.client.1.smithi203.stdout:2/18: write f3 [293636,48921] 0 2024-04-05T03:02:25.877 INFO:tasks.workunit.client.1.smithi203.stdout:1/7: truncate f0 48733 0 2024-04-05T03:02:25.878 INFO:tasks.workunit.client.1.smithi203.stdout:5/7: truncate f1 4670080 0 2024-04-05T03:02:25.884 INFO:tasks.workunit.client.1.smithi203.stdout:4/9: rename f0 to f3 0 2024-04-05T03:02:25.885 INFO:tasks.workunit.client.1.smithi203.stdout:4/10: chown f2 2288154 1 2024-04-05T03:02:25.889 INFO:tasks.workunit.client.1.smithi203.stdout:2/19: rename f0 to d5/f9 0 2024-04-05T03:02:25.890 INFO:tasks.workunit.client.1.smithi203.stdout:2/20: stat l2 0 2024-04-05T03:02:25.890 INFO:tasks.workunit.client.1.smithi203.stdout:5/8: getdents d3/d4 0 2024-04-05T03:02:25.891 INFO:tasks.workunit.client.1.smithi203.stdout:1/8: link c1 c3 0 2024-04-05T03:02:25.893 INFO:tasks.workunit.client.1.smithi203.stdout:2/21: unlink d5/l7 0 2024-04-05T03:02:25.894 INFO:tasks.workunit.client.1.smithi203.stdout:5/9: dread f1 [0,4194304] 0 2024-04-05T03:02:25.895 INFO:tasks.workunit.client.1.smithi203.stdout:5/10: chown d3 128966893 1 2024-04-05T03:02:25.896 INFO:tasks.workunit.client.1.smithi203.stdout:4/11: link f2 f4 0 2024-04-05T03:02:25.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/22: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:25.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/23: write d5/f8 [606322,94827] 0 2024-04-05T03:02:25.898 INFO:tasks.workunit.client.1.smithi203.stdout:4/12: unlink f4 0 2024-04-05T03:02:25.899 INFO:tasks.workunit.client.1.smithi203.stdout:5/11: creat d3/d4/f5 x:0 0 0 2024-04-05T03:02:25.899 INFO:tasks.workunit.client.1.smithi203.stdout:1/9: stat c3 0 2024-04-05T03:02:25.900 INFO:tasks.workunit.client.1.smithi203.stdout:5/12: truncate f2 388954 0 2024-04-05T03:02:25.900 INFO:tasks.workunit.client.1.smithi203.stdout:4/13: creat f5 x:0 0 0 2024-04-05T03:02:25.901 INFO:tasks.workunit.client.1.smithi203.stdout:5/13: write d3/d4/f5 [264771,9364] 0 2024-04-05T03:02:25.902 INFO:tasks.workunit.client.1.smithi203.stdout:5/14: rename d3/d4 to d3/d4/d6 22 2024-04-05T03:02:25.903 INFO:tasks.workunit.client.1.smithi203.stdout:1/10: rename c3 to c4 0 2024-04-05T03:02:25.907 INFO:tasks.workunit.client.1.smithi203.stdout:5/15: creat d3/f7 x:0 0 0 2024-04-05T03:02:25.908 INFO:tasks.workunit.client.1.smithi203.stdout:2/24: fsync d5/f9 0 2024-04-05T03:02:25.910 INFO:tasks.workunit.client.1.smithi203.stdout:1/11: chown c1 882286 1 2024-04-05T03:02:25.912 INFO:tasks.workunit.client.1.smithi203.stdout:2/25: creat d5/fa x:0 0 0 2024-04-05T03:02:25.914 INFO:tasks.workunit.client.1.smithi203.stdout:1/12: read f2 [5035,68701] 0 2024-04-05T03:02:25.914 INFO:tasks.workunit.client.1.smithi203.stdout:2/26: write f3 [1278838,95108] 0 2024-04-05T03:02:25.916 INFO:tasks.workunit.client.1.smithi203.stdout:2/27: chown l4 159299314 1 2024-04-05T03:02:25.917 INFO:tasks.workunit.client.1.smithi203.stdout:1/13: symlink l5 0 2024-04-05T03:02:25.920 INFO:tasks.workunit.client.1.smithi203.stdout:5/16: dwrite f2 [0,4194304] 0 2024-04-05T03:02:25.921 INFO:tasks.workunit.client.1.smithi203.stdout:4/14: dwrite f2 [0,4194304] 0 2024-04-05T03:02:25.924 INFO:tasks.workunit.client.1.smithi203.stdout:4/15: mknod c6 0 2024-04-05T03:02:25.925 INFO:tasks.workunit.client.1.smithi203.stdout:1/14: rename c4 to c6 0 2024-04-05T03:02:25.927 INFO:tasks.workunit.client.1.smithi203.stdout:5/17: mkdir d3/d4/d8 0 2024-04-05T03:02:25.927 INFO:tasks.workunit.client.1.smithi203.stdout:4/16: creat f7 x:0 0 0 2024-04-05T03:02:25.930 INFO:tasks.workunit.client.1.smithi203.stdout:2/28: dwrite d5/f8 [0,4194304] 0 2024-04-05T03:02:25.931 INFO:tasks.workunit.client.1.smithi203.stdout:2/29: fsync d5/f9 0 2024-04-05T03:02:25.932 INFO:tasks.workunit.client.1.smithi203.stdout:2/30: write d5/fa [39752,5940] 0 2024-04-05T03:02:25.934 INFO:tasks.workunit.client.1.smithi203.stdout:4/17: dread f3 [0,4194304] 0 2024-04-05T03:02:25.936 INFO:tasks.workunit.client.1.smithi203.stdout:2/31: mknod d5/cb 0 2024-04-05T03:02:25.945 INFO:tasks.workunit.client.1.smithi203.stdout:2/32: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:25.963 INFO:tasks.workunit.client.1.smithi203.stdout:5/18: fdatasync f2 0 2024-04-05T03:02:25.965 INFO:tasks.workunit.client.1.smithi203.stdout:5/19: mknod d3/c9 0 2024-04-05T03:02:25.968 INFO:tasks.workunit.client.1.smithi203.stdout:5/20: symlink d3/la 0 2024-04-05T03:02:25.970 INFO:tasks.workunit.client.1.smithi203.stdout:5/21: creat d3/d4/fb x:0 0 0 2024-04-05T03:02:25.970 INFO:tasks.workunit.client.1.smithi203.stdout:4/18: truncate f2 2891841 0 2024-04-05T03:02:25.971 INFO:tasks.workunit.client.1.smithi203.stdout:4/19: dread - f5 zero size 2024-04-05T03:02:25.971 INFO:tasks.workunit.client.1.smithi203.stdout:2/33: truncate f3 987416 0 2024-04-05T03:02:25.972 INFO:tasks.workunit.client.1.smithi203.stdout:5/22: mkdir d3/d4/dc 0 2024-04-05T03:02:25.974 INFO:tasks.workunit.client.1.smithi203.stdout:5/23: creat d3/fd x:0 0 0 2024-04-05T03:02:25.976 INFO:tasks.workunit.client.1.smithi203.stdout:5/24: creat d3/d4/fe x:0 0 0 2024-04-05T03:02:25.981 INFO:tasks.workunit.client.1.smithi203.stdout:5/25: creat d3/d4/dc/ff x:0 0 0 2024-04-05T03:02:25.982 INFO:tasks.workunit.client.1.smithi203.stdout:5/26: chown d3/d4 736456861 1 2024-04-05T03:02:25.983 INFO:tasks.workunit.client.1.smithi203.stdout:4/20: dwrite f2 [0,4194304] 0 2024-04-05T03:02:25.985 INFO:tasks.workunit.client.1.smithi203.stdout:2/34: truncate d5/f9 561286 0 2024-04-05T03:02:25.986 INFO:tasks.workunit.client.1.smithi203.stdout:2/35: chown d5/f8 17 1 2024-04-05T03:02:25.987 INFO:tasks.workunit.client.1.smithi203.stdout:4/21: rename f3 to f8 0 2024-04-05T03:02:25.988 INFO:tasks.workunit.client.1.smithi203.stdout:4/22: write f7 [283917,79211] 0 2024-04-05T03:02:25.989 INFO:tasks.workunit.client.1.smithi203.stdout:5/27: creat d3/d4/d8/f10 x:0 0 0 2024-04-05T03:02:25.994 INFO:tasks.workunit.client.1.smithi203.stdout:4/23: link f7 f9 0 2024-04-05T03:02:25.998 INFO:tasks.workunit.client.1.smithi203.stdout:5/28: dwrite d3/d4/d8/f10 [0,4194304] 0 2024-04-05T03:02:26.000 INFO:tasks.workunit.client.1.smithi203.stdout:4/24: chown f8 1982 1 2024-04-05T03:02:26.000 INFO:tasks.workunit.client.1.smithi203.stdout:4/25: stat f8 0 2024-04-05T03:02:26.002 INFO:tasks.workunit.client.1.smithi203.stdout:5/29: unlink d3/d4/dc/ff 0 2024-04-05T03:02:26.008 INFO:tasks.workunit.client.1.smithi203.stdout:5/30: mknod d3/d4/dc/c11 0 2024-04-05T03:02:26.020 INFO:tasks.workunit.client.1.smithi203.stdout:2/36: dwrite f3 [0,4194304] 0 2024-04-05T03:02:26.025 INFO:tasks.workunit.client.1.smithi203.stdout:2/37: rmdir d5 39 2024-04-05T03:02:26.026 INFO:tasks.workunit.client.1.smithi203.stdout:5/31: dwrite d3/d4/fb [0,4194304] 0 2024-04-05T03:02:26.027 INFO:tasks.workunit.client.1.smithi203.stdout:4/26: dwrite f9 [0,4194304] 0 2024-04-05T03:02:26.029 INFO:tasks.workunit.client.1.smithi203.stdout:5/32: dread - d3/d4/fe zero size 2024-04-05T03:02:26.030 INFO:tasks.workunit.client.1.smithi203.stdout:4/27: mkdir da 0 2024-04-05T03:02:26.030 INFO:tasks.workunit.client.1.smithi203.stdout:5/33: stat d3/d4/d8/f10 0 2024-04-05T03:02:26.031 INFO:tasks.workunit.client.1.smithi203.stdout:2/38: write d5/f9 [5003711,81211] 0 2024-04-05T03:02:26.031 INFO:tasks.workunit.client.1.smithi203.stdout:2/39: stat l2 0 2024-04-05T03:02:26.032 INFO:tasks.workunit.client.1.smithi203.stdout:4/28: write f7 [2678857,21768] 0 2024-04-05T03:02:26.034 INFO:tasks.workunit.client.1.smithi203.stdout:5/34: mkdir d3/d4/d8/d12 0 2024-04-05T03:02:26.035 INFO:tasks.workunit.client.1.smithi203.stdout:5/35: fsync f2 0 2024-04-05T03:02:26.037 INFO:tasks.workunit.client.1.smithi203.stdout:5/36: write d3/f7 [857253,120219] 0 2024-04-05T03:02:26.038 INFO:tasks.workunit.client.1.smithi203.stdout:4/29: dread f2 [0,4194304] 0 2024-04-05T03:02:26.038 INFO:tasks.workunit.client.1.smithi203.stdout:2/40: link d5/cb d5/cc 0 2024-04-05T03:02:26.039 INFO:tasks.workunit.client.1.smithi203.stdout:5/37: write d3/f7 [403463,78341] 0 2024-04-05T03:02:26.041 INFO:tasks.workunit.client.1.smithi203.stdout:4/30: dread - f5 zero size 2024-04-05T03:02:26.043 INFO:tasks.workunit.client.1.smithi203.stdout:5/38: mknod d3/d4/dc/c13 0 2024-04-05T03:02:26.045 INFO:tasks.workunit.client.1.smithi203.stdout:2/41: dread d5/f8 [0,4194304] 0 2024-04-05T03:02:26.051 INFO:tasks.workunit.client.1.smithi203.stdout:2/42: dread f3 [4194304,4194304] 0 2024-04-05T03:02:26.054 INFO:tasks.workunit.client.1.smithi203.stdout:2/43: fsync d5/f9 0 2024-04-05T03:02:26.054 INFO:tasks.workunit.client.1.smithi203.stdout:5/39: dwrite f1 [4194304,4194304] 0 2024-04-05T03:02:26.056 INFO:tasks.workunit.client.1.smithi203.stdout:5/40: truncate d3/d4/fe 895427 0 2024-04-05T03:02:26.056 INFO:tasks.workunit.client.1.smithi203.stdout:2/44: symlink d5/ld 0 2024-04-05T03:02:26.057 INFO:tasks.workunit.client.1.smithi203.stdout:5/41: write d3/d4/fe [1157377,32490] 0 2024-04-05T03:02:26.059 INFO:tasks.workunit.client.1.smithi203.stdout:2/45: mknod d5/ce 0 2024-04-05T03:02:26.062 INFO:tasks.workunit.client.1.smithi203.stdout:2/46: mknod d5/cf 0 2024-04-05T03:02:26.062 INFO:tasks.workunit.client.1.smithi203.stdout:5/42: dread d3/d4/d8/f10 [0,4194304] 0 2024-04-05T03:02:26.065 INFO:tasks.workunit.client.1.smithi203.stdout:5/43: symlink d3/d4/dc/l14 0 2024-04-05T03:02:26.068 INFO:tasks.workunit.client.1.smithi203.stdout:5/44: symlink d3/d4/dc/l15 0 2024-04-05T03:02:26.071 INFO:tasks.workunit.client.1.smithi203.stdout:2/47: dwrite f3 [4194304,4194304] 0 2024-04-05T03:02:26.078 INFO:tasks.workunit.client.1.smithi203.stdout:5/45: dwrite d3/d4/fb [0,4194304] 0 2024-04-05T03:02:26.078 INFO:tasks.workunit.client.1.smithi203.stdout:2/48: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:26.082 INFO:tasks.workunit.client.1.smithi203.stdout:2/49: unlink d5/cf 0 2024-04-05T03:02:26.083 INFO:tasks.workunit.client.1.smithi203.stdout:5/46: rename d3/d4/dc/c13 to d3/d4/dc/c16 0 2024-04-05T03:02:26.084 INFO:tasks.workunit.client.1.smithi203.stdout:2/50: unlink l4 0 2024-04-05T03:02:26.085 INFO:tasks.workunit.client.1.smithi203.stdout:2/51: chown d5/fa 58017217 1 2024-04-05T03:02:26.087 INFO:tasks.workunit.client.1.smithi203.stdout:2/52: mkdir d5/d10 0 2024-04-05T03:02:26.088 INFO:tasks.workunit.client.1.smithi203.stdout:5/47: dread d3/f7 [0,4194304] 0 2024-04-05T03:02:26.090 INFO:tasks.workunit.client.1.smithi203.stdout:5/48: symlink d3/d4/dc/l17 0 2024-04-05T03:02:26.091 INFO:tasks.workunit.client.1.smithi203.stdout:5/49: write d3/f7 [309885,98981] 0 2024-04-05T03:02:26.094 INFO:tasks.workunit.client.1.smithi203.stdout:5/50: rename f1 to d3/f18 0 2024-04-05T03:02:26.095 INFO:tasks.workunit.client.1.smithi203.stdout:5/51: write d3/fd [233058,28155] 0 2024-04-05T03:02:26.100 INFO:tasks.workunit.client.1.smithi203.stdout:5/52: dread d3/d4/d8/f10 [0,4194304] 0 2024-04-05T03:02:26.102 INFO:tasks.workunit.client.1.smithi203.stdout:2/53: rmdir d5 39 2024-04-05T03:02:26.104 INFO:tasks.workunit.client.1.smithi203.stdout:5/53: fsync d3/f18 0 2024-04-05T03:02:26.106 INFO:tasks.workunit.client.1.smithi203.stdout:2/54: symlink d5/l11 0 2024-04-05T03:02:26.107 INFO:tasks.workunit.client.1.smithi203.stdout:5/54: mknod d3/d4/dc/c19 0 2024-04-05T03:02:26.109 INFO:tasks.workunit.client.1.smithi203.stdout:5/55: creat d3/d4/f1a x:0 0 0 2024-04-05T03:02:26.109 INFO:tasks.workunit.client.1.smithi203.stdout:5/56: truncate f2 4903079 0 2024-04-05T03:02:26.110 INFO:tasks.workunit.client.1.smithi203.stdout:2/55: truncate f3 119824 0 2024-04-05T03:02:26.113 INFO:tasks.workunit.client.1.smithi203.stdout:5/57: link d3/d4/f5 d3/d4/d8/f1b 0 2024-04-05T03:02:26.115 INFO:tasks.workunit.client.1.smithi203.stdout:5/58: creat d3/d4/d8/f1c x:0 0 0 2024-04-05T03:02:26.115 INFO:tasks.workunit.client.1.smithi203.stdout:5/59: dread - d3/d4/f1a zero size 2024-04-05T03:02:26.119 INFO:tasks.workunit.client.1.smithi203.stdout:5/60: mkdir d3/d4/d1d 0 2024-04-05T03:02:26.124 INFO:tasks.workunit.client.1.smithi203.stdout:5/61: dwrite d3/d4/d8/f10 [0,4194304] 0 2024-04-05T03:02:26.434 INFO:tasks.workunit.client.1.smithi203.stdout:5/62: creat d3/d4/d1d/f1e x:0 0 0 2024-04-05T03:02:26.441 INFO:tasks.workunit.client.1.smithi203.stdout:2/56: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:26.444 INFO:tasks.workunit.client.1.smithi203.stdout:5/63: dwrite d3/d4/fb [4194304,4194304] 0 2024-04-05T03:02:26.454 INFO:tasks.workunit.client.1.smithi203.stdout:5/64: dwrite d3/fd [0,4194304] 0 2024-04-05T03:02:26.568 INFO:tasks.workunit.client.1.smithi203.stdout:2/57: unlink d5/ld 0 2024-04-05T03:02:26.570 INFO:tasks.workunit.client.1.smithi203.stdout:5/65: rename d3/d4/d1d to d3/d1f 0 2024-04-05T03:02:26.577 INFO:tasks.workunit.client.1.smithi203.stdout:5/66: dwrite d3/f7 [0,4194304] 0 2024-04-05T03:02:26.578 INFO:tasks.workunit.client.1.smithi203.stdout:5/67: write d3/f7 [2975255,63549] 0 2024-04-05T03:02:26.579 INFO:tasks.workunit.client.1.smithi203.stdout:5/68: fsync d3/fd 0 2024-04-05T03:02:26.582 INFO:tasks.workunit.client.1.smithi203.stdout:5/69: creat d3/d4/d8/d12/f20 x:0 0 0 2024-04-05T03:02:26.584 INFO:tasks.workunit.client.1.smithi203.stdout:5/70: truncate d3/d4/f1a 745710 0 2024-04-05T03:02:26.588 INFO:tasks.workunit.client.1.smithi203.stdout:5/71: dread d3/f18 [0,4194304] 0 2024-04-05T03:02:26.636 INFO:tasks.workunit.client.1.smithi203.stdout:5/72: write d3/d4/f5 [560489,45293] 0 2024-04-05T03:02:26.636 INFO:tasks.workunit.client.1.smithi203.stdout:5/73: chown d3 1326299228 1 2024-04-05T03:02:26.637 INFO:tasks.workunit.client.1.smithi203.stdout:5/74: readlink d3/d4/dc/l14 0 2024-04-05T03:02:26.638 INFO:tasks.workunit.client.1.smithi203.stdout:5/75: dread - d3/d4/d8/f1c zero size 2024-04-05T03:02:26.644 INFO:tasks.workunit.client.1.smithi203.stdout:5/76: dwrite d3/f7 [0,4194304] 0 2024-04-05T03:02:26.647 INFO:tasks.workunit.client.1.smithi203.stdout:5/77: mknod d3/d4/c21 0 2024-04-05T03:02:26.652 INFO:tasks.workunit.client.1.smithi203.stdout:5/78: rename d3/d4/dc/c16 to d3/d4/d8/d12/c22 0 2024-04-05T03:02:26.667 INFO:tasks.workunit.client.1.smithi203.stdout:5/79: unlink d3/d4/d8/d12/c22 0 2024-04-05T03:02:26.668 INFO:tasks.workunit.client.1.smithi203.stdout:5/80: read d3/fd [1103045,38532] 0 2024-04-05T03:02:26.671 INFO:tasks.workunit.client.1.smithi203.stdout:5/81: getdents d3/d4/dc 0 2024-04-05T03:02:26.677 INFO:tasks.workunit.client.1.smithi203.stdout:5/82: dwrite d3/d4/fb [4194304,4194304] 0 2024-04-05T03:02:26.681 INFO:tasks.workunit.client.1.smithi203.stdout:5/83: getdents d3/d4/d8/d12 0 2024-04-05T03:02:26.684 INFO:tasks.workunit.client.1.smithi203.stdout:5/84: dread d3/f7 [0,4194304] 0 2024-04-05T03:02:26.685 INFO:tasks.workunit.client.1.smithi203.stdout:5/85: write d3/d4/f1a [684897,58292] 0 2024-04-05T03:02:26.689 INFO:tasks.workunit.client.1.smithi203.stdout:5/86: creat d3/d4/d8/f23 x:0 0 0 2024-04-05T03:02:26.689 INFO:tasks.workunit.client.1.smithi203.stdout:5/87: stat d3/d1f 0 2024-04-05T03:02:26.692 INFO:tasks.workunit.client.1.smithi203.stdout:5/88: dread d3/d4/d8/f10 [0,4194304] 0 2024-04-05T03:02:26.695 INFO:tasks.workunit.client.1.smithi203.stdout:5/89: getdents d3/d1f 0 2024-04-05T03:02:26.696 INFO:tasks.workunit.client.1.smithi203.stdout:5/90: chown d3/d4/f5 3492022 1 2024-04-05T03:02:26.698 INFO:tasks.workunit.client.1.smithi203.stdout:5/91: rmdir d3/d4/d8/d12 39 2024-04-05T03:02:26.698 INFO:tasks.workunit.client.1.smithi203.stdout:5/92: chown d3 357249 1 2024-04-05T03:02:26.699 INFO:tasks.workunit.client.1.smithi203.stdout:5/93: symlink d3/d1f/l24 0 2024-04-05T03:02:26.702 INFO:tasks.workunit.client.1.smithi203.stdout:5/94: creat d3/d4/dc/f25 x:0 0 0 2024-04-05T03:02:26.702 INFO:tasks.workunit.client.1.smithi203.stdout:5/95: read d3/f18 [4403458,58011] 0 2024-04-05T03:02:26.708 INFO:tasks.workunit.client.1.smithi203.stdout:5/96: dwrite d3/d1f/f1e [0,4194304] 0 2024-04-05T03:02:26.712 INFO:tasks.workunit.client.1.smithi203.stdout:5/97: dread f2 [0,4194304] 0 2024-04-05T03:02:26.714 INFO:tasks.workunit.client.1.smithi203.stdout:5/98: write d3/d4/fb [144520,121731] 0 2024-04-05T03:02:26.718 INFO:tasks.workunit.client.1.smithi203.stdout:5/99: creat d3/d4/d8/f26 x:0 0 0 2024-04-05T03:02:26.719 INFO:tasks.workunit.client.1.smithi203.stdout:5/100: write d3/d4/d8/f10 [2086639,111601] 0 2024-04-05T03:02:26.720 INFO:tasks.workunit.client.1.smithi203.stdout:5/101: fdatasync d3/d4/d8/f23 0 2024-04-05T03:02:26.729 INFO:tasks.workunit.client.1.smithi203.stdout:5/102: dwrite d3/d4/d8/f1b [0,4194304] 0 2024-04-05T03:02:26.733 INFO:tasks.workunit.client.1.smithi203.stdout:5/103: getdents d3/d4/d8/d12 0 2024-04-05T03:02:26.736 INFO:tasks.workunit.client.1.smithi203.stdout:5/104: symlink d3/d4/d8/d12/l27 0 2024-04-05T03:02:26.736 INFO:tasks.workunit.client.1.smithi203.stdout:5/105: read - d3/d4/d8/d12/f20 zero size 2024-04-05T03:02:26.747 INFO:tasks.workunit.client.1.smithi203.stdout:5/106: fsync d3/d4/d8/f26 0 2024-04-05T03:02:26.748 INFO:tasks.workunit.client.1.smithi203.stdout:5/107: dread - d3/d4/d8/d12/f20 zero size 2024-04-05T03:02:26.749 INFO:tasks.workunit.client.1.smithi203.stdout:5/108: fdatasync d3/d4/d8/d12/f20 0 2024-04-05T03:02:26.750 INFO:tasks.workunit.client.1.smithi203.stdout:5/109: mknod d3/d1f/c28 0 2024-04-05T03:02:26.753 INFO:tasks.workunit.client.1.smithi203.stdout:5/110: link d3/d4/d8/d12/l27 d3/d4/l29 0 2024-04-05T03:02:26.761 INFO:tasks.workunit.client.1.smithi203.stdout:5/111: dwrite d3/f7 [0,4194304] 0 2024-04-05T03:02:26.762 INFO:tasks.workunit.client.1.smithi203.stdout:5/112: truncate d3/d4/dc/f25 520421 0 2024-04-05T03:02:26.765 INFO:tasks.workunit.client.1.smithi203.stdout:5/113: rename d3/d1f/c28 to d3/d4/dc/c2a 0 2024-04-05T03:02:26.780 INFO:tasks.workunit.client.1.smithi203.stdout:5/114: truncate d3/d4/fb 8097341 0 2024-04-05T03:02:26.780 INFO:tasks.workunit.client.1.smithi203.stdout:5/115: chown d3/d4/f1a 263846269 1 2024-04-05T03:02:26.782 INFO:tasks.workunit.client.1.smithi203.stdout:5/116: symlink d3/d4/dc/l2b 0 2024-04-05T03:02:26.783 INFO:tasks.workunit.client.1.smithi203.stdout:5/117: dread - d3/d4/d8/f23 zero size 2024-04-05T03:02:26.784 INFO:tasks.workunit.client.1.smithi203.stdout:5/118: truncate d3/d4/d8/f23 272504 0 2024-04-05T03:02:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:26 smithi203 ceph-mon[39911]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:26 smithi203 ceph-mon[39911]: pgmap v249: 65 pgs: 65 active+clean; 177 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 479 op/s 2024-04-05T03:02:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:26 smithi153 ceph-mon[29381]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:26 smithi153 ceph-mon[29381]: pgmap v249: 65 pgs: 65 active+clean; 177 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 479 op/s 2024-04-05T03:02:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:28 smithi203 ceph-mon[39911]: pgmap v250: 65 pgs: 65 active+clean; 215 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 459 op/s 2024-04-05T03:02:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:28 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:28 smithi153 ceph-mon[29381]: pgmap v250: 65 pgs: 65 active+clean; 215 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 459 op/s 2024-04-05T03:02:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:28 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:30 smithi203 ceph-mon[39911]: pgmap v251: 65 pgs: 65 active+clean; 257 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 12 MiB/s wr, 457 op/s 2024-04-05T03:02:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:30 smithi153 ceph-mon[29381]: pgmap v251: 65 pgs: 65 active+clean; 257 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 12 MiB/s wr, 457 op/s 2024-04-05T03:02:30.829 INFO:tasks.workunit.client.1.smithi203.stdout:3/5: sync 2024-04-05T03:02:30.829 INFO:tasks.workunit.client.1.smithi203.stdout:0/7: sync 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:9/1: sync 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:6/6: sync 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:0/8: dread - no filename 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:6/7: write - no filename 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:0/9: dread - no filename 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:0/10: dread - no filename 2024-04-05T03:02:30.830 INFO:tasks.workunit.client.1.smithi203.stdout:0/11: write - no filename 2024-04-05T03:02:30.832 INFO:tasks.workunit.client.1.smithi203.stdout:0/12: rename c0 to c1 0 2024-04-05T03:02:30.833 INFO:tasks.workunit.client.1.smithi203.stdout:3/6: mkdir d0 0 2024-04-05T03:02:30.833 INFO:tasks.workunit.client.1.smithi203.stdout:3/7: fdatasync - no filename 2024-04-05T03:02:30.833 INFO:tasks.workunit.client.1.smithi203.stdout:3/8: write - no filename 2024-04-05T03:02:30.833 INFO:tasks.workunit.client.1.smithi203.stdout:3/9: stat d0 0 2024-04-05T03:02:30.834 INFO:tasks.workunit.client.1.smithi203.stdout:9/2: symlink l0 0 2024-04-05T03:02:30.834 INFO:tasks.workunit.client.1.smithi203.stdout:6/8: getdents . 0 2024-04-05T03:02:30.835 INFO:tasks.workunit.client.1.smithi203.stdout:9/3: chown l0 99553 1 2024-04-05T03:02:30.835 INFO:tasks.workunit.client.1.smithi203.stdout:9/4: read - no filename 2024-04-05T03:02:30.835 INFO:tasks.workunit.client.1.smithi203.stdout:9/5: dwrite - no filename 2024-04-05T03:02:30.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/9: mkdir d0 0 2024-04-05T03:02:30.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/10: write - no filename 2024-04-05T03:02:30.838 INFO:tasks.workunit.client.1.smithi203.stdout:9/6: link l0 l1 0 2024-04-05T03:02:30.839 INFO:tasks.workunit.client.1.smithi203.stdout:6/11: symlink d0/l1 0 2024-04-05T03:02:30.840 INFO:tasks.workunit.client.1.smithi203.stdout:9/7: stat l0 0 2024-04-05T03:02:30.841 INFO:tasks.workunit.client.1.smithi203.stdout:6/12: rmdir d0 39 2024-04-05T03:02:30.841 INFO:tasks.workunit.client.1.smithi203.stdout:6/13: write - no filename 2024-04-05T03:02:30.843 INFO:tasks.workunit.client.1.smithi203.stdout:9/8: chown l0 0 1 2024-04-05T03:02:30.843 INFO:tasks.workunit.client.1.smithi203.stdout:9/9: write - no filename 2024-04-05T03:02:30.844 INFO:tasks.workunit.client.1.smithi203.stdout:9/10: readlink l1 0 2024-04-05T03:02:30.845 INFO:tasks.workunit.client.1.smithi203.stdout:8/8: sync 2024-04-05T03:02:30.846 INFO:tasks.workunit.client.1.smithi203.stdout:7/7: sync 2024-04-05T03:02:30.846 INFO:tasks.workunit.client.1.smithi203.stdout:4/31: sync 2024-04-05T03:02:30.846 INFO:tasks.workunit.client.1.smithi203.stdout:5/119: sync 2024-04-05T03:02:30.846 INFO:tasks.workunit.client.1.smithi203.stdout:1/15: sync 2024-04-05T03:02:30.846 INFO:tasks.workunit.client.1.smithi203.stdout:2/58: sync 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/11: chown l1 334044 1 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/12: dwrite - no filename 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/13: dread - no filename 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/14: dwrite - no filename 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/15: truncate - no filename 2024-04-05T03:02:30.847 INFO:tasks.workunit.client.1.smithi203.stdout:9/16: read - no filename 2024-04-05T03:02:30.848 INFO:tasks.workunit.client.1.smithi203.stdout:7/8: creat f4 x:0 0 0 2024-04-05T03:02:30.849 INFO:tasks.workunit.client.1.smithi203.stdout:7/9: stat f0 0 2024-04-05T03:02:30.849 INFO:tasks.workunit.client.1.smithi203.stdout:7/10: chown f3 44873706 1 2024-04-05T03:02:30.850 INFO:tasks.workunit.client.1.smithi203.stdout:7/11: chown f0 44386977 1 2024-04-05T03:02:30.852 INFO:tasks.workunit.client.1.smithi203.stdout:1/16: truncate f0 57317 0 2024-04-05T03:02:30.854 INFO:tasks.workunit.client.1.smithi203.stdout:9/17: creat f2 x:0 0 0 2024-04-05T03:02:30.856 INFO:tasks.workunit.client.1.smithi203.stdout:2/59: mknod d5/d10/c12 0 2024-04-05T03:02:30.856 INFO:tasks.workunit.client.1.smithi203.stdout:4/32: dwrite f2 [4194304,4194304] 0 2024-04-05T03:02:30.856 INFO:tasks.workunit.client.1.smithi203.stdout:7/12: dread f3 [0,4194304] 0 2024-04-05T03:02:30.857 INFO:tasks.workunit.client.1.smithi203.stdout:4/33: dread - f5 zero size 2024-04-05T03:02:30.857 INFO:tasks.workunit.client.1.smithi203.stdout:7/13: fdatasync f2 0 2024-04-05T03:02:30.857 INFO:tasks.workunit.client.1.smithi203.stdout:2/60: stat d5/d10/c12 0 2024-04-05T03:02:30.860 INFO:tasks.workunit.client.1.smithi203.stdout:9/18: readlink l0 0 2024-04-05T03:02:30.861 INFO:tasks.workunit.client.1.smithi203.stdout:4/34: dread f8 [0,4194304] 0 2024-04-05T03:02:30.865 INFO:tasks.workunit.client.1.smithi203.stdout:7/14: write f3 [1804025,61119] 0 2024-04-05T03:02:30.865 INFO:tasks.workunit.client.1.smithi203.stdout:8/9: getdents . 0 2024-04-05T03:02:30.866 INFO:tasks.workunit.client.1.smithi203.stdout:8/10: truncate f0 310350 0 2024-04-05T03:02:30.867 INFO:tasks.workunit.client.1.smithi203.stdout:2/61: write d5/f8 [2343695,27920] 0 2024-04-05T03:02:30.867 INFO:tasks.workunit.client.1.smithi203.stdout:8/11: fsync f0 0 2024-04-05T03:02:30.870 INFO:tasks.workunit.client.1.smithi203.stdout:8/12: write f0 [736970,37770] 0 2024-04-05T03:02:30.873 INFO:tasks.workunit.client.1.smithi203.stdout:9/19: chown l0 9855 1 2024-04-05T03:02:30.875 INFO:tasks.workunit.client.1.smithi203.stdout:2/62: fdatasync d5/f8 0 2024-04-05T03:02:30.876 INFO:tasks.workunit.client.1.smithi203.stdout:7/15: dwrite f4 [0,4194304] 0 2024-04-05T03:02:30.877 INFO:tasks.workunit.client.1.smithi203.stdout:2/63: write d5/f9 [4756731,72179] 0 2024-04-05T03:02:30.878 INFO:tasks.workunit.client.1.smithi203.stdout:2/64: fsync d5/f8 0 2024-04-05T03:02:30.879 INFO:tasks.workunit.client.1.smithi203.stdout:7/16: creat f5 x:0 0 0 2024-04-05T03:02:30.888 INFO:tasks.workunit.client.1.smithi203.stdout:2/65: dwrite d5/fa [0,4194304] 0 2024-04-05T03:02:30.893 INFO:tasks.workunit.client.1.smithi203.stdout:2/66: link f3 d5/d10/f13 0 2024-04-05T03:02:30.893 INFO:tasks.workunit.client.1.smithi203.stdout:2/67: chown l2 75 1 2024-04-05T03:02:30.899 INFO:tasks.workunit.client.1.smithi203.stdout:2/68: rename d5/cb to d5/c14 0 2024-04-05T03:02:30.906 INFO:tasks.workunit.client.1.smithi203.stdout:2/69: fdatasync d5/fa 0 2024-04-05T03:02:30.909 INFO:tasks.workunit.client.1.smithi203.stdout:2/70: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:30.919 INFO:tasks.workunit.client.1.smithi203.stdout:2/71: dwrite d5/d10/f13 [4194304,4194304] 0 2024-04-05T03:02:30.921 INFO:tasks.workunit.client.1.smithi203.stdout:2/72: symlink d5/l15 0 2024-04-05T03:02:30.922 INFO:tasks.workunit.client.1.smithi203.stdout:2/73: write d5/f9 [1188897,55388] 0 2024-04-05T03:02:30.929 INFO:tasks.workunit.client.1.smithi203.stdout:2/74: dread d5/f9 [4194304,4194304] 0 2024-04-05T03:02:30.930 INFO:tasks.workunit.client.1.smithi203.stdout:2/75: symlink d5/l16 0 2024-04-05T03:02:30.963 INFO:tasks.workunit.client.1.smithi203.stdout:0/13: chown c1 996 1 2024-04-05T03:02:30.963 INFO:tasks.workunit.client.1.smithi203.stdout:0/14: write - no filename 2024-04-05T03:02:30.963 INFO:tasks.workunit.client.1.smithi203.stdout:0/15: dread - no filename 2024-04-05T03:02:30.963 INFO:tasks.workunit.client.1.smithi203.stdout:0/16: write - no filename 2024-04-05T03:02:30.963 INFO:tasks.workunit.client.1.smithi203.stdout:0/17: readlink - no filename 2024-04-05T03:02:30.965 INFO:tasks.workunit.client.1.smithi203.stdout:0/18: creat f2 x:0 0 0 2024-04-05T03:02:30.967 INFO:tasks.workunit.client.1.smithi203.stdout:0/19: creat f3 x:0 0 0 2024-04-05T03:02:30.967 INFO:tasks.workunit.client.1.smithi203.stdout:0/20: readlink - no filename 2024-04-05T03:02:30.968 INFO:tasks.workunit.client.1.smithi203.stdout:6/14: symlink d0/l2 0 2024-04-05T03:02:30.970 INFO:tasks.workunit.client.1.smithi203.stdout:1/17: truncate f2 237783 0 2024-04-05T03:02:30.970 INFO:tasks.workunit.client.1.smithi203.stdout:1/18: rmdir - no directory 2024-04-05T03:02:30.973 INFO:tasks.workunit.client.1.smithi203.stdout:1/19: mkdir d7 0 2024-04-05T03:02:30.974 INFO:tasks.workunit.client.1.smithi203.stdout:1/20: chown d7 140170604 1 2024-04-05T03:02:30.976 INFO:tasks.workunit.client.1.smithi203.stdout:8/13: truncate f0 291521 0 2024-04-05T03:02:30.976 INFO:tasks.workunit.client.1.smithi203.stdout:5/120: dwrite d3/d4/d8/f1b [4194304,4194304] 0 2024-04-05T03:02:30.978 INFO:tasks.workunit.client.1.smithi203.stdout:7/17: truncate f3 303289 0 2024-04-05T03:02:30.978 INFO:tasks.workunit.client.1.smithi203.stdout:1/21: mknod d7/c8 0 2024-04-05T03:02:30.984 INFO:tasks.workunit.client.1.smithi203.stdout:7/18: rename f5 to f6 0 2024-04-05T03:02:30.986 INFO:tasks.workunit.client.1.smithi203.stdout:5/121: dwrite d3/d4/d8/f1b [0,4194304] 0 2024-04-05T03:02:30.986 INFO:tasks.workunit.client.1.smithi203.stdout:2/76: getdents d5 0 2024-04-05T03:02:30.987 INFO:tasks.workunit.client.1.smithi203.stdout:7/19: symlink l7 0 2024-04-05T03:02:30.988 INFO:tasks.workunit.client.1.smithi203.stdout:7/20: fsync f4 0 2024-04-05T03:02:30.989 INFO:tasks.workunit.client.1.smithi203.stdout:6/15: getdents d0 0 2024-04-05T03:02:30.989 INFO:tasks.workunit.client.1.smithi203.stdout:6/16: read - no filename 2024-04-05T03:02:30.989 INFO:tasks.workunit.client.1.smithi203.stdout:7/21: truncate f6 485505 0 2024-04-05T03:02:30.989 INFO:tasks.workunit.client.1.smithi203.stdout:1/22: write f2 [1208759,13018] 0 2024-04-05T03:02:30.990 INFO:tasks.workunit.client.1.smithi203.stdout:7/22: fsync f4 0 2024-04-05T03:02:30.991 INFO:tasks.workunit.client.1.smithi203.stdout:5/122: mknod d3/d4/dc/c2c 0 2024-04-05T03:02:30.995 INFO:tasks.workunit.client.1.smithi203.stdout:2/77: rename d5/l15 to d5/d10/l17 0 2024-04-05T03:02:30.995 INFO:tasks.workunit.client.1.smithi203.stdout:6/17: creat d0/f3 x:0 0 0 2024-04-05T03:02:30.996 INFO:tasks.workunit.client.1.smithi203.stdout:6/18: chown d0/l2 7 1 2024-04-05T03:02:30.997 INFO:tasks.workunit.client.1.smithi203.stdout:1/23: symlink d7/l9 0 2024-04-05T03:02:30.997 INFO:tasks.workunit.client.1.smithi203.stdout:6/19: readlink d0/l2 0 2024-04-05T03:02:30.998 INFO:tasks.workunit.client.1.smithi203.stdout:5/123: creat d3/d4/dc/f2d x:0 0 0 2024-04-05T03:02:31.001 INFO:tasks.workunit.client.1.smithi203.stdout:1/24: creat d7/fa x:0 0 0 2024-04-05T03:02:31.002 INFO:tasks.workunit.client.1.smithi203.stdout:7/23: truncate f4 1320950 0 2024-04-05T03:02:31.005 INFO:tasks.workunit.client.1.smithi203.stdout:8/14: truncate f0 1213206 0 2024-04-05T03:02:31.006 INFO:tasks.workunit.client.1.smithi203.stdout:2/78: truncate d5/fa 2961364 0 2024-04-05T03:02:31.007 INFO:tasks.workunit.client.1.smithi203.stdout:7/24: mkdir d8 0 2024-04-05T03:02:31.008 INFO:tasks.workunit.client.1.smithi203.stdout:1/25: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:31.011 INFO:tasks.workunit.client.1.smithi203.stdout:1/26: write f0 [584981,122420] 0 2024-04-05T03:02:31.014 INFO:tasks.workunit.client.1.smithi203.stdout:7/25: mknod d8/c9 0 2024-04-05T03:02:31.015 INFO:tasks.workunit.client.1.smithi203.stdout:7/26: write f0 [368885,71659] 0 2024-04-05T03:02:31.021 INFO:tasks.workunit.client.1.smithi203.stdout:7/27: rmdir d8 39 2024-04-05T03:02:31.024 INFO:tasks.workunit.client.1.smithi203.stdout:2/79: getdents d5 0 2024-04-05T03:02:31.028 INFO:tasks.workunit.client.1.smithi203.stdout:2/80: symlink d5/l18 0 2024-04-05T03:02:31.030 INFO:tasks.workunit.client.1.smithi203.stdout:7/28: write f4 [851078,95804] 0 2024-04-05T03:02:31.031 INFO:tasks.workunit.client.1.smithi203.stdout:7/29: read - f2 zero size 2024-04-05T03:02:31.033 INFO:tasks.workunit.client.1.smithi203.stdout:1/27: truncate f0 1053149 0 2024-04-05T03:02:31.034 INFO:tasks.workunit.client.1.smithi203.stdout:8/15: write f0 [1662336,107741] 0 2024-04-05T03:02:31.036 INFO:tasks.workunit.client.1.smithi203.stdout:7/30: mkdir d8/da 0 2024-04-05T03:02:31.037 INFO:tasks.workunit.client.1.smithi203.stdout:5/124: dwrite d3/d4/fb [0,4194304] 0 2024-04-05T03:02:31.037 INFO:tasks.workunit.client.1.smithi203.stdout:2/81: dread d5/d10/f13 [0,4194304] 0 2024-04-05T03:02:31.037 INFO:tasks.workunit.client.1.smithi203.stdout:8/16: write f0 [51267,76310] 0 2024-04-05T03:02:31.039 INFO:tasks.workunit.client.1.smithi203.stdout:5/125: write d3/d4/d8/f1b [7425696,53455] 0 2024-04-05T03:02:31.043 INFO:tasks.workunit.client.1.smithi203.stdout:7/31: mknod d8/da/cb 0 2024-04-05T03:02:31.047 INFO:tasks.workunit.client.1.smithi203.stdout:1/28: truncate d7/fa 475574 0 2024-04-05T03:02:31.048 INFO:tasks.workunit.client.1.smithi203.stdout:7/32: read f6 [302479,16147] 0 2024-04-05T03:02:31.048 INFO:tasks.workunit.client.1.smithi203.stdout:2/82: link d5/ce d5/d10/c19 0 2024-04-05T03:02:31.049 INFO:tasks.workunit.client.1.smithi203.stdout:2/83: readlink d5/l11 0 2024-04-05T03:02:31.050 INFO:tasks.workunit.client.1.smithi203.stdout:7/33: chown d8/da/cb 6167 1 2024-04-05T03:02:31.050 INFO:tasks.workunit.client.1.smithi203.stdout:5/126: dwrite d3/d4/d8/f1b [8388608,4194304] 0 2024-04-05T03:02:31.053 INFO:tasks.workunit.client.1.smithi203.stdout:2/84: write f3 [9287721,41185] 0 2024-04-05T03:02:31.058 INFO:tasks.workunit.client.1.smithi203.stdout:2/85: mkdir d5/d10/d1a 0 2024-04-05T03:02:31.061 INFO:tasks.workunit.client.1.smithi203.stdout:5/127: dread d3/f7 [0,4194304] 0 2024-04-05T03:02:31.061 INFO:tasks.workunit.client.1.smithi203.stdout:1/29: dread d7/fa [0,4194304] 0 2024-04-05T03:02:31.062 INFO:tasks.workunit.client.1.smithi203.stdout:7/34: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.064 INFO:tasks.workunit.client.1.smithi203.stdout:1/30: chown c1 10203030 1 2024-04-05T03:02:31.065 INFO:tasks.workunit.client.1.smithi203.stdout:7/35: mknod d8/da/cc 0 2024-04-05T03:02:31.069 INFO:tasks.workunit.client.1.smithi203.stdout:7/36: creat d8/da/fd x:0 0 0 2024-04-05T03:02:31.071 INFO:tasks.workunit.client.1.smithi203.stdout:5/128: rename d3/fd to d3/d4/d8/f2e 0 2024-04-05T03:02:31.148 INFO:tasks.workunit.client.1.smithi203.stdout:2/86: link d5/fa d5/d10/d1a/f1b 0 2024-04-05T03:02:31.149 INFO:tasks.workunit.client.1.smithi203.stdout:7/37: chown f3 115670 1 2024-04-05T03:02:31.150 INFO:tasks.workunit.client.1.smithi203.stdout:2/87: chown d5/c14 32830950 1 2024-04-05T03:02:31.150 INFO:tasks.workunit.client.1.smithi203.stdout:7/38: creat d8/fe x:0 0 0 2024-04-05T03:02:31.151 INFO:tasks.workunit.client.1.smithi203.stdout:1/31: dread f0 [0,4194304] 0 2024-04-05T03:02:31.151 INFO:tasks.workunit.client.1.smithi203.stdout:5/129: mknod d3/d4/d8/c2f 0 2024-04-05T03:02:31.152 INFO:tasks.workunit.client.1.smithi203.stdout:1/32: chown d7 0 1 2024-04-05T03:02:31.152 INFO:tasks.workunit.client.1.smithi203.stdout:5/130: chown d3/d4/dc 101072599 1 2024-04-05T03:02:31.192 INFO:tasks.workunit.client.1.smithi203.stdout:3/10: sync 2024-04-05T03:02:31.193 INFO:tasks.workunit.client.1.smithi203.stdout:1/33: symlink d7/lb 0 2024-04-05T03:02:31.193 INFO:tasks.workunit.client.1.smithi203.stdout:7/39: truncate d8/da/fd 145145 0 2024-04-05T03:02:31.195 INFO:tasks.workunit.client.1.smithi203.stdout:7/40: fsync f6 0 2024-04-05T03:02:31.195 INFO:tasks.workunit.client.1.smithi203.stdout:5/131: write d3/f7 [1613419,12124] 0 2024-04-05T03:02:31.196 INFO:tasks.workunit.client.1.smithi203.stdout:2/88: dread d5/d10/f13 [4194304,4194304] 0 2024-04-05T03:02:31.196 INFO:tasks.workunit.client.1.smithi203.stdout:1/34: mknod d7/cc 0 2024-04-05T03:02:31.197 INFO:tasks.workunit.client.1.smithi203.stdout:5/132: fsync d3/d4/fe 0 2024-04-05T03:02:31.197 INFO:tasks.workunit.client.1.smithi203.stdout:3/11: mknod d0/c1 0 2024-04-05T03:02:31.197 INFO:tasks.workunit.client.1.smithi203.stdout:3/12: write - no filename 2024-04-05T03:02:31.198 INFO:tasks.workunit.client.1.smithi203.stdout:2/89: fsync f3 0 2024-04-05T03:02:31.199 INFO:tasks.workunit.client.1.smithi203.stdout:7/41: rename f0 to d8/ff 0 2024-04-05T03:02:31.199 INFO:tasks.workunit.client.1.smithi203.stdout:5/133: truncate d3/d4/d8/f1c 859779 0 2024-04-05T03:02:31.200 INFO:tasks.workunit.client.1.smithi203.stdout:5/134: dread - d3/d4/d8/f26 zero size 2024-04-05T03:02:31.200 INFO:tasks.workunit.client.1.smithi203.stdout:7/42: truncate d8/fe 820639 0 2024-04-05T03:02:31.201 INFO:tasks.workunit.client.1.smithi203.stdout:3/13: mknod d0/c2 0 2024-04-05T03:02:31.201 INFO:tasks.workunit.client.1.smithi203.stdout:3/14: write - no filename 2024-04-05T03:02:31.201 INFO:tasks.workunit.client.1.smithi203.stdout:7/43: dread - f2 zero size 2024-04-05T03:02:31.204 INFO:tasks.workunit.client.1.smithi203.stdout:3/15: unlink d0/c2 0 2024-04-05T03:02:31.204 INFO:tasks.workunit.client.1.smithi203.stdout:7/44: dread d8/fe [0,4194304] 0 2024-04-05T03:02:31.204 INFO:tasks.workunit.client.1.smithi203.stdout:3/16: chown d0 3219 1 2024-04-05T03:02:31.204 INFO:tasks.workunit.client.1.smithi203.stdout:3/17: write - no filename 2024-04-05T03:02:31.205 INFO:tasks.workunit.client.1.smithi203.stdout:2/90: creat d5/d10/d1a/f1c x:0 0 0 2024-04-05T03:02:31.205 INFO:tasks.workunit.client.1.smithi203.stdout:2/91: write f3 [6380223,42375] 0 2024-04-05T03:02:31.208 INFO:tasks.workunit.client.1.smithi203.stdout:5/135: mknod d3/d4/d8/c30 0 2024-04-05T03:02:31.209 INFO:tasks.workunit.client.1.smithi203.stdout:3/18: creat d0/f3 x:0 0 0 2024-04-05T03:02:31.210 INFO:tasks.workunit.client.1.smithi203.stdout:3/19: dread - d0/f3 zero size 2024-04-05T03:02:31.210 INFO:tasks.workunit.client.1.smithi203.stdout:3/20: dread - d0/f3 zero size 2024-04-05T03:02:31.211 INFO:tasks.workunit.client.1.smithi203.stdout:7/45: creat d8/da/f10 x:0 0 0 2024-04-05T03:02:31.213 INFO:tasks.workunit.client.1.smithi203.stdout:2/92: creat d5/d10/f1d x:0 0 0 2024-04-05T03:02:31.215 INFO:tasks.workunit.client.1.smithi203.stdout:5/136: rename d3/d1f/l24 to d3/d4/dc/l31 0 2024-04-05T03:02:31.216 INFO:tasks.workunit.client.1.smithi203.stdout:3/21: creat d0/f4 x:0 0 0 2024-04-05T03:02:31.216 INFO:tasks.workunit.client.1.smithi203.stdout:2/93: mkdir d5/d1e 0 2024-04-05T03:02:31.218 INFO:tasks.workunit.client.1.smithi203.stdout:3/22: creat d0/f5 x:0 0 0 2024-04-05T03:02:31.218 INFO:tasks.workunit.client.1.smithi203.stdout:1/35: write f2 [1322060,34633] 0 2024-04-05T03:02:31.220 INFO:tasks.workunit.client.1.smithi203.stdout:5/137: rename d3/d4/d8/d12/f20 to d3/f32 0 2024-04-05T03:02:31.221 INFO:tasks.workunit.client.1.smithi203.stdout:2/94: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:31.221 INFO:tasks.workunit.client.1.smithi203.stdout:2/95: fdatasync d5/f8 0 2024-04-05T03:02:31.224 INFO:tasks.workunit.client.1.smithi203.stdout:5/138: creat d3/d1f/f33 x:0 0 0 2024-04-05T03:02:31.225 INFO:tasks.workunit.client.1.smithi203.stdout:2/96: rmdir d5 39 2024-04-05T03:02:31.230 INFO:tasks.workunit.client.1.smithi203.stdout:5/139: symlink d3/d4/d8/l34 0 2024-04-05T03:02:31.231 INFO:tasks.workunit.client.1.smithi203.stdout:5/140: read d3/d4/f5 [1727856,26889] 0 2024-04-05T03:02:31.232 INFO:tasks.workunit.client.1.smithi203.stdout:2/97: mkdir d5/d1f 0 2024-04-05T03:02:31.233 INFO:tasks.workunit.client.1.smithi203.stdout:2/98: write d5/d10/d1a/f1c [854817,70557] 0 2024-04-05T03:02:31.235 INFO:tasks.workunit.client.1.smithi203.stdout:5/141: creat d3/d4/dc/f35 x:0 0 0 2024-04-05T03:02:31.236 INFO:tasks.workunit.client.1.smithi203.stdout:3/23: rmdir d0 39 2024-04-05T03:02:31.237 INFO:tasks.workunit.client.1.smithi203.stdout:2/99: dread f3 [0,4194304] 0 2024-04-05T03:02:31.239 INFO:tasks.workunit.client.1.smithi203.stdout:1/36: truncate f2 1179988 0 2024-04-05T03:02:31.244 INFO:tasks.workunit.client.1.smithi203.stdout:1/37: chown f0 0 1 2024-04-05T03:02:31.246 INFO:tasks.workunit.client.1.smithi203.stdout:5/142: dwrite d3/d4/d8/f1c [0,4194304] 0 2024-04-05T03:02:31.252 INFO:tasks.workunit.client.1.smithi203.stdout:5/143: dread - d3/d4/d8/f26 zero size 2024-04-05T03:02:31.254 INFO:tasks.workunit.client.1.smithi203.stdout:3/24: dwrite d0/f4 [0,4194304] 0 2024-04-05T03:02:31.254 INFO:tasks.workunit.client.1.smithi203.stdout:2/100: dwrite d5/d10/f13 [0,4194304] 0 2024-04-05T03:02:31.256 INFO:tasks.workunit.client.1.smithi203.stdout:1/38: write f0 [154920,15324] 0 2024-04-05T03:02:31.257 INFO:tasks.workunit.client.1.smithi203.stdout:3/25: write d0/f5 [66048,101359] 0 2024-04-05T03:02:31.258 INFO:tasks.workunit.client.1.smithi203.stdout:5/144: rename d3/d4/f5 to d3/f36 0 2024-04-05T03:02:31.260 INFO:tasks.workunit.client.1.smithi203.stdout:1/39: mknod d7/cd 0 2024-04-05T03:02:31.262 INFO:tasks.workunit.client.1.smithi203.stdout:5/145: creat d3/d4/dc/f37 x:0 0 0 2024-04-05T03:02:31.263 INFO:tasks.workunit.client.1.smithi203.stdout:1/40: creat d7/fe x:0 0 0 2024-04-05T03:02:31.264 INFO:tasks.workunit.client.1.smithi203.stdout:5/146: stat d3/la 0 2024-04-05T03:02:31.270 INFO:tasks.workunit.client.1.smithi203.stdout:3/26: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:31.270 INFO:tasks.workunit.client.1.smithi203.stdout:1/41: link d7/l9 d7/lf 0 2024-04-05T03:02:31.275 INFO:tasks.workunit.client.1.smithi203.stdout:1/42: link d7/l9 d7/l10 0 2024-04-05T03:02:31.276 INFO:tasks.workunit.client.1.smithi203.stdout:3/27: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:31.278 INFO:tasks.workunit.client.1.smithi203.stdout:1/43: mknod d7/c11 0 2024-04-05T03:02:31.279 INFO:tasks.workunit.client.1.smithi203.stdout:3/28: creat d0/f6 x:0 0 0 2024-04-05T03:02:31.282 INFO:tasks.workunit.client.1.smithi203.stdout:1/44: rename d7/c11 to d7/c12 0 2024-04-05T03:02:31.284 INFO:tasks.workunit.client.1.smithi203.stdout:3/29: unlink d0/f3 0 2024-04-05T03:02:31.286 INFO:tasks.workunit.client.1.smithi203.stdout:3/30: read d0/f5 [127823,119948] 0 2024-04-05T03:02:31.289 INFO:tasks.workunit.client.1.smithi203.stdout:3/31: creat d0/f7 x:0 0 0 2024-04-05T03:02:31.290 INFO:tasks.workunit.client.1.smithi203.stdout:1/45: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.296 INFO:tasks.workunit.client.1.smithi203.stdout:3/32: getdents d0 0 2024-04-05T03:02:31.298 INFO:tasks.workunit.client.1.smithi203.stdout:3/33: write d0/f7 [865954,43965] 0 2024-04-05T03:02:31.300 INFO:tasks.workunit.client.1.smithi203.stdout:5/147: dwrite d3/f36 [12582912,4194304] 0 2024-04-05T03:02:31.308 INFO:tasks.workunit.client.1.smithi203.stdout:3/34: dwrite d0/f7 [0,4194304] 0 2024-04-05T03:02:31.310 INFO:tasks.workunit.client.1.smithi203.stdout:3/35: truncate d0/f6 248369 0 2024-04-05T03:02:31.311 INFO:tasks.workunit.client.1.smithi203.stdout:5/148: dwrite d3/d4/fe [0,4194304] 0 2024-04-05T03:02:31.312 INFO:tasks.workunit.client.1.smithi203.stdout:3/36: write d0/f6 [222764,56841] 0 2024-04-05T03:02:31.315 INFO:tasks.workunit.client.1.smithi203.stdout:5/149: symlink d3/d4/dc/l38 0 2024-04-05T03:02:31.317 INFO:tasks.workunit.client.1.smithi203.stdout:3/37: mknod d0/c8 0 2024-04-05T03:02:31.318 INFO:tasks.workunit.client.1.smithi203.stdout:5/150: creat d3/d1f/f39 x:0 0 0 2024-04-05T03:02:31.322 INFO:tasks.workunit.client.1.smithi203.stdout:5/151: creat d3/d4/d8/f3a x:0 0 0 2024-04-05T03:02:31.327 INFO:tasks.workunit.client.1.smithi203.stdout:3/38: getdents d0 0 2024-04-05T03:02:31.329 INFO:tasks.workunit.client.1.smithi203.stdout:3/39: creat d0/f9 x:0 0 0 2024-04-05T03:02:31.330 INFO:tasks.workunit.client.1.smithi203.stdout:5/152: truncate d3/d1f/f1e 2156928 0 2024-04-05T03:02:31.331 INFO:tasks.workunit.client.1.smithi203.stdout:9/20: sync 2024-04-05T03:02:31.331 INFO:tasks.workunit.client.1.smithi203.stdout:4/35: sync 2024-04-05T03:02:31.331 INFO:tasks.workunit.client.1.smithi203.stdout:0/21: sync 2024-04-05T03:02:31.331 INFO:tasks.workunit.client.1.smithi203.stdout:0/22: readlink - no filename 2024-04-05T03:02:31.331 INFO:tasks.workunit.client.1.smithi203.stdout:4/36: stat f5 0 2024-04-05T03:02:31.332 INFO:tasks.workunit.client.1.smithi203.stdout:9/21: dread - f2 zero size 2024-04-05T03:02:31.333 INFO:tasks.workunit.client.1.smithi203.stdout:9/22: dread - f2 zero size 2024-04-05T03:02:31.334 INFO:tasks.workunit.client.1.smithi203.stdout:5/153: unlink d3/d4/d8/f1b 0 2024-04-05T03:02:31.338 INFO:tasks.workunit.client.1.smithi203.stdout:9/23: write f2 [742504,73338] 0 2024-04-05T03:02:31.341 INFO:tasks.workunit.client.1.smithi203.stdout:0/23: getdents . 0 2024-04-05T03:02:31.344 INFO:tasks.workunit.client.1.smithi203.stdout:3/40: dwrite d0/f6 [0,4194304] 0 2024-04-05T03:02:31.344 INFO:tasks.workunit.client.1.smithi203.stdout:4/37: dread f7 [0,4194304] 0 2024-04-05T03:02:31.345 INFO:tasks.workunit.client.1.smithi203.stdout:0/24: write f2 [824292,82364] 0 2024-04-05T03:02:31.355 INFO:tasks.workunit.client.1.smithi203.stdout:9/24: dread f2 [0,4194304] 0 2024-04-05T03:02:31.356 INFO:tasks.workunit.client.1.smithi203.stdout:5/154: dwrite d3/d4/d8/f23 [0,4194304] 0 2024-04-05T03:02:31.363 INFO:tasks.workunit.client.1.smithi203.stdout:9/25: dread f2 [0,4194304] 0 2024-04-05T03:02:31.364 INFO:tasks.workunit.client.1.smithi203.stdout:5/155: creat d3/d1f/f3b x:0 0 0 2024-04-05T03:02:31.365 INFO:tasks.workunit.client.1.smithi203.stdout:0/25: dwrite f2 [0,4194304] 0 2024-04-05T03:02:31.367 INFO:tasks.workunit.client.1.smithi203.stdout:9/26: creat f3 x:0 0 0 2024-04-05T03:02:31.368 INFO:tasks.workunit.client.1.smithi203.stdout:3/41: dwrite d0/f6 [0,4194304] 0 2024-04-05T03:02:31.370 INFO:tasks.workunit.client.1.smithi203.stdout:5/156: rmdir d3/d4/d8 39 2024-04-05T03:02:31.371 INFO:tasks.workunit.client.1.smithi203.stdout:4/38: dwrite f9 [0,4194304] 0 2024-04-05T03:02:31.373 INFO:tasks.workunit.client.1.smithi203.stdout:5/157: dread - d3/d1f/f33 zero size 2024-04-05T03:02:31.373 INFO:tasks.workunit.client.1.smithi203.stdout:9/27: symlink l4 0 2024-04-05T03:02:31.383 INFO:tasks.workunit.client.1.smithi203.stdout:5/158: chown d3/d4/d8/f2e 31 1 2024-04-05T03:02:31.385 INFO:tasks.workunit.client.1.smithi203.stdout:4/39: mknod da/cb 0 2024-04-05T03:02:31.387 INFO:tasks.workunit.client.1.smithi203.stdout:3/42: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:31.388 INFO:tasks.workunit.client.1.smithi203.stdout:5/159: chown d3/d4/d8/c2f 1315690 1 2024-04-05T03:02:31.388 INFO:tasks.workunit.client.1.smithi203.stdout:0/26: dwrite f3 [0,4194304] 0 2024-04-05T03:02:31.390 INFO:tasks.workunit.client.1.smithi203.stdout:3/43: truncate d0/f5 890852 0 2024-04-05T03:02:31.393 INFO:tasks.workunit.client.1.smithi203.stdout:4/40: rename f7 to da/fc 0 2024-04-05T03:02:31.395 INFO:tasks.workunit.client.1.smithi203.stdout:5/160: read d3/d1f/f1e [2013741,38366] 0 2024-04-05T03:02:31.396 INFO:tasks.workunit.client.1.smithi203.stdout:5/161: dread - d3/d4/dc/f35 zero size 2024-04-05T03:02:31.398 INFO:tasks.workunit.client.1.smithi203.stdout:4/41: symlink da/ld 0 2024-04-05T03:02:31.400 INFO:tasks.workunit.client.1.smithi203.stdout:3/44: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:31.401 INFO:tasks.workunit.client.1.smithi203.stdout:5/162: chown d3/d4/dc/l2b 922 1 2024-04-05T03:02:31.403 INFO:tasks.workunit.client.1.smithi203.stdout:0/27: dwrite f2 [0,4194304] 0 2024-04-05T03:02:31.405 INFO:tasks.workunit.client.1.smithi203.stdout:3/45: mknod d0/ca 0 2024-04-05T03:02:31.409 INFO:tasks.workunit.client.1.smithi203.stdout:0/28: getdents . 0 2024-04-05T03:02:31.409 INFO:tasks.workunit.client.1.smithi203.stdout:0/29: rmdir - no directory 2024-04-05T03:02:31.410 INFO:tasks.workunit.client.1.smithi203.stdout:4/42: link da/cb da/ce 0 2024-04-05T03:02:31.410 INFO:tasks.workunit.client.1.smithi203.stdout:9/28: truncate f2 737643 0 2024-04-05T03:02:31.413 INFO:tasks.workunit.client.1.smithi203.stdout:0/30: write f2 [4206913,76983] 0 2024-04-05T03:02:31.416 INFO:tasks.workunit.client.1.smithi203.stdout:4/43: write f9 [4100846,12436] 0 2024-04-05T03:02:31.417 INFO:tasks.workunit.client.1.smithi203.stdout:5/163: dwrite d3/d4/dc/f35 [0,4194304] 0 2024-04-05T03:02:31.417 INFO:tasks.workunit.client.1.smithi203.stdout:0/31: symlink l4 0 2024-04-05T03:02:31.421 INFO:tasks.workunit.client.1.smithi203.stdout:4/44: unlink f2 0 2024-04-05T03:02:31.423 INFO:tasks.workunit.client.1.smithi203.stdout:5/164: symlink d3/l3c 0 2024-04-05T03:02:31.426 INFO:tasks.workunit.client.1.smithi203.stdout:0/32: dwrite f2 [0,4194304] 0 2024-04-05T03:02:31.428 INFO:tasks.workunit.client.1.smithi203.stdout:5/165: creat d3/d4/d8/f3d x:0 0 0 2024-04-05T03:02:31.430 INFO:tasks.workunit.client.1.smithi203.stdout:4/45: dread da/fc [0,4194304] 0 2024-04-05T03:02:31.433 INFO:tasks.workunit.client.1.smithi203.stdout:4/46: symlink da/lf 0 2024-04-05T03:02:31.436 INFO:tasks.workunit.client.1.smithi203.stdout:4/47: creat da/f10 x:0 0 0 2024-04-05T03:02:31.437 INFO:tasks.workunit.client.1.smithi203.stdout:3/46: rmdir d0 39 2024-04-05T03:02:31.439 INFO:tasks.workunit.client.1.smithi203.stdout:5/166: dwrite d3/d4/d8/f26 [0,4194304] 0 2024-04-05T03:02:31.442 INFO:tasks.workunit.client.1.smithi203.stdout:4/48: write da/fc [2243983,43851] 0 2024-04-05T03:02:31.442 INFO:tasks.workunit.client.1.smithi203.stdout:5/167: symlink d3/d4/dc/l3e 0 2024-04-05T03:02:31.447 INFO:tasks.workunit.client.1.smithi203.stdout:5/168: unlink d3/d4/dc/l3e 0 2024-04-05T03:02:31.447 INFO:tasks.workunit.client.1.smithi203.stdout:0/33: getdents . 0 2024-04-05T03:02:31.448 INFO:tasks.workunit.client.1.smithi203.stdout:0/34: chown f3 122880 1 2024-04-05T03:02:31.450 INFO:tasks.workunit.client.1.smithi203.stdout:0/35: chown c1 1888 1 2024-04-05T03:02:31.451 INFO:tasks.workunit.client.1.smithi203.stdout:5/169: dread d3/d1f/f1e [0,4194304] 0 2024-04-05T03:02:31.452 INFO:tasks.workunit.client.1.smithi203.stdout:0/36: creat f5 x:0 0 0 2024-04-05T03:02:31.453 INFO:tasks.workunit.client.1.smithi203.stdout:4/49: getdents da 0 2024-04-05T03:02:31.455 INFO:tasks.workunit.client.1.smithi203.stdout:5/170: creat d3/d4/d8/d12/f3f x:0 0 0 2024-04-05T03:02:31.458 INFO:tasks.workunit.client.1.smithi203.stdout:9/29: write f2 [1587325,30154] 0 2024-04-05T03:02:31.459 INFO:tasks.workunit.client.1.smithi203.stdout:3/47: truncate d0/f6 3684197 0 2024-04-05T03:02:31.461 INFO:tasks.workunit.client.1.smithi203.stdout:5/171: chown d3/d4/d8/l34 526000928 1 2024-04-05T03:02:31.462 INFO:tasks.workunit.client.1.smithi203.stdout:3/48: creat d0/fb x:0 0 0 2024-04-05T03:02:31.462 INFO:tasks.workunit.client.1.smithi203.stdout:9/30: mknod c5 0 2024-04-05T03:02:31.466 INFO:tasks.workunit.client.1.smithi203.stdout:3/49: write d0/f7 [883534,20351] 0 2024-04-05T03:02:31.466 INFO:tasks.workunit.client.1.smithi203.stdout:9/31: write f2 [1912895,74406] 0 2024-04-05T03:02:31.466 INFO:tasks.workunit.client.1.smithi203.stdout:0/37: dwrite f2 [0,4194304] 0 2024-04-05T03:02:31.468 INFO:tasks.workunit.client.1.smithi203.stdout:9/32: stat l0 0 2024-04-05T03:02:31.468 INFO:tasks.workunit.client.1.smithi203.stdout:3/50: stat d0/fb 0 2024-04-05T03:02:31.468 INFO:tasks.workunit.client.1.smithi203.stdout:8/17: sync 2024-04-05T03:02:31.468 INFO:tasks.workunit.client.1.smithi203.stdout:6/20: sync 2024-04-05T03:02:31.470 INFO:tasks.workunit.client.1.smithi203.stdout:0/38: chown c1 1 1 2024-04-05T03:02:31.471 INFO:tasks.workunit.client.1.smithi203.stdout:9/33: symlink l6 0 2024-04-05T03:02:31.472 INFO:tasks.workunit.client.1.smithi203.stdout:9/34: write f3 [465501,73946] 0 2024-04-05T03:02:31.474 INFO:tasks.workunit.client.1.smithi203.stdout:6/21: read - d0/f3 zero size 2024-04-05T03:02:31.474 INFO:tasks.workunit.client.1.smithi203.stdout:3/51: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:31.475 INFO:tasks.workunit.client.1.smithi203.stdout:8/18: mkdir d1 0 2024-04-05T03:02:31.476 INFO:tasks.workunit.client.1.smithi203.stdout:4/50: dwrite f8 [0,4194304] 0 2024-04-05T03:02:31.479 INFO:tasks.workunit.client.1.smithi203.stdout:9/35: rename f3 to f7 0 2024-04-05T03:02:31.482 INFO:tasks.workunit.client.1.smithi203.stdout:3/52: mkdir d0/dc 0 2024-04-05T03:02:31.482 INFO:tasks.workunit.client.1.smithi203.stdout:4/51: write f8 [4184962,97958] 0 2024-04-05T03:02:31.483 INFO:tasks.workunit.client.1.smithi203.stdout:8/19: symlink d1/l2 0 2024-04-05T03:02:31.485 INFO:tasks.workunit.client.1.smithi203.stdout:0/39: getdents . 0 2024-04-05T03:02:31.488 INFO:tasks.workunit.client.1.smithi203.stdout:9/36: creat f8 x:0 0 0 2024-04-05T03:02:31.491 INFO:tasks.workunit.client.1.smithi203.stdout:5/172: dwrite d3/f36 [8388608,4194304] 0 2024-04-05T03:02:31.493 INFO:tasks.workunit.client.1.smithi203.stdout:6/22: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:31.497 INFO:tasks.workunit.client.1.smithi203.stdout:8/20: creat d1/f3 x:0 0 0 2024-04-05T03:02:31.501 INFO:tasks.workunit.client.1.smithi203.stdout:4/52: dread f8 [0,4194304] 0 2024-04-05T03:02:31.501 INFO:tasks.workunit.client.1.smithi203.stdout:2/101: sync 2024-04-05T03:02:31.501 INFO:tasks.workunit.client.1.smithi203.stdout:7/46: sync 2024-04-05T03:02:31.501 INFO:tasks.workunit.client.1.smithi203.stdout:4/53: stat da/fc 0 2024-04-05T03:02:31.501 INFO:tasks.workunit.client.1.smithi203.stdout:1/46: sync 2024-04-05T03:02:31.502 INFO:tasks.workunit.client.1.smithi203.stdout:0/40: truncate f3 2570321 0 2024-04-05T03:02:31.505 INFO:tasks.workunit.client.1.smithi203.stdout:5/173: mkdir d3/d4/d8/d12/d40 0 2024-04-05T03:02:31.508 INFO:tasks.workunit.client.1.smithi203.stdout:2/102: creat d5/d10/f20 x:0 0 0 2024-04-05T03:02:31.512 INFO:tasks.workunit.client.1.smithi203.stdout:8/21: dread f0 [0,4194304] 0 2024-04-05T03:02:31.514 INFO:tasks.workunit.client.1.smithi203.stdout:4/54: unlink c6 0 2024-04-05T03:02:31.514 INFO:tasks.workunit.client.1.smithi203.stdout:9/37: dwrite f8 [0,4194304] 0 2024-04-05T03:02:31.517 INFO:tasks.workunit.client.1.smithi203.stdout:6/23: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:31.517 INFO:tasks.workunit.client.1.smithi203.stdout:5/174: creat d3/d1f/f41 x:0 0 0 2024-04-05T03:02:31.520 INFO:tasks.workunit.client.1.smithi203.stdout:1/47: dread f2 [0,4194304] 0 2024-04-05T03:02:31.520 INFO:tasks.workunit.client.1.smithi203.stdout:3/53: dread d0/f6 [0,4194304] 0 2024-04-05T03:02:31.521 INFO:tasks.workunit.client.1.smithi203.stdout:6/24: rename d0 to d0/d4 22 2024-04-05T03:02:31.523 INFO:tasks.workunit.client.1.smithi203.stdout:6/25: chown d0/f3 0 1 2024-04-05T03:02:31.523 INFO:tasks.workunit.client.1.smithi203.stdout:0/41: unlink f3 0 2024-04-05T03:02:31.525 INFO:tasks.workunit.client.1.smithi203.stdout:2/103: rmdir d5/d1f 0 2024-04-05T03:02:31.528 INFO:tasks.workunit.client.1.smithi203.stdout:4/55: link f8 da/f11 0 2024-04-05T03:02:31.530 INFO:tasks.workunit.client.1.smithi203.stdout:6/26: creat d0/f5 x:0 0 0 2024-04-05T03:02:31.531 INFO:tasks.workunit.client.1.smithi203.stdout:0/42: creat f6 x:0 0 0 2024-04-05T03:02:31.532 INFO:tasks.workunit.client.1.smithi203.stdout:5/175: mknod d3/d4/d8/c42 0 2024-04-05T03:02:31.532 INFO:tasks.workunit.client.1.smithi203.stdout:9/38: dwrite f8 [0,4194304] 0 2024-04-05T03:02:31.538 INFO:tasks.workunit.client.1.smithi203.stdout:8/22: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.539 INFO:tasks.workunit.client.1.smithi203.stdout:0/43: truncate f6 924385 0 2024-04-05T03:02:31.548 INFO:tasks.workunit.client.1.smithi203.stdout:6/27: write d0/f3 [4959880,39354] 0 2024-04-05T03:02:31.549 INFO:tasks.workunit.client.1.smithi203.stdout:5/176: write d3/d4/dc/f25 [1209794,2831] 0 2024-04-05T03:02:31.550 INFO:tasks.workunit.client.1.smithi203.stdout:1/48: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.556 INFO:tasks.workunit.client.1.smithi203.stdout:2/104: dread f3 [0,4194304] 0 2024-04-05T03:02:31.559 INFO:tasks.workunit.client.1.smithi203.stdout:1/49: creat d7/f13 x:0 0 0 2024-04-05T03:02:31.559 INFO:tasks.workunit.client.1.smithi203.stdout:9/39: dwrite f2 [0,4194304] 0 2024-04-05T03:02:31.560 INFO:tasks.workunit.client.1.smithi203.stdout:6/28: write d0/f3 [1759153,6759] 0 2024-04-05T03:02:31.563 INFO:tasks.workunit.client.1.smithi203.stdout:1/50: read f0 [4060992,124130] 0 2024-04-05T03:02:31.563 INFO:tasks.workunit.client.1.smithi203.stdout:4/56: dwrite f5 [0,4194304] 0 2024-04-05T03:02:31.565 INFO:tasks.workunit.client.1.smithi203.stdout:8/23: dread f0 [0,4194304] 0 2024-04-05T03:02:31.566 INFO:tasks.workunit.client.1.smithi203.stdout:1/51: chown d7/lb 320718374 1 2024-04-05T03:02:31.566 INFO:tasks.workunit.client.1.smithi203.stdout:9/40: mknod c9 0 2024-04-05T03:02:31.567 INFO:tasks.workunit.client.1.smithi203.stdout:8/24: dread - d1/f3 zero size 2024-04-05T03:02:31.567 INFO:tasks.workunit.client.1.smithi203.stdout:0/44: dread f2 [0,4194304] 0 2024-04-05T03:02:31.568 INFO:tasks.workunit.client.1.smithi203.stdout:2/105: mknod d5/c21 0 2024-04-05T03:02:31.569 INFO:tasks.workunit.client.1.smithi203.stdout:0/45: dread - f5 zero size 2024-04-05T03:02:31.570 INFO:tasks.workunit.client.1.smithi203.stdout:6/29: mkdir d0/d6 0 2024-04-05T03:02:31.575 INFO:tasks.workunit.client.1.smithi203.stdout:5/177: rename d3/d4/fe to d3/d4/d8/d12/f43 0 2024-04-05T03:02:31.575 INFO:tasks.workunit.client.1.smithi203.stdout:4/57: mknod da/c12 0 2024-04-05T03:02:31.576 INFO:tasks.workunit.client.1.smithi203.stdout:9/41: creat fa x:0 0 0 2024-04-05T03:02:31.576 INFO:tasks.workunit.client.1.smithi203.stdout:1/52: mkdir d7/d14 0 2024-04-05T03:02:31.577 INFO:tasks.workunit.client.1.smithi203.stdout:5/178: truncate d3/f32 8366 0 2024-04-05T03:02:31.579 INFO:tasks.workunit.client.1.smithi203.stdout:2/106: write d5/d10/f13 [9652716,23732] 0 2024-04-05T03:02:31.582 INFO:tasks.workunit.client.1.smithi203.stdout:9/42: creat fb x:0 0 0 2024-04-05T03:02:31.582 INFO:tasks.workunit.client.1.smithi203.stdout:9/43: rmdir - no directory 2024-04-05T03:02:31.582 INFO:tasks.workunit.client.1.smithi203.stdout:1/53: mknod d7/c15 0 2024-04-05T03:02:31.584 INFO:tasks.workunit.client.1.smithi203.stdout:1/54: stat d7/f13 0 2024-04-05T03:02:31.587 INFO:tasks.workunit.client.1.smithi203.stdout:2/107: truncate d5/d10/f13 9965108 0 2024-04-05T03:02:31.590 INFO:tasks.workunit.client.1.smithi203.stdout:5/179: rename d3/d1f/f41 to d3/d4/d8/f44 0 2024-04-05T03:02:31.590 INFO:tasks.workunit.client.1.smithi203.stdout:0/46: dwrite f5 [0,4194304] 0 2024-04-05T03:02:31.593 INFO:tasks.workunit.client.1.smithi203.stdout:1/55: getdents d7/d14 0 2024-04-05T03:02:31.593 INFO:tasks.workunit.client.1.smithi203.stdout:5/180: rename d3/d1f to d3/d1f/d45 22 2024-04-05T03:02:31.595 INFO:tasks.workunit.client.1.smithi203.stdout:8/25: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.598 INFO:tasks.workunit.client.1.smithi203.stdout:5/181: write d3/d4/fb [7100307,81301] 0 2024-04-05T03:02:31.604 INFO:tasks.workunit.client.1.smithi203.stdout:1/56: creat d7/d14/f16 x:0 0 0 2024-04-05T03:02:31.605 INFO:tasks.workunit.client.1.smithi203.stdout:4/58: dwrite f8 [0,4194304] 0 2024-04-05T03:02:31.605 INFO:tasks.workunit.client.1.smithi203.stdout:5/182: dread - d3/d1f/f39 zero size 2024-04-05T03:02:31.606 INFO:tasks.workunit.client.1.smithi203.stdout:2/108: truncate f3 5948251 0 2024-04-05T03:02:31.611 INFO:tasks.workunit.client.1.smithi203.stdout:9/44: dwrite f7 [0,4194304] 0 2024-04-05T03:02:31.611 INFO:tasks.workunit.client.1.smithi203.stdout:4/59: chown da/ld 109 1 2024-04-05T03:02:31.615 INFO:tasks.workunit.client.1.smithi203.stdout:1/57: dread d7/fa [0,4194304] 0 2024-04-05T03:02:31.617 INFO:tasks.workunit.client.1.smithi203.stdout:5/183: creat d3/d4/d8/d12/f46 x:0 0 0 2024-04-05T03:02:31.618 INFO:tasks.workunit.client.1.smithi203.stdout:0/47: dread f2 [0,4194304] 0 2024-04-05T03:02:31.622 INFO:tasks.workunit.client.1.smithi203.stdout:1/58: dread - d7/d14/f16 zero size 2024-04-05T03:02:31.628 INFO:tasks.workunit.client.1.smithi203.stdout:8/26: dwrite d1/f3 [0,4194304] 0 2024-04-05T03:02:31.630 INFO:tasks.workunit.client.1.smithi203.stdout:1/59: creat d7/f17 x:0 0 0 2024-04-05T03:02:31.632 INFO:tasks.workunit.client.1.smithi203.stdout:5/184: symlink d3/d4/d8/l47 0 2024-04-05T03:02:31.633 INFO:tasks.workunit.client.1.smithi203.stdout:2/109: dwrite d5/d10/f20 [0,4194304] 0 2024-04-05T03:02:31.634 INFO:tasks.workunit.client.1.smithi203.stdout:8/27: rename d1 to d1/d4 22 2024-04-05T03:02:31.636 INFO:tasks.workunit.client.1.smithi203.stdout:5/185: chown d3/d4/dc 0 1 2024-04-05T03:02:31.637 INFO:tasks.workunit.client.1.smithi203.stdout:4/60: dwrite f5 [0,4194304] 0 2024-04-05T03:02:31.638 INFO:tasks.workunit.client.1.smithi203.stdout:1/60: dread - d7/fe zero size 2024-04-05T03:02:31.639 INFO:tasks.workunit.client.1.smithi203.stdout:9/45: write f2 [4928361,38488] 0 2024-04-05T03:02:31.642 INFO:tasks.workunit.client.1.smithi203.stdout:0/48: dread f5 [0,4194304] 0 2024-04-05T03:02:31.642 INFO:tasks.workunit.client.1.smithi203.stdout:1/61: mknod d7/c18 0 2024-04-05T03:02:31.643 INFO:tasks.workunit.client.1.smithi203.stdout:9/46: mknod cc 0 2024-04-05T03:02:31.644 INFO:tasks.workunit.client.1.smithi203.stdout:9/47: fdatasync f8 0 2024-04-05T03:02:31.645 INFO:tasks.workunit.client.1.smithi203.stdout:4/61: creat da/f13 x:0 0 0 2024-04-05T03:02:31.646 INFO:tasks.workunit.client.1.smithi203.stdout:2/110: mkdir d5/d1e/d22 0 2024-04-05T03:02:31.646 INFO:tasks.workunit.client.1.smithi203.stdout:8/28: dread f0 [0,4194304] 0 2024-04-05T03:02:31.646 INFO:tasks.workunit.client.1.smithi203.stdout:4/62: fsync da/f10 0 2024-04-05T03:02:31.648 INFO:tasks.workunit.client.1.smithi203.stdout:0/49: write f2 [2555614,41287] 0 2024-04-05T03:02:31.649 INFO:tasks.workunit.client.1.smithi203.stdout:1/62: mknod d7/d14/c19 0 2024-04-05T03:02:31.652 INFO:tasks.workunit.client.1.smithi203.stdout:9/48: creat fd x:0 0 0 2024-04-05T03:02:31.652 INFO:tasks.workunit.client.1.smithi203.stdout:1/63: chown d7/d14 520 1 2024-04-05T03:02:31.654 INFO:tasks.workunit.client.1.smithi203.stdout:4/63: write da/f13 [305935,46454] 0 2024-04-05T03:02:31.656 INFO:tasks.workunit.client.1.smithi203.stdout:9/49: symlink le 0 2024-04-05T03:02:31.658 INFO:tasks.workunit.client.1.smithi203.stdout:5/186: link d3/d4/dc/l15 d3/d4/dc/l48 0 2024-04-05T03:02:31.659 INFO:tasks.workunit.client.1.smithi203.stdout:1/64: fsync d7/fa 0 2024-04-05T03:02:31.659 INFO:tasks.workunit.client.1.smithi203.stdout:9/50: truncate fa 262882 0 2024-04-05T03:02:31.659 INFO:tasks.workunit.client.1.smithi203.stdout:0/50: creat f7 x:0 0 0 2024-04-05T03:02:31.663 INFO:tasks.workunit.client.1.smithi203.stdout:0/51: creat f8 x:0 0 0 2024-04-05T03:02:31.664 INFO:tasks.workunit.client.1.smithi203.stdout:8/29: dwrite d1/f3 [0,4194304] 0 2024-04-05T03:02:31.666 INFO:tasks.workunit.client.1.smithi203.stdout:2/111: dread d5/d10/f20 [0,4194304] 0 2024-04-05T03:02:31.666 INFO:tasks.workunit.client.1.smithi203.stdout:9/51: creat ff x:0 0 0 2024-04-05T03:02:31.666 INFO:tasks.workunit.client.1.smithi203.stdout:0/52: write f6 [1485378,122526] 0 2024-04-05T03:02:31.667 INFO:tasks.workunit.client.1.smithi203.stdout:4/64: link da/f10 da/f14 0 2024-04-05T03:02:31.669 INFO:tasks.workunit.client.1.smithi203.stdout:2/112: dread - d5/d10/f1d zero size 2024-04-05T03:02:31.669 INFO:tasks.workunit.client.1.smithi203.stdout:0/53: chown f7 741823449 1 2024-04-05T03:02:31.669 INFO:tasks.workunit.client.1.smithi203.stdout:5/187: unlink d3/d4/d8/d12/f3f 0 2024-04-05T03:02:31.670 INFO:tasks.workunit.client.1.smithi203.stdout:9/52: truncate fd 559399 0 2024-04-05T03:02:31.671 INFO:tasks.workunit.client.1.smithi203.stdout:1/65: rename c6 to d7/d14/c1a 0 2024-04-05T03:02:31.672 INFO:tasks.workunit.client.1.smithi203.stdout:1/66: dread - d7/d14/f16 zero size 2024-04-05T03:02:31.681 INFO:tasks.workunit.client.1.smithi203.stdout:9/53: dwrite fd [0,4194304] 0 2024-04-05T03:02:31.686 INFO:tasks.workunit.client.1.smithi203.stdout:8/30: fdatasync d1/f3 0 2024-04-05T03:02:31.687 INFO:tasks.workunit.client.1.smithi203.stdout:5/188: getdents d3/d1f 0 2024-04-05T03:02:31.687 INFO:tasks.workunit.client.1.smithi203.stdout:9/54: fdatasync f7 0 2024-04-05T03:02:31.690 INFO:tasks.workunit.client.1.smithi203.stdout:1/67: dwrite d7/f17 [0,4194304] 0 2024-04-05T03:02:31.690 INFO:tasks.workunit.client.1.smithi203.stdout:8/31: write d1/f3 [170114,86079] 0 2024-04-05T03:02:31.691 INFO:tasks.workunit.client.1.smithi203.stdout:9/55: creat f10 x:0 0 0 2024-04-05T03:02:31.692 INFO:tasks.workunit.client.1.smithi203.stdout:5/189: write d3/d4/d8/f3a [159031,99344] 0 2024-04-05T03:02:31.693 INFO:tasks.workunit.client.1.smithi203.stdout:9/56: write fa [297156,62797] 0 2024-04-05T03:02:31.693 INFO:tasks.workunit.client.1.smithi203.stdout:8/32: chown d1 1802293 1 2024-04-05T03:02:31.697 INFO:tasks.workunit.client.1.smithi203.stdout:4/65: truncate da/fc 1151511 0 2024-04-05T03:02:31.699 INFO:tasks.workunit.client.1.smithi203.stdout:9/57: dread - fb zero size 2024-04-05T03:02:31.700 INFO:tasks.workunit.client.1.smithi203.stdout:0/54: truncate f6 50299 0 2024-04-05T03:02:31.702 INFO:tasks.workunit.client.1.smithi203.stdout:5/190: mkdir d3/d1f/d49 0 2024-04-05T03:02:31.702 INFO:tasks.workunit.client.1.smithi203.stdout:9/58: chown f8 15283881 1 2024-04-05T03:02:31.703 INFO:tasks.workunit.client.1.smithi203.stdout:9/59: chown le 964 1 2024-04-05T03:02:31.703 INFO:tasks.workunit.client.1.smithi203.stdout:1/68: dread f0 [0,4194304] 0 2024-04-05T03:02:31.705 INFO:tasks.workunit.client.1.smithi203.stdout:1/69: dread - d7/fe zero size 2024-04-05T03:02:31.705 INFO:tasks.workunit.client.1.smithi203.stdout:9/60: write fd [3824233,25313] 0 2024-04-05T03:02:31.706 INFO:tasks.workunit.client.1.smithi203.stdout:5/191: mknod d3/d1f/c4a 0 2024-04-05T03:02:31.707 INFO:tasks.workunit.client.1.smithi203.stdout:5/192: stat d3/d1f/f33 0 2024-04-05T03:02:31.708 INFO:tasks.workunit.client.1.smithi203.stdout:1/70: unlink f0 0 2024-04-05T03:02:31.709 INFO:tasks.workunit.client.1.smithi203.stdout:5/193: creat d3/d4/dc/f4b x:0 0 0 2024-04-05T03:02:31.710 INFO:tasks.workunit.client.1.smithi203.stdout:5/194: chown d3/d4/dc/l14 2951294 1 2024-04-05T03:02:31.710 INFO:tasks.workunit.client.1.smithi203.stdout:1/71: creat d7/d14/f1b x:0 0 0 2024-04-05T03:02:31.711 INFO:tasks.workunit.client.1.smithi203.stdout:9/61: dread f7 [0,4194304] 0 2024-04-05T03:02:31.711 INFO:tasks.workunit.client.1.smithi203.stdout:1/72: dread - d7/fe zero size 2024-04-05T03:02:31.711 INFO:tasks.workunit.client.1.smithi203.stdout:5/195: dread - d3/d4/d8/f44 zero size 2024-04-05T03:02:31.712 INFO:tasks.workunit.client.1.smithi203.stdout:9/62: chown l4 5702143 1 2024-04-05T03:02:31.712 INFO:tasks.workunit.client.1.smithi203.stdout:9/63: fdatasync fb 0 2024-04-05T03:02:31.713 INFO:tasks.workunit.client.1.smithi203.stdout:9/64: dread - f10 zero size 2024-04-05T03:02:31.717 INFO:tasks.workunit.client.1.smithi203.stdout:9/65: mknod c11 0 2024-04-05T03:02:31.721 INFO:tasks.workunit.client.1.smithi203.stdout:8/33: truncate d1/f3 3830256 0 2024-04-05T03:02:31.724 INFO:tasks.workunit.client.1.smithi203.stdout:9/66: link f8 f12 0 2024-04-05T03:02:31.725 INFO:tasks.workunit.client.1.smithi203.stdout:8/34: mknod d1/c5 0 2024-04-05T03:02:31.727 INFO:tasks.workunit.client.1.smithi203.stdout:5/196: dwrite d3/d4/d8/f44 [0,4194304] 0 2024-04-05T03:02:31.727 INFO:tasks.workunit.client.1.smithi203.stdout:9/67: mkdir d13 0 2024-04-05T03:02:31.728 INFO:tasks.workunit.client.1.smithi203.stdout:9/68: chown fb 818938 1 2024-04-05T03:02:31.730 INFO:tasks.workunit.client.1.smithi203.stdout:1/73: getdents d7/d14 0 2024-04-05T03:02:31.730 INFO:tasks.workunit.client.1.smithi203.stdout:5/197: write d3/d4/d8/f3d [741620,4907] 0 2024-04-05T03:02:31.735 INFO:tasks.workunit.client.1.smithi203.stdout:0/55: write f6 [421633,114957] 0 2024-04-05T03:02:31.737 INFO:tasks.workunit.client.1.smithi203.stdout:0/56: chown l4 60671 1 2024-04-05T03:02:31.737 INFO:tasks.workunit.client.1.smithi203.stdout:0/57: stat c1 0 2024-04-05T03:02:31.741 INFO:tasks.workunit.client.1.smithi203.stdout:0/58: write f6 [144220,26013] 0 2024-04-05T03:02:31.742 INFO:tasks.workunit.client.1.smithi203.stdout:9/69: mknod d13/c14 0 2024-04-05T03:02:31.744 INFO:tasks.workunit.client.1.smithi203.stdout:4/66: dwrite f9 [0,4194304] 0 2024-04-05T03:02:31.750 INFO:tasks.workunit.client.1.smithi203.stdout:1/74: rmdir d7 39 2024-04-05T03:02:31.751 INFO:tasks.workunit.client.1.smithi203.stdout:8/35: getdents d1 0 2024-04-05T03:02:31.754 INFO:tasks.workunit.client.1.smithi203.stdout:4/67: unlink da/f14 0 2024-04-05T03:02:31.756 INFO:tasks.workunit.client.1.smithi203.stdout:0/59: link f6 f9 0 2024-04-05T03:02:31.757 INFO:tasks.workunit.client.1.smithi203.stdout:0/60: chown f5 65985280 1 2024-04-05T03:02:31.758 INFO:tasks.workunit.client.1.smithi203.stdout:8/36: dread f0 [0,4194304] 0 2024-04-05T03:02:31.758 INFO:tasks.workunit.client.1.smithi203.stdout:1/75: chown d7/cc 2739 1 2024-04-05T03:02:31.759 INFO:tasks.workunit.client.1.smithi203.stdout:5/198: link d3/d4/dc/c2a d3/d4/d8/d12/c4c 0 2024-04-05T03:02:31.759 INFO:tasks.workunit.client.1.smithi203.stdout:4/68: creat da/f15 x:0 0 0 2024-04-05T03:02:31.760 INFO:tasks.workunit.client.1.smithi203.stdout:0/61: mknod ca 0 2024-04-05T03:02:31.762 INFO:tasks.workunit.client.1.smithi203.stdout:1/76: dread d7/fa [0,4194304] 0 2024-04-05T03:02:31.762 INFO:tasks.workunit.client.1.smithi203.stdout:8/37: write f0 [1843031,26356] 0 2024-04-05T03:02:31.764 INFO:tasks.workunit.client.1.smithi203.stdout:0/62: mknod cb 0 2024-04-05T03:02:31.765 INFO:tasks.workunit.client.1.smithi203.stdout:8/38: mkdir d1/d6 0 2024-04-05T03:02:31.767 INFO:tasks.workunit.client.1.smithi203.stdout:0/63: rename c1 to cc 0 2024-04-05T03:02:31.770 INFO:tasks.workunit.client.1.smithi203.stdout:0/64: fsync f5 0 2024-04-05T03:02:31.774 INFO:tasks.workunit.client.1.smithi203.stdout:5/199: dwrite d3/d1f/f39 [0,4194304] 0 2024-04-05T03:02:31.782 INFO:tasks.workunit.client.1.smithi203.stdout:1/77: dwrite d7/d14/f1b [0,4194304] 0 2024-04-05T03:02:31.782 INFO:tasks.workunit.client.1.smithi203.stdout:8/39: dwrite f0 [0,4194304] 0 2024-04-05T03:02:31.789 INFO:tasks.workunit.client.1.smithi203.stdout:1/78: readlink d7/l9 0 2024-04-05T03:02:31.790 INFO:tasks.workunit.client.1.smithi203.stdout:8/40: creat d1/f7 x:0 0 0 2024-04-05T03:02:31.790 INFO:tasks.workunit.client.1.smithi203.stdout:0/65: dwrite f7 [0,4194304] 0 2024-04-05T03:02:31.795 INFO:tasks.workunit.client.1.smithi203.stdout:0/66: read f6 [220247,23939] 0 2024-04-05T03:02:31.796 INFO:tasks.workunit.client.1.smithi203.stdout:8/41: stat d1/l2 0 2024-04-05T03:02:31.800 INFO:tasks.workunit.client.1.smithi203.stdout:0/67: fdatasync f9 0 2024-04-05T03:02:31.800 INFO:tasks.workunit.client.1.smithi203.stdout:5/200: dwrite d3/d4/d8/f3a [0,4194304] 0 2024-04-05T03:02:31.800 INFO:tasks.workunit.client.1.smithi203.stdout:1/79: chown d7/d14/f16 1913887247 1 2024-04-05T03:02:31.801 INFO:tasks.workunit.client.1.smithi203.stdout:8/42: read d1/f3 [77782,73275] 0 2024-04-05T03:02:31.802 INFO:tasks.workunit.client.1.smithi203.stdout:1/80: chown d7/cd 39 1 2024-04-05T03:02:31.807 INFO:tasks.workunit.client.1.smithi203.stdout:8/43: mknod d1/d6/c8 0 2024-04-05T03:02:31.810 INFO:tasks.workunit.client.1.smithi203.stdout:8/44: mkdir d1/d9 0 2024-04-05T03:02:31.811 INFO:tasks.workunit.client.1.smithi203.stdout:5/201: getdents d3/d4/d8/d12/d40 0 2024-04-05T03:02:31.812 INFO:tasks.workunit.client.1.smithi203.stdout:8/45: write f0 [3048505,87538] 0 2024-04-05T03:02:31.812 INFO:tasks.workunit.client.1.smithi203.stdout:1/81: getdents d7/d14 0 2024-04-05T03:02:31.814 INFO:tasks.workunit.client.1.smithi203.stdout:5/202: creat d3/f4d x:0 0 0 2024-04-05T03:02:31.816 INFO:tasks.workunit.client.1.smithi203.stdout:0/68: dwrite f8 [0,4194304] 0 2024-04-05T03:02:31.816 INFO:tasks.workunit.client.1.smithi203.stdout:1/82: truncate d7/fe 539026 0 2024-04-05T03:02:31.818 INFO:tasks.workunit.client.1.smithi203.stdout:0/69: chown l4 39 1 2024-04-05T03:02:31.818 INFO:tasks.workunit.client.1.smithi203.stdout:1/83: fdatasync d7/fe 0 2024-04-05T03:02:31.820 INFO:tasks.workunit.client.1.smithi203.stdout:1/84: creat d7/f1c x:0 0 0 2024-04-05T03:02:31.829 INFO:tasks.workunit.client.1.smithi203.stdout:1/85: mknod d7/c1d 0 2024-04-05T03:02:31.831 INFO:tasks.workunit.client.1.smithi203.stdout:8/46: getdents d1 0 2024-04-05T03:02:31.832 INFO:tasks.workunit.client.1.smithi203.stdout:5/203: dread d3/d4/d8/d12/f43 [0,4194304] 0 2024-04-05T03:02:31.836 INFO:tasks.workunit.client.1.smithi203.stdout:5/204: creat d3/d4/d8/d12/f4e x:0 0 0 2024-04-05T03:02:31.836 INFO:tasks.workunit.client.1.smithi203.stdout:8/47: mknod d1/d9/ca 0 2024-04-05T03:02:31.837 INFO:tasks.workunit.client.1.smithi203.stdout:8/48: rename d1 to d1/d6/db 22 2024-04-05T03:02:31.839 INFO:tasks.workunit.client.1.smithi203.stdout:5/205: chown d3/d4/d8/f44 1638 1 2024-04-05T03:02:31.839 INFO:tasks.workunit.client.1.smithi203.stdout:0/70: dwrite f9 [0,4194304] 0 2024-04-05T03:02:31.841 INFO:tasks.workunit.client.1.smithi203.stdout:8/49: write d1/f7 [575608,84787] 0 2024-04-05T03:02:31.842 INFO:tasks.workunit.client.1.smithi203.stdout:0/71: mkdir dd 0 2024-04-05T03:02:31.847 INFO:tasks.workunit.client.1.smithi203.stdout:0/72: symlink dd/le 0 2024-04-05T03:02:31.848 INFO:tasks.workunit.client.1.smithi203.stdout:1/86: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:31.849 INFO:tasks.workunit.client.1.smithi203.stdout:0/73: chown dd 1619837099 1 2024-04-05T03:02:31.850 INFO:tasks.workunit.client.1.smithi203.stdout:5/206: creat d3/d4/d8/f4f x:0 0 0 2024-04-05T03:02:31.854 INFO:tasks.workunit.client.1.smithi203.stdout:1/87: rename d7/f17 to d7/d14/f1e 0 2024-04-05T03:02:31.854 INFO:tasks.workunit.client.1.smithi203.stdout:8/50: dread d1/f3 [0,4194304] 0 2024-04-05T03:02:31.857 INFO:tasks.workunit.client.1.smithi203.stdout:1/88: read d7/fe [383713,120667] 0 2024-04-05T03:02:31.858 INFO:tasks.workunit.client.1.smithi203.stdout:8/51: mknod d1/d6/cc 0 2024-04-05T03:02:31.859 INFO:tasks.workunit.client.1.smithi203.stdout:5/207: getdents d3/d1f/d49 0 2024-04-05T03:02:31.864 INFO:tasks.workunit.client.1.smithi203.stdout:5/208: mkdir d3/d4/d50 0 2024-04-05T03:02:31.864 INFO:tasks.workunit.client.1.smithi203.stdout:0/74: dwrite f9 [4194304,4194304] 0 2024-04-05T03:02:31.866 INFO:tasks.workunit.client.1.smithi203.stdout:0/75: fdatasync f7 0 2024-04-05T03:02:31.867 INFO:tasks.workunit.client.1.smithi203.stdout:8/52: getdents d1/d6 0 2024-04-05T03:02:31.869 INFO:tasks.workunit.client.1.smithi203.stdout:0/76: symlink dd/lf 0 2024-04-05T03:02:31.871 INFO:tasks.workunit.client.1.smithi203.stdout:1/89: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:31.872 INFO:tasks.workunit.client.1.smithi203.stdout:8/53: symlink d1/ld 0 2024-04-05T03:02:31.874 INFO:tasks.workunit.client.1.smithi203.stdout:8/54: write d1/f7 [954667,105077] 0 2024-04-05T03:02:31.874 INFO:tasks.workunit.client.1.smithi203.stdout:0/77: mknod dd/c10 0 2024-04-05T03:02:31.875 INFO:tasks.workunit.client.1.smithi203.stdout:5/209: rename d3/d4/l29 to d3/d4/d8/d12/l51 0 2024-04-05T03:02:31.877 INFO:tasks.workunit.client.1.smithi203.stdout:0/78: symlink dd/l11 0 2024-04-05T03:02:31.878 INFO:tasks.workunit.client.1.smithi203.stdout:0/79: write f8 [2125958,116346] 0 2024-04-05T03:02:31.879 INFO:tasks.workunit.client.1.smithi203.stdout:8/55: mknod d1/d6/ce 0 2024-04-05T03:02:31.879 INFO:tasks.workunit.client.1.smithi203.stdout:0/80: chown l4 74 1 2024-04-05T03:02:31.881 INFO:tasks.workunit.client.1.smithi203.stdout:5/210: symlink d3/d4/d8/d12/l52 0 2024-04-05T03:02:31.882 INFO:tasks.workunit.client.1.smithi203.stdout:1/90: link d7/c18 d7/c1f 0 2024-04-05T03:02:31.883 INFO:tasks.workunit.client.1.smithi203.stdout:0/81: rmdir dd 39 2024-04-05T03:02:31.883 INFO:tasks.workunit.client.1.smithi203.stdout:1/91: chown d7/lf 33665946 1 2024-04-05T03:02:31.887 INFO:tasks.workunit.client.1.smithi203.stdout:1/92: symlink d7/d14/l20 0 2024-04-05T03:02:31.890 INFO:tasks.workunit.client.1.smithi203.stdout:8/56: dread d1/f7 [0,4194304] 0 2024-04-05T03:02:31.891 INFO:tasks.workunit.client.1.smithi203.stdout:5/211: dwrite d3/d4/dc/f25 [0,4194304] 0 2024-04-05T03:02:31.891 INFO:tasks.workunit.client.1.smithi203.stdout:0/82: rename f9 to dd/f12 0 2024-04-05T03:02:31.892 INFO:tasks.workunit.client.1.smithi203.stdout:8/57: chown d1/c5 15918259 1 2024-04-05T03:02:31.893 INFO:tasks.workunit.client.1.smithi203.stdout:5/212: chown d3/d4/d8/l34 48 1 2024-04-05T03:02:31.894 INFO:tasks.workunit.client.1.smithi203.stdout:0/83: write dd/f12 [4573637,123103] 0 2024-04-05T03:02:31.896 INFO:tasks.workunit.client.1.smithi203.stdout:8/58: creat d1/d6/ff x:0 0 0 2024-04-05T03:02:31.966 INFO:tasks.workunit.client.1.smithi203.stdout:5/213: unlink d3/d4/dc/f35 0 2024-04-05T03:02:31.968 INFO:tasks.workunit.client.1.smithi203.stdout:8/59: mkdir d1/d9/d10 0 2024-04-05T03:02:31.968 INFO:tasks.workunit.client.1.smithi203.stdout:0/84: link dd/l11 dd/l13 0 2024-04-05T03:02:31.970 INFO:tasks.workunit.client.1.smithi203.stdout:8/60: mknod d1/c11 0 2024-04-05T03:02:31.971 INFO:tasks.workunit.client.1.smithi203.stdout:5/214: mknod d3/d4/c53 0 2024-04-05T03:02:31.971 INFO:tasks.workunit.client.1.smithi203.stdout:8/61: chown d1/c11 118734643 1 2024-04-05T03:02:31.974 INFO:tasks.workunit.client.1.smithi203.stdout:8/62: mkdir d1/d12 0 2024-04-05T03:02:31.976 INFO:tasks.workunit.client.1.smithi203.stdout:0/85: dwrite f5 [0,4194304] 0 2024-04-05T03:02:31.978 INFO:tasks.workunit.client.1.smithi203.stdout:5/215: dread d3/f7 [0,4194304] 0 2024-04-05T03:02:31.979 INFO:tasks.workunit.client.1.smithi203.stdout:0/86: creat dd/f14 x:0 0 0 2024-04-05T03:02:31.980 INFO:tasks.workunit.client.1.smithi203.stdout:8/63: dread d1/f3 [0,4194304] 0 2024-04-05T03:02:31.981 INFO:tasks.workunit.client.1.smithi203.stdout:0/87: creat dd/f15 x:0 0 0 2024-04-05T03:02:31.982 INFO:tasks.workunit.client.1.smithi203.stdout:8/64: mknod d1/c13 0 2024-04-05T03:02:31.984 INFO:tasks.workunit.client.1.smithi203.stdout:0/88: creat dd/f16 x:0 0 0 2024-04-05T03:02:31.986 INFO:tasks.workunit.client.1.smithi203.stdout:8/65: rename d1/d6/ce to d1/d9/d10/c14 0 2024-04-05T03:02:31.986 INFO:tasks.workunit.client.1.smithi203.stdout:5/216: dwrite d3/f32 [0,4194304] 0 2024-04-05T03:02:31.987 INFO:tasks.workunit.client.1.smithi203.stdout:0/89: write f6 [444285,78516] 0 2024-04-05T03:02:31.987 INFO:tasks.workunit.client.1.smithi203.stdout:8/66: readlink d1/l2 0 2024-04-05T03:02:31.994 INFO:tasks.workunit.client.1.smithi203.stdout:8/67: rename d1/d6/ff to d1/d9/d10/f15 0 2024-04-05T03:02:31.996 INFO:tasks.workunit.client.1.smithi203.stdout:8/68: stat d1/d6/c8 0 2024-04-05T03:02:31.998 INFO:tasks.workunit.client.1.smithi203.stdout:5/217: dwrite d3/d1f/f3b [0,4194304] 0 2024-04-05T03:02:32.000 INFO:tasks.workunit.client.1.smithi203.stdout:0/90: dwrite dd/f12 [4194304,4194304] 0 2024-04-05T03:02:32.001 INFO:tasks.workunit.client.1.smithi203.stdout:8/69: write d1/f7 [1873364,107421] 0 2024-04-05T03:02:32.003 INFO:tasks.workunit.client.1.smithi203.stdout:0/91: chown dd/c10 2325057 1 2024-04-05T03:02:32.009 INFO:tasks.workunit.client.1.smithi203.stdout:8/70: dread d1/f3 [0,4194304] 0 2024-04-05T03:02:32.010 INFO:tasks.workunit.client.1.smithi203.stdout:5/218: dwrite d3/d1f/f3b [0,4194304] 0 2024-04-05T03:02:32.010 INFO:tasks.workunit.client.1.smithi203.stdout:0/92: dread f5 [0,4194304] 0 2024-04-05T03:02:32.012 INFO:tasks.workunit.client.1.smithi203.stdout:0/93: truncate dd/f15 314077 0 2024-04-05T03:02:32.014 INFO:tasks.workunit.client.1.smithi203.stdout:5/219: creat d3/d4/dc/f54 x:0 0 0 2024-04-05T03:02:32.015 INFO:tasks.workunit.client.1.smithi203.stdout:8/71: dread f0 [0,4194304] 0 2024-04-05T03:02:32.015 INFO:tasks.workunit.client.1.smithi203.stdout:0/94: mkdir dd/d17 0 2024-04-05T03:02:32.081 INFO:tasks.workunit.client.1.smithi203.stdout:5/220: link d3/d4/dc/l38 d3/d4/d8/l55 0 2024-04-05T03:02:32.081 INFO:tasks.workunit.client.1.smithi203.stdout:0/95: dwrite f2 [0,4194304] 0 2024-04-05T03:02:32.084 INFO:tasks.workunit.client.1.smithi203.stdout:5/221: fsync d3/d1f/f39 0 2024-04-05T03:02:32.084 INFO:tasks.workunit.client.1.smithi203.stdout:0/96: creat dd/d17/f18 x:0 0 0 2024-04-05T03:02:32.086 INFO:tasks.workunit.client.1.smithi203.stdout:0/97: mknod dd/c19 0 2024-04-05T03:02:32.090 INFO:tasks.workunit.client.1.smithi203.stdout:8/72: dwrite d1/f3 [0,4194304] 0 2024-04-05T03:02:32.101 INFO:tasks.workunit.client.1.smithi203.stdout:5/222: dwrite d3/d4/d8/f44 [0,4194304] 0 2024-04-05T03:02:32.102 INFO:tasks.workunit.client.1.smithi203.stdout:0/98: dwrite f5 [0,4194304] 0 2024-04-05T03:02:32.106 INFO:tasks.workunit.client.1.smithi203.stdout:0/99: symlink dd/d17/l1a 0 2024-04-05T03:02:32.108 INFO:tasks.workunit.client.1.smithi203.stdout:8/73: dwrite f0 [0,4194304] 0 2024-04-05T03:02:32.109 INFO:tasks.workunit.client.1.smithi203.stdout:0/100: mknod dd/c1b 0 2024-04-05T03:02:32.110 INFO:tasks.workunit.client.1.smithi203.stdout:0/101: readlink dd/d17/l1a 0 2024-04-05T03:02:32.111 INFO:tasks.workunit.client.1.smithi203.stdout:0/102: write f8 [2244662,15300] 0 2024-04-05T03:02:32.113 INFO:tasks.workunit.client.1.smithi203.stdout:8/74: unlink f0 0 2024-04-05T03:02:32.114 INFO:tasks.workunit.client.1.smithi203.stdout:0/103: chown dd/f16 33 1 2024-04-05T03:02:32.115 INFO:tasks.workunit.client.1.smithi203.stdout:8/75: write d1/f7 [1620202,64181] 0 2024-04-05T03:02:32.117 INFO:tasks.workunit.client.1.smithi203.stdout:8/76: stat d1/d6/cc 0 2024-04-05T03:02:32.120 INFO:tasks.workunit.client.1.smithi203.stdout:8/77: mkdir d1/d6/d16 0 2024-04-05T03:02:32.121 INFO:tasks.workunit.client.1.smithi203.stdout:8/78: chown d1/d6/d16 52 1 2024-04-05T03:02:32.121 INFO:tasks.workunit.client.1.smithi203.stdout:0/104: rmdir dd/d17 39 2024-04-05T03:02:32.123 INFO:tasks.workunit.client.1.smithi203.stdout:8/79: write d1/f7 [651465,118443] 0 2024-04-05T03:02:32.124 INFO:tasks.workunit.client.1.smithi203.stdout:8/80: stat d1/d6/c8 0 2024-04-05T03:02:32.125 INFO:tasks.workunit.client.1.smithi203.stdout:0/105: rmdir dd 39 2024-04-05T03:02:32.127 INFO:tasks.workunit.client.1.smithi203.stdout:0/106: truncate dd/f16 869407 0 2024-04-05T03:02:32.129 INFO:tasks.workunit.client.1.smithi203.stdout:0/107: mknod dd/d17/c1c 0 2024-04-05T03:02:32.131 INFO:tasks.workunit.client.1.smithi203.stdout:0/108: mkdir dd/d17/d1d 0 2024-04-05T03:02:32.133 INFO:tasks.workunit.client.1.smithi203.stdout:0/109: dread dd/f16 [0,4194304] 0 2024-04-05T03:02:32.134 INFO:tasks.workunit.client.1.smithi203.stdout:0/110: write dd/f12 [3704066,59381] 0 2024-04-05T03:02:32.139 INFO:tasks.workunit.client.1.smithi203.stdout:8/81: truncate d1/f7 1958939 0 2024-04-05T03:02:32.139 INFO:tasks.workunit.client.1.smithi203.stdout:0/111: dread f8 [0,4194304] 0 2024-04-05T03:02:32.143 INFO:tasks.workunit.client.1.smithi203.stdout:8/82: rename d1/d9 to d1/d12/d17 0 2024-04-05T03:02:32.143 INFO:tasks.workunit.client.1.smithi203.stdout:8/83: chown d1 5938982 1 2024-04-05T03:02:32.143 INFO:tasks.workunit.client.1.smithi203.stdout:0/112: creat dd/d17/d1d/f1e x:0 0 0 2024-04-05T03:02:32.144 INFO:tasks.workunit.client.1.smithi203.stdout:0/113: dread - dd/d17/d1d/f1e zero size 2024-04-05T03:02:32.146 INFO:tasks.workunit.client.1.smithi203.stdout:0/114: write f8 [3181920,7663] 0 2024-04-05T03:02:32.148 INFO:tasks.workunit.client.1.smithi203.stdout:0/115: truncate dd/d17/f18 547138 0 2024-04-05T03:02:32.148 INFO:tasks.workunit.client.1.smithi203.stdout:0/116: fsync f2 0 2024-04-05T03:02:32.150 INFO:tasks.workunit.client.1.smithi203.stdout:0/117: symlink dd/d17/d1d/l1f 0 2024-04-05T03:02:32.151 INFO:tasks.workunit.client.1.smithi203.stdout:0/118: write f6 [514438,81666] 0 2024-04-05T03:02:32.155 INFO:tasks.workunit.client.1.smithi203.stdout:8/84: dread d1/f7 [0,4194304] 0 2024-04-05T03:02:32.158 INFO:tasks.workunit.client.1.smithi203.stdout:8/85: dread d1/f7 [0,4194304] 0 2024-04-05T03:02:32.162 INFO:tasks.workunit.client.1.smithi203.stdout:8/86: write d1/f3 [1098863,127449] 0 2024-04-05T03:02:32.162 INFO:tasks.workunit.client.1.smithi203.stdout:8/87: truncate d1/f3 5152162 0 2024-04-05T03:02:32.162 INFO:tasks.workunit.client.1.smithi203.stdout:0/119: truncate dd/d17/f18 128599 0 2024-04-05T03:02:32.164 INFO:tasks.workunit.client.1.smithi203.stdout:0/120: creat dd/f20 x:0 0 0 2024-04-05T03:02:32.165 INFO:tasks.workunit.client.1.smithi203.stdout:8/88: getdents d1/d12/d17 0 2024-04-05T03:02:32.167 INFO:tasks.workunit.client.1.smithi203.stdout:0/121: mkdir dd/d21 0 2024-04-05T03:02:32.168 INFO:tasks.workunit.client.1.smithi203.stdout:8/89: rename d1/c5 to d1/d6/c18 0 2024-04-05T03:02:32.169 INFO:tasks.workunit.client.1.smithi203.stdout:8/90: chown d1/d6/c8 787 1 2024-04-05T03:02:32.176 INFO:tasks.workunit.client.1.smithi203.stdout:8/91: dwrite d1/f7 [0,4194304] 0 2024-04-05T03:02:32.177 INFO:tasks.workunit.client.1.smithi203.stdout:8/92: chown d1/d12 7187 1 2024-04-05T03:02:32.187 INFO:tasks.workunit.client.1.smithi203.stdout:8/93: truncate d1/f3 3426853 0 2024-04-05T03:02:32.188 INFO:tasks.workunit.client.1.smithi203.stdout:8/94: chown d1/d12/d17/d10 619 1 2024-04-05T03:02:32.188 INFO:tasks.workunit.client.1.smithi203.stdout:8/95: write d1/f7 [2164421,87355] 0 2024-04-05T03:02:32.189 INFO:tasks.workunit.client.1.smithi203.stdout:8/96: stat d1/d12 0 2024-04-05T03:02:32.196 INFO:tasks.workunit.client.1.smithi203.stdout:8/97: write d1/f3 [1824659,52438] 0 2024-04-05T03:02:32.198 INFO:tasks.workunit.client.1.smithi203.stdout:8/98: truncate d1/d12/d17/d10/f15 883501 0 2024-04-05T03:02:32.198 INFO:tasks.workunit.client.1.smithi203.stdout:8/99: readlink d1/ld 0 2024-04-05T03:02:32.201 INFO:tasks.workunit.client.1.smithi203.stdout:8/100: dread d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:32.209 INFO:tasks.workunit.client.1.smithi203.stdout:8/101: dwrite d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:32.214 INFO:tasks.workunit.client.1.smithi203.stdout:8/102: dread d1/f7 [0,4194304] 0 2024-04-05T03:02:32.217 INFO:tasks.workunit.client.1.smithi203.stdout:8/103: write d1/f7 [3750168,22679] 0 2024-04-05T03:02:32.218 INFO:tasks.workunit.client.1.smithi203.stdout:8/104: read d1/f7 [4183236,63279] 0 2024-04-05T03:02:32.219 INFO:tasks.workunit.client.1.smithi203.stdout:8/105: chown d1/f3 9486165 1 2024-04-05T03:02:32.219 INFO:tasks.workunit.client.1.smithi203.stdout:8/106: stat d1/f7 0 2024-04-05T03:02:32.222 INFO:tasks.workunit.client.1.smithi203.stdout:8/107: rmdir d1/d6 39 2024-04-05T03:02:32.229 INFO:tasks.workunit.client.1.smithi203.stdout:8/108: dwrite d1/f3 [4194304,4194304] 0 2024-04-05T03:02:32.237 INFO:tasks.workunit.client.1.smithi203.stdout:8/109: dwrite d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:32.248 INFO:tasks.workunit.client.1.smithi203.stdout:8/110: dwrite d1/f7 [0,4194304] 0 2024-04-05T03:02:32.253 INFO:tasks.workunit.client.1.smithi203.stdout:8/111: dread d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:32.254 INFO:tasks.workunit.client.1.smithi203.stdout:8/112: write d1/f7 [3013230,77787] 0 2024-04-05T03:02:32.257 INFO:tasks.workunit.client.1.smithi203.stdout:8/113: symlink d1/d12/l19 0 2024-04-05T03:02:32.265 INFO:tasks.workunit.client.1.smithi203.stdout:8/114: dwrite d1/f7 [4194304,4194304] 0 2024-04-05T03:02:32.266 INFO:tasks.workunit.client.1.smithi203.stdout:8/115: write d1/f7 [9304507,23806] 0 2024-04-05T03:02:32.269 INFO:tasks.workunit.client.1.smithi203.stdout:8/116: mkdir d1/d12/d1a 0 2024-04-05T03:02:32.270 INFO:tasks.workunit.client.1.smithi203.stdout:8/117: truncate d1/f3 8445640 0 2024-04-05T03:02:32.270 INFO:tasks.workunit.client.1.smithi203.stdout:8/118: readlink d1/d12/l19 0 2024-04-05T03:02:32.277 INFO:tasks.workunit.client.1.smithi203.stdout:8/119: dwrite d1/f3 [0,4194304] 0 2024-04-05T03:02:32.280 INFO:tasks.workunit.client.1.smithi203.stdout:8/120: unlink d1/f7 0 2024-04-05T03:02:32.282 INFO:tasks.workunit.client.1.smithi203.stdout:8/121: write d1/f3 [784790,106239] 0 2024-04-05T03:02:32.287 INFO:tasks.workunit.client.1.smithi203.stdout:8/122: symlink d1/d6/d16/l1b 0 2024-04-05T03:02:32.291 INFO:tasks.workunit.client.1.smithi203.stdout:8/123: creat d1/f1c x:0 0 0 2024-04-05T03:02:32.294 INFO:tasks.workunit.client.1.smithi203.stdout:8/124: creat d1/d12/d1a/f1d x:0 0 0 2024-04-05T03:02:32.301 INFO:tasks.workunit.client.1.smithi203.stdout:8/125: dwrite d1/f1c [0,4194304] 0 2024-04-05T03:02:32.304 INFO:tasks.workunit.client.1.smithi203.stdout:8/126: getdents d1/d12/d17 0 2024-04-05T03:02:32.305 INFO:tasks.workunit.client.1.smithi203.stdout:8/127: chown d1/d12/d17/ca 20193123 1 2024-04-05T03:02:32.319 INFO:tasks.workunit.client.1.smithi203.stdout:8/128: fdatasync d1/f1c 0 2024-04-05T03:02:32.322 INFO:tasks.workunit.client.1.smithi203.stdout:8/129: getdents d1/d12/d17/d10 0 2024-04-05T03:02:32.323 INFO:tasks.workunit.client.1.smithi203.stdout:8/130: write d1/d12/d1a/f1d [281501,87305] 0 2024-04-05T03:02:32.327 INFO:tasks.workunit.client.1.smithi203.stdout:8/131: truncate d1/f3 1182341 0 2024-04-05T03:02:32.333 INFO:tasks.workunit.client.1.smithi203.stdout:8/132: dread d1/f3 [0,4194304] 0 2024-04-05T03:02:32.343 INFO:tasks.workunit.client.1.smithi203.stdout:8/133: dwrite d1/f3 [0,4194304] 0 2024-04-05T03:02:32.345 INFO:tasks.workunit.client.1.smithi203.stdout:8/134: creat d1/d12/d1a/f1e x:0 0 0 2024-04-05T03:02:32.349 INFO:tasks.workunit.client.1.smithi203.stdout:8/135: mkdir d1/d6/d16/d1f 0 2024-04-05T03:02:32.355 INFO:tasks.workunit.client.1.smithi203.stdout:8/136: dwrite d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:32.355 INFO:tasks.workunit.client.1.smithi203.stdout:8/137: readlink d1/d12/l19 0 2024-04-05T03:02:32.356 INFO:tasks.workunit.client.1.smithi203.stdout:8/138: chown d1/d6/d16 884341466 1 2024-04-05T03:02:32.357 INFO:tasks.workunit.client.1.smithi203.stdout:8/139: chown d1/d6/c8 453 1 2024-04-05T03:02:32.357 INFO:tasks.workunit.client.1.smithi203.stdout:8/140: fdatasync d1/f3 0 2024-04-05T03:02:32.358 INFO:tasks.workunit.client.1.smithi203.stdout:8/141: readlink d1/d12/l19 0 2024-04-05T03:02:32.363 INFO:tasks.workunit.client.1.smithi203.stdout:8/142: dread d1/f3 [0,4194304] 0 2024-04-05T03:02:32.366 INFO:tasks.workunit.client.1.smithi203.stdout:8/143: unlink d1/d12/d17/d10/c14 0 2024-04-05T03:02:32.367 INFO:tasks.workunit.client.1.smithi203.stdout:8/144: read - d1/d12/d1a/f1e zero size 2024-04-05T03:02:32.496 INFO:tasks.workunit.client.1.smithi203.stdout:7/47: sync 2024-04-05T03:02:32.498 INFO:tasks.workunit.client.1.smithi203.stdout:7/48: mknod d8/c11 0 2024-04-05T03:02:32.500 INFO:tasks.workunit.client.1.smithi203.stdout:7/49: mkdir d8/d12 0 2024-04-05T03:02:32.502 INFO:tasks.workunit.client.1.smithi203.stdout:7/50: dread f6 [0,4194304] 0 2024-04-05T03:02:32.505 INFO:tasks.workunit.client.1.smithi203.stdout:7/51: link d8/da/f10 d8/d12/f13 0 2024-04-05T03:02:32.509 INFO:tasks.workunit.client.1.smithi203.stdout:7/52: dread d8/ff [0,4194304] 0 2024-04-05T03:02:32.511 INFO:tasks.workunit.client.1.smithi203.stdout:7/53: creat d8/da/f14 x:0 0 0 2024-04-05T03:02:32.518 INFO:tasks.workunit.client.1.smithi203.stdout:7/54: dwrite f6 [0,4194304] 0 2024-04-05T03:02:32.520 INFO:tasks.workunit.client.1.smithi203.stdout:7/55: creat d8/da/f15 x:0 0 0 2024-04-05T03:02:32.532 INFO:tasks.workunit.client.1.smithi203.stdout:7/56: dwrite f3 [0,4194304] 0 2024-04-05T03:02:32.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/57: mkdir d8/d12/d16 0 2024-04-05T03:02:32.537 INFO:tasks.workunit.client.1.smithi203.stdout:7/58: rmdir d8/da 39 2024-04-05T03:02:32.548 INFO:tasks.workunit.client.1.smithi203.stdout:7/59: dwrite d8/fe [0,4194304] 0 2024-04-05T03:02:32.553 INFO:tasks.workunit.client.1.smithi203.stdout:7/60: fsync f3 0 2024-04-05T03:02:32.557 INFO:tasks.workunit.client.1.smithi203.stdout:7/61: dread d8/fe [0,4194304] 0 2024-04-05T03:02:32.563 INFO:tasks.workunit.client.1.smithi203.stdout:7/62: rename d8/da/f10 to d8/f17 0 2024-04-05T03:02:32.563 INFO:tasks.workunit.client.1.smithi203.stdout:7/63: write f3 [4407517,113864] 0 2024-04-05T03:02:32.568 INFO:tasks.workunit.client.1.smithi203.stdout:7/64: write d8/f17 [781604,88965] 0 2024-04-05T03:02:32.568 INFO:tasks.workunit.client.1.smithi203.stdout:7/65: chown f3 787 1 2024-04-05T03:02:32.571 INFO:tasks.workunit.client.1.smithi203.stdout:7/66: rename f2 to d8/d12/f18 0 2024-04-05T03:02:32.574 INFO:tasks.workunit.client.1.smithi203.stdout:7/67: getdents d8 0 2024-04-05T03:02:32.578 INFO:tasks.workunit.client.1.smithi203.stdout:7/68: mkdir d8/da/d19 0 2024-04-05T03:02:32.579 INFO:tasks.workunit.client.1.smithi203.stdout:7/69: truncate d8/da/f14 950989 0 2024-04-05T03:02:32.581 INFO:tasks.workunit.client.1.smithi203.stdout:7/70: mkdir d8/d12/d16/d1a 0 2024-04-05T03:02:32.587 INFO:tasks.workunit.client.1.smithi203.stdout:7/71: dwrite d8/d12/f18 [0,4194304] 0 2024-04-05T03:02:32.587 INFO:tasks.workunit.client.1.smithi203.stdout:7/72: readlink l7 0 2024-04-05T03:02:32.589 INFO:tasks.workunit.client.1.smithi203.stdout:7/73: mknod d8/c1b 0 2024-04-05T03:02:32.602 INFO:tasks.workunit.client.1.smithi203.stdout:7/74: truncate f4 683837 0 2024-04-05T03:02:32.602 INFO:tasks.workunit.client.1.smithi203.stdout:7/75: readlink l7 0 2024-04-05T03:02:32.604 INFO:tasks.workunit.client.1.smithi203.stdout:7/76: chown d8/da/cb 58 1 2024-04-05T03:02:32.610 INFO:tasks.workunit.client.1.smithi203.stdout:7/77: dwrite d8/da/fd [0,4194304] 0 2024-04-05T03:02:32.617 INFO:tasks.workunit.client.1.smithi203.stdout:7/78: link d8/c1b d8/c1c 0 2024-04-05T03:02:32.622 INFO:tasks.workunit.client.1.smithi203.stdout:7/79: dwrite f3 [0,4194304] 0 2024-04-05T03:02:32.626 INFO:tasks.workunit.client.1.smithi203.stdout:7/80: rename f6 to d8/da/f1d 0 2024-04-05T03:02:32.633 INFO:tasks.workunit.client.1.smithi203.stdout:7/81: dwrite d8/fe [0,4194304] 0 2024-04-05T03:02:32.638 INFO:tasks.workunit.client.1.smithi203.stdout:7/82: unlink d8/c1b 0 2024-04-05T03:02:32.645 INFO:tasks.workunit.client.1.smithi203.stdout:7/83: dwrite d8/d12/f18 [0,4194304] 0 2024-04-05T03:02:32.658 INFO:tasks.workunit.client.1.smithi203.stdout:7/84: rename d8/d12/f13 to d8/d12/f1e 0 2024-04-05T03:02:32.662 INFO:tasks.workunit.client.1.smithi203.stdout:7/85: rmdir d8/da/d19 0 2024-04-05T03:02:32.681 INFO:tasks.workunit.client.1.smithi203.stdout:3/54: sync 2024-04-05T03:02:32.692 INFO:tasks.workunit.client.1.smithi203.stdout:7/86: truncate f3 1935750 0 2024-04-05T03:02:32.693 INFO:tasks.workunit.client.1.smithi203.stdout:3/55: truncate d0/f7 2034255 0 2024-04-05T03:02:32.700 INFO:tasks.workunit.client.1.smithi203.stdout:3/56: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:32.703 INFO:tasks.workunit.client.1.smithi203.stdout:3/57: unlink d0/f6 0 2024-04-05T03:02:32.718 INFO:tasks.workunit.client.1.smithi203.stdout:3/58: rename d0/f7 to d0/fd 0 2024-04-05T03:02:32.719 INFO:tasks.workunit.client.1.smithi203.stdout:3/59: write d0/f5 [3798836,16352] 0 2024-04-05T03:02:32.747 INFO:tasks.workunit.client.1.smithi203.stdout:3/60: dread d0/fd [0,4194304] 0 2024-04-05T03:02:32.751 INFO:tasks.workunit.client.1.smithi203.stdout:3/61: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:32.755 INFO:tasks.workunit.client.1.smithi203.stdout:3/62: dread d0/f4 [0,4194304] 0 2024-04-05T03:02:32.759 INFO:tasks.workunit.client.1.smithi203.stdout:3/63: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:32.762 INFO:tasks.workunit.client.1.smithi203.stdout:3/64: symlink d0/dc/le 0 2024-04-05T03:02:32.764 INFO:tasks.workunit.client.1.smithi203.stdout:3/65: creat d0/ff x:0 0 0 2024-04-05T03:02:32.764 INFO:tasks.workunit.client.1.smithi203.stdout:3/66: stat d0/fd 0 2024-04-05T03:02:32.765 INFO:tasks.workunit.client.1.smithi203.stdout:3/67: chown d0/fd 160201534 1 2024-04-05T03:02:32.765 INFO:tasks.workunit.client.1.smithi203.stdout:3/68: dread - d0/ff zero size 2024-04-05T03:02:32.769 INFO:tasks.workunit.client.1.smithi203.stdout:3/69: getdents d0/dc 0 2024-04-05T03:02:32.775 INFO:tasks.workunit.client.1.smithi203.stdout:3/70: dwrite d0/ff [0,4194304] 0 2024-04-05T03:02:32.777 INFO:tasks.workunit.client.1.smithi203.stdout:3/71: getdents d0/dc 0 2024-04-05T03:02:32.785 INFO:tasks.workunit.client.1.smithi203.stdout:3/72: dwrite d0/f4 [0,4194304] 0 2024-04-05T03:02:32.785 INFO:tasks.workunit.client.1.smithi203.stdout:3/73: stat d0/c8 0 2024-04-05T03:02:32.785 INFO:tasks.workunit.client.1.smithi203.stdout:3/74: stat d0/fb 0 2024-04-05T03:02:32.788 INFO:tasks.workunit.client.1.smithi203.stdout:3/75: unlink d0/fb 0 2024-04-05T03:02:32.794 INFO:tasks.workunit.client.1.smithi203.stdout:3/76: dwrite d0/ff [0,4194304] 0 2024-04-05T03:02:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:32 smithi203 ceph-mon[39911]: pgmap v252: 65 pgs: 65 active+clean; 257 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 11 MiB/s wr, 347 op/s 2024-04-05T03:02:32.798 INFO:tasks.workunit.client.1.smithi203.stdout:3/77: rename d0/c8 to d0/dc/c10 0 2024-04-05T03:02:32.820 INFO:tasks.workunit.client.1.smithi203.stdout:3/78: dwrite d0/fd [0,4194304] 0 2024-04-05T03:02:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:32 smithi153 ceph-mon[29381]: pgmap v252: 65 pgs: 65 active+clean; 257 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 11 MiB/s wr, 347 op/s 2024-04-05T03:02:32.822 INFO:tasks.workunit.client.1.smithi203.stdout:3/79: creat d0/dc/f11 x:0 0 0 2024-04-05T03:02:32.877 INFO:tasks.workunit.client.1.smithi203.stdout:6/30: sync 2024-04-05T03:02:32.877 INFO:tasks.workunit.client.1.smithi203.stdout:2/113: sync 2024-04-05T03:02:32.878 INFO:tasks.workunit.client.1.smithi203.stdout:6/31: dread - d0/f5 zero size 2024-04-05T03:02:32.878 INFO:tasks.workunit.client.1.smithi203.stdout:6/32: write d0/f5 [485594,78308] 0 2024-04-05T03:02:32.880 INFO:tasks.workunit.client.1.smithi203.stdout:2/114: rename d5/d1e to d5/d10/d23 0 2024-04-05T03:02:32.881 INFO:tasks.workunit.client.1.smithi203.stdout:6/33: read d0/f5 [36505,114257] 0 2024-04-05T03:02:32.882 INFO:tasks.workunit.client.1.smithi203.stdout:2/115: mkdir d5/d10/d1a/d24 0 2024-04-05T03:02:32.883 INFO:tasks.workunit.client.1.smithi203.stdout:6/34: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:32.884 INFO:tasks.workunit.client.1.smithi203.stdout:2/116: write d5/d10/d1a/f1c [836802,53181] 0 2024-04-05T03:02:32.885 INFO:tasks.workunit.client.1.smithi203.stdout:6/35: symlink d0/d6/l7 0 2024-04-05T03:02:32.890 INFO:tasks.workunit.client.1.smithi203.stdout:2/117: link d5/f9 d5/d10/d23/f25 0 2024-04-05T03:02:32.892 INFO:tasks.workunit.client.1.smithi203.stdout:6/36: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:32.895 INFO:tasks.workunit.client.1.smithi203.stdout:2/118: rename d5/ce to d5/d10/d23/c26 0 2024-04-05T03:02:32.896 INFO:tasks.workunit.client.1.smithi203.stdout:2/119: read d5/d10/d1a/f1c [848552,53830] 0 2024-04-05T03:02:32.896 INFO:tasks.workunit.client.1.smithi203.stdout:2/120: readlink l2 0 2024-04-05T03:02:32.897 INFO:tasks.workunit.client.1.smithi203.stdout:6/37: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:32.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/121: write d5/f8 [3457080,98995] 0 2024-04-05T03:02:32.899 INFO:tasks.workunit.client.1.smithi203.stdout:6/38: write d0/f5 [1513987,122797] 0 2024-04-05T03:02:32.903 INFO:tasks.workunit.client.1.smithi203.stdout:6/39: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:32.905 INFO:tasks.workunit.client.1.smithi203.stdout:2/122: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:32.906 INFO:tasks.workunit.client.1.smithi203.stdout:2/123: stat d5/d10 0 2024-04-05T03:02:32.906 INFO:tasks.workunit.client.1.smithi203.stdout:6/40: unlink d0/f5 0 2024-04-05T03:02:32.910 INFO:tasks.workunit.client.1.smithi203.stdout:2/124: link d5/d10/d23/f25 d5/d10/d1a/d24/f27 0 2024-04-05T03:02:32.912 INFO:tasks.workunit.client.1.smithi203.stdout:2/125: creat d5/d10/d1a/d24/f28 x:0 0 0 2024-04-05T03:02:32.928 INFO:tasks.workunit.client.1.smithi203.stdout:2/126: dwrite f3 [0,4194304] 0 2024-04-05T03:02:32.939 INFO:tasks.workunit.client.1.smithi203.stdout:2/127: dwrite d5/d10/d1a/f1b [0,4194304] 0 2024-04-05T03:02:32.942 INFO:tasks.workunit.client.1.smithi203.stdout:2/128: mkdir d5/d10/d23/d29 0 2024-04-05T03:02:32.946 INFO:tasks.workunit.client.1.smithi203.stdout:2/129: mknod d5/d10/d23/d29/c2a 0 2024-04-05T03:02:32.947 INFO:tasks.workunit.client.1.smithi203.stdout:2/130: dread - d5/d10/f1d zero size 2024-04-05T03:02:32.952 INFO:tasks.workunit.client.1.smithi203.stdout:2/131: getdents d5/d10/d23 0 2024-04-05T03:02:32.959 INFO:tasks.workunit.client.1.smithi203.stdout:2/132: dwrite d5/f9 [4194304,4194304] 0 2024-04-05T03:02:32.961 INFO:tasks.workunit.client.1.smithi203.stdout:2/133: creat d5/d10/d1a/d24/f2b x:0 0 0 2024-04-05T03:02:32.964 INFO:tasks.workunit.client.1.smithi203.stdout:2/134: creat d5/d10/d23/f2c x:0 0 0 2024-04-05T03:02:32.968 INFO:tasks.workunit.client.1.smithi203.stdout:2/135: dread d5/d10/d23/f25 [0,4194304] 0 2024-04-05T03:02:32.969 INFO:tasks.workunit.client.1.smithi203.stdout:2/136: stat d5/d10/f13 0 2024-04-05T03:02:32.971 INFO:tasks.workunit.client.1.smithi203.stdout:2/137: mkdir d5/d10/d23/d22/d2d 0 2024-04-05T03:02:32.978 INFO:tasks.workunit.client.1.smithi203.stdout:2/138: dwrite d5/d10/d23/f25 [4194304,4194304] 0 2024-04-05T03:02:32.981 INFO:tasks.workunit.client.1.smithi203.stdout:2/139: creat d5/d10/d23/d22/d2d/f2e x:0 0 0 2024-04-05T03:02:32.982 INFO:tasks.workunit.client.1.smithi203.stdout:2/140: chown d5/d10/d1a/f1b 5 1 2024-04-05T03:02:32.982 INFO:tasks.workunit.client.1.smithi203.stdout:2/141: readlink d5/l11 0 2024-04-05T03:02:32.983 INFO:tasks.workunit.client.1.smithi203.stdout:2/142: dread - d5/d10/d1a/d24/f28 zero size 2024-04-05T03:02:32.983 INFO:tasks.workunit.client.1.smithi203.stdout:2/143: chown d5/d10/d23/d22 3236754 1 2024-04-05T03:02:33.050 INFO:tasks.workunit.client.1.smithi203.stdout:9/70: sync 2024-04-05T03:02:33.053 INFO:tasks.workunit.client.1.smithi203.stdout:9/71: dread f2 [0,4194304] 0 2024-04-05T03:02:33.057 INFO:tasks.workunit.client.1.smithi203.stdout:9/72: dread f8 [0,4194304] 0 2024-04-05T03:02:33.060 INFO:tasks.workunit.client.1.smithi203.stdout:9/73: dread fd [0,4194304] 0 2024-04-05T03:02:33.062 INFO:tasks.workunit.client.1.smithi203.stdout:9/74: creat d13/f15 x:0 0 0 2024-04-05T03:02:33.070 INFO:tasks.workunit.client.1.smithi203.stdout:9/75: dwrite f12 [0,4194304] 0 2024-04-05T03:02:33.074 INFO:tasks.workunit.client.1.smithi203.stdout:9/76: unlink ff 0 2024-04-05T03:02:33.076 INFO:tasks.workunit.client.1.smithi203.stdout:9/77: dread fa [0,4194304] 0 2024-04-05T03:02:33.078 INFO:tasks.workunit.client.1.smithi203.stdout:9/78: write fa [572355,30089] 0 2024-04-05T03:02:33.080 INFO:tasks.workunit.client.1.smithi203.stdout:9/79: mknod d13/c16 0 2024-04-05T03:02:33.084 INFO:tasks.workunit.client.1.smithi203.stdout:9/80: link f8 d13/f17 0 2024-04-05T03:02:33.085 INFO:tasks.workunit.client.1.smithi203.stdout:9/81: symlink d13/l18 0 2024-04-05T03:02:33.086 INFO:tasks.workunit.client.1.smithi203.stdout:9/82: chown f7 59453676 1 2024-04-05T03:02:33.092 INFO:tasks.workunit.client.1.smithi203.stdout:9/83: dwrite f12 [0,4194304] 0 2024-04-05T03:02:33.095 INFO:tasks.workunit.client.1.smithi203.stdout:9/84: mkdir d13/d19 0 2024-04-05T03:02:33.100 INFO:tasks.workunit.client.1.smithi203.stdout:9/85: dwrite fa [0,4194304] 0 2024-04-05T03:02:33.101 INFO:tasks.workunit.client.1.smithi203.stdout:9/86: stat f7 0 2024-04-05T03:02:33.111 INFO:tasks.workunit.client.1.smithi203.stdout:9/87: truncate fd 858807 0 2024-04-05T03:02:33.113 INFO:tasks.workunit.client.1.smithi203.stdout:9/88: symlink d13/l1a 0 2024-04-05T03:02:33.116 INFO:tasks.workunit.client.1.smithi203.stdout:9/89: dread f7 [0,4194304] 0 2024-04-05T03:02:33.119 INFO:tasks.workunit.client.1.smithi203.stdout:9/90: chown fd 2598 1 2024-04-05T03:02:33.121 INFO:tasks.workunit.client.1.smithi203.stdout:9/91: fdatasync fd 0 2024-04-05T03:02:33.123 INFO:tasks.workunit.client.1.smithi203.stdout:9/92: mkdir d13/d1b 0 2024-04-05T03:02:33.123 INFO:tasks.workunit.client.1.smithi203.stdout:9/93: dread - d13/f15 zero size 2024-04-05T03:02:33.123 INFO:tasks.workunit.client.1.smithi203.stdout:9/94: readlink l1 0 2024-04-05T03:02:33.126 INFO:tasks.workunit.client.1.smithi203.stdout:9/95: rmdir d13 39 2024-04-05T03:02:33.131 INFO:tasks.workunit.client.1.smithi203.stdout:9/96: dwrite fd [0,4194304] 0 2024-04-05T03:02:33.135 INFO:tasks.workunit.client.1.smithi203.stdout:1/93: sync 2024-04-05T03:02:33.135 INFO:tasks.workunit.client.1.smithi203.stdout:4/69: sync 2024-04-05T03:02:33.135 INFO:tasks.workunit.client.1.smithi203.stdout:5/223: sync 2024-04-05T03:02:33.137 INFO:tasks.workunit.client.1.smithi203.stdout:9/97: dread d13/f17 [0,4194304] 0 2024-04-05T03:02:33.138 INFO:tasks.workunit.client.1.smithi203.stdout:1/94: stat d7/d14/c1a 0 2024-04-05T03:02:33.140 INFO:tasks.workunit.client.1.smithi203.stdout:1/95: write d7/d14/f16 [358378,13060] 0 2024-04-05T03:02:33.141 INFO:tasks.workunit.client.1.smithi203.stdout:9/98: getdents d13/d1b 0 2024-04-05T03:02:33.142 INFO:tasks.workunit.client.1.smithi203.stdout:4/70: dwrite f8 [0,4194304] 0 2024-04-05T03:02:33.143 INFO:tasks.workunit.client.1.smithi203.stdout:4/71: truncate f5 4985312 0 2024-04-05T03:02:33.144 INFO:tasks.workunit.client.1.smithi203.stdout:1/96: creat d7/d14/f21 x:0 0 0 2024-04-05T03:02:33.145 INFO:tasks.workunit.client.1.smithi203.stdout:4/72: write da/fc [5036693,8285] 0 2024-04-05T03:02:33.147 INFO:tasks.workunit.client.1.smithi203.stdout:1/97: mknod d7/c22 0 2024-04-05T03:02:33.147 INFO:tasks.workunit.client.1.smithi203.stdout:9/99: link c5 d13/c1c 0 2024-04-05T03:02:33.148 INFO:tasks.workunit.client.1.smithi203.stdout:4/73: write da/f15 [27517,87626] 0 2024-04-05T03:02:33.148 INFO:tasks.workunit.client.1.smithi203.stdout:9/100: unlink l1 0 2024-04-05T03:02:33.149 INFO:tasks.workunit.client.1.smithi203.stdout:4/74: write f8 [4325531,48615] 0 2024-04-05T03:02:33.150 INFO:tasks.workunit.client.1.smithi203.stdout:1/98: creat d7/f23 x:0 0 0 2024-04-05T03:02:33.153 INFO:tasks.workunit.client.1.smithi203.stdout:4/75: symlink da/l16 0 2024-04-05T03:02:33.154 INFO:tasks.workunit.client.1.smithi203.stdout:1/99: mkdir d7/d24 0 2024-04-05T03:02:33.157 INFO:tasks.workunit.client.1.smithi203.stdout:9/101: rmdir d13/d19 0 2024-04-05T03:02:33.157 INFO:tasks.workunit.client.1.smithi203.stdout:1/100: mknod d7/d14/c25 0 2024-04-05T03:02:33.158 INFO:tasks.workunit.client.1.smithi203.stdout:9/102: write f2 [2694216,62218] 0 2024-04-05T03:02:33.159 INFO:tasks.workunit.client.1.smithi203.stdout:1/101: truncate d7/d14/f21 696485 0 2024-04-05T03:02:33.159 INFO:tasks.workunit.client.1.smithi203.stdout:1/102: stat d7/cd 0 2024-04-05T03:02:33.163 INFO:tasks.workunit.client.1.smithi203.stdout:9/103: unlink l0 0 2024-04-05T03:02:33.166 INFO:tasks.workunit.client.1.smithi203.stdout:1/103: dwrite d7/fe [0,4194304] 0 2024-04-05T03:02:33.167 INFO:tasks.workunit.client.1.smithi203.stdout:4/76: truncate f8 886443 0 2024-04-05T03:02:33.169 INFO:tasks.workunit.client.1.smithi203.stdout:1/104: symlink d7/d14/l26 0 2024-04-05T03:02:33.173 INFO:tasks.workunit.client.1.smithi203.stdout:1/105: link d7/d14/f1e d7/f27 0 2024-04-05T03:02:33.174 INFO:tasks.workunit.client.1.smithi203.stdout:1/106: chown d7/cd 511 1 2024-04-05T03:02:33.176 INFO:tasks.workunit.client.1.smithi203.stdout:1/107: read d7/fa [2122234,1803] 0 2024-04-05T03:02:33.176 INFO:tasks.workunit.client.1.smithi203.stdout:4/77: dwrite da/f13 [0,4194304] 0 2024-04-05T03:02:33.180 INFO:tasks.workunit.client.1.smithi203.stdout:1/108: link d7/c22 d7/d24/c28 0 2024-04-05T03:02:33.181 INFO:tasks.workunit.client.1.smithi203.stdout:1/109: chown d7/l10 7 1 2024-04-05T03:02:33.182 INFO:tasks.workunit.client.1.smithi203.stdout:1/110: write d7/d14/f16 [682352,106125] 0 2024-04-05T03:02:33.183 INFO:tasks.workunit.client.1.smithi203.stdout:4/78: getdents da 0 2024-04-05T03:02:33.185 INFO:tasks.workunit.client.1.smithi203.stdout:4/79: mknod da/c17 0 2024-04-05T03:02:33.190 INFO:tasks.workunit.client.1.smithi203.stdout:4/80: dwrite f5 [0,4194304] 0 2024-04-05T03:02:33.192 INFO:tasks.workunit.client.1.smithi203.stdout:4/81: creat da/f18 x:0 0 0 2024-04-05T03:02:33.194 INFO:tasks.workunit.client.1.smithi203.stdout:4/82: mknod da/c19 0 2024-04-05T03:02:33.195 INFO:tasks.workunit.client.1.smithi203.stdout:4/83: rename da to da/d1a 22 2024-04-05T03:02:33.201 INFO:tasks.workunit.client.1.smithi203.stdout:4/84: dwrite da/f15 [0,4194304] 0 2024-04-05T03:02:33.204 INFO:tasks.workunit.client.1.smithi203.stdout:4/85: creat da/f1b x:0 0 0 2024-04-05T03:02:33.207 INFO:tasks.workunit.client.1.smithi203.stdout:4/86: rename da/c17 to da/c1c 0 2024-04-05T03:02:33.208 INFO:tasks.workunit.client.1.smithi203.stdout:4/87: chown da/f10 8 1 2024-04-05T03:02:33.234 INFO:tasks.workunit.client.1.smithi203.stdout:8/145: sync 2024-04-05T03:02:33.234 INFO:tasks.workunit.client.1.smithi203.stdout:7/87: sync 2024-04-05T03:02:33.234 INFO:tasks.workunit.client.1.smithi203.stdout:2/144: sync 2024-04-05T03:02:33.234 INFO:tasks.workunit.client.1.smithi203.stdout:3/80: sync 2024-04-05T03:02:33.234 INFO:tasks.workunit.client.1.smithi203.stdout:0/122: sync 2024-04-05T03:02:33.235 INFO:tasks.workunit.client.1.smithi203.stdout:6/41: sync 2024-04-05T03:02:33.236 INFO:tasks.workunit.client.1.smithi203.stdout:3/81: read - d0/dc/f11 zero size 2024-04-05T03:02:33.237 INFO:tasks.workunit.client.1.smithi203.stdout:2/145: mkdir d5/d10/d2f 0 2024-04-05T03:02:33.238 INFO:tasks.workunit.client.1.smithi203.stdout:6/42: creat d0/f8 x:0 0 0 2024-04-05T03:02:33.240 INFO:tasks.workunit.client.1.smithi203.stdout:8/146: unlink d1/f1c 0 2024-04-05T03:02:33.243 INFO:tasks.workunit.client.1.smithi203.stdout:2/146: mknod d5/d10/d23/c30 0 2024-04-05T03:02:33.246 INFO:tasks.workunit.client.1.smithi203.stdout:7/88: creat d8/d12/d16/d1a/f1f x:0 0 0 2024-04-05T03:02:33.246 INFO:tasks.workunit.client.1.smithi203.stdout:1/111: dread d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:33.246 INFO:tasks.workunit.client.1.smithi203.stdout:8/147: creat d1/d12/d1a/f20 x:0 0 0 2024-04-05T03:02:33.248 INFO:tasks.workunit.client.1.smithi203.stdout:4/88: truncate f8 1546409 0 2024-04-05T03:02:33.249 INFO:tasks.workunit.client.1.smithi203.stdout:3/82: dwrite d0/dc/f11 [0,4194304] 0 2024-04-05T03:02:33.249 INFO:tasks.workunit.client.1.smithi203.stdout:4/89: stat da/f1b 0 2024-04-05T03:02:33.251 INFO:tasks.workunit.client.1.smithi203.stdout:8/148: truncate d1/d12/d1a/f1d 4907114 0 2024-04-05T03:02:33.251 INFO:tasks.workunit.client.1.smithi203.stdout:0/123: getdents dd/d17 0 2024-04-05T03:02:33.251 INFO:tasks.workunit.client.1.smithi203.stdout:6/43: getdents d0/d6 0 2024-04-05T03:02:33.252 INFO:tasks.workunit.client.1.smithi203.stdout:0/124: stat dd/c10 0 2024-04-05T03:02:33.255 INFO:tasks.workunit.client.1.smithi203.stdout:4/90: symlink da/l1d 0 2024-04-05T03:02:33.257 INFO:tasks.workunit.client.1.smithi203.stdout:3/83: rmdir d0/dc 39 2024-04-05T03:02:33.260 INFO:tasks.workunit.client.1.smithi203.stdout:8/149: mknod d1/d12/d1a/c21 0 2024-04-05T03:02:33.261 INFO:tasks.workunit.client.1.smithi203.stdout:8/150: write d1/d12/d1a/f1d [1069698,81718] 0 2024-04-05T03:02:33.262 INFO:tasks.workunit.client.1.smithi203.stdout:6/44: creat d0/f9 x:0 0 0 2024-04-05T03:02:33.263 INFO:tasks.workunit.client.1.smithi203.stdout:8/151: truncate d1/d12/d1a/f1e 67692 0 2024-04-05T03:02:33.263 INFO:tasks.workunit.client.1.smithi203.stdout:6/45: chown d0/l2 244002 1 2024-04-05T03:02:33.263 INFO:tasks.workunit.client.1.smithi203.stdout:6/46: chown d0 115278 1 2024-04-05T03:02:33.264 INFO:tasks.workunit.client.1.smithi203.stdout:0/125: mkdir dd/d17/d1d/d22 0 2024-04-05T03:02:33.264 INFO:tasks.workunit.client.1.smithi203.stdout:0/126: fsync f6 0 2024-04-05T03:02:33.265 INFO:tasks.workunit.client.1.smithi203.stdout:2/147: link l2 d5/d10/d2f/l31 0 2024-04-05T03:02:33.265 INFO:tasks.workunit.client.1.smithi203.stdout:0/127: write dd/f20 [283449,125472] 0 2024-04-05T03:02:33.266 INFO:tasks.workunit.client.1.smithi203.stdout:0/128: dread - dd/f14 zero size 2024-04-05T03:02:33.266 INFO:tasks.workunit.client.1.smithi203.stdout:0/129: fsync f2 0 2024-04-05T03:02:33.267 INFO:tasks.workunit.client.1.smithi203.stdout:4/91: creat da/f1e x:0 0 0 2024-04-05T03:02:33.271 INFO:tasks.workunit.client.1.smithi203.stdout:8/152: unlink d1/d12/d1a/f20 0 2024-04-05T03:02:33.273 INFO:tasks.workunit.client.1.smithi203.stdout:6/47: mkdir d0/da 0 2024-04-05T03:02:33.273 INFO:tasks.workunit.client.1.smithi203.stdout:6/48: chown d0/l2 89 1 2024-04-05T03:02:33.273 INFO:tasks.workunit.client.1.smithi203.stdout:6/49: dread - d0/f8 zero size 2024-04-05T03:02:33.274 INFO:tasks.workunit.client.1.smithi203.stdout:6/50: write d0/f8 [160683,8741] 0 2024-04-05T03:02:33.275 INFO:tasks.workunit.client.1.smithi203.stdout:2/148: symlink d5/d10/d23/d22/d2d/l32 0 2024-04-05T03:02:33.277 INFO:tasks.workunit.client.1.smithi203.stdout:3/84: mknod d0/c12 0 2024-04-05T03:02:33.278 INFO:tasks.workunit.client.1.smithi203.stdout:4/92: symlink da/l1f 0 2024-04-05T03:02:33.278 INFO:tasks.workunit.client.1.smithi203.stdout:4/93: fsync da/f18 0 2024-04-05T03:02:33.282 INFO:tasks.workunit.client.1.smithi203.stdout:2/149: creat d5/d10/d23/d22/d2d/f33 x:0 0 0 2024-04-05T03:02:33.283 INFO:tasks.workunit.client.1.smithi203.stdout:2/150: fdatasync d5/fa 0 2024-04-05T03:02:33.284 INFO:tasks.workunit.client.1.smithi203.stdout:0/130: creat dd/d17/d1d/d22/f23 x:0 0 0 2024-04-05T03:02:33.285 INFO:tasks.workunit.client.1.smithi203.stdout:0/131: stat dd/d17/d1d 0 2024-04-05T03:02:33.285 INFO:tasks.workunit.client.1.smithi203.stdout:7/89: truncate d8/da/fd 4159138 0 2024-04-05T03:02:33.286 INFO:tasks.workunit.client.1.smithi203.stdout:0/132: dread - dd/d17/d1d/f1e zero size 2024-04-05T03:02:33.286 INFO:tasks.workunit.client.1.smithi203.stdout:4/94: symlink da/l20 0 2024-04-05T03:02:33.287 INFO:tasks.workunit.client.1.smithi203.stdout:8/153: creat d1/d6/d16/d1f/f22 x:0 0 0 2024-04-05T03:02:33.288 INFO:tasks.workunit.client.1.smithi203.stdout:6/51: mknod d0/da/cb 0 2024-04-05T03:02:33.292 INFO:tasks.workunit.client.1.smithi203.stdout:2/151: rmdir d5 39 2024-04-05T03:02:33.292 INFO:tasks.workunit.client.1.smithi203.stdout:3/85: mkdir d0/d13 0 2024-04-05T03:02:33.293 INFO:tasks.workunit.client.1.smithi203.stdout:3/86: read d0/ff [324335,118173] 0 2024-04-05T03:02:33.295 INFO:tasks.workunit.client.1.smithi203.stdout:4/95: creat da/f21 x:0 0 0 2024-04-05T03:02:33.297 INFO:tasks.workunit.client.1.smithi203.stdout:6/52: write d0/f3 [1055687,67694] 0 2024-04-05T03:02:33.299 INFO:tasks.workunit.client.1.smithi203.stdout:6/53: write d0/f9 [703196,118740] 0 2024-04-05T03:02:33.300 INFO:tasks.workunit.client.1.smithi203.stdout:2/152: fdatasync d5/d10/d1a/f1b 0 2024-04-05T03:02:33.301 INFO:tasks.workunit.client.1.smithi203.stdout:2/153: chown d5/d10/f1d 30858 1 2024-04-05T03:02:33.307 INFO:tasks.workunit.client.1.smithi203.stdout:6/54: symlink d0/d6/lc 0 2024-04-05T03:02:33.308 INFO:tasks.workunit.client.1.smithi203.stdout:7/90: dwrite d8/f17 [0,4194304] 0 2024-04-05T03:02:33.309 INFO:tasks.workunit.client.1.smithi203.stdout:2/154: rmdir d5/d10/d23/d29 39 2024-04-05T03:02:33.311 INFO:tasks.workunit.client.1.smithi203.stdout:0/133: rmdir dd/d21 0 2024-04-05T03:02:33.313 INFO:tasks.workunit.client.1.smithi203.stdout:4/96: rmdir da 39 2024-04-05T03:02:33.317 INFO:tasks.workunit.client.1.smithi203.stdout:6/55: creat d0/da/fd x:0 0 0 2024-04-05T03:02:33.318 INFO:tasks.workunit.client.1.smithi203.stdout:3/87: rmdir d0/d13 0 2024-04-05T03:02:33.320 INFO:tasks.workunit.client.1.smithi203.stdout:8/154: getdents d1 0 2024-04-05T03:02:33.320 INFO:tasks.workunit.client.1.smithi203.stdout:6/56: chown d0/f8 832 1 2024-04-05T03:02:33.320 INFO:tasks.workunit.client.1.smithi203.stdout:2/155: dwrite d5/d10/d1a/f1b [0,4194304] 0 2024-04-05T03:02:33.322 INFO:tasks.workunit.client.1.smithi203.stdout:0/134: dwrite f5 [0,4194304] 0 2024-04-05T03:02:33.324 INFO:tasks.workunit.client.1.smithi203.stdout:6/57: write d0/f8 [453278,62880] 0 2024-04-05T03:02:33.325 INFO:tasks.workunit.client.1.smithi203.stdout:0/135: chown dd/f16 73776 1 2024-04-05T03:02:33.325 INFO:tasks.workunit.client.1.smithi203.stdout:8/155: read d1/d12/d17/d10/f15 [508885,1876] 0 2024-04-05T03:02:33.328 INFO:tasks.workunit.client.1.smithi203.stdout:7/91: creat d8/f20 x:0 0 0 2024-04-05T03:02:33.329 INFO:tasks.workunit.client.1.smithi203.stdout:3/88: creat d0/dc/f14 x:0 0 0 2024-04-05T03:02:33.329 INFO:tasks.workunit.client.1.smithi203.stdout:0/136: dread - dd/d17/d1d/d22/f23 zero size 2024-04-05T03:02:33.329 INFO:tasks.workunit.client.1.smithi203.stdout:4/97: dread da/f11 [0,4194304] 0 2024-04-05T03:02:33.330 INFO:tasks.workunit.client.1.smithi203.stdout:7/92: fsync d8/da/f14 0 2024-04-05T03:02:33.333 INFO:tasks.workunit.client.1.smithi203.stdout:6/58: creat d0/d6/fe x:0 0 0 2024-04-05T03:02:33.334 INFO:tasks.workunit.client.1.smithi203.stdout:8/156: creat d1/d6/d16/d1f/f23 x:0 0 0 2024-04-05T03:02:33.336 INFO:tasks.workunit.client.1.smithi203.stdout:2/156: dwrite d5/d10/d23/d22/d2d/f33 [0,4194304] 0 2024-04-05T03:02:33.337 INFO:tasks.workunit.client.1.smithi203.stdout:4/98: rename da/f15 to da/f22 0 2024-04-05T03:02:33.337 INFO:tasks.workunit.client.1.smithi203.stdout:4/99: dread - da/f21 zero size 2024-04-05T03:02:33.340 INFO:tasks.workunit.client.1.smithi203.stdout:4/100: readlink da/l1f 0 2024-04-05T03:02:33.340 INFO:tasks.workunit.client.1.smithi203.stdout:7/93: mkdir d8/d12/d16/d1a/d21 0 2024-04-05T03:02:33.341 INFO:tasks.workunit.client.1.smithi203.stdout:6/59: creat d0/ff x:0 0 0 2024-04-05T03:02:33.342 INFO:tasks.workunit.client.1.smithi203.stdout:7/94: chown d8 3 1 2024-04-05T03:02:33.343 INFO:tasks.workunit.client.1.smithi203.stdout:0/137: mknod dd/c24 0 2024-04-05T03:02:33.343 INFO:tasks.workunit.client.1.smithi203.stdout:0/138: stat dd 0 2024-04-05T03:02:33.344 INFO:tasks.workunit.client.1.smithi203.stdout:4/101: chown da/f22 193149 1 2024-04-05T03:02:33.345 INFO:tasks.workunit.client.1.smithi203.stdout:6/60: creat d0/da/f10 x:0 0 0 2024-04-05T03:02:33.345 INFO:tasks.workunit.client.1.smithi203.stdout:4/102: stat f5 0 2024-04-05T03:02:33.350 INFO:tasks.workunit.client.1.smithi203.stdout:6/61: mkdir d0/da/d11 0 2024-04-05T03:02:33.352 INFO:tasks.workunit.client.1.smithi203.stdout:4/103: dwrite da/f1b [0,4194304] 0 2024-04-05T03:02:33.353 INFO:tasks.workunit.client.1.smithi203.stdout:0/139: unlink dd/l11 0 2024-04-05T03:02:33.354 INFO:tasks.workunit.client.1.smithi203.stdout:0/140: write dd/f15 [1053569,124819] 0 2024-04-05T03:02:33.355 INFO:tasks.workunit.client.1.smithi203.stdout:2/157: getdents d5/d10/d23/d22 0 2024-04-05T03:02:33.355 INFO:tasks.workunit.client.1.smithi203.stdout:0/141: stat f7 0 2024-04-05T03:02:33.357 INFO:tasks.workunit.client.1.smithi203.stdout:0/142: write f8 [1669596,50179] 0 2024-04-05T03:02:33.357 INFO:tasks.workunit.client.1.smithi203.stdout:6/62: symlink d0/da/l12 0 2024-04-05T03:02:33.358 INFO:tasks.workunit.client.1.smithi203.stdout:6/63: chown d0/f9 0 1 2024-04-05T03:02:33.360 INFO:tasks.workunit.client.1.smithi203.stdout:7/95: getdents d8/d12 0 2024-04-05T03:02:33.365 INFO:tasks.workunit.client.1.smithi203.stdout:0/143: unlink dd/le 0 2024-04-05T03:02:33.367 INFO:tasks.workunit.client.1.smithi203.stdout:2/158: rename d5/d10/d1a/f1c to d5/d10/d1a/f34 0 2024-04-05T03:02:33.368 INFO:tasks.workunit.client.1.smithi203.stdout:2/159: write f3 [8596044,18996] 0 2024-04-05T03:02:33.369 INFO:tasks.workunit.client.1.smithi203.stdout:6/64: creat d0/da/f13 x:0 0 0 2024-04-05T03:02:33.370 INFO:tasks.workunit.client.1.smithi203.stdout:3/89: truncate d0/f5 2571863 0 2024-04-05T03:02:33.370 INFO:tasks.workunit.client.1.smithi203.stdout:3/90: chown d0/ca 410 1 2024-04-05T03:02:33.372 INFO:tasks.workunit.client.1.smithi203.stdout:0/144: creat dd/d17/d1d/d22/f25 x:0 0 0 2024-04-05T03:02:33.373 INFO:tasks.workunit.client.1.smithi203.stdout:8/157: truncate d1/d12/d1a/f1d 547451 0 2024-04-05T03:02:33.373 INFO:tasks.workunit.client.1.smithi203.stdout:8/158: readlink d1/ld 0 2024-04-05T03:02:33.374 INFO:tasks.workunit.client.1.smithi203.stdout:6/65: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:33.374 INFO:tasks.workunit.client.1.smithi203.stdout:5/224: sync 2024-04-05T03:02:33.378 INFO:tasks.workunit.client.1.smithi203.stdout:2/160: rename d5/d10/d23/d22 to d5/d10/d35 0 2024-04-05T03:02:33.380 INFO:tasks.workunit.client.1.smithi203.stdout:5/225: dread d3/d4/d8/f3a [0,4194304] 0 2024-04-05T03:02:33.380 INFO:tasks.workunit.client.1.smithi203.stdout:8/159: dwrite d1/d6/d16/d1f/f22 [0,4194304] 0 2024-04-05T03:02:33.381 INFO:tasks.workunit.client.1.smithi203.stdout:7/96: creat d8/d12/d16/d1a/d21/f22 x:0 0 0 2024-04-05T03:02:33.385 INFO:tasks.workunit.client.1.smithi203.stdout:3/91: fdatasync d0/ff 0 2024-04-05T03:02:33.385 INFO:tasks.workunit.client.1.smithi203.stdout:0/145: mkdir dd/d17/d1d/d26 0 2024-04-05T03:02:33.386 INFO:tasks.workunit.client.1.smithi203.stdout:6/66: symlink d0/d6/l14 0 2024-04-05T03:02:33.394 INFO:tasks.workunit.client.1.smithi203.stdout:6/67: mknod d0/da/c15 0 2024-04-05T03:02:33.396 INFO:tasks.workunit.client.1.smithi203.stdout:3/92: mknod d0/c15 0 2024-04-05T03:02:33.397 INFO:tasks.workunit.client.1.smithi203.stdout:3/93: truncate d0/dc/f14 184187 0 2024-04-05T03:02:33.398 INFO:tasks.workunit.client.1.smithi203.stdout:6/68: rmdir d0/da/d11 0 2024-04-05T03:02:33.399 INFO:tasks.workunit.client.1.smithi203.stdout:6/69: mknod d0/c16 0 2024-04-05T03:02:33.400 INFO:tasks.workunit.client.1.smithi203.stdout:6/70: truncate d0/d6/fe 246193 0 2024-04-05T03:02:33.401 INFO:tasks.workunit.client.1.smithi203.stdout:6/71: write d0/da/f13 [446852,79420] 0 2024-04-05T03:02:33.401 INFO:tasks.workunit.client.1.smithi203.stdout:6/72: truncate d0/f8 1498836 0 2024-04-05T03:02:33.402 INFO:tasks.workunit.client.1.smithi203.stdout:6/73: chown d0/l2 32427 1 2024-04-05T03:02:33.403 INFO:tasks.workunit.client.1.smithi203.stdout:6/74: mknod d0/da/c17 0 2024-04-05T03:02:33.405 INFO:tasks.workunit.client.1.smithi203.stdout:6/75: mkdir d0/da/d18 0 2024-04-05T03:02:33.405 INFO:tasks.workunit.client.1.smithi203.stdout:6/76: readlink d0/d6/lc 0 2024-04-05T03:02:33.487 INFO:tasks.workunit.client.1.smithi203.stdout:6/77: dread d0/d6/fe [0,4194304] 0 2024-04-05T03:02:33.488 INFO:tasks.workunit.client.1.smithi203.stdout:6/78: write d0/da/f13 [478596,119349] 0 2024-04-05T03:02:33.490 INFO:tasks.workunit.client.1.smithi203.stdout:6/79: truncate d0/ff 568654 0 2024-04-05T03:02:33.497 INFO:tasks.workunit.client.1.smithi203.stdout:5/226: dread d3/d1f/f1e [0,4194304] 0 2024-04-05T03:02:33.497 INFO:tasks.workunit.client.1.smithi203.stdout:8/160: dread d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:33.497 INFO:tasks.workunit.client.1.smithi203.stdout:6/80: dwrite d0/f8 [0,4194304] 0 2024-04-05T03:02:33.500 INFO:tasks.workunit.client.1.smithi203.stdout:3/94: truncate d0/ff 2643946 0 2024-04-05T03:02:33.501 INFO:tasks.workunit.client.1.smithi203.stdout:0/146: fdatasync dd/d17/d1d/d22/f23 0 2024-04-05T03:02:33.502 INFO:tasks.workunit.client.1.smithi203.stdout:8/161: unlink d1/d12/d17/ca 0 2024-04-05T03:02:33.502 INFO:tasks.workunit.client.1.smithi203.stdout:7/97: truncate d8/d12/f18 634727 0 2024-04-05T03:02:33.507 INFO:tasks.workunit.client.1.smithi203.stdout:6/81: dread d0/f8 [0,4194304] 0 2024-04-05T03:02:33.509 INFO:tasks.workunit.client.1.smithi203.stdout:6/82: chown d0/d6/l7 192 1 2024-04-05T03:02:33.509 INFO:tasks.workunit.client.1.smithi203.stdout:3/95: dwrite d0/dc/f11 [0,4194304] 0 2024-04-05T03:02:33.513 INFO:tasks.workunit.client.1.smithi203.stdout:8/162: write d1/d12/d1a/f1d [320104,30757] 0 2024-04-05T03:02:33.513 INFO:tasks.workunit.client.1.smithi203.stdout:7/98: creat d8/d12/d16/d1a/d21/f23 x:0 0 0 2024-04-05T03:02:33.514 INFO:tasks.workunit.client.1.smithi203.stdout:5/227: rmdir d3/d4/d50 0 2024-04-05T03:02:33.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/83: rename d0/d6/fe to d0/f19 0 2024-04-05T03:02:33.519 INFO:tasks.workunit.client.1.smithi203.stdout:9/104: sync 2024-04-05T03:02:33.519 INFO:tasks.workunit.client.1.smithi203.stdout:5/228: dread d3/d1f/f3b [0,4194304] 0 2024-04-05T03:02:33.525 INFO:tasks.workunit.client.1.smithi203.stdout:8/163: unlink d1/f3 0 2024-04-05T03:02:33.526 INFO:tasks.workunit.client.1.smithi203.stdout:5/229: dwrite d3/d4/d8/d12/f4e [0,4194304] 0 2024-04-05T03:02:33.528 INFO:tasks.workunit.client.1.smithi203.stdout:9/105: rename d13/f15 to d13/f1d 0 2024-04-05T03:02:33.529 INFO:tasks.workunit.client.1.smithi203.stdout:9/106: stat d13/l18 0 2024-04-05T03:02:33.530 INFO:tasks.workunit.client.1.smithi203.stdout:0/147: creat dd/f27 x:0 0 0 2024-04-05T03:02:33.530 INFO:tasks.workunit.client.1.smithi203.stdout:0/148: stat f6 0 2024-04-05T03:02:33.532 INFO:tasks.workunit.client.1.smithi203.stdout:3/96: link d0/dc/f14 d0/dc/f16 0 2024-04-05T03:02:33.533 INFO:tasks.workunit.client.1.smithi203.stdout:7/99: write d8/ff [2769332,119232] 0 2024-04-05T03:02:33.535 INFO:tasks.workunit.client.1.smithi203.stdout:7/100: readlink l7 0 2024-04-05T03:02:33.537 INFO:tasks.workunit.client.1.smithi203.stdout:6/84: write d0/f3 [1936963,80504] 0 2024-04-05T03:02:33.538 INFO:tasks.workunit.client.1.smithi203.stdout:9/107: write d13/f17 [4682872,9580] 0 2024-04-05T03:02:33.539 INFO:tasks.workunit.client.1.smithi203.stdout:6/85: dread - d0/da/fd zero size 2024-04-05T03:02:33.541 INFO:tasks.workunit.client.1.smithi203.stdout:3/97: dwrite d0/dc/f11 [0,4194304] 0 2024-04-05T03:02:33.543 INFO:tasks.workunit.client.1.smithi203.stdout:6/86: dread d0/f19 [0,4194304] 0 2024-04-05T03:02:33.544 INFO:tasks.workunit.client.1.smithi203.stdout:8/164: creat d1/d12/d17/d10/f24 x:0 0 0 2024-04-05T03:02:33.547 INFO:tasks.workunit.client.1.smithi203.stdout:0/149: readlink dd/d17/l1a 0 2024-04-05T03:02:33.547 INFO:tasks.workunit.client.1.smithi203.stdout:9/108: dread f8 [0,4194304] 0 2024-04-05T03:02:33.548 INFO:tasks.workunit.client.1.smithi203.stdout:8/165: readlink d1/l2 0 2024-04-05T03:02:33.549 INFO:tasks.workunit.client.1.smithi203.stdout:3/98: symlink d0/l17 0 2024-04-05T03:02:33.550 INFO:tasks.workunit.client.1.smithi203.stdout:7/101: getdents d8/da 0 2024-04-05T03:02:33.550 INFO:tasks.workunit.client.1.smithi203.stdout:0/150: truncate dd/d17/d1d/f1e 564592 0 2024-04-05T03:02:33.550 INFO:tasks.workunit.client.1.smithi203.stdout:6/87: link d0/ff d0/f1a 0 2024-04-05T03:02:33.551 INFO:tasks.workunit.client.1.smithi203.stdout:7/102: fdatasync d8/d12/f1e 0 2024-04-05T03:02:33.553 INFO:tasks.workunit.client.1.smithi203.stdout:8/166: dread - d1/d6/d16/d1f/f23 zero size 2024-04-05T03:02:33.553 INFO:tasks.workunit.client.1.smithi203.stdout:0/151: dread - dd/f14 zero size 2024-04-05T03:02:33.553 INFO:tasks.workunit.client.1.smithi203.stdout:6/88: dread d0/f19 [0,4194304] 0 2024-04-05T03:02:33.555 INFO:tasks.workunit.client.1.smithi203.stdout:3/99: mknod d0/c18 0 2024-04-05T03:02:33.556 INFO:tasks.workunit.client.1.smithi203.stdout:7/103: write d8/d12/d16/d1a/d21/f22 [196027,113219] 0 2024-04-05T03:02:33.557 INFO:tasks.workunit.client.1.smithi203.stdout:7/104: write d8/f20 [262079,35252] 0 2024-04-05T03:02:33.558 INFO:tasks.workunit.client.1.smithi203.stdout:0/152: read - dd/d17/d1d/d22/f25 zero size 2024-04-05T03:02:33.563 INFO:tasks.workunit.client.1.smithi203.stdout:0/153: unlink dd/c1b 0 2024-04-05T03:02:33.564 INFO:tasks.workunit.client.1.smithi203.stdout:9/109: dwrite f8 [0,4194304] 0 2024-04-05T03:02:33.564 INFO:tasks.workunit.client.1.smithi203.stdout:7/105: write d8/d12/f1e [315425,130721] 0 2024-04-05T03:02:33.567 INFO:tasks.workunit.client.1.smithi203.stdout:7/106: fdatasync d8/d12/d16/d1a/f1f 0 2024-04-05T03:02:33.569 INFO:tasks.workunit.client.1.smithi203.stdout:9/110: symlink d13/l1e 0 2024-04-05T03:02:33.570 INFO:tasks.workunit.client.1.smithi203.stdout:3/100: truncate d0/f9 854032 0 2024-04-05T03:02:33.571 INFO:tasks.workunit.client.1.smithi203.stdout:8/167: dwrite d1/d6/d16/d1f/f22 [0,4194304] 0 2024-04-05T03:02:33.571 INFO:tasks.workunit.client.1.smithi203.stdout:0/154: mkdir dd/d17/d1d/d26/d28 0 2024-04-05T03:02:33.573 INFO:tasks.workunit.client.1.smithi203.stdout:6/89: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:33.573 INFO:tasks.workunit.client.1.smithi203.stdout:3/101: truncate d0/f4 4615950 0 2024-04-05T03:02:33.574 INFO:tasks.workunit.client.1.smithi203.stdout:8/168: chown d1/d6/d16/l1b 51444965 1 2024-04-05T03:02:33.577 INFO:tasks.workunit.client.1.smithi203.stdout:7/107: rename d8/d12/f1e to d8/d12/d16/d1a/f24 0 2024-04-05T03:02:33.577 INFO:tasks.workunit.client.1.smithi203.stdout:8/169: chown d1/d12/d17 2 1 2024-04-05T03:02:33.580 INFO:tasks.workunit.client.1.smithi203.stdout:8/170: write d1/d12/d1a/f1d [1035138,51610] 0 2024-04-05T03:02:33.581 INFO:tasks.workunit.client.1.smithi203.stdout:7/108: fdatasync d8/d12/d16/d1a/f1f 0 2024-04-05T03:02:33.582 INFO:tasks.workunit.client.1.smithi203.stdout:0/155: unlink dd/f20 0 2024-04-05T03:02:33.585 INFO:tasks.workunit.client.1.smithi203.stdout:6/90: dread d0/f8 [0,4194304] 0 2024-04-05T03:02:33.586 INFO:tasks.workunit.client.1.smithi203.stdout:6/91: read - d0/da/f10 zero size 2024-04-05T03:02:33.593 INFO:tasks.workunit.client.1.smithi203.stdout:0/156: dwrite dd/d17/d1d/d22/f23 [0,4194304] 0 2024-04-05T03:02:33.593 INFO:tasks.workunit.client.1.smithi203.stdout:0/157: dread - dd/f27 zero size 2024-04-05T03:02:33.595 INFO:tasks.workunit.client.1.smithi203.stdout:6/92: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:33.596 INFO:tasks.workunit.client.1.smithi203.stdout:0/158: dread dd/d17/f18 [0,4194304] 0 2024-04-05T03:02:33.599 INFO:tasks.workunit.client.1.smithi203.stdout:0/159: mkdir dd/d17/d29 0 2024-04-05T03:02:33.600 INFO:tasks.workunit.client.1.smithi203.stdout:6/93: dread d0/f8 [0,4194304] 0 2024-04-05T03:02:33.603 INFO:tasks.workunit.client.1.smithi203.stdout:0/160: unlink dd/f12 0 2024-04-05T03:02:33.607 INFO:tasks.workunit.client.1.smithi203.stdout:0/161: unlink cc 0 2024-04-05T03:02:33.608 INFO:tasks.workunit.client.1.smithi203.stdout:6/94: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:33.611 INFO:tasks.workunit.client.1.smithi203.stdout:3/102: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:33.612 INFO:tasks.workunit.client.1.smithi203.stdout:9/111: fdatasync f12 0 2024-04-05T03:02:33.617 INFO:tasks.workunit.client.1.smithi203.stdout:3/103: rename d0 to d0/d19 22 2024-04-05T03:02:33.620 INFO:tasks.workunit.client.1.smithi203.stdout:0/162: link ca dd/d17/d1d/c2a 0 2024-04-05T03:02:33.622 INFO:tasks.workunit.client.1.smithi203.stdout:3/104: mkdir d0/dc/d1a 0 2024-04-05T03:02:33.622 INFO:tasks.workunit.client.1.smithi203.stdout:6/95: link d0/d6/lc d0/da/d18/l1b 0 2024-04-05T03:02:33.625 INFO:tasks.workunit.client.1.smithi203.stdout:9/112: rename d13/c1c to d13/d1b/c1f 0 2024-04-05T03:02:33.626 INFO:tasks.workunit.client.1.smithi203.stdout:0/163: getdents dd/d17/d29 0 2024-04-05T03:02:33.627 INFO:tasks.workunit.client.1.smithi203.stdout:9/113: creat d13/f20 x:0 0 0 2024-04-05T03:02:33.628 INFO:tasks.workunit.client.1.smithi203.stdout:3/105: symlink d0/l1b 0 2024-04-05T03:02:33.628 INFO:tasks.workunit.client.1.smithi203.stdout:6/96: symlink d0/da/d18/l1c 0 2024-04-05T03:02:33.630 INFO:tasks.workunit.client.1.smithi203.stdout:9/114: creat d13/d1b/f21 x:0 0 0 2024-04-05T03:02:33.630 INFO:tasks.workunit.client.1.smithi203.stdout:0/164: creat dd/d17/d1d/d26/d28/f2b x:0 0 0 2024-04-05T03:02:33.631 INFO:tasks.workunit.client.1.smithi203.stdout:3/106: symlink d0/dc/l1c 0 2024-04-05T03:02:33.631 INFO:tasks.workunit.client.1.smithi203.stdout:9/115: write f2 [5202436,33402] 0 2024-04-05T03:02:33.635 INFO:tasks.workunit.client.1.smithi203.stdout:9/116: symlink d13/d1b/l22 0 2024-04-05T03:02:33.635 INFO:tasks.workunit.client.1.smithi203.stdout:1/112: sync 2024-04-05T03:02:33.637 INFO:tasks.workunit.client.1.smithi203.stdout:9/117: chown d13/l1a 15 1 2024-04-05T03:02:33.637 INFO:tasks.workunit.client.1.smithi203.stdout:0/165: link l4 dd/d17/d29/l2c 0 2024-04-05T03:02:33.637 INFO:tasks.workunit.client.1.smithi203.stdout:1/113: write d7/d14/f1b [1643317,60172] 0 2024-04-05T03:02:33.640 INFO:tasks.workunit.client.1.smithi203.stdout:9/118: mknod d13/d1b/c23 0 2024-04-05T03:02:33.640 INFO:tasks.workunit.client.1.smithi203.stdout:9/119: write f12 [501673,126631] 0 2024-04-05T03:02:33.641 INFO:tasks.workunit.client.1.smithi203.stdout:1/114: creat d7/d14/f29 x:0 0 0 2024-04-05T03:02:33.641 INFO:tasks.workunit.client.1.smithi203.stdout:9/120: chown le 61674 1 2024-04-05T03:02:33.642 INFO:tasks.workunit.client.1.smithi203.stdout:9/121: truncate fd 4380611 0 2024-04-05T03:02:33.643 INFO:tasks.workunit.client.1.smithi203.stdout:0/166: rename dd/d17/f18 to dd/d17/d1d/d26/d28/f2d 0 2024-04-05T03:02:33.643 INFO:tasks.workunit.client.1.smithi203.stdout:1/115: creat d7/f2a x:0 0 0 2024-04-05T03:02:33.644 INFO:tasks.workunit.client.1.smithi203.stdout:9/122: creat d13/f24 x:0 0 0 2024-04-05T03:02:33.645 INFO:tasks.workunit.client.1.smithi203.stdout:9/123: readlink d13/l1a 0 2024-04-05T03:02:33.647 INFO:tasks.workunit.client.1.smithi203.stdout:1/116: mkdir d7/d14/d2b 0 2024-04-05T03:02:33.647 INFO:tasks.workunit.client.1.smithi203.stdout:1/117: dread - d7/d14/f29 zero size 2024-04-05T03:02:33.648 INFO:tasks.workunit.client.1.smithi203.stdout:9/124: creat d13/f25 x:0 0 0 2024-04-05T03:02:33.648 INFO:tasks.workunit.client.1.smithi203.stdout:1/118: chown d7/d14/f16 0 1 2024-04-05T03:02:33.649 INFO:tasks.workunit.client.1.smithi203.stdout:6/97: write d0/ff [1614588,26552] 0 2024-04-05T03:02:33.649 INFO:tasks.workunit.client.1.smithi203.stdout:9/125: fsync f7 0 2024-04-05T03:02:33.652 INFO:tasks.workunit.client.1.smithi203.stdout:6/98: readlink d0/d6/lc 0 2024-04-05T03:02:33.652 INFO:tasks.workunit.client.1.smithi203.stdout:1/119: truncate d7/fa 3188822 0 2024-04-05T03:02:33.653 INFO:tasks.workunit.client.1.smithi203.stdout:0/167: rmdir dd/d17/d1d/d26/d28 39 2024-04-05T03:02:33.656 INFO:tasks.workunit.client.1.smithi203.stdout:0/168: dread - dd/f27 zero size 2024-04-05T03:02:33.656 INFO:tasks.workunit.client.1.smithi203.stdout:3/107: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:33.656 INFO:tasks.workunit.client.1.smithi203.stdout:6/99: write d0/da/f10 [770247,118405] 0 2024-04-05T03:02:33.661 INFO:tasks.workunit.client.1.smithi203.stdout:3/108: chown d0/ff 0 1 2024-04-05T03:02:33.661 INFO:tasks.workunit.client.1.smithi203.stdout:9/126: dwrite f2 [0,4194304] 0 2024-04-05T03:02:33.661 INFO:tasks.workunit.client.1.smithi203.stdout:6/100: creat d0/da/d18/f1d x:0 0 0 2024-04-05T03:02:33.662 INFO:tasks.workunit.client.1.smithi203.stdout:3/109: chown d0/c12 13 1 2024-04-05T03:02:33.663 INFO:tasks.workunit.client.1.smithi203.stdout:0/169: truncate dd/d17/d1d/d26/d28/f2b 679551 0 2024-04-05T03:02:33.666 INFO:tasks.workunit.client.1.smithi203.stdout:3/110: chown d0/dc/c10 84986 1 2024-04-05T03:02:33.670 INFO:tasks.workunit.client.1.smithi203.stdout:3/111: mkdir d0/dc/d1a/d1d 0 2024-04-05T03:02:33.671 INFO:tasks.workunit.client.1.smithi203.stdout:9/127: dwrite d13/d1b/f21 [0,4194304] 0 2024-04-05T03:02:33.674 INFO:tasks.workunit.client.1.smithi203.stdout:6/101: truncate d0/ff 206973 0 2024-04-05T03:02:33.675 INFO:tasks.workunit.client.1.smithi203.stdout:9/128: mkdir d13/d1b/d26 0 2024-04-05T03:02:33.675 INFO:tasks.workunit.client.1.smithi203.stdout:0/170: link dd/d17/d1d/d26/d28/f2d dd/d17/d1d/f2e 0 2024-04-05T03:02:33.675 INFO:tasks.workunit.client.1.smithi203.stdout:6/102: fsync d0/da/d18/f1d 0 2024-04-05T03:02:33.676 INFO:tasks.workunit.client.1.smithi203.stdout:0/171: write f7 [2409787,66642] 0 2024-04-05T03:02:33.678 INFO:tasks.workunit.client.1.smithi203.stdout:9/129: truncate d13/f1d 92249 0 2024-04-05T03:02:33.678 INFO:tasks.workunit.client.1.smithi203.stdout:6/103: rename d0/l1 to d0/d6/l1e 0 2024-04-05T03:02:33.680 INFO:tasks.workunit.client.1.smithi203.stdout:9/130: creat d13/f27 x:0 0 0 2024-04-05T03:02:33.680 INFO:tasks.workunit.client.1.smithi203.stdout:0/172: fsync dd/f16 0 2024-04-05T03:02:33.683 INFO:tasks.workunit.client.1.smithi203.stdout:0/173: unlink dd/d17/d1d/d22/f25 0 2024-04-05T03:02:33.686 INFO:tasks.workunit.client.1.smithi203.stdout:0/174: getdents dd/d17/d1d/d26 0 2024-04-05T03:02:33.689 INFO:tasks.workunit.client.1.smithi203.stdout:0/175: getdents dd 0 2024-04-05T03:02:33.690 INFO:tasks.workunit.client.1.smithi203.stdout:0/176: write dd/d17/d1d/f1e [938890,32975] 0 2024-04-05T03:02:33.690 INFO:tasks.workunit.client.1.smithi203.stdout:0/177: write f2 [4677925,41927] 0 2024-04-05T03:02:33.692 INFO:tasks.workunit.client.1.smithi203.stdout:0/178: mknod dd/c2f 0 2024-04-05T03:02:33.699 INFO:tasks.workunit.client.1.smithi203.stdout:9/131: getdents d13 0 2024-04-05T03:02:33.700 INFO:tasks.workunit.client.1.smithi203.stdout:9/132: write fa [1406213,70918] 0 2024-04-05T03:02:33.701 INFO:tasks.workunit.client.1.smithi203.stdout:6/104: unlink d0/d6/l1e 0 2024-04-05T03:02:33.704 INFO:tasks.workunit.client.1.smithi203.stdout:6/105: mknod d0/da/d18/c1f 0 2024-04-05T03:02:33.710 INFO:tasks.workunit.client.1.smithi203.stdout:9/133: truncate fa 3613171 0 2024-04-05T03:02:33.710 INFO:tasks.workunit.client.1.smithi203.stdout:6/106: dwrite d0/da/f10 [0,4194304] 0 2024-04-05T03:02:33.714 INFO:tasks.workunit.client.1.smithi203.stdout:6/107: symlink d0/d6/l20 0 2024-04-05T03:02:33.714 INFO:tasks.workunit.client.1.smithi203.stdout:6/108: chown d0 1834270938 1 2024-04-05T03:02:33.714 INFO:tasks.workunit.client.1.smithi203.stdout:9/134: symlink d13/d1b/d26/l28 0 2024-04-05T03:02:33.717 INFO:tasks.workunit.client.1.smithi203.stdout:9/135: rename l6 to d13/d1b/l29 0 2024-04-05T03:02:33.721 INFO:tasks.workunit.client.1.smithi203.stdout:9/136: rename d13/f17 to d13/f2a 0 2024-04-05T03:02:33.722 INFO:tasks.workunit.client.1.smithi203.stdout:9/137: dread - d13/f25 zero size 2024-04-05T03:02:33.734 INFO:tasks.workunit.client.1.smithi203.stdout:9/138: dwrite fa [0,4194304] 0 2024-04-05T03:02:33.735 INFO:tasks.workunit.client.1.smithi203.stdout:9/139: chown d13/d1b/f21 99960 1 2024-04-05T03:02:33.735 INFO:tasks.workunit.client.1.smithi203.stdout:9/140: read f12 [4324151,98467] 0 2024-04-05T03:02:33.740 INFO:tasks.workunit.client.1.smithi203.stdout:9/141: getdents d13 0 2024-04-05T03:02:33.742 INFO:tasks.workunit.client.1.smithi203.stdout:9/142: unlink d13/f27 0 2024-04-05T03:02:33.743 INFO:tasks.workunit.client.1.smithi203.stdout:9/143: mknod d13/c2b 0 2024-04-05T03:02:33.839 INFO:tasks.workunit.client.1.smithi203.stdout:4/104: sync 2024-04-05T03:02:33.840 INFO:tasks.workunit.client.1.smithi203.stdout:2/161: sync 2024-04-05T03:02:33.841 INFO:tasks.workunit.client.1.smithi203.stdout:2/162: write d5/d10/d1a/d24/f28 [638121,2887] 0 2024-04-05T03:02:33.842 INFO:tasks.workunit.client.1.smithi203.stdout:4/105: symlink da/l23 0 2024-04-05T03:02:33.844 INFO:tasks.workunit.client.1.smithi203.stdout:2/163: mkdir d5/d10/d35/d36 0 2024-04-05T03:02:33.845 INFO:tasks.workunit.client.1.smithi203.stdout:4/106: rename da/lf to da/l24 0 2024-04-05T03:02:33.846 INFO:tasks.workunit.client.1.smithi203.stdout:2/164: chown d5/d10/d2f/l31 24 1 2024-04-05T03:02:33.846 INFO:tasks.workunit.client.1.smithi203.stdout:4/107: write da/f1e [119623,114182] 0 2024-04-05T03:02:33.848 INFO:tasks.workunit.client.1.smithi203.stdout:4/108: symlink da/l25 0 2024-04-05T03:02:33.848 INFO:tasks.workunit.client.1.smithi203.stdout:4/109: stat f8 0 2024-04-05T03:02:33.849 INFO:tasks.workunit.client.1.smithi203.stdout:2/165: creat d5/d10/d1a/d24/f37 x:0 0 0 2024-04-05T03:02:33.850 INFO:tasks.workunit.client.1.smithi203.stdout:4/110: creat da/f26 x:0 0 0 2024-04-05T03:02:33.851 INFO:tasks.workunit.client.1.smithi203.stdout:4/111: truncate da/f21 39613 0 2024-04-05T03:02:33.853 INFO:tasks.workunit.client.1.smithi203.stdout:2/166: getdents d5/d10 0 2024-04-05T03:02:33.853 INFO:tasks.workunit.client.1.smithi203.stdout:2/167: fsync d5/d10/d1a/d24/f27 0 2024-04-05T03:02:33.854 INFO:tasks.workunit.client.1.smithi203.stdout:2/168: stat d5/d10/d35/d2d/f33 0 2024-04-05T03:02:33.855 INFO:tasks.workunit.client.1.smithi203.stdout:2/169: fdatasync d5/d10/d1a/d24/f27 0 2024-04-05T03:02:33.857 INFO:tasks.workunit.client.1.smithi203.stdout:2/170: creat d5/d10/d35/d2d/f38 x:0 0 0 2024-04-05T03:02:33.859 INFO:tasks.workunit.client.1.smithi203.stdout:2/171: creat d5/d10/d23/f39 x:0 0 0 2024-04-05T03:02:33.861 INFO:tasks.workunit.client.1.smithi203.stdout:2/172: symlink d5/d10/d1a/l3a 0 2024-04-05T03:02:33.863 INFO:tasks.workunit.client.1.smithi203.stdout:2/173: rename d5/d10/d23/f39 to d5/d10/d35/d36/f3b 0 2024-04-05T03:02:33.865 INFO:tasks.workunit.client.1.smithi203.stdout:2/174: mknod d5/d10/d1a/c3c 0 2024-04-05T03:02:33.867 INFO:tasks.workunit.client.1.smithi203.stdout:2/175: symlink d5/d10/d35/d2d/l3d 0 2024-04-05T03:02:33.868 INFO:tasks.workunit.client.1.smithi203.stdout:2/176: write d5/d10/d1a/d24/f28 [300888,12003] 0 2024-04-05T03:02:33.868 INFO:tasks.workunit.client.1.smithi203.stdout:2/177: stat d5/d10/d35/d2d/f33 0 2024-04-05T03:02:33.869 INFO:tasks.workunit.client.1.smithi203.stdout:2/178: readlink d5/d10/d2f/l31 0 2024-04-05T03:02:33.869 INFO:tasks.workunit.client.1.smithi203.stdout:2/179: chown d5/d10/d35/d2d/f38 15799 1 2024-04-05T03:02:33.871 INFO:tasks.workunit.client.1.smithi203.stdout:2/180: unlink d5/c21 0 2024-04-05T03:02:33.873 INFO:tasks.workunit.client.1.smithi203.stdout:2/181: mkdir d5/d10/d35/d3e 0 2024-04-05T03:02:33.874 INFO:tasks.workunit.client.1.smithi203.stdout:2/182: write d5/d10/d1a/d24/f28 [1479774,98141] 0 2024-04-05T03:02:33.876 INFO:tasks.workunit.client.1.smithi203.stdout:2/183: creat d5/d10/d1a/d24/f3f x:0 0 0 2024-04-05T03:02:33.877 INFO:tasks.workunit.client.1.smithi203.stdout:2/184: read d5/d10/d1a/d24/f27 [1920133,48264] 0 2024-04-05T03:02:33.878 INFO:tasks.workunit.client.1.smithi203.stdout:2/185: readlink l2 0 2024-04-05T03:02:33.881 INFO:tasks.workunit.client.1.smithi203.stdout:2/186: rename d5/d10/f13 to d5/f40 0 2024-04-05T03:02:33.882 INFO:tasks.workunit.client.1.smithi203.stdout:2/187: fdatasync d5/d10/d23/f25 0 2024-04-05T03:02:33.913 INFO:tasks.workunit.client.1.smithi203.stdout:4/112: dread da/fc [0,4194304] 0 2024-04-05T03:02:33.914 INFO:tasks.workunit.client.1.smithi203.stdout:4/113: creat da/f27 x:0 0 0 2024-04-05T03:02:33.919 INFO:tasks.workunit.client.1.smithi203.stdout:4/114: dwrite da/f27 [0,4194304] 0 2024-04-05T03:02:33.930 INFO:tasks.workunit.client.1.smithi203.stdout:4/115: getdents da 0 2024-04-05T03:02:33.969 INFO:tasks.workunit.client.1.smithi203.stdout:8/171: sync 2024-04-05T03:02:33.969 INFO:tasks.workunit.client.1.smithi203.stdout:5/230: sync 2024-04-05T03:02:33.969 INFO:tasks.workunit.client.1.smithi203.stdout:7/109: sync 2024-04-05T03:02:33.971 INFO:tasks.workunit.client.1.smithi203.stdout:5/231: rmdir d3/d4 39 2024-04-05T03:02:33.971 INFO:tasks.workunit.client.1.smithi203.stdout:5/232: chown d3/f7 116707 1 2024-04-05T03:02:33.972 INFO:tasks.workunit.client.1.smithi203.stdout:8/172: write d1/d12/d17/d10/f15 [804331,130746] 0 2024-04-05T03:02:33.974 INFO:tasks.workunit.client.1.smithi203.stdout:7/110: fsync f3 0 2024-04-05T03:02:33.978 INFO:tasks.workunit.client.1.smithi203.stdout:7/111: symlink d8/d12/d16/d1a/l25 0 2024-04-05T03:02:33.980 INFO:tasks.workunit.client.1.smithi203.stdout:8/173: dread d1/d6/d16/d1f/f22 [0,4194304] 0 2024-04-05T03:02:33.981 INFO:tasks.workunit.client.1.smithi203.stdout:7/112: unlink d8/da/f15 0 2024-04-05T03:02:33.981 INFO:tasks.workunit.client.1.smithi203.stdout:8/174: chown d1/d6/d16/l1b 1009695935 1 2024-04-05T03:02:33.982 INFO:tasks.workunit.client.1.smithi203.stdout:7/113: chown d8 5072 1 2024-04-05T03:02:33.982 INFO:tasks.workunit.client.1.smithi203.stdout:5/233: dwrite d3/d4/dc/f54 [0,4194304] 0 2024-04-05T03:02:33.987 INFO:tasks.workunit.client.1.smithi203.stdout:7/114: rename d8/d12/d16/d1a/l25 to d8/d12/d16/d1a/d21/l26 0 2024-04-05T03:02:33.989 INFO:tasks.workunit.client.1.smithi203.stdout:8/175: link d1/d6/c8 d1/d6/c25 0 2024-04-05T03:02:33.991 INFO:tasks.workunit.client.1.smithi203.stdout:7/115: creat d8/d12/d16/d1a/d21/f27 x:0 0 0 2024-04-05T03:02:33.992 INFO:tasks.workunit.client.1.smithi203.stdout:7/116: stat d8/da/cc 0 2024-04-05T03:02:33.992 INFO:tasks.workunit.client.1.smithi203.stdout:8/176: dread d1/d6/d16/d1f/f22 [0,4194304] 0 2024-04-05T03:02:33.993 INFO:tasks.workunit.client.1.smithi203.stdout:5/234: write f2 [3985110,65643] 0 2024-04-05T03:02:33.995 INFO:tasks.workunit.client.1.smithi203.stdout:8/177: mknod d1/d6/d16/c26 0 2024-04-05T03:02:33.998 INFO:tasks.workunit.client.1.smithi203.stdout:8/178: rename d1/d6/d16/c26 to d1/d12/d1a/c27 0 2024-04-05T03:02:34.000 INFO:tasks.workunit.client.1.smithi203.stdout:8/179: creat d1/d12/f28 x:0 0 0 2024-04-05T03:02:34.001 INFO:tasks.workunit.client.1.smithi203.stdout:7/117: dread f4 [0,4194304] 0 2024-04-05T03:02:34.002 INFO:tasks.workunit.client.1.smithi203.stdout:8/180: symlink d1/d6/l29 0 2024-04-05T03:02:34.005 INFO:tasks.workunit.client.1.smithi203.stdout:7/118: mknod d8/da/c28 0 2024-04-05T03:02:34.005 INFO:tasks.workunit.client.1.smithi203.stdout:8/181: write d1/d6/d16/d1f/f23 [632361,46051] 0 2024-04-05T03:02:34.006 INFO:tasks.workunit.client.1.smithi203.stdout:8/182: truncate d1/d12/d17/d10/f24 883836 0 2024-04-05T03:02:34.007 INFO:tasks.workunit.client.1.smithi203.stdout:8/183: readlink d1/l2 0 2024-04-05T03:02:34.009 INFO:tasks.workunit.client.1.smithi203.stdout:8/184: creat d1/d6/d16/f2a x:0 0 0 2024-04-05T03:02:34.013 INFO:tasks.workunit.client.1.smithi203.stdout:8/185: rmdir d1/d12/d1a 39 2024-04-05T03:02:34.013 INFO:tasks.workunit.client.1.smithi203.stdout:5/235: dread d3/d4/d8/d12/f43 [0,4194304] 0 2024-04-05T03:02:34.015 INFO:tasks.workunit.client.1.smithi203.stdout:7/119: truncate d8/fe 2356146 0 2024-04-05T03:02:34.016 INFO:tasks.workunit.client.1.smithi203.stdout:5/236: chown d3/d4/d8/d12/l27 417 1 2024-04-05T03:02:34.016 INFO:tasks.workunit.client.1.smithi203.stdout:5/237: chown d3/d4/d8/d12/l52 6393759 1 2024-04-05T03:02:34.017 INFO:tasks.workunit.client.1.smithi203.stdout:8/186: rename d1/d12/l19 to d1/d6/d16/d1f/l2b 0 2024-04-05T03:02:34.018 INFO:tasks.workunit.client.1.smithi203.stdout:7/120: mkdir d8/d12/d16/d1a/d29 0 2024-04-05T03:02:34.018 INFO:tasks.workunit.client.1.smithi203.stdout:7/121: chown d8/ff 457 1 2024-04-05T03:02:34.020 INFO:tasks.workunit.client.1.smithi203.stdout:7/122: rmdir d8/d12 39 2024-04-05T03:02:34.024 INFO:tasks.workunit.client.1.smithi203.stdout:7/123: write d8/d12/d16/d1a/f1f [99423,26082] 0 2024-04-05T03:02:34.025 INFO:tasks.workunit.client.1.smithi203.stdout:5/238: write d3/d1f/f1e [1930410,109524] 0 2024-04-05T03:02:34.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/187: dwrite d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:34.029 INFO:tasks.workunit.client.1.smithi203.stdout:7/124: mkdir d8/d2a 0 2024-04-05T03:02:34.032 INFO:tasks.workunit.client.1.smithi203.stdout:5/239: dread d3/f32 [0,4194304] 0 2024-04-05T03:02:34.032 INFO:tasks.workunit.client.1.smithi203.stdout:8/188: rename d1/d12/d17/d10/f24 to d1/d12/d1a/f2c 0 2024-04-05T03:02:34.033 INFO:tasks.workunit.client.1.smithi203.stdout:5/240: stat d3/d4/d8/f3d 0 2024-04-05T03:02:34.034 INFO:tasks.workunit.client.1.smithi203.stdout:7/125: symlink d8/d12/d16/d1a/l2b 0 2024-04-05T03:02:34.037 INFO:tasks.workunit.client.1.smithi203.stdout:8/189: symlink d1/d6/d16/d1f/l2d 0 2024-04-05T03:02:34.040 INFO:tasks.workunit.client.1.smithi203.stdout:8/190: mkdir d1/d6/d2e 0 2024-04-05T03:02:34.042 INFO:tasks.workunit.client.1.smithi203.stdout:5/241: dwrite d3/f36 [16777216,4194304] 0 2024-04-05T03:02:34.043 INFO:tasks.workunit.client.1.smithi203.stdout:8/191: chown d1/d12/d17/d10/f15 0 1 2024-04-05T03:02:34.044 INFO:tasks.workunit.client.1.smithi203.stdout:8/192: chown d1/d6/d16/d1f/f23 1 1 2024-04-05T03:02:34.044 INFO:tasks.workunit.client.1.smithi203.stdout:7/126: link d8/c1c d8/d12/d16/d1a/d29/c2c 0 2024-04-05T03:02:34.045 INFO:tasks.workunit.client.1.smithi203.stdout:5/242: mknod d3/d4/d8/d12/c56 0 2024-04-05T03:02:34.047 INFO:tasks.workunit.client.1.smithi203.stdout:7/127: symlink d8/da/l2d 0 2024-04-05T03:02:34.049 INFO:tasks.workunit.client.1.smithi203.stdout:8/193: dread d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:34.050 INFO:tasks.workunit.client.1.smithi203.stdout:5/243: creat d3/d1f/d49/f57 x:0 0 0 2024-04-05T03:02:34.052 INFO:tasks.workunit.client.1.smithi203.stdout:8/194: fdatasync d1/d12/f28 0 2024-04-05T03:02:34.053 INFO:tasks.workunit.client.1.smithi203.stdout:7/128: fdatasync d8/d12/f18 0 2024-04-05T03:02:34.054 INFO:tasks.workunit.client.1.smithi203.stdout:8/195: stat d1/c13 0 2024-04-05T03:02:34.056 INFO:tasks.workunit.client.1.smithi203.stdout:5/244: mkdir d3/d4/d8/d12/d40/d58 0 2024-04-05T03:02:34.056 INFO:tasks.workunit.client.1.smithi203.stdout:8/196: dread d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:34.142 INFO:tasks.workunit.client.1.smithi203.stdout:7/129: dread d8/ff [0,4194304] 0 2024-04-05T03:02:34.142 INFO:tasks.workunit.client.1.smithi203.stdout:5/245: mkdir d3/d1f/d59 0 2024-04-05T03:02:34.143 INFO:tasks.workunit.client.1.smithi203.stdout:8/197: write d1/d12/d17/d10/f15 [4832288,105944] 0 2024-04-05T03:02:34.143 INFO:tasks.workunit.client.1.smithi203.stdout:5/246: readlink d3/d4/d8/l34 0 2024-04-05T03:02:34.143 INFO:tasks.workunit.client.1.smithi203.stdout:1/120: sync 2024-04-05T03:02:34.148 INFO:tasks.workunit.client.1.smithi203.stdout:5/247: rmdir d3/d4/d8/d12/d40/d58 0 2024-04-05T03:02:34.149 INFO:tasks.workunit.client.1.smithi203.stdout:8/198: getdents d1/d12/d17/d10 0 2024-04-05T03:02:34.153 INFO:tasks.workunit.client.1.smithi203.stdout:7/130: dwrite d8/d12/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:34.156 INFO:tasks.workunit.client.1.smithi203.stdout:7/131: unlink f3 0 2024-04-05T03:02:34.156 INFO:tasks.workunit.client.1.smithi203.stdout:1/121: dwrite f2 [4194304,4194304] 0 2024-04-05T03:02:34.160 INFO:tasks.workunit.client.1.smithi203.stdout:7/132: write d8/f17 [4975639,107499] 0 2024-04-05T03:02:34.163 INFO:tasks.workunit.client.1.smithi203.stdout:7/133: readlink d8/d12/d16/d1a/l2b 0 2024-04-05T03:02:34.163 INFO:tasks.workunit.client.1.smithi203.stdout:8/199: dwrite d1/d6/d16/f2a [0,4194304] 0 2024-04-05T03:02:34.171 INFO:tasks.workunit.client.1.smithi203.stdout:1/122: dwrite d7/f1c [0,4194304] 0 2024-04-05T03:02:34.174 INFO:tasks.workunit.client.1.smithi203.stdout:8/200: rename d1/d6/d16 to d1/d12/d2f 0 2024-04-05T03:02:34.175 INFO:tasks.workunit.client.1.smithi203.stdout:7/134: getdents d8/d12/d16/d1a 0 2024-04-05T03:02:34.175 INFO:tasks.workunit.client.1.smithi203.stdout:7/135: chown d8/ff 49461091 1 2024-04-05T03:02:34.177 INFO:tasks.workunit.client.1.smithi203.stdout:1/123: mknod d7/d24/c2c 0 2024-04-05T03:02:34.180 INFO:tasks.workunit.client.1.smithi203.stdout:8/201: creat d1/d6/d2e/f30 x:0 0 0 2024-04-05T03:02:34.184 INFO:tasks.workunit.client.1.smithi203.stdout:7/136: link f4 d8/d12/d16/d1a/d29/f2e 0 2024-04-05T03:02:34.185 INFO:tasks.workunit.client.1.smithi203.stdout:8/202: dread d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:34.186 INFO:tasks.workunit.client.1.smithi203.stdout:7/137: chown d8/fe 7 1 2024-04-05T03:02:34.188 INFO:tasks.workunit.client.1.smithi203.stdout:8/203: creat d1/d12/f31 x:0 0 0 2024-04-05T03:02:34.189 INFO:tasks.workunit.client.1.smithi203.stdout:8/204: dread - d1/d12/f31 zero size 2024-04-05T03:02:34.191 INFO:tasks.workunit.client.1.smithi203.stdout:8/205: symlink d1/d12/l32 0 2024-04-05T03:02:34.193 INFO:tasks.workunit.client.1.smithi203.stdout:8/206: chown d1/d6/c8 200 1 2024-04-05T03:02:34.195 INFO:tasks.workunit.client.1.smithi203.stdout:8/207: truncate d1/d12/f31 248947 0 2024-04-05T03:02:34.196 INFO:tasks.workunit.client.1.smithi203.stdout:7/138: rename d8/d12/d16/d1a/d29 to d8/da/d2f 0 2024-04-05T03:02:34.197 INFO:tasks.workunit.client.1.smithi203.stdout:8/208: fsync d1/d12/d2f/d1f/f22 0 2024-04-05T03:02:34.199 INFO:tasks.workunit.client.1.smithi203.stdout:7/139: write d8/da/f1d [2664980,117950] 0 2024-04-05T03:02:34.201 INFO:tasks.workunit.client.1.smithi203.stdout:1/124: dread f2 [0,4194304] 0 2024-04-05T03:02:34.202 INFO:tasks.workunit.client.1.smithi203.stdout:7/140: readlink d8/d12/d16/d1a/d21/l26 0 2024-04-05T03:02:34.205 INFO:tasks.workunit.client.1.smithi203.stdout:1/125: creat d7/f2d x:0 0 0 2024-04-05T03:02:34.205 INFO:tasks.workunit.client.1.smithi203.stdout:8/209: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:34.206 INFO:tasks.workunit.client.1.smithi203.stdout:7/141: mkdir d8/d12/d16/d1a/d30 0 2024-04-05T03:02:34.206 INFO:tasks.workunit.client.1.smithi203.stdout:1/126: creat d7/f2e x:0 0 0 2024-04-05T03:02:34.208 INFO:tasks.workunit.client.1.smithi203.stdout:8/210: write d1/d6/d2e/f30 [294706,48803] 0 2024-04-05T03:02:34.210 INFO:tasks.workunit.client.1.smithi203.stdout:7/142: dread d8/fe [0,4194304] 0 2024-04-05T03:02:34.211 INFO:tasks.workunit.client.1.smithi203.stdout:8/211: write d1/d12/f28 [4049592,79139] 0 2024-04-05T03:02:34.212 INFO:tasks.workunit.client.1.smithi203.stdout:1/127: link d7/cd d7/d14/c2f 0 2024-04-05T03:02:34.214 INFO:tasks.workunit.client.1.smithi203.stdout:7/143: dread d8/d12/f18 [0,4194304] 0 2024-04-05T03:02:34.218 INFO:tasks.workunit.client.1.smithi203.stdout:8/212: getdents d1/d6 0 2024-04-05T03:02:34.221 INFO:tasks.workunit.client.1.smithi203.stdout:1/128: getdents d7 0 2024-04-05T03:02:34.223 INFO:tasks.workunit.client.1.smithi203.stdout:8/213: mknod d1/c33 0 2024-04-05T03:02:34.226 INFO:tasks.workunit.client.1.smithi203.stdout:1/129: unlink d7/d14/c2f 0 2024-04-05T03:02:34.227 INFO:tasks.workunit.client.1.smithi203.stdout:8/214: dread d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:34.229 INFO:tasks.workunit.client.1.smithi203.stdout:7/144: dwrite d8/da/d2f/f2e [0,4194304] 0 2024-04-05T03:02:34.231 INFO:tasks.workunit.client.1.smithi203.stdout:7/145: write d8/da/f14 [1391591,2450] 0 2024-04-05T03:02:34.232 INFO:tasks.workunit.client.1.smithi203.stdout:1/130: getdents d7/d24 0 2024-04-05T03:02:34.237 INFO:tasks.workunit.client.1.smithi203.stdout:1/131: dread d7/d14/f21 [0,4194304] 0 2024-04-05T03:02:34.237 INFO:tasks.workunit.client.1.smithi203.stdout:8/215: write d1/d12/d1a/f1e [983989,35486] 0 2024-04-05T03:02:34.239 INFO:tasks.workunit.client.1.smithi203.stdout:8/216: chown d1/d12/d2f 1571593918 1 2024-04-05T03:02:34.241 INFO:tasks.workunit.client.1.smithi203.stdout:7/146: dwrite d8/d12/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:34.242 INFO:tasks.workunit.client.1.smithi203.stdout:7/147: write d8/d12/d16/d1a/f24 [2748088,26573] 0 2024-04-05T03:02:34.243 INFO:tasks.workunit.client.1.smithi203.stdout:1/132: write d7/f27 [4981286,75956] 0 2024-04-05T03:02:34.249 INFO:tasks.workunit.client.1.smithi203.stdout:7/148: creat d8/f31 x:0 0 0 2024-04-05T03:02:34.250 INFO:tasks.workunit.client.1.smithi203.stdout:1/133: getdents d7 0 2024-04-05T03:02:34.251 INFO:tasks.workunit.client.1.smithi203.stdout:8/217: dwrite d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:34.254 INFO:tasks.workunit.client.1.smithi203.stdout:1/134: creat d7/d24/f30 x:0 0 0 2024-04-05T03:02:34.256 INFO:tasks.workunit.client.1.smithi203.stdout:1/135: chown d7/f27 671190825 1 2024-04-05T03:02:34.256 INFO:tasks.workunit.client.1.smithi203.stdout:8/218: chown d1/d12/d2f/d1f/l2d 0 1 2024-04-05T03:02:34.257 INFO:tasks.workunit.client.1.smithi203.stdout:8/219: stat d1/c33 0 2024-04-05T03:02:34.259 INFO:tasks.workunit.client.1.smithi203.stdout:7/149: dwrite d8/f20 [0,4194304] 0 2024-04-05T03:02:34.263 INFO:tasks.workunit.client.1.smithi203.stdout:7/150: rmdir d8/d12/d16/d1a/d21 39 2024-04-05T03:02:34.263 INFO:tasks.workunit.client.1.smithi203.stdout:1/136: getdents d7/d14 0 2024-04-05T03:02:34.268 INFO:tasks.workunit.client.1.smithi203.stdout:7/151: rename d8/d12 to d8/d32 0 2024-04-05T03:02:34.270 INFO:tasks.workunit.client.1.smithi203.stdout:7/152: truncate d8/fe 286009 0 2024-04-05T03:02:34.276 INFO:tasks.workunit.client.1.smithi203.stdout:7/153: rmdir d8/d32 39 2024-04-05T03:02:34.279 INFO:tasks.workunit.client.1.smithi203.stdout:7/154: chown d8/d32/d16 7787 1 2024-04-05T03:02:34.281 INFO:tasks.workunit.client.1.smithi203.stdout:7/155: write d8/d32/d16/d1a/d21/f22 [964738,96163] 0 2024-04-05T03:02:34.285 INFO:tasks.workunit.client.1.smithi203.stdout:7/156: rename d8/d32/d16/d1a/f24 to d8/f33 0 2024-04-05T03:02:34.292 INFO:tasks.workunit.client.1.smithi203.stdout:7/157: write d8/fe [691867,51005] 0 2024-04-05T03:02:34.366 INFO:tasks.workunit.client.1.smithi203.stdout:3/112: sync 2024-04-05T03:02:34.368 INFO:tasks.workunit.client.1.smithi203.stdout:3/113: rmdir d0 39 2024-04-05T03:02:34.370 INFO:tasks.workunit.client.1.smithi203.stdout:3/114: fdatasync d0/dc/f14 0 2024-04-05T03:02:34.372 INFO:tasks.workunit.client.1.smithi203.stdout:3/115: mknod d0/dc/d1a/d1d/c1e 0 2024-04-05T03:02:34.373 INFO:tasks.workunit.client.1.smithi203.stdout:3/116: write d0/fd [2866685,22383] 0 2024-04-05T03:02:34.374 INFO:tasks.workunit.client.1.smithi203.stdout:3/117: fdatasync d0/f4 0 2024-04-05T03:02:34.376 INFO:tasks.workunit.client.1.smithi203.stdout:3/118: mkdir d0/dc/d1a/d1f 0 2024-04-05T03:02:34.382 INFO:tasks.workunit.client.1.smithi203.stdout:3/119: truncate d0/f4 4095276 0 2024-04-05T03:02:34.384 INFO:tasks.workunit.client.1.smithi203.stdout:3/120: unlink d0/c12 0 2024-04-05T03:02:34.386 INFO:tasks.workunit.client.1.smithi203.stdout:3/121: symlink d0/dc/d1a/l20 0 2024-04-05T03:02:34.386 INFO:tasks.workunit.client.1.smithi203.stdout:3/122: readlink d0/l1b 0 2024-04-05T03:02:34.387 INFO:tasks.workunit.client.1.smithi203.stdout:3/123: read d0/dc/f16 [35989,107716] 0 2024-04-05T03:02:34.388 INFO:tasks.workunit.client.1.smithi203.stdout:3/124: chown d0/dc/f14 63618707 1 2024-04-05T03:02:34.390 INFO:tasks.workunit.client.1.smithi203.stdout:3/125: mkdir d0/d21 0 2024-04-05T03:02:34.392 INFO:tasks.workunit.client.1.smithi203.stdout:3/126: truncate d0/dc/f14 591202 0 2024-04-05T03:02:34.392 INFO:tasks.workunit.client.1.smithi203.stdout:3/127: read d0/f9 [3235921,94194] 0 2024-04-05T03:02:34.395 INFO:tasks.workunit.client.1.smithi203.stdout:3/128: fdatasync d0/f5 0 2024-04-05T03:02:34.585 INFO:tasks.workunit.client.1.smithi203.stdout:6/109: sync 2024-04-05T03:02:34.585 INFO:tasks.workunit.client.1.smithi203.stdout:0/179: sync 2024-04-05T03:02:34.586 INFO:tasks.workunit.client.1.smithi203.stdout:6/110: write d0/da/d18/f1d [486678,28851] 0 2024-04-05T03:02:34.587 INFO:tasks.workunit.client.1.smithi203.stdout:0/180: rename dd/d17 to dd/d17/d30 22 2024-04-05T03:02:34.589 INFO:tasks.workunit.client.1.smithi203.stdout:0/181: mknod dd/d17/d29/c31 0 2024-04-05T03:02:34.590 INFO:tasks.workunit.client.1.smithi203.stdout:6/111: dread d0/f8 [0,4194304] 0 2024-04-05T03:02:34.591 INFO:tasks.workunit.client.1.smithi203.stdout:6/112: truncate d0/f3 5356474 0 2024-04-05T03:02:34.591 INFO:tasks.workunit.client.1.smithi203.stdout:0/182: write dd/f14 [92109,65927] 0 2024-04-05T03:02:34.600 INFO:tasks.workunit.client.1.smithi203.stdout:0/183: dwrite dd/d17/d1d/d22/f23 [0,4194304] 0 2024-04-05T03:02:34.600 INFO:tasks.workunit.client.1.smithi203.stdout:6/113: dwrite d0/f8 [4194304,4194304] 0 2024-04-05T03:02:34.601 INFO:tasks.workunit.client.1.smithi203.stdout:0/184: dread - dd/f27 zero size 2024-04-05T03:02:34.602 INFO:tasks.workunit.client.1.smithi203.stdout:0/185: write f8 [4621713,100904] 0 2024-04-05T03:02:34.609 INFO:tasks.workunit.client.1.smithi203.stdout:6/114: dwrite d0/da/f10 [0,4194304] 0 2024-04-05T03:02:34.611 INFO:tasks.workunit.client.1.smithi203.stdout:6/115: rename d0/da to d0/da/d21 22 2024-04-05T03:02:34.615 INFO:tasks.workunit.client.1.smithi203.stdout:6/116: mknod d0/d6/c22 0 2024-04-05T03:02:34.616 INFO:tasks.workunit.client.1.smithi203.stdout:0/186: dwrite dd/f15 [0,4194304] 0 2024-04-05T03:02:34.619 INFO:tasks.workunit.client.1.smithi203.stdout:6/117: mkdir d0/d23 0 2024-04-05T03:02:34.621 INFO:tasks.workunit.client.1.smithi203.stdout:6/118: fsync d0/da/f13 0 2024-04-05T03:02:34.624 INFO:tasks.workunit.client.1.smithi203.stdout:6/119: mknod d0/d23/c24 0 2024-04-05T03:02:34.625 INFO:tasks.workunit.client.1.smithi203.stdout:6/120: write d0/f8 [7810542,120766] 0 2024-04-05T03:02:34.637 INFO:tasks.workunit.client.1.smithi203.stdout:6/121: getdents d0 0 2024-04-05T03:02:34.639 INFO:tasks.workunit.client.1.smithi203.stdout:6/122: rmdir d0/d6 39 2024-04-05T03:02:34.641 INFO:tasks.workunit.client.1.smithi203.stdout:6/123: creat d0/da/d18/f25 x:0 0 0 2024-04-05T03:02:34.643 INFO:tasks.workunit.client.1.smithi203.stdout:0/187: dwrite dd/d17/d1d/f2e [0,4194304] 0 2024-04-05T03:02:34.646 INFO:tasks.workunit.client.1.smithi203.stdout:0/188: creat dd/d17/d1d/d22/f32 x:0 0 0 2024-04-05T03:02:34.648 INFO:tasks.workunit.client.1.smithi203.stdout:6/124: read d0/da/d18/f1d [437415,71772] 0 2024-04-05T03:02:34.649 INFO:tasks.workunit.client.1.smithi203.stdout:0/189: unlink dd/d17/c1c 0 2024-04-05T03:02:34.650 INFO:tasks.workunit.client.1.smithi203.stdout:6/125: write d0/da/d18/f1d [548493,84594] 0 2024-04-05T03:02:34.654 INFO:tasks.workunit.client.1.smithi203.stdout:6/126: rename d0/f19 to d0/da/d18/f26 0 2024-04-05T03:02:34.656 INFO:tasks.workunit.client.1.smithi203.stdout:0/190: link dd/d17/d1d/c2a dd/d17/d1d/d26/d28/c33 0 2024-04-05T03:02:34.659 INFO:tasks.workunit.client.1.smithi203.stdout:6/127: getdents d0/da 0 2024-04-05T03:02:34.660 INFO:tasks.workunit.client.1.smithi203.stdout:6/128: read d0/f3 [4642294,5108] 0 2024-04-05T03:02:34.662 INFO:tasks.workunit.client.1.smithi203.stdout:6/129: mknod d0/c27 0 2024-04-05T03:02:34.664 INFO:tasks.workunit.client.1.smithi203.stdout:6/130: creat d0/d23/f28 x:0 0 0 2024-04-05T03:02:34.666 INFO:tasks.workunit.client.1.smithi203.stdout:6/131: unlink d0/da/l12 0 2024-04-05T03:02:34.669 INFO:tasks.workunit.client.1.smithi203.stdout:6/132: dread d0/da/f13 [0,4194304] 0 2024-04-05T03:02:34.673 INFO:tasks.workunit.client.1.smithi203.stdout:6/133: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:34.674 INFO:tasks.workunit.client.1.smithi203.stdout:6/134: readlink d0/d6/l7 0 2024-04-05T03:02:34.677 INFO:tasks.workunit.client.1.smithi203.stdout:6/135: write d0/f3 [5045427,69971] 0 2024-04-05T03:02:34.679 INFO:tasks.workunit.client.1.smithi203.stdout:6/136: symlink d0/l29 0 2024-04-05T03:02:34.688 INFO:tasks.workunit.client.1.smithi203.stdout:6/137: dwrite d0/ff [0,4194304] 0 2024-04-05T03:02:34.696 INFO:tasks.workunit.client.1.smithi203.stdout:6/138: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:34.698 INFO:tasks.workunit.client.1.smithi203.stdout:6/139: write d0/d23/f28 [621636,16449] 0 2024-04-05T03:02:34.698 INFO:tasks.workunit.client.1.smithi203.stdout:6/140: chown d0/d6/c22 5948 1 2024-04-05T03:02:34.700 INFO:tasks.workunit.client.1.smithi203.stdout:6/141: truncate d0/da/d18/f25 532893 0 2024-04-05T03:02:34.702 INFO:tasks.workunit.client.1.smithi203.stdout:6/142: mkdir d0/d6/d2a 0 2024-04-05T03:02:34.705 INFO:tasks.workunit.client.1.smithi203.stdout:6/143: fsync d0/d23/f28 0 2024-04-05T03:02:34.709 INFO:tasks.workunit.client.1.smithi203.stdout:6/144: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:34.710 INFO:tasks.workunit.client.1.smithi203.stdout:6/145: write d0/d23/f28 [1357683,79426] 0 2024-04-05T03:02:34.716 INFO:tasks.workunit.client.1.smithi203.stdout:6/146: truncate d0/d23/f28 617994 0 2024-04-05T03:02:34.717 INFO:tasks.workunit.client.1.smithi203.stdout:6/147: write d0/da/f10 [2911825,92927] 0 2024-04-05T03:02:34.721 INFO:tasks.workunit.client.1.smithi203.stdout:6/148: truncate d0/d23/f28 969202 0 2024-04-05T03:02:34.724 INFO:tasks.workunit.client.1.smithi203.stdout:6/149: mkdir d0/d6/d2b 0 2024-04-05T03:02:34.726 INFO:tasks.workunit.client.1.smithi203.stdout:6/150: mkdir d0/d6/d2a/d2c 0 2024-04-05T03:02:34.774 INFO:tasks.workunit.client.1.smithi203.stdout:2/188: sync 2024-04-05T03:02:34.774 INFO:tasks.workunit.client.1.smithi203.stdout:9/144: sync 2024-04-05T03:02:34.781 INFO:tasks.workunit.client.1.smithi203.stdout:9/145: dwrite d13/f20 [0,4194304] 0 2024-04-05T03:02:34.782 INFO:tasks.workunit.client.1.smithi203.stdout:2/189: dwrite d5/d10/d1a/d24/f2b [0,4194304] 0 2024-04-05T03:02:34.787 INFO:tasks.workunit.client.1.smithi203.stdout:9/146: rename d13/d1b/l22 to d13/l2c 0 2024-04-05T03:02:34.788 INFO:tasks.workunit.client.1.smithi203.stdout:9/147: dread - d13/f24 zero size 2024-04-05T03:02:34.790 INFO:tasks.workunit.client.1.smithi203.stdout:2/190: getdents d5/d10/d35 0 2024-04-05T03:02:34.791 INFO:tasks.workunit.client.1.smithi203.stdout:2/191: mkdir d5/d10/d35/d36/d41 0 2024-04-05T03:02:34.792 INFO:tasks.workunit.client.1.smithi203.stdout:2/192: readlink d5/d10/d2f/l31 0 2024-04-05T03:02:34.796 INFO:tasks.workunit.client.1.smithi203.stdout:9/148: dread fd [4194304,4194304] 0 2024-04-05T03:02:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:34 smithi203 ceph-mon[39911]: pgmap v253: 65 pgs: 65 active+clean; 324 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 21 MiB/s wr, 398 op/s 2024-04-05T03:02:34.797 INFO:tasks.workunit.client.1.smithi203.stdout:9/149: write d13/f24 [508446,85052] 0 2024-04-05T03:02:34.817 INFO:tasks.workunit.client.1.smithi203.stdout:2/193: dread d5/f8 [0,4194304] 0 2024-04-05T03:02:34.818 INFO:tasks.workunit.client.1.smithi203.stdout:2/194: chown d5/l11 1 1 2024-04-05T03:02:34.818 INFO:tasks.workunit.client.1.smithi203.stdout:2/195: fdatasync d5/d10/d23/f25 0 2024-04-05T03:02:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:34 smithi153 ceph-mon[29381]: pgmap v253: 65 pgs: 65 active+clean; 324 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 21 MiB/s wr, 398 op/s 2024-04-05T03:02:34.944 INFO:tasks.workunit.client.1.smithi203.stdout:4/116: sync 2024-04-05T03:02:34.945 INFO:tasks.workunit.client.1.smithi203.stdout:4/117: write da/f13 [2480983,73319] 0 2024-04-05T03:02:34.945 INFO:tasks.workunit.client.1.smithi203.stdout:4/118: stat da/f26 0 2024-04-05T03:02:34.948 INFO:tasks.workunit.client.1.smithi203.stdout:4/119: creat da/f28 x:0 0 0 2024-04-05T03:02:34.948 INFO:tasks.workunit.client.1.smithi203.stdout:4/120: write f5 [2455320,1786] 0 2024-04-05T03:02:34.950 INFO:tasks.workunit.client.1.smithi203.stdout:4/121: mkdir da/d29 0 2024-04-05T03:02:34.951 INFO:tasks.workunit.client.1.smithi203.stdout:4/122: truncate da/f26 50491 0 2024-04-05T03:02:35.050 INFO:tasks.workunit.client.0.smithi153.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-05T03:02:35.058 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T03:02:35.058 INFO:tasks.workunit.client.0.smithi153.stderr:+ make 2024-04-05T03:02:35.077 INFO:tasks.workunit.client.1.smithi203.stdout:8/220: sync 2024-04-05T03:02:35.077 INFO:tasks.workunit.client.1.smithi203.stdout:5/248: sync 2024-04-05T03:02:35.077 INFO:tasks.workunit.client.1.smithi203.stdout:8/221: chown d1 8 1 2024-04-05T03:02:35.081 INFO:tasks.workunit.client.1.smithi203.stdout:8/222: creat d1/f34 x:0 0 0 2024-04-05T03:02:35.083 INFO:tasks.workunit.client.1.smithi203.stdout:8/223: read d1/d12/f31 [206139,47199] 0 2024-04-05T03:02:35.086 INFO:tasks.workunit.client.1.smithi203.stdout:8/224: mknod d1/d6/d2e/c35 0 2024-04-05T03:02:35.088 INFO:tasks.workunit.client.1.smithi203.stdout:5/249: dwrite d3/d4/d8/f3a [0,4194304] 0 2024-04-05T03:02:35.088 INFO:tasks.workunit.client.1.smithi203.stdout:8/225: symlink d1/d12/d17/d10/l36 0 2024-04-05T03:02:35.089 INFO:tasks.workunit.client.1.smithi203.stdout:8/226: fsync d1/d12/d2f/f2a 0 2024-04-05T03:02:35.097 INFO:tasks.workunit.client.1.smithi203.stdout:8/227: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:35.105 INFO:tasks.workunit.client.1.smithi203.stdout:8/228: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:35.106 INFO:tasks.workunit.client.1.smithi203.stdout:8/229: rename d1 to d1/d12/d1a/d37 22 2024-04-05T03:02:35.110 INFO:tasks.workunit.client.1.smithi203.stdout:8/230: rename d1/d6/cc to d1/d6/d2e/c38 0 2024-04-05T03:02:35.114 INFO:tasks.workunit.client.1.smithi203.stdout:8/231: rename d1/c11 to d1/d6/d2e/c39 0 2024-04-05T03:02:35.116 INFO:tasks.workunit.client.1.smithi203.stdout:8/232: dread d1/d12/d1a/f2c [0,4194304] 0 2024-04-05T03:02:35.118 INFO:tasks.workunit.client.1.smithi203.stdout:8/233: mknod d1/d12/d2f/c3a 0 2024-04-05T03:02:35.123 INFO:tasks.workunit.client.1.smithi203.stdout:8/234: getdents d1/d12/d2f 0 2024-04-05T03:02:35.128 INFO:tasks.workunit.client.1.smithi203.stdout:8/235: truncate d1/d12/d17/d10/f15 936822 0 2024-04-05T03:02:35.132 INFO:tasks.workunit.client.1.smithi203.stdout:8/236: dread d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:35.133 INFO:tasks.workunit.client.1.smithi203.stdout:8/237: symlink d1/d6/d2e/l3b 0 2024-04-05T03:02:35.135 INFO:tasks.workunit.client.1.smithi203.stdout:8/238: symlink d1/d12/l3c 0 2024-04-05T03:02:35.141 INFO:tasks.workunit.client.1.smithi203.stdout:8/239: truncate d1/f34 2313252 0 2024-04-05T03:02:35.144 INFO:tasks.workunit.client.1.smithi203.stdout:8/240: dread d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:35.147 INFO:tasks.workunit.client.1.smithi203.stdout:8/241: dread d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:35.153 INFO:tasks.workunit.client.1.smithi203.stdout:8/242: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:35.159 INFO:tasks.workunit.client.1.smithi203.stdout:8/243: chown d1/d12/d17/d10/f15 637391068 1 2024-04-05T03:02:35.161 INFO:tasks.workunit.client.1.smithi203.stdout:8/244: dread d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:35.164 INFO:tasks.workunit.client.1.smithi203.stdout:8/245: dread d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:35.170 INFO:tasks.workunit.client.1.smithi203.stdout:8/246: link d1/d6/d2e/c38 d1/d12/d17/c3d 0 2024-04-05T03:02:35.172 INFO:tasks.workunit.client.1.smithi203.stdout:8/247: dread d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:35.173 INFO:tasks.workunit.client.1.smithi203.stdout:8/248: fdatasync d1/d12/d2f/d1f/f22 0 2024-04-05T03:02:35.178 INFO:tasks.workunit.client.1.smithi203.stdout:8/249: dread d1/f34 [0,4194304] 0 2024-04-05T03:02:35.181 INFO:tasks.workunit.client.1.smithi203.stdout:8/250: unlink d1/d12/d2f/d1f/f22 0 2024-04-05T03:02:35.185 INFO:tasks.workunit.client.1.smithi203.stdout:8/251: write d1/f34 [2464617,48122] 0 2024-04-05T03:02:35.192 INFO:tasks.workunit.client.1.smithi203.stdout:8/252: dwrite d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:35.195 INFO:tasks.workunit.client.1.smithi203.stdout:8/253: mknod d1/d12/d17/c3e 0 2024-04-05T03:02:35.195 INFO:tasks.workunit.client.1.smithi203.stdout:8/254: rename d1 to d1/d12/d2f/d1f/d3f 22 2024-04-05T03:02:35.198 INFO:tasks.workunit.client.1.smithi203.stdout:8/255: creat d1/d12/d17/d10/f40 x:0 0 0 2024-04-05T03:02:35.199 INFO:tasks.workunit.client.1.smithi203.stdout:8/256: read d1/d6/d2e/f30 [54523,124908] 0 2024-04-05T03:02:35.200 INFO:tasks.workunit.client.1.smithi203.stdout:8/257: fdatasync d1/d12/d2f/f2a 0 2024-04-05T03:02:35.203 INFO:tasks.workunit.client.1.smithi203.stdout:8/258: dread d1/f34 [0,4194304] 0 2024-04-05T03:02:35.205 INFO:tasks.workunit.client.1.smithi203.stdout:8/259: creat d1/d6/f41 x:0 0 0 2024-04-05T03:02:35.207 INFO:tasks.workunit.client.1.smithi203.stdout:8/260: truncate d1/d12/f31 181104 0 2024-04-05T03:02:35.209 INFO:tasks.workunit.client.1.smithi203.stdout:8/261: dread d1/d12/d1a/f2c [0,4194304] 0 2024-04-05T03:02:35.231 INFO:tasks.workunit.client.0.smithi153.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-05T03:02:35.262 INFO:tasks.workunit.client.1.smithi203.stdout:1/137: sync 2024-04-05T03:02:35.262 INFO:tasks.workunit.client.1.smithi203.stdout:7/158: sync 2024-04-05T03:02:35.263 INFO:tasks.workunit.client.1.smithi203.stdout:1/138: chown d7/d14/f1b 125804921 1 2024-04-05T03:02:35.265 INFO:tasks.workunit.client.1.smithi203.stdout:7/159: mknod d8/d32/d16/d1a/c34 0 2024-04-05T03:02:35.267 INFO:tasks.workunit.client.1.smithi203.stdout:7/160: mkdir d8/d32/d16/d35 0 2024-04-05T03:02:35.269 INFO:tasks.workunit.client.1.smithi203.stdout:1/139: dwrite f2 [0,4194304] 0 2024-04-05T03:02:35.270 INFO:tasks.workunit.client.1.smithi203.stdout:1/140: stat d7/d14/c19 0 2024-04-05T03:02:35.274 INFO:tasks.workunit.client.1.smithi203.stdout:1/141: rename d7/d24/c2c to d7/d14/d2b/c31 0 2024-04-05T03:02:35.274 INFO:tasks.workunit.client.1.smithi203.stdout:1/142: readlink d7/l10 0 2024-04-05T03:02:35.275 INFO:tasks.workunit.client.1.smithi203.stdout:7/161: dwrite d8/ff [0,4194304] 0 2024-04-05T03:02:35.279 INFO:tasks.workunit.client.1.smithi203.stdout:1/143: link d7/l9 d7/d24/l32 0 2024-04-05T03:02:35.279 INFO:tasks.workunit.client.1.smithi203.stdout:1/144: readlink d7/d14/l20 0 2024-04-05T03:02:35.280 INFO:tasks.workunit.client.1.smithi203.stdout:1/145: chown d7/d14/f29 1194 1 2024-04-05T03:02:35.282 INFO:tasks.workunit.client.1.smithi203.stdout:1/146: dread d7/d14/f21 [0,4194304] 0 2024-04-05T03:02:35.283 INFO:tasks.workunit.client.1.smithi203.stdout:1/147: mkdir d7/d24/d33 0 2024-04-05T03:02:35.285 INFO:tasks.workunit.client.1.smithi203.stdout:1/148: creat d7/d24/f34 x:0 0 0 2024-04-05T03:02:35.285 INFO:tasks.workunit.client.1.smithi203.stdout:1/149: chown d7/d14 1556229 1 2024-04-05T03:02:35.295 INFO:tasks.workunit.client.1.smithi203.stdout:1/150: rmdir d7/d24 39 2024-04-05T03:02:35.298 INFO:tasks.workunit.client.1.smithi203.stdout:1/151: write d7/d14/f21 [1036522,43841] 0 2024-04-05T03:02:35.301 INFO:tasks.workunit.client.1.smithi203.stdout:1/152: creat d7/d24/f35 x:0 0 0 2024-04-05T03:02:35.303 INFO:tasks.workunit.client.1.smithi203.stdout:1/153: rename d7/d24/f30 to d7/d24/d33/f36 0 2024-04-05T03:02:35.309 INFO:tasks.workunit.client.1.smithi203.stdout:1/154: dwrite d7/fe [4194304,4194304] 0 2024-04-05T03:02:35.310 INFO:tasks.workunit.client.1.smithi203.stdout:1/155: write d7/d14/f29 [259280,37913] 0 2024-04-05T03:02:35.313 INFO:tasks.workunit.client.1.smithi203.stdout:1/156: symlink d7/d24/d33/l37 0 2024-04-05T03:02:35.314 INFO:tasks.workunit.client.1.smithi203.stdout:1/157: read d7/f27 [3681411,103973] 0 2024-04-05T03:02:35.316 INFO:tasks.workunit.client.1.smithi203.stdout:1/158: creat d7/d24/f38 x:0 0 0 2024-04-05T03:02:35.322 INFO:tasks.workunit.client.1.smithi203.stdout:1/159: dwrite d7/d24/f35 [0,4194304] 0 2024-04-05T03:02:35.326 INFO:tasks.workunit.client.1.smithi203.stdout:1/160: creat d7/d14/f39 x:0 0 0 2024-04-05T03:02:35.327 INFO:tasks.workunit.client.1.smithi203.stdout:1/161: write d7/f2e [264402,125305] 0 2024-04-05T03:02:35.333 INFO:tasks.workunit.client.1.smithi203.stdout:1/162: dwrite d7/d24/f34 [0,4194304] 0 2024-04-05T03:02:35.334 INFO:tasks.workunit.client.1.smithi203.stdout:1/163: chown d7/lb 2180794 1 2024-04-05T03:02:35.340 INFO:tasks.workunit.client.1.smithi203.stdout:1/164: dwrite d7/f1c [0,4194304] 0 2024-04-05T03:02:35.343 INFO:tasks.workunit.client.1.smithi203.stdout:1/165: mknod d7/d14/d2b/c3a 0 2024-04-05T03:02:35.350 INFO:tasks.workunit.client.1.smithi203.stdout:1/166: dwrite d7/d24/f38 [0,4194304] 0 2024-04-05T03:02:35.352 INFO:tasks.workunit.client.1.smithi203.stdout:1/167: chown d7/c18 175 1 2024-04-05T03:02:35.354 INFO:tasks.workunit.client.1.smithi203.stdout:1/168: read d7/d14/f29 [259419,98802] 0 2024-04-05T03:02:35.359 INFO:tasks.workunit.client.1.smithi203.stdout:1/169: link d7/cd d7/d14/c3b 0 2024-04-05T03:02:35.364 INFO:tasks.workunit.client.1.smithi203.stdout:1/170: dwrite d7/d14/f1b [0,4194304] 0 2024-04-05T03:02:35.366 INFO:tasks.workunit.client.1.smithi203.stdout:1/171: mknod d7/d14/d2b/c3c 0 2024-04-05T03:02:35.372 INFO:tasks.workunit.client.1.smithi203.stdout:1/172: write d7/fa [3620814,116742] 0 2024-04-05T03:02:35.376 INFO:tasks.workunit.client.1.smithi203.stdout:1/173: rename d7/d14/f39 to d7/f3d 0 2024-04-05T03:02:35.378 INFO:tasks.workunit.client.1.smithi203.stdout:1/174: creat d7/d24/f3e x:0 0 0 2024-04-05T03:02:35.378 INFO:tasks.workunit.client.1.smithi203.stdout:1/175: chown l5 2 1 2024-04-05T03:02:35.379 INFO:tasks.workunit.client.1.smithi203.stdout:1/176: dread - d7/d24/d33/f36 zero size 2024-04-05T03:02:35.382 INFO:tasks.workunit.client.1.smithi203.stdout:1/177: dread f2 [0,4194304] 0 2024-04-05T03:02:35.383 INFO:tasks.workunit.client.1.smithi203.stdout:1/178: creat d7/d24/d33/f3f x:0 0 0 2024-04-05T03:02:35.385 INFO:tasks.workunit.client.1.smithi203.stdout:1/179: symlink d7/d14/d2b/l40 0 2024-04-05T03:02:35.385 INFO:tasks.workunit.client.1.smithi203.stdout:1/180: read d7/d14/f1b [1987326,3721] 0 2024-04-05T03:02:35.391 INFO:tasks.workunit.client.1.smithi203.stdout:1/181: getdents d7/d14/d2b 0 2024-04-05T03:02:35.392 INFO:tasks.workunit.client.1.smithi203.stdout:1/182: write d7/d24/f38 [189809,29393] 0 2024-04-05T03:02:35.393 INFO:tasks.workunit.client.1.smithi203.stdout:1/183: rmdir d7/d14/d2b 39 2024-04-05T03:02:35.394 INFO:tasks.workunit.client.1.smithi203.stdout:1/184: write d7/d24/d33/f3f [854190,116964] 0 2024-04-05T03:02:35.402 INFO:tasks.workunit.client.1.smithi203.stdout:6/151: sync 2024-04-05T03:02:35.402 INFO:tasks.workunit.client.1.smithi203.stdout:9/150: sync 2024-04-05T03:02:35.402 INFO:tasks.workunit.client.1.smithi203.stdout:3/129: sync 2024-04-05T03:02:35.402 INFO:tasks.workunit.client.1.smithi203.stdout:0/191: sync 2024-04-05T03:02:35.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/192: stat dd/d17/d29/l2c 0 2024-04-05T03:02:35.408 INFO:tasks.workunit.client.1.smithi203.stdout:0/193: fdatasync dd/d17/d1d/d26/d28/f2d 0 2024-04-05T03:02:35.409 INFO:tasks.workunit.client.1.smithi203.stdout:6/152: dwrite d0/f8 [4194304,4194304] 0 2024-04-05T03:02:35.410 INFO:tasks.workunit.client.1.smithi203.stdout:6/153: fdatasync d0/da/f10 0 2024-04-05T03:02:35.411 INFO:tasks.workunit.client.1.smithi203.stdout:0/194: fdatasync dd/d17/d1d/f2e 0 2024-04-05T03:02:35.413 INFO:tasks.workunit.client.1.smithi203.stdout:6/154: write d0/da/fd [1023102,71526] 0 2024-04-05T03:02:35.413 INFO:tasks.workunit.client.1.smithi203.stdout:3/130: write d0/f9 [3838803,30584] 0 2024-04-05T03:02:35.415 INFO:tasks.workunit.client.1.smithi203.stdout:9/151: truncate f8 35717 0 2024-04-05T03:02:35.416 INFO:tasks.workunit.client.1.smithi203.stdout:9/152: chown d13/f1d 14896 1 2024-04-05T03:02:35.416 INFO:tasks.workunit.client.1.smithi203.stdout:0/195: dread dd/d17/d1d/d26/d28/f2b [0,4194304] 0 2024-04-05T03:02:35.420 INFO:tasks.workunit.client.1.smithi203.stdout:3/131: symlink d0/l22 0 2024-04-05T03:02:35.421 INFO:tasks.workunit.client.1.smithi203.stdout:6/155: dread d0/ff [0,4194304] 0 2024-04-05T03:02:35.423 INFO:tasks.workunit.client.1.smithi203.stdout:6/156: write d0/da/d18/f25 [1284174,15713] 0 2024-04-05T03:02:35.425 INFO:tasks.workunit.client.1.smithi203.stdout:0/196: dread dd/d17/d1d/d26/d28/f2d [0,4194304] 0 2024-04-05T03:02:35.427 INFO:tasks.workunit.client.1.smithi203.stdout:6/157: rmdir d0/d6/d2a 39 2024-04-05T03:02:35.429 INFO:tasks.workunit.client.1.smithi203.stdout:6/158: fsync d0/f9 0 2024-04-05T03:02:35.433 INFO:tasks.workunit.client.1.smithi203.stdout:0/197: mkdir dd/d17/d1d/d26/d34 0 2024-04-05T03:02:35.434 INFO:tasks.workunit.client.1.smithi203.stdout:6/159: creat d0/d6/d2b/f2d x:0 0 0 2024-04-05T03:02:35.434 INFO:tasks.workunit.client.1.smithi203.stdout:0/198: write dd/d17/d1d/d22/f23 [811302,62669] 0 2024-04-05T03:02:35.435 INFO:tasks.workunit.client.1.smithi203.stdout:0/199: chown dd/d17/d1d/d22/f23 1954607 1 2024-04-05T03:02:35.442 INFO:tasks.workunit.client.1.smithi203.stdout:3/132: dwrite d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:35.451 INFO:tasks.workunit.client.1.smithi203.stdout:0/200: link f8 dd/d17/d1d/d26/d34/f35 0 2024-04-05T03:02:35.456 INFO:tasks.workunit.client.1.smithi203.stdout:6/160: dwrite d0/d23/f28 [0,4194304] 0 2024-04-05T03:02:35.459 INFO:tasks.workunit.client.1.smithi203.stdout:3/133: dwrite d0/f4 [0,4194304] 0 2024-04-05T03:02:35.460 INFO:tasks.workunit.client.1.smithi203.stdout:3/134: stat d0/dc/d1a/l20 0 2024-04-05T03:02:35.461 INFO:tasks.workunit.client.1.smithi203.stdout:3/135: chown d0/dc/c10 43 1 2024-04-05T03:02:35.467 INFO:tasks.workunit.client.1.smithi203.stdout:6/161: dwrite d0/f1a [0,4194304] 0 2024-04-05T03:02:35.469 INFO:tasks.workunit.client.1.smithi203.stdout:3/136: truncate d0/fd 2573664 0 2024-04-05T03:02:35.470 INFO:tasks.workunit.client.1.smithi203.stdout:3/137: chown d0/c15 2830 1 2024-04-05T03:02:35.471 INFO:tasks.workunit.client.1.smithi203.stdout:6/162: unlink d0/d23/c24 0 2024-04-05T03:02:35.474 INFO:tasks.workunit.client.1.smithi203.stdout:3/138: creat d0/f23 x:0 0 0 2024-04-05T03:02:35.477 INFO:tasks.workunit.client.1.smithi203.stdout:3/139: mknod d0/d21/c24 0 2024-04-05T03:02:35.480 INFO:tasks.workunit.client.1.smithi203.stdout:3/140: mkdir d0/dc/d1a/d1f/d25 0 2024-04-05T03:02:35.480 INFO:tasks.workunit.client.1.smithi203.stdout:3/141: readlink d0/l22 0 2024-04-05T03:02:35.483 INFO:tasks.workunit.client.1.smithi203.stdout:3/142: truncate d0/f5 473266 0 2024-04-05T03:02:35.486 INFO:tasks.workunit.client.1.smithi203.stdout:6/163: truncate d0/ff 2194329 0 2024-04-05T03:02:35.489 INFO:tasks.workunit.client.1.smithi203.stdout:6/164: rmdir d0/d6/d2a 39 2024-04-05T03:02:35.489 INFO:tasks.workunit.client.1.smithi203.stdout:6/165: chown d0/f3 1935647850 1 2024-04-05T03:02:35.490 INFO:tasks.workunit.client.1.smithi203.stdout:6/166: write d0/f8 [8820371,124028] 0 2024-04-05T03:02:35.491 INFO:tasks.workunit.client.1.smithi203.stdout:6/167: fsync d0/da/d18/f1d 0 2024-04-05T03:02:35.492 INFO:tasks.workunit.client.1.smithi203.stdout:6/168: chown d0/d6/d2b/f2d 457869 1 2024-04-05T03:02:35.493 INFO:tasks.workunit.client.1.smithi203.stdout:6/169: fsync d0/f8 0 2024-04-05T03:02:35.500 INFO:tasks.workunit.client.1.smithi203.stdout:6/170: rename d0/da/c15 to d0/d6/d2a/c2e 0 2024-04-05T03:02:35.500 INFO:tasks.workunit.client.1.smithi203.stdout:3/143: truncate d0/dc/f16 2917442 0 2024-04-05T03:02:35.501 INFO:tasks.workunit.client.1.smithi203.stdout:6/171: read d0/da/d18/f1d [354926,14708] 0 2024-04-05T03:02:35.506 INFO:tasks.workunit.client.1.smithi203.stdout:6/172: truncate d0/da/d18/f26 495218 0 2024-04-05T03:02:35.511 INFO:tasks.workunit.client.1.smithi203.stdout:6/173: rename d0/c27 to d0/d6/d2b/c2f 0 2024-04-05T03:02:35.512 INFO:tasks.workunit.client.1.smithi203.stdout:6/174: write d0/f1a [3135184,30975] 0 2024-04-05T03:02:35.513 INFO:tasks.workunit.client.1.smithi203.stdout:6/175: fdatasync d0/da/d18/f25 0 2024-04-05T03:02:35.517 INFO:tasks.workunit.client.1.smithi203.stdout:6/176: dread d0/ff [0,4194304] 0 2024-04-05T03:02:35.519 INFO:tasks.workunit.client.1.smithi203.stdout:6/177: creat d0/da/d18/f30 x:0 0 0 2024-04-05T03:02:35.520 INFO:tasks.workunit.client.1.smithi203.stdout:6/178: stat d0/da/f10 0 2024-04-05T03:02:35.523 INFO:tasks.workunit.client.1.smithi203.stdout:3/144: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:35.523 INFO:tasks.workunit.client.1.smithi203.stdout:6/179: symlink d0/da/d18/l31 0 2024-04-05T03:02:35.531 INFO:tasks.workunit.client.1.smithi203.stdout:3/145: rmdir d0/dc/d1a/d1f/d25 0 2024-04-05T03:02:35.533 INFO:tasks.workunit.client.1.smithi203.stdout:3/146: symlink d0/dc/d1a/d1f/l26 0 2024-04-05T03:02:35.535 INFO:tasks.workunit.client.1.smithi203.stdout:3/147: write d0/f5 [3036394,5783] 0 2024-04-05T03:02:35.537 INFO:tasks.workunit.client.1.smithi203.stdout:6/180: dwrite d0/da/d18/f26 [0,4194304] 0 2024-04-05T03:02:35.541 INFO:tasks.workunit.client.1.smithi203.stdout:2/196: sync 2024-04-05T03:02:35.541 INFO:tasks.workunit.client.1.smithi203.stdout:6/181: mknod d0/da/c32 0 2024-04-05T03:02:35.543 INFO:tasks.workunit.client.1.smithi203.stdout:6/182: chown d0/da/c17 16023107 1 2024-04-05T03:02:35.543 INFO:tasks.workunit.client.1.smithi203.stdout:3/148: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:35.543 INFO:tasks.workunit.client.1.smithi203.stdout:2/197: fsync d5/d10/d1a/d24/f2b 0 2024-04-05T03:02:35.545 INFO:tasks.workunit.client.1.smithi203.stdout:2/198: stat d5/d10/d35/d2d/f33 0 2024-04-05T03:02:35.548 INFO:tasks.workunit.client.1.smithi203.stdout:3/149: unlink d0/dc/d1a/l20 0 2024-04-05T03:02:35.549 INFO:tasks.workunit.client.1.smithi203.stdout:6/183: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:35.552 INFO:tasks.workunit.client.1.smithi203.stdout:6/184: write d0/da/d18/f30 [283001,110849] 0 2024-04-05T03:02:35.554 INFO:tasks.workunit.client.1.smithi203.stdout:2/199: dwrite d5/d10/d35/d2d/f38 [0,4194304] 0 2024-04-05T03:02:35.558 INFO:tasks.workunit.client.1.smithi203.stdout:6/185: mknod d0/da/c33 0 2024-04-05T03:02:35.559 INFO:tasks.workunit.client.1.smithi203.stdout:2/200: fsync d5/d10/f20 0 2024-04-05T03:02:35.560 INFO:tasks.workunit.client.1.smithi203.stdout:6/186: chown d0/d6/l20 13 1 2024-04-05T03:02:35.561 INFO:tasks.workunit.client.1.smithi203.stdout:2/201: mknod d5/c42 0 2024-04-05T03:02:35.564 INFO:tasks.workunit.client.1.smithi203.stdout:2/202: creat d5/f43 x:0 0 0 2024-04-05T03:02:35.565 INFO:tasks.workunit.client.1.smithi203.stdout:3/150: dwrite d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:35.565 INFO:tasks.workunit.client.1.smithi203.stdout:6/187: truncate d0/f8 776314 0 2024-04-05T03:02:35.567 INFO:tasks.workunit.client.1.smithi203.stdout:2/203: mkdir d5/d10/d44 0 2024-04-05T03:02:35.569 INFO:tasks.workunit.client.1.smithi203.stdout:3/151: symlink d0/d21/l27 0 2024-04-05T03:02:35.575 INFO:tasks.workunit.client.1.smithi203.stdout:3/152: truncate d0/ff 3565730 0 2024-04-05T03:02:35.576 INFO:tasks.workunit.client.1.smithi203.stdout:6/188: dwrite d0/ff [0,4194304] 0 2024-04-05T03:02:35.576 INFO:tasks.workunit.client.1.smithi203.stdout:2/204: dwrite d5/d10/d1a/d24/f2b [0,4194304] 0 2024-04-05T03:02:35.577 INFO:tasks.workunit.client.1.smithi203.stdout:3/153: chown d0/l22 889 1 2024-04-05T03:02:35.577 INFO:tasks.workunit.client.1.smithi203.stdout:3/154: rename d0 to d0/dc/d28 22 2024-04-05T03:02:35.588 INFO:tasks.workunit.client.1.smithi203.stdout:2/205: mknod d5/d10/d1a/c45 0 2024-04-05T03:02:35.591 INFO:tasks.workunit.client.1.smithi203.stdout:3/155: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:35.594 INFO:tasks.workunit.client.1.smithi203.stdout:6/189: dwrite d0/da/f10 [0,4194304] 0 2024-04-05T03:02:35.597 INFO:tasks.workunit.client.1.smithi203.stdout:2/206: link l2 d5/d10/d35/d36/l46 0 2024-04-05T03:02:35.597 INFO:tasks.workunit.client.1.smithi203.stdout:3/156: creat d0/dc/d1a/d1d/f29 x:0 0 0 2024-04-05T03:02:35.599 INFO:tasks.workunit.client.1.smithi203.stdout:6/190: symlink d0/d6/d2a/d2c/l34 0 2024-04-05T03:02:35.600 INFO:tasks.workunit.client.1.smithi203.stdout:2/207: mknod d5/c47 0 2024-04-05T03:02:35.600 INFO:tasks.workunit.client.1.smithi203.stdout:6/191: fsync d0/da/d18/f25 0 2024-04-05T03:02:35.603 INFO:tasks.workunit.client.1.smithi203.stdout:3/157: getdents d0/dc/d1a/d1f 0 2024-04-05T03:02:35.605 INFO:tasks.workunit.client.1.smithi203.stdout:6/192: rename d0/d6/lc to d0/d23/l35 0 2024-04-05T03:02:35.606 INFO:tasks.workunit.client.1.smithi203.stdout:2/208: symlink d5/d10/d35/d36/d41/l48 0 2024-04-05T03:02:35.608 INFO:tasks.workunit.client.1.smithi203.stdout:2/209: rmdir d5/d10/d2f 39 2024-04-05T03:02:35.610 INFO:tasks.workunit.client.1.smithi203.stdout:2/210: write d5/d10/d1a/d24/f2b [2843359,119286] 0 2024-04-05T03:02:35.611 INFO:tasks.workunit.client.1.smithi203.stdout:6/193: link d0/l29 d0/d23/l36 0 2024-04-05T03:02:35.611 INFO:tasks.workunit.client.1.smithi203.stdout:6/194: chown d0 4422284 1 2024-04-05T03:02:35.620 INFO:tasks.workunit.client.1.smithi203.stdout:6/195: readlink d0/d23/l35 0 2024-04-05T03:02:35.620 INFO:tasks.workunit.client.1.smithi203.stdout:2/211: dwrite d5/d10/d35/d2d/f2e [0,4194304] 0 2024-04-05T03:02:35.622 INFO:tasks.workunit.client.1.smithi203.stdout:2/212: mknod d5/c49 0 2024-04-05T03:02:35.623 INFO:tasks.workunit.client.1.smithi203.stdout:6/196: chown d0/d6/d2b/c2f 3885104 1 2024-04-05T03:02:35.626 INFO:tasks.workunit.client.1.smithi203.stdout:6/197: write d0/da/fd [353933,85333] 0 2024-04-05T03:02:35.627 INFO:tasks.workunit.client.1.smithi203.stdout:6/198: chown d0/da/c33 3502 1 2024-04-05T03:02:35.631 INFO:tasks.workunit.client.1.smithi203.stdout:6/199: creat d0/f37 x:0 0 0 2024-04-05T03:02:35.632 INFO:tasks.workunit.client.1.smithi203.stdout:2/213: dwrite d5/d10/d35/d2d/f38 [0,4194304] 0 2024-04-05T03:02:35.632 INFO:tasks.workunit.client.1.smithi203.stdout:3/158: dwrite d0/ff [0,4194304] 0 2024-04-05T03:02:35.634 INFO:tasks.workunit.client.1.smithi203.stdout:2/214: chown d5/d10/d23/f25 49 1 2024-04-05T03:02:35.637 INFO:tasks.workunit.client.1.smithi203.stdout:3/159: unlink d0/dc/f11 0 2024-04-05T03:02:35.646 INFO:tasks.workunit.client.1.smithi203.stdout:2/215: dwrite d5/f8 [0,4194304] 0 2024-04-05T03:02:35.649 INFO:tasks.workunit.client.1.smithi203.stdout:2/216: symlink d5/d10/d44/l4a 0 2024-04-05T03:02:35.651 INFO:tasks.workunit.client.1.smithi203.stdout:2/217: creat d5/d10/d1a/f4b x:0 0 0 2024-04-05T03:02:35.653 INFO:tasks.workunit.client.1.smithi203.stdout:2/218: write d5/d10/d35/d36/f3b [898604,102449] 0 2024-04-05T03:02:35.654 INFO:tasks.workunit.client.1.smithi203.stdout:2/219: fdatasync d5/d10/d1a/f4b 0 2024-04-05T03:02:35.658 INFO:tasks.workunit.client.1.smithi203.stdout:2/220: link d5/c42 d5/d10/d35/c4c 0 2024-04-05T03:02:35.658 INFO:tasks.workunit.client.1.smithi203.stdout:2/221: chown d5/f40 2086 1 2024-04-05T03:02:35.661 INFO:tasks.workunit.client.1.smithi203.stdout:6/200: write d0/f8 [273853,109006] 0 2024-04-05T03:02:35.661 INFO:tasks.workunit.client.1.smithi203.stdout:3/160: truncate d0/f5 2691154 0 2024-04-05T03:02:35.662 INFO:tasks.workunit.client.1.smithi203.stdout:6/201: write d0/f3 [1317459,83281] 0 2024-04-05T03:02:35.663 INFO:tasks.workunit.client.1.smithi203.stdout:6/202: fsync d0/da/f13 0 2024-04-05T03:02:35.665 INFO:tasks.workunit.client.1.smithi203.stdout:3/161: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:35.667 INFO:tasks.workunit.client.1.smithi203.stdout:6/203: rename d0/d6 to d0/da/d18/d38 0 2024-04-05T03:02:35.668 INFO:tasks.workunit.client.1.smithi203.stdout:2/222: fdatasync d5/f8 0 2024-04-05T03:02:35.668 INFO:tasks.workunit.client.1.smithi203.stdout:6/204: chown d0/f3 237699852 1 2024-04-05T03:02:35.675 INFO:tasks.workunit.client.1.smithi203.stdout:6/205: unlink d0/da/d18/d38/d2a/c2e 0 2024-04-05T03:02:35.678 INFO:tasks.workunit.client.1.smithi203.stdout:6/206: write d0/da/d18/f26 [2539430,113016] 0 2024-04-05T03:02:35.679 INFO:tasks.workunit.client.1.smithi203.stdout:2/223: dwrite d5/d10/f1d [0,4194304] 0 2024-04-05T03:02:35.681 INFO:tasks.workunit.client.1.smithi203.stdout:6/207: write d0/da/d18/f30 [660780,73909] 0 2024-04-05T03:02:35.685 INFO:tasks.workunit.client.1.smithi203.stdout:2/224: chown d5/d10/d1a/f34 1131104 1 2024-04-05T03:02:35.685 INFO:tasks.workunit.client.0.smithi153.stderr:++ readlink -f fsstress 2024-04-05T03:02:35.688 INFO:tasks.workunit.client.1.smithi203.stdout:6/208: creat d0/da/d18/d38/d2a/d2c/f39 x:0 0 0 2024-04-05T03:02:35.689 INFO:tasks.workunit.client.0.smithi153.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-05T03:02:35.689 INFO:tasks.workunit.client.0.smithi153.stderr:+ popd 2024-04-05T03:02:35.689 INFO:tasks.workunit.client.1.smithi203.stdout:6/209: dread - d0/f37 zero size 2024-04-05T03:02:35.690 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T03:02:35.691 INFO:tasks.workunit.client.0.smithi153.stderr:+ popd 2024-04-05T03:02:35.691 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-05T03:02:35.692 INFO:tasks.workunit.client.0.smithi153.stderr:++ mktemp -d -p . 2024-04-05T03:02:35.693 INFO:tasks.workunit.client.1.smithi203.stdout:2/225: mknod d5/d10/c4d 0 2024-04-05T03:02:35.696 INFO:tasks.workunit.client.0.smithi153.stderr:+ T=./tmp.poLuWgRt5g 2024-04-05T03:02:35.696 INFO:tasks.workunit.client.0.smithi153.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.poLuWgRt5g -l 1 -n 1000 -p 10 -v 2024-04-05T03:02:35.696 INFO:tasks.workunit.client.1.smithi203.stdout:2/226: rmdir d5/d10/d35/d2d 39 2024-04-05T03:02:35.698 INFO:tasks.workunit.client.1.smithi203.stdout:2/227: mknod d5/d10/d2f/c4e 0 2024-04-05T03:02:35.698 INFO:tasks.workunit.client.1.smithi203.stdout:6/210: getdents d0/da/d18/d38/d2a 0 2024-04-05T03:02:35.699 INFO:tasks.workunit.client.0.smithi153.stdout:seed = 1711659713 2024-04-05T03:02:35.700 INFO:tasks.workunit.client.1.smithi203.stdout:6/211: creat d0/da/d18/d38/d2a/d2c/f3a x:0 0 0 2024-04-05T03:02:35.703 INFO:tasks.workunit.client.1.smithi203.stdout:6/212: symlink d0/da/d18/d38/d2a/d2c/l3b 0 2024-04-05T03:02:35.704 INFO:tasks.workunit.client.1.smithi203.stdout:2/228: getdents d5/d10/d35/d2d 0 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/0: dread - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/1: dread - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/2: dwrite - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/3: write - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/4: chown . 41463319 1 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/5: link - no file 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/6: rmdir - no directory 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/7: truncate - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/8: fsync - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/9: rename - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/10: dread - no filename 2024-04-05T03:02:35.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/11: write - no filename 2024-04-05T03:02:35.706 INFO:tasks.workunit.client.0.smithi153.stdout:0/12: dwrite - no filename 2024-04-05T03:02:35.706 INFO:tasks.workunit.client.1.smithi203.stdout:6/213: symlink d0/da/d18/d38/d2a/l3c 0 2024-04-05T03:02:35.706 INFO:tasks.workunit.client.0.smithi153.stdout:1/0: dread - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/1: fsync - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/2: truncate - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/3: dwrite - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/4: rmdir - no directory 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/5: truncate - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/6: readlink - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/7: dread - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/8: readlink - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/9: dwrite - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/10: write - no filename 2024-04-05T03:02:35.707 INFO:tasks.workunit.client.0.smithi153.stdout:0/13: creat f0 x:0 0 0 2024-04-05T03:02:35.708 INFO:tasks.workunit.client.1.smithi203.stdout:6/214: fdatasync d0/da/d18/d38/d2a/d2c/f3a 0 2024-04-05T03:02:35.709 INFO:tasks.workunit.client.0.smithi153.stdout:0/14: symlink l1 0 2024-04-05T03:02:35.709 INFO:tasks.workunit.client.0.smithi153.stdout:1/11: symlink l0 0 2024-04-05T03:02:35.710 INFO:tasks.workunit.client.0.smithi153.stdout:0/15: chown l1 1 1 2024-04-05T03:02:35.710 INFO:tasks.workunit.client.0.smithi153.stdout:3/0: rmdir - no directory 2024-04-05T03:02:35.711 INFO:tasks.workunit.client.0.smithi153.stdout:2/0: symlink l0 0 2024-04-05T03:02:35.711 INFO:tasks.workunit.client.0.smithi153.stdout:2/1: write - no filename 2024-04-05T03:02:35.711 INFO:tasks.workunit.client.1.smithi203.stdout:6/215: rmdir d0/da/d18/d38/d2a 39 2024-04-05T03:02:35.711 INFO:tasks.workunit.client.0.smithi153.stdout:1/12: mknod c1 0 2024-04-05T03:02:35.713 INFO:tasks.workunit.client.0.smithi153.stdout:4/0: rename - no filename 2024-04-05T03:02:35.713 INFO:tasks.workunit.client.0.smithi153.stdout:0/16: creat f2 x:0 0 0 2024-04-05T03:02:35.713 INFO:tasks.workunit.client.0.smithi153.stdout:4/1: chown . 14839298 1 2024-04-05T03:02:35.714 INFO:tasks.workunit.client.0.smithi153.stdout:0/17: dread - f2 zero size 2024-04-05T03:02:35.714 INFO:tasks.workunit.client.0.smithi153.stdout:2/2: symlink l1 0 2024-04-05T03:02:35.715 INFO:tasks.workunit.client.1.smithi203.stdout:2/229: dread d5/d10/d1a/d24/f28 [0,4194304] 0 2024-04-05T03:02:35.715 INFO:tasks.workunit.client.0.smithi153.stdout:1/13: creat f2 x:0 0 0 2024-04-05T03:02:35.717 INFO:tasks.workunit.client.0.smithi153.stdout:3/1: mknod c0 0 2024-04-05T03:02:35.717 INFO:tasks.workunit.client.0.smithi153.stdout:3/2: stat c0 0 2024-04-05T03:02:35.719 INFO:tasks.workunit.client.1.smithi203.stdout:2/230: dread d5/f8 [0,4194304] 0 2024-04-05T03:02:35.721 INFO:tasks.workunit.client.0.smithi153.stdout:2/3: symlink l2 0 2024-04-05T03:02:35.722 INFO:tasks.workunit.client.0.smithi153.stdout:0/18: creat f3 x:0 0 0 2024-04-05T03:02:35.722 INFO:tasks.workunit.client.1.smithi203.stdout:6/216: getdents d0/da/d18/d38/d2a/d2c 0 2024-04-05T03:02:35.723 INFO:tasks.workunit.client.0.smithi153.stdout:0/19: read - f3 zero size 2024-04-05T03:02:35.723 INFO:tasks.workunit.client.0.smithi153.stdout:0/20: chown l1 13121 1 2024-04-05T03:02:35.723 INFO:tasks.workunit.client.0.smithi153.stdout:0/21: fdatasync f3 0 2024-04-05T03:02:35.724 INFO:tasks.workunit.client.0.smithi153.stdout:0/22: readlink l1 0 2024-04-05T03:02:35.724 INFO:tasks.workunit.client.0.smithi153.stdout:0/23: readlink l1 0 2024-04-05T03:02:35.726 INFO:tasks.workunit.client.1.smithi203.stdout:6/217: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:35.727 INFO:tasks.workunit.client.0.smithi153.stdout:6/0: rename - no filename 2024-04-05T03:02:35.727 INFO:tasks.workunit.client.0.smithi153.stdout:6/1: dread - no filename 2024-04-05T03:02:35.728 INFO:tasks.workunit.client.0.smithi153.stdout:3/3: rename c0 to c1 0 2024-04-05T03:02:35.728 INFO:tasks.workunit.client.0.smithi153.stdout:3/4: dwrite - no filename 2024-04-05T03:02:35.728 INFO:tasks.workunit.client.0.smithi153.stdout:3/5: dwrite - no filename 2024-04-05T03:02:35.728 INFO:tasks.workunit.client.0.smithi153.stdout:4/2: symlink l0 0 2024-04-05T03:02:35.729 INFO:tasks.workunit.client.0.smithi153.stdout:2/4: symlink l3 0 2024-04-05T03:02:35.730 INFO:tasks.workunit.client.1.smithi203.stdout:6/218: mkdir d0/da/d18/d38/d2a/d2c/d3d 0 2024-04-05T03:02:35.730 INFO:tasks.workunit.client.1.smithi203.stdout:2/231: truncate d5/d10/d1a/d24/f2b 1241437 0 2024-04-05T03:02:35.731 INFO:tasks.workunit.client.0.smithi153.stdout:0/24: symlink l4 0 2024-04-05T03:02:35.731 INFO:tasks.workunit.client.1.smithi203.stdout:2/232: chown d5/d10/d1a/l3a 6476146 1 2024-04-05T03:02:35.732 INFO:tasks.workunit.client.0.smithi153.stdout:5/0: mkdir d0 0 2024-04-05T03:02:35.732 INFO:tasks.workunit.client.0.smithi153.stdout:5/1: dread - no filename 2024-04-05T03:02:35.732 INFO:tasks.workunit.client.0.smithi153.stdout:5/2: write - no filename 2024-04-05T03:02:35.733 INFO:tasks.workunit.client.1.smithi203.stdout:6/219: mknod d0/da/d18/d38/d2a/d2c/d3d/c3e 0 2024-04-05T03:02:35.734 INFO:tasks.workunit.client.0.smithi153.stdout:7/0: fsync - no filename 2024-04-05T03:02:35.735 INFO:tasks.workunit.client.0.smithi153.stdout:7/1: chown . 660782903 1 2024-04-05T03:02:35.735 INFO:tasks.workunit.client.0.smithi153.stdout:7/2: dwrite - no filename 2024-04-05T03:02:35.735 INFO:tasks.workunit.client.0.smithi153.stdout:7/3: dread - no filename 2024-04-05T03:02:35.735 INFO:tasks.workunit.client.1.smithi203.stdout:2/233: creat d5/d10/d2f/f4f x:0 0 0 2024-04-05T03:02:35.738 INFO:tasks.workunit.client.1.smithi203.stdout:6/220: rename d0/da/d18/d38/d2a/d2c/f39 to d0/da/d18/d38/d2a/d2c/d3d/f3f 0 2024-04-05T03:02:35.738 INFO:tasks.workunit.client.0.smithi153.stdout:4/3: rename l0 to l1 0 2024-04-05T03:02:35.738 INFO:tasks.workunit.client.0.smithi153.stdout:4/4: truncate - no filename 2024-04-05T03:02:35.738 INFO:tasks.workunit.client.0.smithi153.stdout:4/5: dread - no filename 2024-04-05T03:02:35.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/6: write - no filename 2024-04-05T03:02:35.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/7: dread - no filename 2024-04-05T03:02:35.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/8: dread - no filename 2024-04-05T03:02:35.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/9: rmdir - no directory 2024-04-05T03:02:35.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/10: dread - no filename 2024-04-05T03:02:35.740 INFO:tasks.workunit.client.0.smithi153.stdout:2/5: rename l0 to l4 0 2024-04-05T03:02:35.741 INFO:tasks.workunit.client.0.smithi153.stdout:2/6: chown l2 858592546 1 2024-04-05T03:02:35.741 INFO:tasks.workunit.client.0.smithi153.stdout:2/7: dread - no filename 2024-04-05T03:02:35.741 INFO:tasks.workunit.client.0.smithi153.stdout:0/25: creat f5 x:0 0 0 2024-04-05T03:02:35.742 INFO:tasks.workunit.client.0.smithi153.stdout:0/26: read - f0 zero size 2024-04-05T03:02:35.742 INFO:tasks.workunit.client.1.smithi203.stdout:2/234: rename d5/f40 to d5/d10/d35/d2d/f50 0 2024-04-05T03:02:35.743 INFO:tasks.workunit.client.0.smithi153.stdout:0/27: read - f0 zero size 2024-04-05T03:02:35.743 INFO:tasks.workunit.client.0.smithi153.stdout:0/28: rmdir - no directory 2024-04-05T03:02:35.743 INFO:tasks.workunit.client.0.smithi153.stdout:0/29: truncate f2 1009895 0 2024-04-05T03:02:35.744 INFO:tasks.workunit.client.0.smithi153.stdout:8/0: read - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/1: write - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/2: stat - no entries 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/3: fsync - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/4: fsync - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/5: write - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/6: rename - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/7: chown . 6986073 1 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/8: dwrite - no filename 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/9: rmdir - no directory 2024-04-05T03:02:35.745 INFO:tasks.workunit.client.1.smithi203.stdout:2/235: write d5/d10/d1a/d24/f27 [6350907,44466] 0 2024-04-05T03:02:35.746 INFO:tasks.workunit.client.0.smithi153.stdout:6/2: creat f0 x:0 0 0 2024-04-05T03:02:35.747 INFO:tasks.workunit.client.1.smithi203.stdout:6/221: rename d0/da/d18/d38/l14 to d0/da/l40 0 2024-04-05T03:02:35.747 INFO:tasks.workunit.client.0.smithi153.stdout:6/3: dread - f0 zero size 2024-04-05T03:02:35.748 INFO:tasks.workunit.client.0.smithi153.stdout:6/4: truncate f0 178951 0 2024-04-05T03:02:35.749 INFO:tasks.workunit.client.0.smithi153.stdout:6/5: dread f0 [0,4194304] 0 2024-04-05T03:02:35.750 INFO:tasks.workunit.client.0.smithi153.stdout:6/6: write f0 [640767,18729] 0 2024-04-05T03:02:35.750 INFO:tasks.workunit.client.1.smithi203.stdout:6/222: creat d0/f41 x:0 0 0 2024-04-05T03:02:35.752 INFO:tasks.workunit.client.0.smithi153.stdout:4/11: mknod c2 0 2024-04-05T03:02:35.753 INFO:tasks.workunit.client.1.smithi203.stdout:2/236: dwrite d5/d10/f1d [0,4194304] 0 2024-04-05T03:02:35.757 INFO:tasks.workunit.client.0.smithi153.stdout:2/8: mknod c5 0 2024-04-05T03:02:35.757 INFO:tasks.workunit.client.0.smithi153.stdout:2/9: dwrite - no filename 2024-04-05T03:02:35.757 INFO:tasks.workunit.client.1.smithi203.stdout:6/223: symlink d0/da/d18/d38/d2a/d2c/l42 0 2024-04-05T03:02:35.759 INFO:tasks.workunit.client.0.smithi153.stdout:5/3: creat d0/f1 x:0 0 0 2024-04-05T03:02:35.762 INFO:tasks.workunit.client.1.smithi203.stdout:2/237: dread d5/d10/d23/f25 [0,4194304] 0 2024-04-05T03:02:35.764 INFO:tasks.workunit.client.0.smithi153.stdout:0/30: symlink l6 0 2024-04-05T03:02:35.768 INFO:tasks.workunit.client.0.smithi153.stdout:9/0: rename - no filename 2024-04-05T03:02:35.768 INFO:tasks.workunit.client.1.smithi203.stdout:6/224: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:35.776 INFO:tasks.workunit.client.0.smithi153.stdout:7/4: symlink l0 0 2024-04-05T03:02:35.777 INFO:tasks.workunit.client.1.smithi203.stdout:2/238: dwrite d5/d10/d1a/d24/f3f [0,4194304] 0 2024-04-05T03:02:35.777 INFO:tasks.workunit.client.1.smithi203.stdout:6/225: creat d0/da/f43 x:0 0 0 2024-04-05T03:02:35.780 INFO:tasks.workunit.client.1.smithi203.stdout:6/226: write d0/f1a [3194447,50778] 0 2024-04-05T03:02:35.781 INFO:tasks.workunit.client.0.smithi153.stdout:6/7: mknod c1 0 2024-04-05T03:02:35.784 INFO:tasks.workunit.client.1.smithi203.stdout:2/239: unlink d5/c14 0 2024-04-05T03:02:35.785 INFO:tasks.workunit.client.0.smithi153.stdout:2/10: mknod c6 0 2024-04-05T03:02:35.785 INFO:tasks.workunit.client.0.smithi153.stdout:2/11: rmdir - no directory 2024-04-05T03:02:35.787 INFO:tasks.workunit.client.1.smithi203.stdout:6/227: mkdir d0/da/d18/d38/d2a/d44 0 2024-04-05T03:02:35.788 INFO:tasks.workunit.client.1.smithi203.stdout:6/228: fdatasync d0/da/d18/d38/d2a/d2c/f3a 0 2024-04-05T03:02:35.790 INFO:tasks.workunit.client.0.smithi153.stdout:5/4: symlink d0/l2 0 2024-04-05T03:02:35.792 INFO:tasks.workunit.client.0.smithi153.stdout:5/5: chown d0/f1 6 1 2024-04-05T03:02:35.793 INFO:tasks.workunit.client.1.smithi203.stdout:2/240: mknod d5/d10/d1a/d24/c51 0 2024-04-05T03:02:35.793 INFO:tasks.workunit.client.1.smithi203.stdout:6/229: unlink d0/da/c33 0 2024-04-05T03:02:35.794 INFO:tasks.workunit.client.0.smithi153.stdout:7/5: symlink l1 0 2024-04-05T03:02:35.794 INFO:tasks.workunit.client.0.smithi153.stdout:7/6: fdatasync - no filename 2024-04-05T03:02:35.795 INFO:tasks.workunit.client.1.smithi203.stdout:2/241: write d5/d10/d35/d2d/f50 [8535308,129889] 0 2024-04-05T03:02:35.796 INFO:tasks.workunit.client.1.smithi203.stdout:6/230: creat d0/da/d18/d38/d2a/d2c/d3d/f45 x:0 0 0 2024-04-05T03:02:35.796 INFO:tasks.workunit.client.0.smithi153.stdout:8/10: creat f0 x:0 0 0 2024-04-05T03:02:35.797 INFO:tasks.workunit.client.1.smithi203.stdout:2/242: mknod d5/d10/d44/c52 0 2024-04-05T03:02:35.797 INFO:tasks.workunit.client.0.smithi153.stdout:6/8: mknod c2 0 2024-04-05T03:02:35.798 INFO:tasks.workunit.client.1.smithi203.stdout:2/243: chown d5/d10/d35/d2d/f2e 176892479 1 2024-04-05T03:02:35.799 INFO:tasks.workunit.client.1.smithi203.stdout:6/231: readlink d0/da/l40 0 2024-04-05T03:02:35.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/12: mknod c7 0 2024-04-05T03:02:35.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/13: read - no filename 2024-04-05T03:02:35.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/14: dread - no filename 2024-04-05T03:02:35.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/15: fdatasync - no filename 2024-04-05T03:02:35.802 INFO:tasks.workunit.client.1.smithi203.stdout:6/232: creat d0/da/d18/d38/d2a/d2c/f46 x:0 0 0 2024-04-05T03:02:35.802 INFO:tasks.workunit.client.1.smithi203.stdout:2/244: dread d5/d10/d35/d2d/f2e [0,4194304] 0 2024-04-05T03:02:35.803 INFO:tasks.workunit.client.0.smithi153.stdout:5/6: creat d0/f3 x:0 0 0 2024-04-05T03:02:35.803 INFO:tasks.workunit.client.0.smithi153.stdout:5/7: dread - d0/f1 zero size 2024-04-05T03:02:35.806 INFO:tasks.workunit.client.0.smithi153.stdout:9/1: mknod c0 0 2024-04-05T03:02:35.806 INFO:tasks.workunit.client.0.smithi153.stdout:9/2: write - no filename 2024-04-05T03:02:35.806 INFO:tasks.workunit.client.0.smithi153.stdout:9/3: chown c0 2 1 2024-04-05T03:02:35.808 INFO:tasks.workunit.client.0.smithi153.stdout:7/7: mkdir d2 0 2024-04-05T03:02:35.808 INFO:tasks.workunit.client.0.smithi153.stdout:7/8: write - no filename 2024-04-05T03:02:35.809 INFO:tasks.workunit.client.1.smithi203.stdout:6/233: rename d0/f8 to d0/da/d18/d38/f47 0 2024-04-05T03:02:35.810 INFO:tasks.workunit.client.1.smithi203.stdout:2/245: dwrite d5/f43 [0,4194304] 0 2024-04-05T03:02:35.814 INFO:tasks.workunit.client.1.smithi203.stdout:6/234: creat d0/da/d18/f48 x:0 0 0 2024-04-05T03:02:35.815 INFO:tasks.workunit.client.0.smithi153.stdout:8/11: mkdir d1 0 2024-04-05T03:02:35.816 INFO:tasks.workunit.client.1.smithi203.stdout:6/235: write d0/da/d18/f25 [1497041,74799] 0 2024-04-05T03:02:35.817 INFO:tasks.workunit.client.1.smithi203.stdout:2/246: dread d5/d10/d1a/d24/f27 [4194304,4194304] 0 2024-04-05T03:02:35.820 INFO:tasks.workunit.client.0.smithi153.stdout:2/16: creat f8 x:0 0 0 2024-04-05T03:02:35.822 INFO:tasks.workunit.client.1.smithi203.stdout:6/236: mknod d0/da/d18/d38/d2a/d44/c49 0 2024-04-05T03:02:35.822 INFO:tasks.workunit.client.0.smithi153.stdout:5/8: symlink d0/l4 0 2024-04-05T03:02:35.823 INFO:tasks.workunit.client.0.smithi153.stdout:9/4: mkdir d1 0 2024-04-05T03:02:35.824 INFO:tasks.workunit.client.0.smithi153.stdout:9/5: chown d1 1638176936 1 2024-04-05T03:02:35.826 INFO:tasks.workunit.client.0.smithi153.stdout:8/12: dwrite f0 [0,4194304] 0 2024-04-05T03:02:35.826 INFO:tasks.workunit.client.1.smithi203.stdout:6/237: creat d0/da/d18/d38/d2a/d44/f4a x:0 0 0 2024-04-05T03:02:35.826 INFO:tasks.workunit.client.1.smithi203.stdout:6/238: dread - d0/da/d18/d38/d2a/d2c/d3d/f45 zero size 2024-04-05T03:02:35.827 INFO:tasks.workunit.client.1.smithi203.stdout:6/239: dread - d0/da/d18/d38/d2b/f2d zero size 2024-04-05T03:02:35.827 INFO:tasks.workunit.client.1.smithi203.stdout:6/240: stat d0/f9 0 2024-04-05T03:02:35.829 INFO:tasks.workunit.client.0.smithi153.stdout:2/17: mkdir d9 0 2024-04-05T03:02:35.832 INFO:tasks.workunit.client.0.smithi153.stdout:7/9: creat d2/f3 x:0 0 0 2024-04-05T03:02:35.835 INFO:tasks.workunit.client.0.smithi153.stdout:8/13: dwrite f0 [0,4194304] 0 2024-04-05T03:02:35.836 INFO:tasks.workunit.client.0.smithi153.stdout:8/14: rename d1 to d1/d2 22 2024-04-05T03:02:35.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/241: link d0/ff d0/da/d18/d38/d2a/d2c/d3d/f4b 0 2024-04-05T03:02:35.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/242: chown d0/f1a 30140284 1 2024-04-05T03:02:35.840 INFO:tasks.workunit.client.0.smithi153.stdout:7/10: mknod d2/c4 0 2024-04-05T03:02:35.841 INFO:tasks.workunit.client.0.smithi153.stdout:7/11: dread - d2/f3 zero size 2024-04-05T03:02:35.841 INFO:tasks.workunit.client.0.smithi153.stdout:7/12: dread - d2/f3 zero size 2024-04-05T03:02:35.842 INFO:tasks.workunit.client.0.smithi153.stdout:7/13: write d2/f3 [238423,21475] 0 2024-04-05T03:02:35.843 INFO:tasks.workunit.client.0.smithi153.stdout:7/14: stat d2 0 2024-04-05T03:02:35.844 INFO:tasks.workunit.client.1.smithi203.stdout:6/243: getdents d0/da/d18/d38/d2b 0 2024-04-05T03:02:35.844 INFO:tasks.workunit.client.1.smithi203.stdout:8/262: sync 2024-04-05T03:02:35.844 INFO:tasks.workunit.client.1.smithi203.stdout:5/250: sync 2024-04-05T03:02:35.845 INFO:tasks.workunit.client.1.smithi203.stdout:4/123: sync 2024-04-05T03:02:35.845 INFO:tasks.workunit.client.0.smithi153.stdout:7/15: write d2/f3 [789107,125642] 0 2024-04-05T03:02:35.845 INFO:tasks.workunit.client.0.smithi153.stdout:2/18: mkdir d9/da 0 2024-04-05T03:02:35.846 INFO:tasks.workunit.client.0.smithi153.stdout:2/19: write f8 [771211,80225] 0 2024-04-05T03:02:35.847 INFO:tasks.workunit.client.0.smithi153.stdout:9/6: link c0 d1/c2 0 2024-04-05T03:02:35.850 INFO:tasks.workunit.client.0.smithi153.stdout:7/16: creat d2/f5 x:0 0 0 2024-04-05T03:02:35.853 INFO:tasks.workunit.client.0.smithi153.stdout:7/17: creat d2/f6 x:0 0 0 2024-04-05T03:02:35.855 INFO:tasks.workunit.client.1.smithi203.stdout:6/244: getdents d0/da/d18/d38/d2a 0 2024-04-05T03:02:35.856 INFO:tasks.workunit.client.1.smithi203.stdout:6/245: read - d0/da/d18/d38/d2a/d2c/d3d/f45 zero size 2024-04-05T03:02:35.860 INFO:tasks.workunit.client.1.smithi203.stdout:6/246: mkdir d0/da/d18/d38/d2a/d2c/d3d/d4c 0 2024-04-05T03:02:35.862 INFO:tasks.workunit.client.0.smithi153.stdout:2/20: creat d9/da/fb x:0 0 0 2024-04-05T03:02:35.863 INFO:tasks.workunit.client.1.smithi203.stdout:6/247: mknod d0/da/d18/d38/d2a/c4d 0 2024-04-05T03:02:35.864 INFO:tasks.workunit.client.0.smithi153.stdout:9/7: mknod d1/c3 0 2024-04-05T03:02:35.865 INFO:tasks.workunit.client.1.smithi203.stdout:6/248: write d0/da/fd [969153,46045] 0 2024-04-05T03:02:35.867 INFO:tasks.workunit.client.1.smithi203.stdout:4/124: dwrite da/f28 [0,4194304] 0 2024-04-05T03:02:35.869 INFO:tasks.workunit.client.1.smithi203.stdout:4/125: read da/f28 [2520682,40666] 0 2024-04-05T03:02:35.871 INFO:tasks.workunit.client.0.smithi153.stdout:2/21: creat d9/da/fc x:0 0 0 2024-04-05T03:02:35.872 INFO:tasks.workunit.client.0.smithi153.stdout:2/22: write d9/da/fb [218455,69152] 0 2024-04-05T03:02:35.873 INFO:tasks.workunit.client.0.smithi153.stdout:9/8: creat d1/f4 x:0 0 0 2024-04-05T03:02:35.877 INFO:tasks.workunit.client.1.smithi203.stdout:4/126: mknod da/c2a 0 2024-04-05T03:02:35.879 INFO:tasks.workunit.client.1.smithi203.stdout:8/263: write d1/d12/d17/d10/f15 [1722199,123741] 0 2024-04-05T03:02:35.879 INFO:tasks.workunit.client.0.smithi153.stdout:7/18: link l0 d2/l7 0 2024-04-05T03:02:35.883 INFO:tasks.workunit.client.0.smithi153.stdout:7/19: write d2/f3 [1487340,122220] 0 2024-04-05T03:02:35.884 INFO:tasks.workunit.client.0.smithi153.stdout:2/23: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:35.885 INFO:tasks.workunit.client.0.smithi153.stdout:7/20: truncate d2/f6 509276 0 2024-04-05T03:02:35.885 INFO:tasks.workunit.client.1.smithi203.stdout:8/264: creat d1/d12/d1a/f42 x:0 0 0 2024-04-05T03:02:35.885 INFO:tasks.workunit.client.0.smithi153.stdout:7/21: chown d2/c4 543793 1 2024-04-05T03:02:35.887 INFO:tasks.workunit.client.1.smithi203.stdout:4/127: dwrite da/f27 [0,4194304] 0 2024-04-05T03:02:35.887 INFO:tasks.workunit.client.1.smithi203.stdout:5/251: dread d3/f7 [0,4194304] 0 2024-04-05T03:02:35.889 INFO:tasks.workunit.client.1.smithi203.stdout:6/249: truncate d0/ff 829010 0 2024-04-05T03:02:35.889 INFO:tasks.workunit.client.0.smithi153.stdout:9/9: rename c0 to d1/c5 0 2024-04-05T03:02:35.890 INFO:tasks.workunit.client.1.smithi203.stdout:5/252: write d3/d4/dc/f37 [202113,86677] 0 2024-04-05T03:02:35.891 INFO:tasks.workunit.client.1.smithi203.stdout:6/250: write d0/da/f43 [124508,89127] 0 2024-04-05T03:02:35.891 INFO:tasks.workunit.client.1.smithi203.stdout:8/265: symlink d1/d12/d2f/d1f/l43 0 2024-04-05T03:02:35.893 INFO:tasks.workunit.client.1.smithi203.stdout:8/266: stat d1/d12/d17 0 2024-04-05T03:02:35.893 INFO:tasks.workunit.client.1.smithi203.stdout:5/253: write d3/d4/dc/f2d [813099,56797] 0 2024-04-05T03:02:35.895 INFO:tasks.workunit.client.0.smithi153.stdout:9/10: dwrite d1/f4 [0,4194304] 0 2024-04-05T03:02:35.896 INFO:tasks.workunit.client.0.smithi153.stdout:9/11: readlink - no filename 2024-04-05T03:02:35.901 INFO:tasks.workunit.client.1.smithi203.stdout:4/128: read da/f22 [4114054,50058] 0 2024-04-05T03:02:35.902 INFO:tasks.workunit.client.1.smithi203.stdout:4/129: dread - da/f18 zero size 2024-04-05T03:02:35.904 INFO:tasks.workunit.client.1.smithi203.stdout:6/251: dwrite d0/f41 [0,4194304] 0 2024-04-05T03:02:35.907 INFO:tasks.workunit.client.0.smithi153.stdout:9/12: mknod d1/c6 0 2024-04-05T03:02:35.908 INFO:tasks.workunit.client.1.smithi203.stdout:8/267: dwrite d1/d12/d17/d10/f40 [0,4194304] 0 2024-04-05T03:02:35.911 INFO:tasks.workunit.client.0.smithi153.stdout:9/13: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:35.912 INFO:tasks.workunit.client.0.smithi153.stdout:9/14: write d1/f4 [3320792,33996] 0 2024-04-05T03:02:35.914 INFO:tasks.workunit.client.1.smithi203.stdout:5/254: dwrite d3/d1f/f1e [0,4194304] 0 2024-04-05T03:02:35.915 INFO:tasks.workunit.client.1.smithi203.stdout:8/268: write d1/f34 [664142,34031] 0 2024-04-05T03:02:35.927 INFO:tasks.workunit.client.1.smithi203.stdout:8/269: symlink d1/l44 0 2024-04-05T03:02:35.928 INFO:tasks.workunit.client.1.smithi203.stdout:8/270: stat d1/d6/d2e/c35 0 2024-04-05T03:02:35.949 INFO:tasks.workunit.client.0.smithi153.stdout:1/14: fsync f2 0 2024-04-05T03:02:35.951 INFO:tasks.workunit.client.0.smithi153.stdout:1/15: creat f3 x:0 0 0 2024-04-05T03:02:35.951 INFO:tasks.workunit.client.0.smithi153.stdout:1/16: dread - f2 zero size 2024-04-05T03:02:35.952 INFO:tasks.workunit.client.0.smithi153.stdout:1/17: truncate f3 1015690 0 2024-04-05T03:02:35.953 INFO:tasks.workunit.client.0.smithi153.stdout:3/6: getdents . 0 2024-04-05T03:02:35.958 INFO:tasks.workunit.client.0.smithi153.stdout:4/12: readlink l1 0 2024-04-05T03:02:35.958 INFO:tasks.workunit.client.0.smithi153.stdout:3/7: mknod c2 0 2024-04-05T03:02:35.959 INFO:tasks.workunit.client.0.smithi153.stdout:3/8: read - no filename 2024-04-05T03:02:35.959 INFO:tasks.workunit.client.0.smithi153.stdout:3/9: write - no filename 2024-04-05T03:02:35.959 INFO:tasks.workunit.client.0.smithi153.stdout:3/10: dwrite - no filename 2024-04-05T03:02:35.962 INFO:tasks.workunit.client.0.smithi153.stdout:4/13: creat f3 x:0 0 0 2024-04-05T03:02:35.963 INFO:tasks.workunit.client.0.smithi153.stdout:3/11: creat f3 x:0 0 0 2024-04-05T03:02:35.966 INFO:tasks.workunit.client.0.smithi153.stdout:2/24: rename l4 to d9/ld 0 2024-04-05T03:02:35.966 INFO:tasks.workunit.client.0.smithi153.stdout:2/25: stat l3 0 2024-04-05T03:02:35.967 INFO:tasks.workunit.client.0.smithi153.stdout:4/14: creat f4 x:0 0 0 2024-04-05T03:02:35.969 INFO:tasks.workunit.client.0.smithi153.stdout:2/26: creat d9/da/fe x:0 0 0 2024-04-05T03:02:35.970 INFO:tasks.workunit.client.0.smithi153.stdout:3/12: dwrite f3 [0,4194304] 0 2024-04-05T03:02:35.973 INFO:tasks.workunit.client.0.smithi153.stdout:4/15: unlink f3 0 2024-04-05T03:02:35.978 INFO:tasks.workunit.client.0.smithi153.stdout:4/16: symlink l5 0 2024-04-05T03:02:35.978 INFO:tasks.workunit.client.0.smithi153.stdout:4/17: dread - f4 zero size 2024-04-05T03:02:35.982 INFO:tasks.workunit.client.0.smithi153.stdout:6/9: getdents . 0 2024-04-05T03:02:35.984 INFO:tasks.workunit.client.0.smithi153.stdout:6/10: dread f0 [0,4194304] 0 2024-04-05T03:02:35.985 INFO:tasks.workunit.client.0.smithi153.stdout:3/13: link c1 c4 0 2024-04-05T03:02:35.988 INFO:tasks.workunit.client.0.smithi153.stdout:5/9: getdents d0 0 2024-04-05T03:02:35.989 INFO:tasks.workunit.client.0.smithi153.stdout:3/14: symlink l5 0 2024-04-05T03:02:35.990 INFO:tasks.workunit.client.0.smithi153.stdout:3/15: chown c2 432909007 1 2024-04-05T03:02:35.990 INFO:tasks.workunit.client.0.smithi153.stdout:6/11: getdents . 0 2024-04-05T03:02:35.992 INFO:tasks.workunit.client.0.smithi153.stdout:5/10: creat d0/f5 x:0 0 0 2024-04-05T03:02:35.993 INFO:tasks.workunit.client.0.smithi153.stdout:3/16: dread f3 [0,4194304] 0 2024-04-05T03:02:36.000 INFO:tasks.workunit.client.1.smithi203.stdout:1/185: sync 2024-04-05T03:02:36.000 INFO:tasks.workunit.client.1.smithi203.stdout:9/153: sync 2024-04-05T03:02:36.000 INFO:tasks.workunit.client.1.smithi203.stdout:0/201: sync 2024-04-05T03:02:36.000 INFO:tasks.workunit.client.1.smithi203.stdout:7/162: sync 2024-04-05T03:02:36.003 INFO:tasks.workunit.client.0.smithi153.stdout:5/11: link d0/l4 d0/l6 0 2024-04-05T03:02:36.010 INFO:tasks.workunit.client.0.smithi153.stdout:7/22: truncate d2/f3 172907 0 2024-04-05T03:02:36.013 INFO:tasks.workunit.client.0.smithi153.stdout:5/12: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:36.013 INFO:tasks.workunit.client.0.smithi153.stdout:7/23: dread d2/f6 [0,4194304] 0 2024-04-05T03:02:36.015 INFO:tasks.workunit.client.1.smithi203.stdout:8/271: read d1/d12/d1a/f1e [539052,51085] 0 2024-04-05T03:02:36.016 INFO:tasks.workunit.client.1.smithi203.stdout:4/130: rmdir da 39 2024-04-05T03:02:36.017 INFO:tasks.workunit.client.0.smithi153.stdout:3/17: fsync f3 0 2024-04-05T03:02:36.018 INFO:tasks.workunit.client.1.smithi203.stdout:9/154: chown d13/d1b/c23 51389378 1 2024-04-05T03:02:36.023 INFO:tasks.workunit.client.0.smithi153.stdout:4/18: fsync f4 0 2024-04-05T03:02:36.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/272: creat d1/d6/f45 x:0 0 0 2024-04-05T03:02:36.025 INFO:tasks.workunit.client.0.smithi153.stdout:4/19: read - f4 zero size 2024-04-05T03:02:36.025 INFO:tasks.workunit.client.0.smithi153.stdout:3/18: symlink l6 0 2024-04-05T03:02:36.026 INFO:tasks.workunit.client.0.smithi153.stdout:5/13: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:36.027 INFO:tasks.workunit.client.1.smithi203.stdout:5/255: dwrite d3/d1f/f3b [4194304,4194304] 0 2024-04-05T03:02:36.027 INFO:tasks.workunit.client.1.smithi203.stdout:1/186: chown d7/lf 16305 1 2024-04-05T03:02:36.027 INFO:tasks.workunit.client.1.smithi203.stdout:1/187: stat d7/lb 0 2024-04-05T03:02:36.028 INFO:tasks.workunit.client.0.smithi153.stdout:4/20: fdatasync f4 0 2024-04-05T03:02:36.029 INFO:tasks.workunit.client.1.smithi203.stdout:7/163: creat d8/d2a/f36 x:0 0 0 2024-04-05T03:02:36.029 INFO:tasks.workunit.client.1.smithi203.stdout:1/188: fsync d7/f13 0 2024-04-05T03:02:36.030 INFO:tasks.workunit.client.0.smithi153.stdout:5/14: read d0/f3 [3008644,106276] 0 2024-04-05T03:02:36.030 INFO:tasks.workunit.client.0.smithi153.stdout:3/19: symlink l7 0 2024-04-05T03:02:36.030 INFO:tasks.workunit.client.1.smithi203.stdout:4/131: truncate da/f11 266610 0 2024-04-05T03:02:36.030 INFO:tasks.workunit.client.0.smithi153.stdout:4/21: truncate f4 854639 0 2024-04-05T03:02:36.032 INFO:tasks.workunit.client.0.smithi153.stdout:4/22: chown f4 389412888 1 2024-04-05T03:02:36.035 INFO:tasks.workunit.client.1.smithi203.stdout:4/132: write da/f26 [679033,32389] 0 2024-04-05T03:02:36.036 INFO:tasks.workunit.client.1.smithi203.stdout:9/155: dread d13/f20 [0,4194304] 0 2024-04-05T03:02:36.037 INFO:tasks.workunit.client.1.smithi203.stdout:4/133: truncate da/f10 24473 0 2024-04-05T03:02:36.041 INFO:tasks.workunit.client.0.smithi153.stdout:3/20: mknod c8 0 2024-04-05T03:02:36.041 INFO:tasks.workunit.client.0.smithi153.stdout:7/24: dwrite d2/f6 [0,4194304] 0 2024-04-05T03:02:36.043 INFO:tasks.workunit.client.0.smithi153.stdout:3/21: stat f3 0 2024-04-05T03:02:36.044 INFO:tasks.workunit.client.1.smithi203.stdout:9/156: dwrite d13/f25 [0,4194304] 0 2024-04-05T03:02:36.045 INFO:tasks.workunit.client.0.smithi153.stdout:7/25: dread - d2/f5 zero size 2024-04-05T03:02:36.046 INFO:tasks.workunit.client.1.smithi203.stdout:0/202: link ca dd/d17/d29/c36 0 2024-04-05T03:02:36.046 INFO:tasks.workunit.client.0.smithi153.stdout:4/23: rename l5 to l6 0 2024-04-05T03:02:36.047 INFO:tasks.workunit.client.0.smithi153.stdout:5/15: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:36.049 INFO:tasks.workunit.client.1.smithi203.stdout:5/256: symlink d3/d1f/l5a 0 2024-04-05T03:02:36.049 INFO:tasks.workunit.client.1.smithi203.stdout:8/273: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:36.050 INFO:tasks.workunit.client.0.smithi153.stdout:7/26: mknod d2/c8 0 2024-04-05T03:02:36.054 INFO:tasks.workunit.client.0.smithi153.stdout:7/27: truncate d2/f5 1022332 0 2024-04-05T03:02:36.056 INFO:tasks.workunit.client.0.smithi153.stdout:5/16: symlink d0/l7 0 2024-04-05T03:02:36.056 INFO:tasks.workunit.client.0.smithi153.stdout:6/12: dwrite f0 [0,4194304] 0 2024-04-05T03:02:36.056 INFO:tasks.workunit.client.1.smithi203.stdout:7/164: creat d8/f37 x:0 0 0 2024-04-05T03:02:36.058 INFO:tasks.workunit.client.0.smithi153.stdout:5/17: dread - d0/f1 zero size 2024-04-05T03:02:36.059 INFO:tasks.workunit.client.1.smithi203.stdout:8/274: chown d1/d12/d17/d10/l36 5 1 2024-04-05T03:02:36.060 INFO:tasks.workunit.client.1.smithi203.stdout:7/165: stat d8/da 0 2024-04-05T03:02:36.060 INFO:tasks.workunit.client.0.smithi153.stdout:3/22: dread f3 [0,4194304] 0 2024-04-05T03:02:36.061 INFO:tasks.workunit.client.0.smithi153.stdout:5/18: chown d0/f5 4797185 1 2024-04-05T03:02:36.065 INFO:tasks.workunit.client.0.smithi153.stdout:4/24: dwrite f4 [0,4194304] 0 2024-04-05T03:02:36.066 INFO:tasks.workunit.client.0.smithi153.stdout:4/25: stat c2 0 2024-04-05T03:02:36.069 INFO:tasks.workunit.client.1.smithi203.stdout:9/157: mknod d13/d1b/d26/c2d 0 2024-04-05T03:02:36.069 INFO:tasks.workunit.client.1.smithi203.stdout:1/189: link d7/d14/l26 d7/d24/d33/l41 0 2024-04-05T03:02:36.071 INFO:tasks.workunit.client.1.smithi203.stdout:4/134: dwrite f5 [4194304,4194304] 0 2024-04-05T03:02:36.076 INFO:tasks.workunit.client.0.smithi153.stdout:3/23: mknod c9 0 2024-04-05T03:02:36.078 INFO:tasks.workunit.client.1.smithi203.stdout:9/158: chown fa 1013281423 1 2024-04-05T03:02:36.078 INFO:tasks.workunit.client.1.smithi203.stdout:5/257: dwrite d3/d4/fb [8388608,4194304] 0 2024-04-05T03:02:36.082 INFO:tasks.workunit.client.1.smithi203.stdout:1/190: read d7/fa [1619613,127106] 0 2024-04-05T03:02:36.083 INFO:tasks.workunit.client.1.smithi203.stdout:0/203: truncate dd/d17/d1d/d26/d34/f35 3721282 0 2024-04-05T03:02:36.083 INFO:tasks.workunit.client.1.smithi203.stdout:8/275: dread d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:36.086 INFO:tasks.workunit.client.0.smithi153.stdout:5/19: creat d0/f8 x:0 0 0 2024-04-05T03:02:36.087 INFO:tasks.workunit.client.1.smithi203.stdout:9/159: symlink d13/d1b/d26/l2e 0 2024-04-05T03:02:36.087 INFO:tasks.workunit.client.0.smithi153.stdout:5/20: readlink d0/l2 0 2024-04-05T03:02:36.090 INFO:tasks.workunit.client.1.smithi203.stdout:4/135: link da/f1b da/d29/f2b 0 2024-04-05T03:02:36.094 INFO:tasks.workunit.client.1.smithi203.stdout:1/191: write d7/d24/f35 [4722418,9809] 0 2024-04-05T03:02:36.100 INFO:tasks.workunit.client.1.smithi203.stdout:5/258: dwrite d3/d4/d8/d12/f4e [0,4194304] 0 2024-04-05T03:02:36.102 INFO:tasks.workunit.client.1.smithi203.stdout:9/160: truncate f7 2674801 0 2024-04-05T03:02:36.103 INFO:tasks.workunit.client.1.smithi203.stdout:9/161: readlink le 0 2024-04-05T03:02:36.105 INFO:tasks.workunit.client.1.smithi203.stdout:8/276: dread d1/f34 [0,4194304] 0 2024-04-05T03:02:36.105 INFO:tasks.workunit.client.1.smithi203.stdout:1/192: write d7/d24/d33/f3f [1520425,62885] 0 2024-04-05T03:02:36.109 INFO:tasks.workunit.client.1.smithi203.stdout:0/204: rename dd/d17/d1d/d22/f23 to dd/d17/d1d/f37 0 2024-04-05T03:02:36.110 INFO:tasks.workunit.client.1.smithi203.stdout:7/166: dwrite d8/da/f1d [0,4194304] 0 2024-04-05T03:02:36.112 INFO:tasks.workunit.client.1.smithi203.stdout:0/205: read dd/d17/d1d/f37 [2348010,89645] 0 2024-04-05T03:02:36.116 INFO:tasks.workunit.client.0.smithi153.stdout:3/24: rename c4 to ca 0 2024-04-05T03:02:36.119 INFO:tasks.workunit.client.1.smithi203.stdout:5/259: write d3/d4/dc/f37 [940473,91475] 0 2024-04-05T03:02:36.120 INFO:tasks.workunit.client.0.smithi153.stdout:4/26: fsync f4 0 2024-04-05T03:02:36.120 INFO:tasks.workunit.client.0.smithi153.stdout:4/27: chown l1 531613788 1 2024-04-05T03:02:36.121 INFO:tasks.workunit.client.0.smithi153.stdout:3/25: symlink lb 0 2024-04-05T03:02:36.127 INFO:tasks.workunit.client.0.smithi153.stdout:4/28: dread f4 [0,4194304] 0 2024-04-05T03:02:36.128 INFO:tasks.workunit.client.1.smithi203.stdout:8/277: dwrite d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:36.131 INFO:tasks.workunit.client.0.smithi153.stdout:3/26: dwrite f3 [0,4194304] 0 2024-04-05T03:02:36.132 INFO:tasks.workunit.client.1.smithi203.stdout:5/260: rmdir d3/d4/d8 39 2024-04-05T03:02:36.133 INFO:tasks.workunit.client.1.smithi203.stdout:5/261: readlink d3/la 0 2024-04-05T03:02:36.135 INFO:tasks.workunit.client.1.smithi203.stdout:7/167: dread d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:36.135 INFO:tasks.workunit.client.1.smithi203.stdout:1/193: dread d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:36.137 INFO:tasks.workunit.client.1.smithi203.stdout:1/194: fsync d7/fe 0 2024-04-05T03:02:36.138 INFO:tasks.workunit.client.1.smithi203.stdout:7/168: stat d8/da/f1d 0 2024-04-05T03:02:36.138 INFO:tasks.workunit.client.0.smithi153.stdout:7/28: dwrite d2/f6 [4194304,4194304] 0 2024-04-05T03:02:36.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/29: dread f4 [0,4194304] 0 2024-04-05T03:02:36.139 INFO:tasks.workunit.client.0.smithi153.stdout:7/29: fsync d2/f5 0 2024-04-05T03:02:36.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/30: chown c2 27 1 2024-04-05T03:02:36.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/31: fsync f4 0 2024-04-05T03:02:36.140 INFO:tasks.workunit.client.1.smithi203.stdout:5/262: stat d3/d4/d8/f26 0 2024-04-05T03:02:36.142 INFO:tasks.workunit.client.1.smithi203.stdout:8/278: write d1/d12/f28 [2578275,4851] 0 2024-04-05T03:02:36.145 INFO:tasks.workunit.client.0.smithi153.stdout:7/30: readlink l0 0 2024-04-05T03:02:36.147 INFO:tasks.workunit.client.1.smithi203.stdout:5/263: dread d3/d1f/f1e [0,4194304] 0 2024-04-05T03:02:36.147 INFO:tasks.workunit.client.1.smithi203.stdout:5/264: readlink d3/l3c 0 2024-04-05T03:02:36.148 INFO:tasks.workunit.client.1.smithi203.stdout:8/279: creat d1/d12/d1a/f46 x:0 0 0 2024-04-05T03:02:36.149 INFO:tasks.workunit.client.1.smithi203.stdout:7/169: read d8/d32/f18 [399289,99439] 0 2024-04-05T03:02:36.153 INFO:tasks.workunit.client.1.smithi203.stdout:5/265: write d3/d1f/d49/f57 [83617,53850] 0 2024-04-05T03:02:36.153 INFO:tasks.workunit.client.1.smithi203.stdout:8/280: mkdir d1/d12/d17/d47 0 2024-04-05T03:02:36.154 INFO:tasks.workunit.client.0.smithi153.stdout:4/32: creat f7 x:0 0 0 2024-04-05T03:02:36.154 INFO:tasks.workunit.client.0.smithi153.stdout:5/21: fsync d0/f8 0 2024-04-05T03:02:36.155 INFO:tasks.workunit.client.0.smithi153.stdout:7/31: dwrite d2/f5 [0,4194304] 0 2024-04-05T03:02:36.155 INFO:tasks.workunit.client.0.smithi153.stdout:4/33: chown l1 26235262 1 2024-04-05T03:02:36.158 INFO:tasks.workunit.client.0.smithi153.stdout:6/13: truncate f0 264539 0 2024-04-05T03:02:36.159 INFO:tasks.workunit.client.1.smithi203.stdout:7/170: creat d8/d32/d16/d1a/d30/f38 x:0 0 0 2024-04-05T03:02:36.160 INFO:tasks.workunit.client.1.smithi203.stdout:7/171: chown d8/d2a/f36 2459 1 2024-04-05T03:02:36.160 INFO:tasks.workunit.client.1.smithi203.stdout:4/136: dread da/d29/f2b [0,4194304] 0 2024-04-05T03:02:36.160 INFO:tasks.workunit.client.1.smithi203.stdout:5/266: rename d3/d1f/c4a to d3/d4/dc/c5b 0 2024-04-05T03:02:36.167 INFO:tasks.workunit.client.1.smithi203.stdout:7/172: creat d8/d32/d16/f39 x:0 0 0 2024-04-05T03:02:36.170 INFO:tasks.workunit.client.0.smithi153.stdout:4/34: link f7 f8 0 2024-04-05T03:02:36.171 INFO:tasks.workunit.client.1.smithi203.stdout:7/173: rename d8/ff to d8/d32/f3a 0 2024-04-05T03:02:36.172 INFO:tasks.workunit.client.0.smithi153.stdout:5/22: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:36.172 INFO:tasks.workunit.client.0.smithi153.stdout:4/35: read - f7 zero size 2024-04-05T03:02:36.173 INFO:tasks.workunit.client.0.smithi153.stdout:7/32: dwrite d2/f3 [0,4194304] 0 2024-04-05T03:02:36.174 INFO:tasks.workunit.client.0.smithi153.stdout:7/33: chown d2/c8 4536 1 2024-04-05T03:02:36.175 INFO:tasks.workunit.client.0.smithi153.stdout:4/36: creat f9 x:0 0 0 2024-04-05T03:02:36.176 INFO:tasks.workunit.client.1.smithi203.stdout:0/206: write dd/d17/d1d/d26/d34/f35 [3940032,34983] 0 2024-04-05T03:02:36.180 INFO:tasks.workunit.client.0.smithi153.stdout:7/34: unlink l0 0 2024-04-05T03:02:36.181 INFO:tasks.workunit.client.1.smithi203.stdout:5/267: rmdir d3/d4/d8/d12/d40 0 2024-04-05T03:02:36.181 INFO:tasks.workunit.client.0.smithi153.stdout:4/37: write f8 [988167,107622] 0 2024-04-05T03:02:36.182 INFO:tasks.workunit.client.0.smithi153.stdout:4/38: rmdir - no directory 2024-04-05T03:02:36.185 INFO:tasks.workunit.client.0.smithi153.stdout:7/35: dread d2/f6 [0,4194304] 0 2024-04-05T03:02:36.185 INFO:tasks.workunit.client.1.smithi203.stdout:5/268: fdatasync d3/d4/d8/f44 0 2024-04-05T03:02:36.188 INFO:tasks.workunit.client.1.smithi203.stdout:7/174: dwrite d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:36.188 INFO:tasks.workunit.client.1.smithi203.stdout:0/207: dwrite f2 [4194304,4194304] 0 2024-04-05T03:02:36.191 INFO:tasks.workunit.client.1.smithi203.stdout:5/269: chown d3/d4/d8/d12/l27 23 1 2024-04-05T03:02:36.195 INFO:tasks.workunit.client.1.smithi203.stdout:7/175: symlink d8/da/d2f/l3b 0 2024-04-05T03:02:36.195 INFO:tasks.workunit.client.0.smithi153.stdout:3/27: truncate f3 3041569 0 2024-04-05T03:02:36.196 INFO:tasks.workunit.client.0.smithi153.stdout:3/28: chown l7 1 1 2024-04-05T03:02:36.197 INFO:tasks.workunit.client.1.smithi203.stdout:7/176: write d8/d2a/f36 [94410,19951] 0 2024-04-05T03:02:36.201 INFO:tasks.workunit.client.1.smithi203.stdout:7/177: write d8/d2a/f36 [1143179,124102] 0 2024-04-05T03:02:36.201 INFO:tasks.workunit.client.1.smithi203.stdout:7/178: readlink l7 0 2024-04-05T03:02:36.202 INFO:tasks.workunit.client.0.smithi153.stdout:7/36: creat d2/f9 x:0 0 0 2024-04-05T03:02:36.203 INFO:tasks.workunit.client.0.smithi153.stdout:5/23: getdents d0 0 2024-04-05T03:02:36.205 INFO:tasks.workunit.client.0.smithi153.stdout:5/24: readlink d0/l7 0 2024-04-05T03:02:36.205 INFO:tasks.workunit.client.0.smithi153.stdout:6/14: dwrite f0 [0,4194304] 0 2024-04-05T03:02:36.207 INFO:tasks.workunit.client.0.smithi153.stdout:7/37: write d2/f5 [628713,110300] 0 2024-04-05T03:02:36.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/38: write d2/f5 [1419933,84062] 0 2024-04-05T03:02:36.211 INFO:tasks.workunit.client.1.smithi203.stdout:7/179: getdents d8/d2a 0 2024-04-05T03:02:36.217 INFO:tasks.workunit.client.1.smithi203.stdout:8/281: fsync d1/d12/d1a/f46 0 2024-04-05T03:02:36.217 INFO:tasks.workunit.client.1.smithi203.stdout:1/195: truncate d7/d24/f38 410630 0 2024-04-05T03:02:36.218 INFO:tasks.workunit.client.0.smithi153.stdout:5/25: dwrite d0/f1 [0,4194304] 0 2024-04-05T03:02:36.220 INFO:tasks.workunit.client.0.smithi153.stdout:6/15: dread f0 [0,4194304] 0 2024-04-05T03:02:36.222 INFO:tasks.workunit.client.1.smithi203.stdout:8/282: readlink d1/l2 0 2024-04-05T03:02:36.228 INFO:tasks.workunit.client.0.smithi153.stdout:5/26: dwrite d0/f1 [0,4194304] 0 2024-04-05T03:02:36.228 INFO:tasks.workunit.client.0.smithi153.stdout:6/16: dread f0 [0,4194304] 0 2024-04-05T03:02:36.232 INFO:tasks.workunit.client.1.smithi203.stdout:8/283: symlink d1/l48 0 2024-04-05T03:02:36.236 INFO:tasks.workunit.client.0.smithi153.stdout:6/17: unlink c1 0 2024-04-05T03:02:36.236 INFO:tasks.workunit.client.0.smithi153.stdout:6/18: chown c2 234 1 2024-04-05T03:02:36.238 INFO:tasks.workunit.client.1.smithi203.stdout:8/284: stat d1/d12/l32 0 2024-04-05T03:02:36.238 INFO:tasks.workunit.client.1.smithi203.stdout:1/196: dwrite d7/f3d [0,4194304] 0 2024-04-05T03:02:36.239 INFO:tasks.workunit.client.0.smithi153.stdout:5/27: mkdir d0/d9 0 2024-04-05T03:02:36.240 INFO:tasks.workunit.client.0.smithi153.stdout:6/19: write f0 [39358,77987] 0 2024-04-05T03:02:36.241 INFO:tasks.workunit.client.1.smithi203.stdout:1/197: chown d7/d24/d33 1 1 2024-04-05T03:02:36.242 INFO:tasks.workunit.client.0.smithi153.stdout:4/39: fsync f8 0 2024-04-05T03:02:36.242 INFO:tasks.workunit.client.0.smithi153.stdout:5/28: chown d0/l6 41897366 1 2024-04-05T03:02:36.243 INFO:tasks.workunit.client.0.smithi153.stdout:4/40: write f4 [1726163,97469] 0 2024-04-05T03:02:36.244 INFO:tasks.workunit.client.0.smithi153.stdout:6/20: creat f3 x:0 0 0 2024-04-05T03:02:36.245 INFO:tasks.workunit.client.0.smithi153.stdout:6/21: stat c2 0 2024-04-05T03:02:36.245 INFO:tasks.workunit.client.0.smithi153.stdout:6/22: readlink - no filename 2024-04-05T03:02:36.245 INFO:tasks.workunit.client.1.smithi203.stdout:4/137: dwrite f8 [0,4194304] 0 2024-04-05T03:02:36.248 INFO:tasks.workunit.client.0.smithi153.stdout:7/39: rmdir d2 39 2024-04-05T03:02:36.250 INFO:tasks.workunit.client.1.smithi203.stdout:1/198: rmdir d7/d14 39 2024-04-05T03:02:36.254 INFO:tasks.workunit.client.1.smithi203.stdout:3/162: sync 2024-04-05T03:02:36.254 INFO:tasks.workunit.client.1.smithi203.stdout:4/138: mkdir da/d29/d2c 0 2024-04-05T03:02:36.255 INFO:tasks.workunit.client.0.smithi153.stdout:4/41: mkdir da 0 2024-04-05T03:02:36.255 INFO:tasks.workunit.client.0.smithi153.stdout:6/23: mkdir d4 0 2024-04-05T03:02:36.256 INFO:tasks.workunit.client.1.smithi203.stdout:4/139: creat da/d29/f2d x:0 0 0 2024-04-05T03:02:36.258 INFO:tasks.workunit.client.1.smithi203.stdout:4/140: write da/f1e [832407,57397] 0 2024-04-05T03:02:36.259 INFO:tasks.workunit.client.1.smithi203.stdout:1/199: rename d7/c12 to d7/c42 0 2024-04-05T03:02:36.260 INFO:tasks.workunit.client.1.smithi203.stdout:8/285: dwrite d1/d12/d1a/f42 [0,4194304] 0 2024-04-05T03:02:36.262 INFO:tasks.workunit.client.0.smithi153.stdout:7/40: chown d2/f3 15363118 1 2024-04-05T03:02:36.264 INFO:tasks.workunit.client.1.smithi203.stdout:5/270: write d3/d4/d8/f10 [238996,18873] 0 2024-04-05T03:02:36.266 INFO:tasks.workunit.client.0.smithi153.stdout:5/29: link d0/l7 d0/la 0 2024-04-05T03:02:36.267 INFO:tasks.workunit.client.0.smithi153.stdout:5/30: chown d0/l2 9938 1 2024-04-05T03:02:36.268 INFO:tasks.workunit.client.0.smithi153.stdout:5/31: write d0/f5 [4861651,36322] 0 2024-04-05T03:02:36.269 INFO:tasks.workunit.client.1.smithi203.stdout:0/208: dread dd/f15 [0,4194304] 0 2024-04-05T03:02:36.270 INFO:tasks.workunit.client.0.smithi153.stdout:4/42: mkdir da/db 0 2024-04-05T03:02:36.271 INFO:tasks.workunit.client.0.smithi153.stdout:6/24: mknod d4/c5 0 2024-04-05T03:02:36.271 INFO:tasks.workunit.client.1.smithi203.stdout:4/141: rmdir da/d29/d2c 0 2024-04-05T03:02:36.274 INFO:tasks.workunit.client.0.smithi153.stdout:4/43: readlink l6 0 2024-04-05T03:02:36.274 INFO:tasks.workunit.client.0.smithi153.stdout:5/32: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:36.276 INFO:tasks.workunit.client.0.smithi153.stdout:6/25: rename f0 to d4/f6 0 2024-04-05T03:02:36.276 INFO:tasks.workunit.client.1.smithi203.stdout:8/286: write d1/d12/d17/d10/f15 [1597138,18431] 0 2024-04-05T03:02:36.278 INFO:tasks.workunit.client.1.smithi203.stdout:4/142: chown da/f27 5 1 2024-04-05T03:02:36.281 INFO:tasks.workunit.client.1.smithi203.stdout:5/271: dwrite d3/d4/d8/f10 [4194304,4194304] 0 2024-04-05T03:02:36.282 INFO:tasks.workunit.client.1.smithi203.stdout:4/143: fsync f5 0 2024-04-05T03:02:36.282 INFO:tasks.workunit.client.1.smithi203.stdout:7/180: dread f4 [0,4194304] 0 2024-04-05T03:02:36.282 INFO:tasks.workunit.client.1.smithi203.stdout:8/287: write d1/d12/d1a/f1d [1835030,47758] 0 2024-04-05T03:02:36.282 INFO:tasks.workunit.client.0.smithi153.stdout:5/33: rename d0/l2 to d0/d9/lb 0 2024-04-05T03:02:36.286 INFO:tasks.workunit.client.1.smithi203.stdout:0/209: write dd/d17/d1d/d26/d28/f2d [3550527,68290] 0 2024-04-05T03:02:36.286 INFO:tasks.workunit.client.0.smithi153.stdout:6/26: creat d4/f7 x:0 0 0 2024-04-05T03:02:36.286 INFO:tasks.workunit.client.0.smithi153.stdout:4/44: dread f7 [0,4194304] 0 2024-04-05T03:02:36.287 INFO:tasks.workunit.client.0.smithi153.stdout:4/45: write f7 [461711,87639] 0 2024-04-05T03:02:36.287 INFO:tasks.workunit.client.1.smithi203.stdout:7/181: write d8/d32/d16/d1a/d21/f23 [395144,45988] 0 2024-04-05T03:02:36.289 INFO:tasks.workunit.client.0.smithi153.stdout:5/34: rename d0/f1 to d0/d9/fc 0 2024-04-05T03:02:36.292 INFO:tasks.workunit.client.1.smithi203.stdout:5/272: symlink d3/d4/dc/l5c 0 2024-04-05T03:02:36.293 INFO:tasks.workunit.client.0.smithi153.stdout:4/46: creat da/fc x:0 0 0 2024-04-05T03:02:36.294 INFO:tasks.workunit.client.1.smithi203.stdout:8/288: write d1/d12/d1a/f1e [1897828,22347] 0 2024-04-05T03:02:36.295 INFO:tasks.workunit.client.1.smithi203.stdout:5/273: fdatasync d3/d4/d8/f23 0 2024-04-05T03:02:36.369 INFO:tasks.workunit.client.0.smithi153.stdout:6/27: write d4/f6 [3340531,37471] 0 2024-04-05T03:02:36.369 INFO:tasks.workunit.client.1.smithi203.stdout:7/182: symlink d8/da/d2f/l3c 0 2024-04-05T03:02:36.370 INFO:tasks.workunit.client.1.smithi203.stdout:8/289: dread d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:36.371 INFO:tasks.workunit.client.0.smithi153.stdout:4/47: write f9 [724053,129571] 0 2024-04-05T03:02:36.371 INFO:tasks.workunit.client.1.smithi203.stdout:8/290: chown d1/d12/l32 1057837 1 2024-04-05T03:02:36.372 INFO:tasks.workunit.client.0.smithi153.stdout:7/41: write d2/f5 [1356669,44763] 0 2024-04-05T03:02:36.372 INFO:tasks.workunit.client.1.smithi203.stdout:8/291: read - d1/d6/f45 zero size 2024-04-05T03:02:36.373 INFO:tasks.workunit.client.1.smithi203.stdout:0/210: mkdir dd/d17/d1d/d26/d38 0 2024-04-05T03:02:36.378 INFO:tasks.workunit.client.0.smithi153.stdout:6/28: dwrite f3 [0,4194304] 0 2024-04-05T03:02:36.379 INFO:tasks.workunit.client.1.smithi203.stdout:7/183: write f4 [2563133,83801] 0 2024-04-05T03:02:36.381 INFO:tasks.workunit.client.1.smithi203.stdout:5/274: mknod d3/d4/d8/c5d 0 2024-04-05T03:02:36.383 INFO:tasks.workunit.client.1.smithi203.stdout:5/275: chown d3/d4/d8/d12/c56 6746027 1 2024-04-05T03:02:36.384 INFO:tasks.workunit.client.1.smithi203.stdout:7/184: write d8/d2a/f36 [1050710,129761] 0 2024-04-05T03:02:36.384 INFO:tasks.workunit.client.0.smithi153.stdout:4/48: unlink l1 0 2024-04-05T03:02:36.384 INFO:tasks.workunit.client.0.smithi153.stdout:4/49: readlink l6 0 2024-04-05T03:02:36.386 INFO:tasks.workunit.client.1.smithi203.stdout:7/185: readlink d8/d32/d16/d1a/l2b 0 2024-04-05T03:02:36.387 INFO:tasks.workunit.client.1.smithi203.stdout:8/292: link d1/d6/d2e/f30 d1/d12/d17/d10/f49 0 2024-04-05T03:02:36.387 INFO:tasks.workunit.client.0.smithi153.stdout:5/35: fsync d0/d9/fc 0 2024-04-05T03:02:36.388 INFO:tasks.workunit.client.1.smithi203.stdout:0/211: dread dd/d17/d1d/d26/d28/f2d [0,4194304] 0 2024-04-05T03:02:36.388 INFO:tasks.workunit.client.0.smithi153.stdout:5/36: write d0/d9/fc [4848941,96722] 0 2024-04-05T03:02:36.388 INFO:tasks.workunit.client.0.smithi153.stdout:7/42: rmdir d2 39 2024-04-05T03:02:36.390 INFO:tasks.workunit.client.0.smithi153.stdout:6/29: symlink d4/l8 0 2024-04-05T03:02:36.393 INFO:tasks.workunit.client.1.smithi203.stdout:1/200: link d7/c42 d7/c43 0 2024-04-05T03:02:36.394 INFO:tasks.workunit.client.0.smithi153.stdout:5/37: creat d0/fd x:0 0 0 2024-04-05T03:02:36.394 INFO:tasks.workunit.client.1.smithi203.stdout:5/276: symlink d3/d1f/d59/l5e 0 2024-04-05T03:02:36.397 INFO:tasks.workunit.client.0.smithi153.stdout:4/50: link f9 da/db/fd 0 2024-04-05T03:02:36.397 INFO:tasks.workunit.client.1.smithi203.stdout:7/186: mkdir d8/d32/d16/d1a/d3d 0 2024-04-05T03:02:36.397 INFO:tasks.workunit.client.1.smithi203.stdout:5/277: write d3/f4d [358758,59145] 0 2024-04-05T03:02:36.397 INFO:tasks.workunit.client.1.smithi203.stdout:4/144: dwrite da/d29/f2b [0,4194304] 0 2024-04-05T03:02:36.398 INFO:tasks.workunit.client.0.smithi153.stdout:4/51: truncate da/db/fd 895623 0 2024-04-05T03:02:36.398 INFO:tasks.workunit.client.1.smithi203.stdout:1/201: creat d7/d24/d33/f44 x:0 0 0 2024-04-05T03:02:36.398 INFO:tasks.workunit.client.0.smithi153.stdout:4/52: read f7 [511189,81598] 0 2024-04-05T03:02:36.399 INFO:tasks.workunit.client.1.smithi203.stdout:5/278: chown d3/d4 467 1 2024-04-05T03:02:36.399 INFO:tasks.workunit.client.0.smithi153.stdout:6/30: rename d4/l8 to d4/l9 0 2024-04-05T03:02:36.400 INFO:tasks.workunit.client.1.smithi203.stdout:6/252: sync 2024-04-05T03:02:36.401 INFO:tasks.workunit.client.1.smithi203.stdout:8/293: getdents d1/d6 0 2024-04-05T03:02:36.402 INFO:tasks.workunit.client.1.smithi203.stdout:2/247: sync 2024-04-05T03:02:36.403 INFO:tasks.workunit.client.1.smithi203.stdout:5/279: rmdir d3/d1f/d49 39 2024-04-05T03:02:36.403 INFO:tasks.workunit.client.0.smithi153.stdout:5/38: mkdir d0/de 0 2024-04-05T03:02:36.406 INFO:tasks.workunit.client.0.smithi153.stdout:7/43: mkdir d2/da 0 2024-04-05T03:02:36.413 INFO:tasks.workunit.client.1.smithi203.stdout:7/187: dwrite d8/da/d2f/f2e [0,4194304] 0 2024-04-05T03:02:36.417 INFO:tasks.workunit.client.1.smithi203.stdout:6/253: rename d0/da/d18/f48 to d0/da/d18/d38/d2a/d2c/f4e 0 2024-04-05T03:02:36.420 INFO:tasks.workunit.client.1.smithi203.stdout:2/248: rmdir d5/d10/d2f 39 2024-04-05T03:02:36.421 INFO:tasks.workunit.client.1.smithi203.stdout:4/145: fsync da/f1b 0 2024-04-05T03:02:36.427 INFO:tasks.workunit.client.1.smithi203.stdout:4/146: mkdir da/d29/d2e 0 2024-04-05T03:02:36.428 INFO:tasks.workunit.client.1.smithi203.stdout:4/147: truncate da/f26 902548 0 2024-04-05T03:02:36.429 INFO:tasks.workunit.client.0.smithi153.stdout:4/53: mkdir da/de 0 2024-04-05T03:02:36.429 INFO:tasks.workunit.client.0.smithi153.stdout:5/39: symlink d0/lf 0 2024-04-05T03:02:36.430 INFO:tasks.workunit.client.1.smithi203.stdout:5/280: getdents d3/d1f 0 2024-04-05T03:02:36.432 INFO:tasks.workunit.client.1.smithi203.stdout:8/294: link d1/d12/d2f/d1f/l2b d1/d12/l4a 0 2024-04-05T03:02:36.433 INFO:tasks.workunit.client.1.smithi203.stdout:7/188: creat d8/f3e x:0 0 0 2024-04-05T03:02:36.434 INFO:tasks.workunit.client.1.smithi203.stdout:1/202: getdents d7/d14 0 2024-04-05T03:02:36.436 INFO:tasks.workunit.client.1.smithi203.stdout:1/203: chown l5 1 1 2024-04-05T03:02:36.436 INFO:tasks.workunit.client.1.smithi203.stdout:5/281: write d3/d4/d8/d12/f43 [4001684,87438] 0 2024-04-05T03:02:36.437 INFO:tasks.workunit.client.1.smithi203.stdout:4/148: symlink da/d29/d2e/l2f 0 2024-04-05T03:02:36.437 INFO:tasks.workunit.client.1.smithi203.stdout:6/254: truncate d0/da/d18/d38/f47 406714 0 2024-04-05T03:02:36.439 INFO:tasks.workunit.client.1.smithi203.stdout:2/249: rename d5/d10/d2f/f4f to d5/d10/d23/d29/f53 0 2024-04-05T03:02:36.443 INFO:tasks.workunit.client.1.smithi203.stdout:1/204: write d7/f27 [800741,71798] 0 2024-04-05T03:02:36.446 INFO:tasks.workunit.client.1.smithi203.stdout:6/255: dread d0/f9 [0,4194304] 0 2024-04-05T03:02:36.447 INFO:tasks.workunit.client.1.smithi203.stdout:5/282: truncate d3/d4/d8/f44 3971972 0 2024-04-05T03:02:36.450 INFO:tasks.workunit.client.1.smithi203.stdout:8/295: getdents d1/d6/d2e 0 2024-04-05T03:02:36.451 INFO:tasks.workunit.client.1.smithi203.stdout:4/149: rename da/ce to da/d29/c30 0 2024-04-05T03:02:36.452 INFO:tasks.workunit.client.1.smithi203.stdout:2/250: dread d5/d10/d35/d2d/f50 [0,4194304] 0 2024-04-05T03:02:36.453 INFO:tasks.workunit.client.1.smithi203.stdout:1/205: creat d7/d24/d33/f45 x:0 0 0 2024-04-05T03:02:36.453 INFO:tasks.workunit.client.1.smithi203.stdout:5/283: read d3/d1f/f39 [4093904,117645] 0 2024-04-05T03:02:36.454 INFO:tasks.workunit.client.1.smithi203.stdout:2/251: chown d5/d10/d35/d36/d41/l48 193 1 2024-04-05T03:02:36.454 INFO:tasks.workunit.client.1.smithi203.stdout:6/256: creat d0/da/d18/d38/d2a/d2c/f4f x:0 0 0 2024-04-05T03:02:36.457 INFO:tasks.workunit.client.1.smithi203.stdout:8/296: rmdir d1/d12/d1a 39 2024-04-05T03:02:36.458 INFO:tasks.workunit.client.1.smithi203.stdout:7/189: dread d8/f20 [0,4194304] 0 2024-04-05T03:02:36.458 INFO:tasks.workunit.client.1.smithi203.stdout:6/257: chown d0/da/d18/f25 108223871 1 2024-04-05T03:02:36.459 INFO:tasks.workunit.client.1.smithi203.stdout:5/284: rmdir d3/d4/dc 39 2024-04-05T03:02:36.462 INFO:tasks.workunit.client.1.smithi203.stdout:2/252: rmdir d5/d10/d2f 39 2024-04-05T03:02:36.464 INFO:tasks.workunit.client.1.smithi203.stdout:7/190: mkdir d8/d2a/d3f 0 2024-04-05T03:02:36.466 INFO:tasks.workunit.client.1.smithi203.stdout:8/297: mkdir d1/d4b 0 2024-04-05T03:02:36.468 INFO:tasks.workunit.client.1.smithi203.stdout:2/253: fsync f3 0 2024-04-05T03:02:36.474 INFO:tasks.workunit.client.1.smithi203.stdout:4/150: dwrite da/fc [4194304,4194304] 0 2024-04-05T03:02:36.477 INFO:tasks.workunit.client.1.smithi203.stdout:1/206: dwrite d7/f27 [4194304,4194304] 0 2024-04-05T03:02:36.478 INFO:tasks.workunit.client.1.smithi203.stdout:6/258: rmdir d0/da/d18/d38/d2a/d2c/d3d 39 2024-04-05T03:02:36.481 INFO:tasks.workunit.client.1.smithi203.stdout:5/285: dread d3/d4/d8/f10 [4194304,4194304] 0 2024-04-05T03:02:36.482 INFO:tasks.workunit.client.1.smithi203.stdout:7/191: creat d8/d2a/f40 x:0 0 0 2024-04-05T03:02:36.489 INFO:tasks.workunit.client.1.smithi203.stdout:2/254: rename d5/d10/d23/f2c to d5/d10/d35/d36/f54 0 2024-04-05T03:02:36.491 INFO:tasks.workunit.client.1.smithi203.stdout:6/259: dread d0/da/f43 [0,4194304] 0 2024-04-05T03:02:36.493 INFO:tasks.workunit.client.1.smithi203.stdout:4/151: dread da/f26 [0,4194304] 0 2024-04-05T03:02:36.496 INFO:tasks.workunit.client.1.smithi203.stdout:4/152: read da/f27 [946552,116775] 0 2024-04-05T03:02:36.500 INFO:tasks.workunit.client.1.smithi203.stdout:8/298: dwrite d1/d6/f41 [0,4194304] 0 2024-04-05T03:02:36.505 INFO:tasks.workunit.client.1.smithi203.stdout:7/192: symlink d8/d32/d16/d35/l41 0 2024-04-05T03:02:36.505 INFO:tasks.workunit.client.1.smithi203.stdout:3/163: sync 2024-04-05T03:02:36.505 INFO:tasks.workunit.client.1.smithi203.stdout:5/286: dwrite d3/d4/d8/f26 [0,4194304] 0 2024-04-05T03:02:36.505 INFO:tasks.workunit.client.1.smithi203.stdout:9/162: sync 2024-04-05T03:02:36.508 INFO:tasks.workunit.client.1.smithi203.stdout:4/153: creat da/d29/f31 x:0 0 0 2024-04-05T03:02:36.509 INFO:tasks.workunit.client.1.smithi203.stdout:4/154: truncate da/f26 1033994 0 2024-04-05T03:02:36.512 INFO:tasks.workunit.client.1.smithi203.stdout:8/299: symlink d1/d12/d2f/l4c 0 2024-04-05T03:02:36.514 INFO:tasks.workunit.client.1.smithi203.stdout:7/193: creat d8/d2a/f42 x:0 0 0 2024-04-05T03:02:36.514 INFO:tasks.workunit.client.1.smithi203.stdout:1/207: dwrite d7/d24/d33/f36 [0,4194304] 0 2024-04-05T03:02:36.516 INFO:tasks.workunit.client.1.smithi203.stdout:1/208: stat d7/c15 0 2024-04-05T03:02:36.522 INFO:tasks.workunit.client.1.smithi203.stdout:8/300: read d1/d6/f41 [1535755,18924] 0 2024-04-05T03:02:36.522 INFO:tasks.workunit.client.1.smithi203.stdout:9/163: dwrite f10 [0,4194304] 0 2024-04-05T03:02:36.526 INFO:tasks.workunit.client.1.smithi203.stdout:6/260: rename d0/da/d18/d38/l7 to d0/da/l50 0 2024-04-05T03:02:36.529 INFO:tasks.workunit.client.1.smithi203.stdout:3/164: symlink d0/dc/d1a/d1d/l2a 0 2024-04-05T03:02:36.529 INFO:tasks.workunit.client.1.smithi203.stdout:8/301: readlink d1/d12/d17/d10/l36 0 2024-04-05T03:02:36.529 INFO:tasks.workunit.client.1.smithi203.stdout:7/194: creat d8/d32/f43 x:0 0 0 2024-04-05T03:02:36.530 INFO:tasks.workunit.client.1.smithi203.stdout:4/155: dread da/f28 [0,4194304] 0 2024-04-05T03:02:36.532 INFO:tasks.workunit.client.1.smithi203.stdout:6/261: write d0/d23/f28 [1975063,59617] 0 2024-04-05T03:02:36.532 INFO:tasks.workunit.client.1.smithi203.stdout:6/262: read d0/da/d18/f26 [4072860,61248] 0 2024-04-05T03:02:36.533 INFO:tasks.workunit.client.1.smithi203.stdout:8/302: write d1/d6/f41 [500404,28456] 0 2024-04-05T03:02:36.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/195: creat d8/d32/f44 x:0 0 0 2024-04-05T03:02:36.535 INFO:tasks.workunit.client.1.smithi203.stdout:4/156: readlink da/ld 0 2024-04-05T03:02:36.545 INFO:tasks.workunit.client.1.smithi203.stdout:3/165: symlink d0/l2b 0 2024-04-05T03:02:36.550 INFO:tasks.workunit.client.1.smithi203.stdout:4/157: dread da/f28 [0,4194304] 0 2024-04-05T03:02:36.551 INFO:tasks.workunit.client.1.smithi203.stdout:8/303: dwrite d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:36.551 INFO:tasks.workunit.client.1.smithi203.stdout:4/158: write da/f21 [43383,119691] 0 2024-04-05T03:02:36.559 INFO:tasks.workunit.client.1.smithi203.stdout:3/166: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:36.560 INFO:tasks.workunit.client.1.smithi203.stdout:1/209: rename d7/d14/c3b to d7/c46 0 2024-04-05T03:02:36.563 INFO:tasks.workunit.client.1.smithi203.stdout:8/304: rmdir d1/d12/d2f 39 2024-04-05T03:02:36.564 INFO:tasks.workunit.client.1.smithi203.stdout:6/263: dread d0/da/d18/d38/f47 [0,4194304] 0 2024-04-05T03:02:36.564 INFO:tasks.workunit.client.1.smithi203.stdout:2/255: dread d5/fa [0,4194304] 0 2024-04-05T03:02:36.566 INFO:tasks.workunit.client.1.smithi203.stdout:4/159: fsync da/f28 0 2024-04-05T03:02:36.569 INFO:tasks.workunit.client.1.smithi203.stdout:9/164: rename le to d13/d1b/l2f 0 2024-04-05T03:02:36.569 INFO:tasks.workunit.client.1.smithi203.stdout:1/210: mkdir d7/d24/d47 0 2024-04-05T03:02:36.571 INFO:tasks.workunit.client.1.smithi203.stdout:9/165: stat d13/d1b/d26/l2e 0 2024-04-05T03:02:36.572 INFO:tasks.workunit.client.1.smithi203.stdout:2/256: fsync d5/d10/d1a/f1b 0 2024-04-05T03:02:36.573 INFO:tasks.workunit.client.1.smithi203.stdout:3/167: symlink d0/l2c 0 2024-04-05T03:02:36.574 INFO:tasks.workunit.client.1.smithi203.stdout:4/160: creat da/d29/f32 x:0 0 0 2024-04-05T03:02:36.579 INFO:tasks.workunit.client.1.smithi203.stdout:4/161: write da/f1e [652088,105236] 0 2024-04-05T03:02:36.579 INFO:tasks.workunit.client.1.smithi203.stdout:9/166: unlink d13/c14 0 2024-04-05T03:02:36.579 INFO:tasks.workunit.client.1.smithi203.stdout:9/167: truncate fb 1034137 0 2024-04-05T03:02:36.579 INFO:tasks.workunit.client.1.smithi203.stdout:6/264: symlink d0/da/d18/d38/d2a/d2c/d3d/l51 0 2024-04-05T03:02:36.579 INFO:tasks.workunit.client.1.smithi203.stdout:7/196: dread d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:36.582 INFO:tasks.workunit.client.1.smithi203.stdout:2/257: write d5/d10/d23/d29/f53 [660619,33531] 0 2024-04-05T03:02:36.585 INFO:tasks.workunit.client.1.smithi203.stdout:3/168: creat d0/dc/d1a/d1f/f2d x:0 0 0 2024-04-05T03:02:36.586 INFO:tasks.workunit.client.1.smithi203.stdout:3/169: stat d0/dc/d1a/d1f/l26 0 2024-04-05T03:02:36.587 INFO:tasks.workunit.client.1.smithi203.stdout:9/168: mknod d13/d1b/c30 0 2024-04-05T03:02:36.588 INFO:tasks.workunit.client.1.smithi203.stdout:1/211: rmdir d7/d24/d47 0 2024-04-05T03:02:36.589 INFO:tasks.workunit.client.1.smithi203.stdout:3/170: chown d0/ca 2678942 1 2024-04-05T03:02:36.595 INFO:tasks.workunit.client.1.smithi203.stdout:2/258: rename d5/d10/d1a/l3a to d5/d10/d35/d36/l55 0 2024-04-05T03:02:36.596 INFO:tasks.workunit.client.1.smithi203.stdout:6/265: rmdir d0/da/d18/d38/d2a/d2c 39 2024-04-05T03:02:36.601 INFO:tasks.workunit.client.1.smithi203.stdout:2/259: rename d5/d10/d1a/d24/f28 to d5/d10/d35/d3e/f56 0 2024-04-05T03:02:36.604 INFO:tasks.workunit.client.1.smithi203.stdout:4/162: getdents da/d29 0 2024-04-05T03:02:36.606 INFO:tasks.workunit.client.1.smithi203.stdout:7/197: truncate d8/f20 323154 0 2024-04-05T03:02:36.608 INFO:tasks.workunit.client.1.smithi203.stdout:2/260: mkdir d5/d10/d1a/d57 0 2024-04-05T03:02:36.609 INFO:tasks.workunit.client.1.smithi203.stdout:9/169: write f7 [3255299,65022] 0 2024-04-05T03:02:36.609 INFO:tasks.workunit.client.1.smithi203.stdout:7/198: creat d8/da/d2f/f45 x:0 0 0 2024-04-05T03:02:36.609 INFO:tasks.workunit.client.1.smithi203.stdout:4/163: chown da/f27 24326 1 2024-04-05T03:02:36.610 INFO:tasks.workunit.client.1.smithi203.stdout:3/171: dwrite d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:36.612 INFO:tasks.workunit.client.1.smithi203.stdout:6/266: mknod d0/da/d18/d38/d2a/d2c/d3d/d4c/c52 0 2024-04-05T03:02:36.612 INFO:tasks.workunit.client.1.smithi203.stdout:1/212: dread f2 [4194304,4194304] 0 2024-04-05T03:02:36.612 INFO:tasks.workunit.client.1.smithi203.stdout:4/164: chown da/f11 63589098 1 2024-04-05T03:02:36.615 INFO:tasks.workunit.client.1.smithi203.stdout:1/213: fsync d7/d24/d33/f44 0 2024-04-05T03:02:36.619 INFO:tasks.workunit.client.1.smithi203.stdout:3/172: chown d0/dc/l1c 33 1 2024-04-05T03:02:36.620 INFO:tasks.workunit.client.1.smithi203.stdout:7/199: chown d8/d32/d16/d1a 64580756 1 2024-04-05T03:02:36.620 INFO:tasks.workunit.client.1.smithi203.stdout:6/267: readlink d0/da/d18/d38/d2a/d2c/l3b 0 2024-04-05T03:02:36.622 INFO:tasks.workunit.client.1.smithi203.stdout:6/268: fsync d0/f3 0 2024-04-05T03:02:36.622 INFO:tasks.workunit.client.1.smithi203.stdout:9/170: mkdir d13/d1b/d26/d31 0 2024-04-05T03:02:36.623 INFO:tasks.workunit.client.1.smithi203.stdout:1/214: mkdir d7/d24/d33/d48 0 2024-04-05T03:02:36.623 INFO:tasks.workunit.client.1.smithi203.stdout:4/165: write da/f22 [2162078,13487] 0 2024-04-05T03:02:36.624 INFO:tasks.workunit.client.1.smithi203.stdout:3/173: symlink d0/dc/d1a/d1f/l2e 0 2024-04-05T03:02:36.624 INFO:tasks.workunit.client.1.smithi203.stdout:2/261: dwrite d5/d10/d35/d2d/f38 [0,4194304] 0 2024-04-05T03:02:36.628 INFO:tasks.workunit.client.1.smithi203.stdout:2/262: readlink d5/d10/l17 0 2024-04-05T03:02:36.629 INFO:tasks.workunit.client.1.smithi203.stdout:1/215: creat d7/d14/d2b/f49 x:0 0 0 2024-04-05T03:02:36.631 INFO:tasks.workunit.client.1.smithi203.stdout:6/269: fsync d0/da/d18/d38/d2a/d2c/f46 0 2024-04-05T03:02:36.631 INFO:tasks.workunit.client.1.smithi203.stdout:9/171: chown d13/d1b/d26/c2d 2889660 1 2024-04-05T03:02:36.634 INFO:tasks.workunit.client.1.smithi203.stdout:2/263: dread - d5/d10/d1a/f4b zero size 2024-04-05T03:02:36.636 INFO:tasks.workunit.client.1.smithi203.stdout:1/216: mknod d7/d24/d33/c4a 0 2024-04-05T03:02:36.637 INFO:tasks.workunit.client.1.smithi203.stdout:3/174: dread d0/dc/f14 [0,4194304] 0 2024-04-05T03:02:36.639 INFO:tasks.workunit.client.1.smithi203.stdout:4/166: dread da/f28 [0,4194304] 0 2024-04-05T03:02:36.640 INFO:tasks.workunit.client.1.smithi203.stdout:9/172: mknod d13/d1b/d26/d31/c32 0 2024-04-05T03:02:36.643 INFO:tasks.workunit.client.1.smithi203.stdout:2/264: creat d5/d10/d35/d36/f58 x:0 0 0 2024-04-05T03:02:36.644 INFO:tasks.workunit.client.1.smithi203.stdout:2/265: write d5/fa [1639656,98335] 0 2024-04-05T03:02:36.644 INFO:tasks.workunit.client.1.smithi203.stdout:9/173: mkdir d13/d1b/d33 0 2024-04-05T03:02:36.645 INFO:tasks.workunit.client.1.smithi203.stdout:6/270: mkdir d0/da/d18/d38/d2a/d2c/d3d/d53 0 2024-04-05T03:02:36.645 INFO:tasks.workunit.client.1.smithi203.stdout:9/174: stat f2 0 2024-04-05T03:02:36.648 INFO:tasks.workunit.client.1.smithi203.stdout:1/217: dread d7/f2e [0,4194304] 0 2024-04-05T03:02:36.648 INFO:tasks.workunit.client.1.smithi203.stdout:9/175: read d13/f25 [2312117,35731] 0 2024-04-05T03:02:36.652 INFO:tasks.workunit.client.1.smithi203.stdout:1/218: mkdir d7/d24/d33/d48/d4b 0 2024-04-05T03:02:36.652 INFO:tasks.workunit.client.1.smithi203.stdout:9/176: creat d13/d1b/d33/f34 x:0 0 0 2024-04-05T03:02:36.659 INFO:tasks.workunit.client.1.smithi203.stdout:3/175: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:36.662 INFO:tasks.workunit.client.1.smithi203.stdout:2/266: fsync d5/d10/d35/d36/f58 0 2024-04-05T03:02:36.664 INFO:tasks.workunit.client.1.smithi203.stdout:6/271: truncate d0/f41 1923498 0 2024-04-05T03:02:36.668 INFO:tasks.workunit.client.1.smithi203.stdout:1/219: rmdir d7/d24 39 2024-04-05T03:02:36.669 INFO:tasks.workunit.client.1.smithi203.stdout:3/176: link d0/f5 d0/dc/d1a/d1d/f2f 0 2024-04-05T03:02:36.674 INFO:tasks.workunit.client.1.smithi203.stdout:2/267: creat d5/d10/d2f/f59 x:0 0 0 2024-04-05T03:02:36.674 INFO:tasks.workunit.client.1.smithi203.stdout:9/177: truncate f2 4794306 0 2024-04-05T03:02:36.674 INFO:tasks.workunit.client.1.smithi203.stdout:6/272: getdents d0/da/d18/d38/d2b 0 2024-04-05T03:02:36.681 INFO:tasks.workunit.client.1.smithi203.stdout:1/220: dwrite d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:36.683 INFO:tasks.workunit.client.1.smithi203.stdout:2/268: creat d5/d10/d1a/f5a x:0 0 0 2024-04-05T03:02:36.685 INFO:tasks.workunit.client.1.smithi203.stdout:1/221: stat d7/d24/d33/d48/d4b 0 2024-04-05T03:02:36.685 INFO:tasks.workunit.client.1.smithi203.stdout:0/212: sync 2024-04-05T03:02:36.686 INFO:tasks.workunit.client.1.smithi203.stdout:6/273: rename d0/da/d18/d38/d2a/d2c/d3d to d0/da/d54 0 2024-04-05T03:02:36.690 INFO:tasks.workunit.client.1.smithi203.stdout:2/269: creat d5/d10/f5b x:0 0 0 2024-04-05T03:02:36.691 INFO:tasks.workunit.client.1.smithi203.stdout:9/178: dwrite f10 [4194304,4194304] 0 2024-04-05T03:02:36.691 INFO:tasks.workunit.client.1.smithi203.stdout:6/274: read d0/d23/f28 [2936803,84061] 0 2024-04-05T03:02:36.693 INFO:tasks.workunit.client.1.smithi203.stdout:6/275: dread d0/da/d18/d38/f47 [0,4194304] 0 2024-04-05T03:02:36.699 INFO:tasks.workunit.client.1.smithi203.stdout:9/179: fdatasync f2 0 2024-04-05T03:02:36.706 INFO:tasks.workunit.client.1.smithi203.stdout:6/276: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:36.708 INFO:tasks.workunit.client.1.smithi203.stdout:2/270: dread d5/d10/d35/d2d/f50 [4194304,4194304] 0 2024-04-05T03:02:36.709 INFO:tasks.workunit.client.1.smithi203.stdout:9/180: getdents d13 0 2024-04-05T03:02:36.713 INFO:tasks.workunit.client.1.smithi203.stdout:9/181: symlink d13/d1b/d26/l35 0 2024-04-05T03:02:36.722 INFO:tasks.workunit.client.1.smithi203.stdout:2/271: rmdir d5/d10/d1a 39 2024-04-05T03:02:36.724 INFO:tasks.workunit.client.1.smithi203.stdout:3/177: write d0/fd [3417848,43268] 0 2024-04-05T03:02:36.725 INFO:tasks.workunit.client.1.smithi203.stdout:2/272: read d5/d10/d35/d2d/f50 [51277,115969] 0 2024-04-05T03:02:36.725 INFO:tasks.workunit.client.1.smithi203.stdout:1/222: write d7/d14/f16 [4410129,5058] 0 2024-04-05T03:02:36.727 INFO:tasks.workunit.client.1.smithi203.stdout:0/213: truncate f8 2092399 0 2024-04-05T03:02:36.729 INFO:tasks.workunit.client.1.smithi203.stdout:2/273: stat d5/c42 0 2024-04-05T03:02:36.730 INFO:tasks.workunit.client.1.smithi203.stdout:0/214: fsync dd/d17/d1d/d22/f32 0 2024-04-05T03:02:36.731 INFO:tasks.workunit.client.1.smithi203.stdout:3/178: unlink d0/c18 0 2024-04-05T03:02:36.733 INFO:tasks.workunit.client.1.smithi203.stdout:2/274: write d5/d10/d35/d36/f58 [76070,89261] 0 2024-04-05T03:02:36.736 INFO:tasks.workunit.client.1.smithi203.stdout:0/215: write dd/d17/d1d/f37 [3459376,113197] 0 2024-04-05T03:02:36.742 INFO:tasks.workunit.client.1.smithi203.stdout:2/275: write d5/d10/d1a/d24/f3f [4244752,129366] 0 2024-04-05T03:02:36.745 INFO:tasks.workunit.client.1.smithi203.stdout:2/276: dread - d5/d10/f5b zero size 2024-04-05T03:02:36.745 INFO:tasks.workunit.client.1.smithi203.stdout:1/223: dwrite d7/f3d [4194304,4194304] 0 2024-04-05T03:02:36.745 INFO:tasks.workunit.client.1.smithi203.stdout:0/216: mknod dd/d17/d1d/c39 0 2024-04-05T03:02:36.748 INFO:tasks.workunit.client.1.smithi203.stdout:1/224: fsync d7/f2d 0 2024-04-05T03:02:36.751 INFO:tasks.workunit.client.1.smithi203.stdout:2/277: write d5/d10/d2f/f59 [428714,13870] 0 2024-04-05T03:02:36.751 INFO:tasks.workunit.client.1.smithi203.stdout:1/225: fdatasync d7/d24/d33/f44 0 2024-04-05T03:02:36.754 INFO:tasks.workunit.client.1.smithi203.stdout:3/179: dwrite d0/dc/d1a/d1f/f2d [0,4194304] 0 2024-04-05T03:02:36.758 INFO:tasks.workunit.client.1.smithi203.stdout:9/182: dread d13/d1b/f21 [0,4194304] 0 2024-04-05T03:02:36.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/180: stat d0/dc/f14 0 2024-04-05T03:02:36.760 INFO:tasks.workunit.client.1.smithi203.stdout:2/278: fdatasync d5/d10/d1a/d24/f3f 0 2024-04-05T03:02:36.761 INFO:tasks.workunit.client.1.smithi203.stdout:0/217: creat dd/d17/d1d/d26/f3a x:0 0 0 2024-04-05T03:02:36.763 INFO:tasks.workunit.client.1.smithi203.stdout:2/279: chown d5/d10/d35/d36 0 1 2024-04-05T03:02:36.764 INFO:tasks.workunit.client.1.smithi203.stdout:2/280: chown d5/c42 3694987 1 2024-04-05T03:02:36.764 INFO:tasks.workunit.client.1.smithi203.stdout:9/183: mknod d13/c36 0 2024-04-05T03:02:36.766 INFO:tasks.workunit.client.1.smithi203.stdout:3/181: fsync d0/f5 0 2024-04-05T03:02:36.768 INFO:tasks.workunit.client.1.smithi203.stdout:2/281: chown d5/d10/f20 927894939 1 2024-04-05T03:02:36.773 INFO:tasks.workunit.client.1.smithi203.stdout:2/282: mknod d5/d10/d23/c5c 0 2024-04-05T03:02:36.774 INFO:tasks.workunit.client.1.smithi203.stdout:9/184: dread d13/f25 [0,4194304] 0 2024-04-05T03:02:36.776 INFO:tasks.workunit.client.1.smithi203.stdout:3/182: dread d0/dc/f14 [0,4194304] 0 2024-04-05T03:02:36.778 INFO:tasks.workunit.client.1.smithi203.stdout:2/283: mkdir d5/d10/d2f/d5d 0 2024-04-05T03:02:36.779 INFO:tasks.workunit.client.1.smithi203.stdout:9/185: symlink d13/d1b/d26/d31/l37 0 2024-04-05T03:02:36.779 INFO:tasks.workunit.client.1.smithi203.stdout:2/284: chown d5/d10/d35/d2d/f2e 39128 1 2024-04-05T03:02:36.780 INFO:tasks.workunit.client.1.smithi203.stdout:0/218: write dd/f16 [945662,35677] 0 2024-04-05T03:02:36.782 INFO:tasks.workunit.client.1.smithi203.stdout:3/183: rename d0/ca to d0/dc/d1a/d1f/c30 0 2024-04-05T03:02:36.792 INFO:tasks.workunit.client.1.smithi203.stdout:2/285: dread d5/d10/d35/d3e/f56 [0,4194304] 0 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr fail", "who": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: osdmap e46: 6 total, 6 up, 6 in 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "mgr fail", "who": "smithi153.npzyqy"}]': finished 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: mgrmap e21: smithi203.iakodz(active, starting, since 0.0242198s) 2024-04-05T03:02:36.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:02:36.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: Manager daemon smithi203.iakodz is now available 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/trash_purge_schedule"}]: dispatch 2024-04-05T03:02:36.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:36 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/trash_purge_schedule"}]: dispatch 2024-04-05T03:02:36.799 INFO:tasks.workunit.client.1.smithi203.stdout:9/186: dwrite d13/f25 [0,4194304] 0 2024-04-05T03:02:36.799 INFO:tasks.workunit.client.1.smithi203.stdout:0/219: dwrite dd/d17/d1d/d26/d28/f2d [4194304,4194304] 0 2024-04-05T03:02:36.800 INFO:tasks.workunit.client.1.smithi203.stdout:3/184: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:36.800 INFO:tasks.workunit.client.1.smithi203.stdout:2/286: creat d5/f5e x:0 0 0 2024-04-05T03:02:36.800 INFO:tasks.workunit.client.1.smithi203.stdout:3/185: fsync d0/ff 0 2024-04-05T03:02:36.805 INFO:tasks.workunit.client.1.smithi203.stdout:0/220: fsync dd/d17/d1d/d26/d28/f2b 0 2024-04-05T03:02:36.806 INFO:tasks.workunit.client.1.smithi203.stdout:2/287: mknod d5/d10/d44/c5f 0 2024-04-05T03:02:36.807 INFO:tasks.workunit.client.0.smithi153.stdout:4/54: creat da/ff x:0 0 0 2024-04-05T03:02:36.808 INFO:tasks.workunit.client.0.smithi153.stdout:4/55: chown f9 1 1 2024-04-05T03:02:36.810 INFO:tasks.workunit.client.0.smithi153.stdout:6/31: link c2 d4/ca 0 2024-04-05T03:02:36.813 INFO:tasks.workunit.client.0.smithi153.stdout:5/40: creat d0/de/f10 x:0 0 0 2024-04-05T03:02:36.813 INFO:tasks.workunit.client.1.smithi203.stdout:2/288: mknod d5/d10/d2f/d5d/c60 0 2024-04-05T03:02:36.815 INFO:tasks.workunit.client.0.smithi153.stdout:7/44: creat d2/da/fb x:0 0 0 2024-04-05T03:02:36.817 INFO:tasks.workunit.client.0.smithi153.stdout:6/32: creat d4/fb x:0 0 0 2024-04-05T03:02:36.818 INFO:tasks.workunit.client.0.smithi153.stdout:6/33: dread - d4/f7 zero size 2024-04-05T03:02:36.818 INFO:tasks.workunit.client.0.smithi153.stdout:4/56: symlink da/de/l10 0 2024-04-05T03:02:36.819 INFO:tasks.workunit.client.0.smithi153.stdout:4/57: chown da/db 134 1 2024-04-05T03:02:36.819 INFO:tasks.workunit.client.1.smithi203.stdout:2/289: rename d5/d10/c4d to d5/d10/c61 0 2024-04-05T03:02:36.820 INFO:tasks.workunit.client.1.smithi203.stdout:9/187: write f8 [85186,67691] 0 2024-04-05T03:02:36.821 INFO:tasks.workunit.client.0.smithi153.stdout:5/41: rename d0/l6 to d0/de/l11 0 2024-04-05T03:02:36.822 INFO:tasks.workunit.client.0.smithi153.stdout:5/42: fdatasync d0/d9/fc 0 2024-04-05T03:02:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr fail", "who": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: osdmap e46: 6 total, 6 up, 6 in 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.14254 172.21.15.153:0/3547808775' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "mgr fail", "who": "smithi153.npzyqy"}]': finished 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: mgrmap e21: smithi203.iakodz(active, starting, since 0.0242198s) 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:02:36.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: Manager daemon smithi203.iakodz is now available 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:02:36.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:02:36.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/trash_purge_schedule"}]: dispatch 2024-04-05T03:02:36.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:36 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi203.iakodz/trash_purge_schedule"}]: dispatch 2024-04-05T03:02:36.825 INFO:tasks.workunit.client.1.smithi203.stdout:9/188: mkdir d13/d1b/d26/d31/d38 0 2024-04-05T03:02:36.825 INFO:tasks.workunit.client.0.smithi153.stdout:6/34: mkdir d4/dc 0 2024-04-05T03:02:36.827 INFO:tasks.workunit.client.0.smithi153.stdout:5/43: dread d0/d9/fc [0,4194304] 0 2024-04-05T03:02:36.831 INFO:tasks.workunit.client.1.smithi203.stdout:0/221: dwrite f6 [4194304,4194304] 0 2024-04-05T03:02:36.833 INFO:tasks.workunit.client.1.smithi203.stdout:3/186: dread d0/ff [0,4194304] 0 2024-04-05T03:02:36.833 INFO:tasks.workunit.client.0.smithi153.stdout:7/45: mknod d2/cc 0 2024-04-05T03:02:36.834 INFO:tasks.workunit.client.1.smithi203.stdout:2/290: truncate d5/d10/d2f/f59 371361 0 2024-04-05T03:02:36.835 INFO:tasks.workunit.client.1.smithi203.stdout:9/189: dread f10 [0,4194304] 0 2024-04-05T03:02:36.837 INFO:tasks.workunit.client.0.smithi153.stdout:6/35: chown d4/ca 12 1 2024-04-05T03:02:36.838 INFO:tasks.workunit.client.0.smithi153.stdout:6/36: chown d4 313876 1 2024-04-05T03:02:36.838 INFO:tasks.workunit.client.0.smithi153.stdout:6/37: chown d4/dc 488238322 1 2024-04-05T03:02:36.839 INFO:tasks.workunit.client.1.smithi203.stdout:3/187: mknod d0/dc/d1a/d1f/c31 0 2024-04-05T03:02:36.839 INFO:tasks.workunit.client.1.smithi203.stdout:2/291: write d5/d10/f5b [468464,109501] 0 2024-04-05T03:02:36.840 INFO:tasks.workunit.client.0.smithi153.stdout:7/46: unlink d2/da/fb 0 2024-04-05T03:02:36.840 INFO:tasks.workunit.client.1.smithi203.stdout:0/222: dread f2 [4194304,4194304] 0 2024-04-05T03:02:36.841 INFO:tasks.workunit.client.1.smithi203.stdout:3/188: write d0/f9 [4225484,107486] 0 2024-04-05T03:02:36.845 INFO:tasks.workunit.client.1.smithi203.stdout:2/292: symlink d5/d10/d35/d36/l62 0 2024-04-05T03:02:36.846 INFO:tasks.workunit.client.1.smithi203.stdout:3/189: write d0/dc/d1a/d1f/f2d [3377239,88946] 0 2024-04-05T03:02:36.848 INFO:tasks.workunit.client.1.smithi203.stdout:0/223: stat dd/d17/d1d/d26/d28/c33 0 2024-04-05T03:02:36.849 INFO:tasks.workunit.client.0.smithi153.stdout:7/47: unlink d2/c4 0 2024-04-05T03:02:36.850 INFO:tasks.workunit.client.1.smithi203.stdout:2/293: truncate f3 8206156 0 2024-04-05T03:02:36.851 INFO:tasks.workunit.client.1.smithi203.stdout:9/190: getdents d13/d1b/d26/d31 0 2024-04-05T03:02:36.851 INFO:tasks.workunit.client.1.smithi203.stdout:2/294: readlink d5/d10/l17 0 2024-04-05T03:02:36.852 INFO:tasks.workunit.client.1.smithi203.stdout:3/190: creat d0/dc/d1a/d1f/f32 x:0 0 0 2024-04-05T03:02:36.853 INFO:tasks.workunit.client.1.smithi203.stdout:0/224: creat dd/d17/d1d/d22/f3b x:0 0 0 2024-04-05T03:02:36.853 INFO:tasks.workunit.client.0.smithi153.stdout:7/48: write d2/f9 [726280,39653] 0 2024-04-05T03:02:36.857 INFO:tasks.workunit.client.1.smithi203.stdout:0/225: readlink dd/d17/d29/l2c 0 2024-04-05T03:02:36.858 INFO:tasks.workunit.client.0.smithi153.stdout:7/49: mknod d2/da/cd 0 2024-04-05T03:02:36.861 INFO:tasks.workunit.client.1.smithi203.stdout:9/191: rename d13/d1b/l29 to d13/d1b/d26/l39 0 2024-04-05T03:02:36.862 INFO:tasks.workunit.client.0.smithi153.stdout:7/50: dread d2/f5 [0,4194304] 0 2024-04-05T03:02:36.862 INFO:tasks.workunit.client.1.smithi203.stdout:0/226: creat dd/d17/d29/f3c x:0 0 0 2024-04-05T03:02:36.863 INFO:tasks.workunit.client.1.smithi203.stdout:3/191: dwrite d0/dc/d1a/d1d/f29 [0,4194304] 0 2024-04-05T03:02:36.863 INFO:tasks.workunit.client.1.smithi203.stdout:0/227: chown dd/f16 4396 1 2024-04-05T03:02:36.865 INFO:tasks.workunit.client.1.smithi203.stdout:3/192: dread - d0/f23 zero size 2024-04-05T03:02:36.866 INFO:tasks.workunit.client.1.smithi203.stdout:9/192: mknod d13/d1b/d33/c3a 0 2024-04-05T03:02:36.866 INFO:tasks.workunit.client.0.smithi153.stdout:7/51: symlink d2/le 0 2024-04-05T03:02:36.875 INFO:tasks.workunit.client.0.smithi153.stdout:7/52: dwrite d2/f5 [0,4194304] 0 2024-04-05T03:02:36.880 INFO:tasks.workunit.client.1.smithi203.stdout:9/193: dwrite fb [0,4194304] 0 2024-04-05T03:02:36.880 INFO:tasks.workunit.client.1.smithi203.stdout:0/228: dread dd/d17/d1d/f37 [0,4194304] 0 2024-04-05T03:02:36.881 INFO:tasks.workunit.client.1.smithi203.stdout:3/193: dwrite d0/dc/d1a/d1f/f2d [4194304,4194304] 0 2024-04-05T03:02:36.884 INFO:tasks.workunit.client.1.smithi203.stdout:3/194: chown d0/c15 7577446 1 2024-04-05T03:02:36.890 INFO:tasks.workunit.client.0.smithi153.stdout:7/53: dread d2/f9 [0,4194304] 0 2024-04-05T03:02:36.891 INFO:tasks.workunit.client.1.smithi203.stdout:3/195: mkdir d0/dc/d1a/d33 0 2024-04-05T03:02:36.893 INFO:tasks.workunit.client.1.smithi203.stdout:2/295: write d5/d10/d35/d2d/f50 [5537564,75292] 0 2024-04-05T03:02:36.897 INFO:tasks.workunit.client.1.smithi203.stdout:0/229: creat dd/d17/d1d/d26/f3d x:0 0 0 2024-04-05T03:02:36.897 INFO:tasks.workunit.client.0.smithi153.stdout:7/54: mkdir d2/da/df 0 2024-04-05T03:02:36.900 INFO:tasks.workunit.client.1.smithi203.stdout:2/296: dread d5/d10/d1a/d24/f27 [4194304,4194304] 0 2024-04-05T03:02:36.904 INFO:tasks.workunit.client.0.smithi153.stdout:7/55: creat d2/da/df/f10 x:0 0 0 2024-04-05T03:02:36.907 INFO:tasks.workunit.client.1.smithi203.stdout:0/230: link dd/d17/d29/c31 dd/d17/d29/c3e 0 2024-04-05T03:02:36.907 INFO:tasks.workunit.client.0.smithi153.stdout:7/56: mkdir d2/da/d11 0 2024-04-05T03:02:36.907 INFO:tasks.workunit.client.1.smithi203.stdout:9/194: dread fa [0,4194304] 0 2024-04-05T03:02:36.908 INFO:tasks.workunit.client.0.smithi153.stdout:7/57: write d2/f5 [748750,53814] 0 2024-04-05T03:02:36.908 INFO:tasks.workunit.client.1.smithi203.stdout:2/297: mkdir d5/d63 0 2024-04-05T03:02:36.910 INFO:tasks.workunit.client.1.smithi203.stdout:2/298: chown d5/f5e 126 1 2024-04-05T03:02:36.910 INFO:tasks.workunit.client.1.smithi203.stdout:2/299: stat d5/d10/d1a/c3c 0 2024-04-05T03:02:36.911 INFO:tasks.workunit.client.0.smithi153.stdout:7/58: mkdir d2/da/d11/d12 0 2024-04-05T03:02:36.912 INFO:tasks.workunit.client.1.smithi203.stdout:0/231: creat dd/d17/d1d/d26/d28/f3f x:0 0 0 2024-04-05T03:02:36.912 INFO:tasks.workunit.client.1.smithi203.stdout:9/195: symlink d13/d1b/d26/d31/d38/l3b 0 2024-04-05T03:02:36.913 INFO:tasks.workunit.client.1.smithi203.stdout:0/232: chown dd/d17/d1d/f37 10202607 1 2024-04-05T03:02:36.914 INFO:tasks.workunit.client.0.smithi153.stdout:7/59: creat d2/da/d11/f13 x:0 0 0 2024-04-05T03:02:36.914 INFO:tasks.workunit.client.1.smithi203.stdout:0/233: chown dd/d17/d1d/d22 1941 1 2024-04-05T03:02:36.914 INFO:tasks.workunit.client.1.smithi203.stdout:9/196: rmdir d13/d1b/d26/d31/d38 39 2024-04-05T03:02:36.915 INFO:tasks.workunit.client.0.smithi153.stdout:7/60: chown d2/da/d11/d12 61000 1 2024-04-05T03:02:36.916 INFO:tasks.workunit.client.1.smithi203.stdout:2/300: fdatasync d5/d10/d2f/f59 0 2024-04-05T03:02:36.921 INFO:tasks.workunit.client.1.smithi203.stdout:0/234: link f8 dd/d17/f40 0 2024-04-05T03:02:36.922 INFO:tasks.workunit.client.0.smithi153.stdout:5/44: rename d0/de/l11 to d0/l12 0 2024-04-05T03:02:36.922 INFO:tasks.workunit.client.0.smithi153.stdout:7/61: rename d2 to d2/da/d11/d12/d14 22 2024-04-05T03:02:36.922 INFO:tasks.workunit.client.0.smithi153.stdout:6/38: rmdir d4 39 2024-04-05T03:02:36.923 INFO:tasks.workunit.client.1.smithi203.stdout:9/197: dwrite d13/f2a [0,4194304] 0 2024-04-05T03:02:36.927 INFO:tasks.workunit.client.1.smithi203.stdout:2/301: getdents d5/d10/d1a 0 2024-04-05T03:02:36.933 INFO:tasks.workunit.client.0.smithi153.stdout:6/39: unlink d4/l9 0 2024-04-05T03:02:36.935 INFO:tasks.workunit.client.0.smithi153.stdout:4/58: dwrite f7 [0,4194304] 0 2024-04-05T03:02:36.936 INFO:tasks.workunit.client.1.smithi203.stdout:2/302: link d5/d10/d23/f25 d5/d10/d2f/f64 0 2024-04-05T03:02:36.936 INFO:tasks.workunit.client.1.smithi203.stdout:2/303: chown d5/d10/d35/d2d 0 1 2024-04-05T03:02:36.936 INFO:tasks.workunit.client.0.smithi153.stdout:7/62: dread d2/f5 [0,4194304] 0 2024-04-05T03:02:36.936 INFO:tasks.workunit.client.0.smithi153.stdout:5/45: dwrite d0/fd [0,4194304] 0 2024-04-05T03:02:36.938 INFO:tasks.workunit.client.1.smithi203.stdout:2/304: chown d5/d10/d2f/l31 772112 1 2024-04-05T03:02:36.939 INFO:tasks.workunit.client.0.smithi153.stdout:7/63: chown d2/f3 4272 1 2024-04-05T03:02:36.943 INFO:tasks.workunit.client.1.smithi203.stdout:0/235: truncate f7 1892888 0 2024-04-05T03:02:36.944 INFO:tasks.workunit.client.0.smithi153.stdout:6/40: dwrite f3 [0,4194304] 0 2024-04-05T03:02:36.944 INFO:tasks.workunit.client.1.smithi203.stdout:2/305: unlink d5/d10/l17 0 2024-04-05T03:02:36.945 INFO:tasks.workunit.client.0.smithi153.stdout:6/41: chown d4/fb 33083705 1 2024-04-05T03:02:36.946 INFO:tasks.workunit.client.0.smithi153.stdout:7/64: rename d2/le to d2/da/d11/d12/l15 0 2024-04-05T03:02:36.950 INFO:tasks.workunit.client.1.smithi203.stdout:9/198: dwrite fd [0,4194304] 0 2024-04-05T03:02:36.956 INFO:tasks.workunit.client.0.smithi153.stdout:6/42: rename c2 to d4/cd 0 2024-04-05T03:02:36.957 INFO:tasks.workunit.client.0.smithi153.stdout:6/43: chown d4/dc 715 1 2024-04-05T03:02:36.960 INFO:tasks.workunit.client.1.smithi203.stdout:0/236: dwrite dd/d17/d1d/d22/f32 [0,4194304] 0 2024-04-05T03:02:36.961 INFO:tasks.workunit.client.0.smithi153.stdout:5/46: chown d0/l12 0 1 2024-04-05T03:02:36.962 INFO:tasks.workunit.client.0.smithi153.stdout:6/44: mknod d4/ce 0 2024-04-05T03:02:36.964 INFO:tasks.workunit.client.1.smithi203.stdout:2/306: truncate d5/f8 1520399 0 2024-04-05T03:02:36.964 INFO:tasks.workunit.client.1.smithi203.stdout:0/237: read dd/f15 [1648916,102031] 0 2024-04-05T03:02:36.966 INFO:tasks.workunit.client.0.smithi153.stdout:5/47: write d0/f3 [3744379,13084] 0 2024-04-05T03:02:36.968 INFO:tasks.workunit.client.0.smithi153.stdout:5/48: symlink d0/d9/l13 0 2024-04-05T03:02:36.969 INFO:tasks.workunit.client.0.smithi153.stdout:6/45: mkdir d4/dc/df 0 2024-04-05T03:02:36.969 INFO:tasks.workunit.client.1.smithi203.stdout:2/307: fdatasync f3 0 2024-04-05T03:02:36.970 INFO:tasks.workunit.client.1.smithi203.stdout:9/199: dwrite fd [0,4194304] 0 2024-04-05T03:02:36.971 INFO:tasks.workunit.client.0.smithi153.stdout:5/49: creat d0/f14 x:0 0 0 2024-04-05T03:02:36.973 INFO:tasks.workunit.client.1.smithi203.stdout:0/238: mkdir dd/d17/d1d/d26/d41 0 2024-04-05T03:02:36.973 INFO:tasks.workunit.client.1.smithi203.stdout:2/308: creat d5/d10/d1a/f65 x:0 0 0 2024-04-05T03:02:36.973 INFO:tasks.workunit.client.0.smithi153.stdout:6/46: symlink d4/dc/l10 0 2024-04-05T03:02:36.977 INFO:tasks.workunit.client.0.smithi153.stdout:6/47: mknod d4/dc/df/c11 0 2024-04-05T03:02:36.980 INFO:tasks.workunit.client.1.smithi203.stdout:0/239: unlink dd/d17/d1d/d26/f3a 0 2024-04-05T03:02:36.981 INFO:tasks.workunit.client.1.smithi203.stdout:2/309: mkdir d5/d66 0 2024-04-05T03:02:36.982 INFO:tasks.workunit.client.0.smithi153.stdout:6/48: dread f3 [0,4194304] 0 2024-04-05T03:02:36.984 INFO:tasks.workunit.client.1.smithi203.stdout:0/240: readlink dd/l13 0 2024-04-05T03:02:36.984 INFO:tasks.workunit.client.1.smithi203.stdout:2/310: mkdir d5/d10/d23/d29/d67 0 2024-04-05T03:02:36.985 INFO:tasks.workunit.client.1.smithi203.stdout:2/311: write d5/d10/f5b [541378,117481] 0 2024-04-05T03:02:36.987 INFO:tasks.workunit.client.1.smithi203.stdout:0/241: creat dd/d17/d1d/d26/f42 x:0 0 0 2024-04-05T03:02:36.988 INFO:tasks.workunit.client.0.smithi153.stdout:4/59: truncate f8 3815322 0 2024-04-05T03:02:36.989 INFO:tasks.workunit.client.1.smithi203.stdout:2/312: creat d5/d10/d2f/d5d/f68 x:0 0 0 2024-04-05T03:02:36.991 INFO:tasks.workunit.client.1.smithi203.stdout:2/313: write d5/fa [4420911,40865] 0 2024-04-05T03:02:36.993 INFO:tasks.workunit.client.0.smithi153.stdout:4/60: creat da/de/f11 x:0 0 0 2024-04-05T03:02:36.994 INFO:tasks.workunit.client.1.smithi203.stdout:0/242: fdatasync dd/d17/f40 0 2024-04-05T03:02:36.995 INFO:tasks.workunit.client.1.smithi203.stdout:0/243: write f6 [7331090,85995] 0 2024-04-05T03:02:36.996 INFO:tasks.workunit.client.1.smithi203.stdout:2/314: symlink d5/l69 0 2024-04-05T03:02:36.997 INFO:tasks.workunit.client.0.smithi153.stdout:7/65: rename d2/da/d11/d12/l15 to d2/da/l16 0 2024-04-05T03:02:36.999 INFO:tasks.workunit.client.1.smithi203.stdout:2/315: write d5/d10/d35/d2d/f50 [4744619,70340] 0 2024-04-05T03:02:36.999 INFO:tasks.workunit.client.0.smithi153.stdout:4/61: mkdir da/db/d12 0 2024-04-05T03:02:37.000 INFO:tasks.workunit.client.0.smithi153.stdout:4/62: truncate da/fc 625076 0 2024-04-05T03:02:37.001 INFO:tasks.workunit.client.0.smithi153.stdout:7/66: mknod d2/da/df/c17 0 2024-04-05T03:02:37.003 INFO:tasks.workunit.client.0.smithi153.stdout:7/67: creat d2/da/d11/f18 x:0 0 0 2024-04-05T03:02:37.003 INFO:tasks.workunit.client.1.smithi203.stdout:0/244: symlink dd/d17/d1d/d26/l43 0 2024-04-05T03:02:37.004 INFO:tasks.workunit.client.1.smithi203.stdout:2/316: readlink d5/d10/d35/d36/l55 0 2024-04-05T03:02:37.005 INFO:tasks.workunit.client.1.smithi203.stdout:2/317: write d5/d10/d1a/f4b [587511,33518] 0 2024-04-05T03:02:37.006 INFO:tasks.workunit.client.0.smithi153.stdout:7/68: creat d2/da/d11/d12/f19 x:0 0 0 2024-04-05T03:02:37.006 INFO:tasks.workunit.client.0.smithi153.stdout:7/69: rename d2/da to d2/da/df/d1a 22 2024-04-05T03:02:37.008 INFO:tasks.workunit.client.0.smithi153.stdout:7/70: fsync d2/f5 0 2024-04-05T03:02:37.009 INFO:tasks.workunit.client.1.smithi203.stdout:2/318: write d5/f9 [6286949,86628] 0 2024-04-05T03:02:37.012 INFO:tasks.workunit.client.0.smithi153.stdout:7/71: unlink d2/cc 0 2024-04-05T03:02:37.015 INFO:tasks.workunit.client.0.smithi153.stdout:4/63: dwrite f8 [0,4194304] 0 2024-04-05T03:02:37.016 INFO:tasks.workunit.client.0.smithi153.stdout:7/72: creat d2/da/d11/d12/f1b x:0 0 0 2024-04-05T03:02:37.018 INFO:tasks.workunit.client.1.smithi203.stdout:2/319: dwrite d5/d10/d35/d2d/f50 [0,4194304] 0 2024-04-05T03:02:37.021 INFO:tasks.workunit.client.1.smithi203.stdout:0/245: dwrite dd/d17/f40 [0,4194304] 0 2024-04-05T03:02:37.022 INFO:tasks.workunit.client.1.smithi203.stdout:2/320: write d5/d10/d23/d29/f53 [1624253,87015] 0 2024-04-05T03:02:37.023 INFO:tasks.workunit.client.0.smithi153.stdout:4/64: rmdir da/db/d12 0 2024-04-05T03:02:37.024 INFO:tasks.workunit.client.0.smithi153.stdout:7/73: link d2/da/cd d2/da/c1c 0 2024-04-05T03:02:37.025 INFO:tasks.workunit.client.0.smithi153.stdout:7/74: dread - d2/da/d11/d12/f19 zero size 2024-04-05T03:02:37.027 INFO:tasks.workunit.client.0.smithi153.stdout:4/65: dread f7 [0,4194304] 0 2024-04-05T03:02:37.028 INFO:tasks.workunit.client.1.smithi203.stdout:0/246: write dd/d17/d1d/d26/d34/f35 [4106298,68427] 0 2024-04-05T03:02:37.032 INFO:tasks.workunit.client.1.smithi203.stdout:0/247: unlink dd/c2f 0 2024-04-05T03:02:37.034 INFO:tasks.workunit.client.0.smithi153.stdout:7/75: dread d2/f6 [4194304,4194304] 0 2024-04-05T03:02:37.038 INFO:tasks.workunit.client.1.smithi203.stdout:2/321: dwrite d5/f5e [0,4194304] 0 2024-04-05T03:02:37.040 INFO:tasks.workunit.client.1.smithi203.stdout:0/248: rmdir dd/d17/d1d/d26/d38 0 2024-04-05T03:02:37.042 INFO:tasks.workunit.client.1.smithi203.stdout:0/249: fdatasync dd/d17/d1d/d26/d28/f3f 0 2024-04-05T03:02:37.042 INFO:tasks.workunit.client.0.smithi153.stdout:7/76: link d2/da/d11/f18 d2/f1d 0 2024-04-05T03:02:37.045 INFO:tasks.workunit.client.1.smithi203.stdout:2/322: truncate d5/d10/f20 2219167 0 2024-04-05T03:02:37.046 INFO:tasks.workunit.client.0.smithi153.stdout:7/77: creat d2/da/d11/d12/f1e x:0 0 0 2024-04-05T03:02:37.047 INFO:tasks.workunit.client.1.smithi203.stdout:0/250: creat dd/f44 x:0 0 0 2024-04-05T03:02:37.054 INFO:tasks.workunit.client.1.smithi203.stdout:0/251: dwrite dd/f14 [0,4194304] 0 2024-04-05T03:02:37.057 INFO:tasks.workunit.client.1.smithi203.stdout:0/252: mkdir dd/d17/d1d/d22/d45 0 2024-04-05T03:02:37.059 INFO:tasks.workunit.client.0.smithi153.stdout:4/66: dwrite f9 [0,4194304] 0 2024-04-05T03:02:37.060 INFO:tasks.workunit.client.0.smithi153.stdout:4/67: write da/db/fd [995593,82546] 0 2024-04-05T03:02:37.065 INFO:tasks.workunit.client.0.smithi153.stdout:4/68: link f7 da/db/f13 0 2024-04-05T03:02:37.067 INFO:tasks.workunit.client.1.smithi203.stdout:2/323: truncate d5/d10/d35/d2d/f50 3712627 0 2024-04-05T03:02:37.068 INFO:tasks.workunit.client.0.smithi153.stdout:4/69: write f7 [182434,41138] 0 2024-04-05T03:02:37.070 INFO:tasks.workunit.client.1.smithi203.stdout:0/253: dwrite dd/d17/d1d/d26/f3d [0,4194304] 0 2024-04-05T03:02:37.073 INFO:tasks.workunit.client.0.smithi153.stdout:4/70: creat da/de/f14 x:0 0 0 2024-04-05T03:02:37.079 INFO:tasks.workunit.client.1.smithi203.stdout:0/254: getdents dd/d17/d1d 0 2024-04-05T03:02:37.082 INFO:tasks.workunit.client.0.smithi153.stdout:4/71: link l6 da/l15 0 2024-04-05T03:02:37.084 INFO:tasks.workunit.client.1.smithi203.stdout:0/255: dread dd/f15 [0,4194304] 0 2024-04-05T03:02:37.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/72: dread f9 [0,4194304] 0 2024-04-05T03:02:37.087 INFO:tasks.workunit.client.1.smithi203.stdout:0/256: chown dd/c24 545397635 1 2024-04-05T03:02:37.088 INFO:tasks.workunit.client.0.smithi153.stdout:4/73: rmdir da/db 39 2024-04-05T03:02:37.088 INFO:tasks.workunit.client.1.smithi203.stdout:2/324: dwrite d5/d10/d1a/f34 [0,4194304] 0 2024-04-05T03:02:37.091 INFO:tasks.workunit.client.1.smithi203.stdout:2/325: symlink d5/d10/d1a/l6a 0 2024-04-05T03:02:37.093 INFO:tasks.workunit.client.0.smithi153.stdout:4/74: fdatasync da/db/f13 0 2024-04-05T03:02:37.095 INFO:tasks.workunit.client.0.smithi153.stdout:4/75: symlink da/db/l16 0 2024-04-05T03:02:37.096 INFO:tasks.workunit.client.1.smithi203.stdout:2/326: creat d5/d10/d1a/d57/f6b x:0 0 0 2024-04-05T03:02:37.096 INFO:tasks.workunit.client.1.smithi203.stdout:2/327: chown d5/d10/d23/f25 37 1 2024-04-05T03:02:37.099 INFO:tasks.workunit.client.0.smithi153.stdout:4/76: creat da/f17 x:0 0 0 2024-04-05T03:02:37.099 INFO:tasks.workunit.client.1.smithi203.stdout:2/328: creat d5/d10/d35/d36/f6c x:0 0 0 2024-04-05T03:02:37.101 INFO:tasks.workunit.client.1.smithi203.stdout:2/329: dread d5/d10/d35/d36/f58 [0,4194304] 0 2024-04-05T03:02:37.104 INFO:tasks.workunit.client.1.smithi203.stdout:2/330: mknod d5/d10/d2f/c6d 0 2024-04-05T03:02:37.110 INFO:tasks.workunit.client.1.smithi203.stdout:0/257: dwrite f7 [0,4194304] 0 2024-04-05T03:02:37.113 INFO:tasks.workunit.client.1.smithi203.stdout:2/331: dwrite d5/d10/f5b [0,4194304] 0 2024-04-05T03:02:37.113 INFO:tasks.workunit.client.0.smithi153.stdout:4/77: rmdir da 39 2024-04-05T03:02:37.114 INFO:tasks.workunit.client.1.smithi203.stdout:0/258: mknod dd/d17/c46 0 2024-04-05T03:02:37.115 INFO:tasks.workunit.client.0.smithi153.stdout:4/78: chown da/de/f11 1 1 2024-04-05T03:02:37.118 INFO:tasks.workunit.client.1.smithi203.stdout:2/332: creat d5/d63/f6e x:0 0 0 2024-04-05T03:02:37.122 INFO:tasks.workunit.client.0.smithi153.stdout:4/79: getdents da/db 0 2024-04-05T03:02:37.125 INFO:tasks.workunit.client.1.smithi203.stdout:0/259: link dd/d17/d1d/d26/d28/f3f dd/d17/d1d/d26/d34/f47 0 2024-04-05T03:02:37.128 INFO:tasks.workunit.client.1.smithi203.stdout:0/260: creat dd/d17/f48 x:0 0 0 2024-04-05T03:02:37.129 INFO:tasks.workunit.client.0.smithi153.stdout:4/80: dwrite da/db/f13 [0,4194304] 0 2024-04-05T03:02:37.129 INFO:tasks.workunit.client.0.smithi153.stdout:4/81: dread - da/ff zero size 2024-04-05T03:02:37.131 INFO:tasks.workunit.client.0.smithi153.stdout:4/82: readlink da/db/l16 0 2024-04-05T03:02:37.132 INFO:tasks.workunit.client.1.smithi203.stdout:0/261: unlink dd/d17/d1d/f2e 0 2024-04-05T03:02:37.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/83: dwrite da/f17 [0,4194304] 0 2024-04-05T03:02:37.141 INFO:tasks.workunit.client.1.smithi203.stdout:2/333: write d5/d10/d2f/f59 [440131,45837] 0 2024-04-05T03:02:37.142 INFO:tasks.workunit.client.0.smithi153.stdout:4/84: mknod da/db/c18 0 2024-04-05T03:02:37.146 INFO:tasks.workunit.client.0.smithi153.stdout:4/85: creat da/de/f19 x:0 0 0 2024-04-05T03:02:37.146 INFO:tasks.workunit.client.1.smithi203.stdout:0/262: write dd/d17/d1d/d26/d28/f2b [1374976,82601] 0 2024-04-05T03:02:37.147 INFO:tasks.workunit.client.1.smithi203.stdout:0/263: fsync f5 0 2024-04-05T03:02:37.149 INFO:tasks.workunit.client.0.smithi153.stdout:4/86: chown da/l15 310631025 1 2024-04-05T03:02:37.152 INFO:tasks.workunit.client.0.smithi153.stdout:4/87: creat da/db/f1a x:0 0 0 2024-04-05T03:02:37.153 INFO:tasks.workunit.client.0.smithi153.stdout:4/88: chown da/db/f1a 252714073 1 2024-04-05T03:02:37.153 INFO:tasks.workunit.client.1.smithi203.stdout:2/334: write d5/d10/d1a/d24/f2b [854148,111082] 0 2024-04-05T03:02:37.154 INFO:tasks.workunit.client.1.smithi203.stdout:2/335: stat f3 0 2024-04-05T03:02:37.156 INFO:tasks.workunit.client.1.smithi203.stdout:0/264: dwrite dd/f15 [0,4194304] 0 2024-04-05T03:02:37.157 INFO:tasks.workunit.client.1.smithi203.stdout:0/265: truncate dd/d17/f48 259297 0 2024-04-05T03:02:37.158 INFO:tasks.workunit.client.1.smithi203.stdout:0/266: fsync dd/d17/d1d/d26/f3d 0 2024-04-05T03:02:37.161 INFO:tasks.workunit.client.1.smithi203.stdout:0/267: write dd/d17/d29/f3c [1011198,8571] 0 2024-04-05T03:02:37.162 INFO:tasks.workunit.client.1.smithi203.stdout:0/268: chown cb 456036 1 2024-04-05T03:02:37.167 INFO:tasks.workunit.client.0.smithi153.stdout:4/89: rename da/db/l16 to da/db/l1b 0 2024-04-05T03:02:37.167 INFO:tasks.workunit.client.1.smithi203.stdout:2/336: dwrite d5/d10/d35/d36/f54 [0,4194304] 0 2024-04-05T03:02:37.168 INFO:tasks.workunit.client.0.smithi153.stdout:4/90: write da/db/f13 [3996922,73299] 0 2024-04-05T03:02:37.169 INFO:tasks.workunit.client.0.smithi153.stdout:4/91: truncate da/de/f14 529001 0 2024-04-05T03:02:37.173 INFO:tasks.workunit.client.0.smithi153.stdout:4/92: dread f9 [0,4194304] 0 2024-04-05T03:02:37.174 INFO:tasks.workunit.client.1.smithi203.stdout:2/337: dwrite d5/d10/d23/d29/f53 [0,4194304] 0 2024-04-05T03:02:37.175 INFO:tasks.workunit.client.0.smithi153.stdout:4/93: creat da/de/f1c x:0 0 0 2024-04-05T03:02:37.176 INFO:tasks.workunit.client.0.smithi153.stdout:4/94: dread - da/db/f1a zero size 2024-04-05T03:02:37.180 INFO:tasks.workunit.client.1.smithi203.stdout:2/338: creat d5/f6f x:0 0 0 2024-04-05T03:02:37.183 INFO:tasks.workunit.client.1.smithi203.stdout:2/339: mkdir d5/d10/d1a/d57/d70 0 2024-04-05T03:02:37.186 INFO:tasks.workunit.client.1.smithi203.stdout:2/340: fsync d5/f8 0 2024-04-05T03:02:37.188 INFO:tasks.workunit.client.0.smithi153.stdout:4/95: getdents da/db 0 2024-04-05T03:02:37.188 INFO:tasks.workunit.client.1.smithi203.stdout:2/341: chown d5/d10/d1a/f5a 7 1 2024-04-05T03:02:37.190 INFO:tasks.workunit.client.1.smithi203.stdout:2/342: rmdir d5/d10/d35/d3e 39 2024-04-05T03:02:37.191 INFO:tasks.workunit.client.0.smithi153.stdout:4/96: getdents da/de 0 2024-04-05T03:02:37.192 INFO:tasks.workunit.client.1.smithi203.stdout:2/343: mkdir d5/d10/d1a/d24/d71 0 2024-04-05T03:02:37.198 INFO:tasks.workunit.client.1.smithi203.stdout:0/269: dread f5 [0,4194304] 0 2024-04-05T03:02:37.198 INFO:tasks.workunit.client.1.smithi203.stdout:0/270: stat dd/d17/d1d/d22 0 2024-04-05T03:02:37.199 INFO:tasks.workunit.client.0.smithi153.stdout:4/97: dwrite f9 [4194304,4194304] 0 2024-04-05T03:02:37.200 INFO:tasks.workunit.client.0.smithi153.stdout:4/98: fdatasync f8 0 2024-04-05T03:02:37.200 INFO:tasks.workunit.client.1.smithi203.stdout:2/344: dread d5/f8 [0,4194304] 0 2024-04-05T03:02:37.201 INFO:tasks.workunit.client.1.smithi203.stdout:2/345: chown d5/d10/d35/d2d/f38 42 1 2024-04-05T03:02:37.201 INFO:tasks.workunit.client.1.smithi203.stdout:0/271: mkdir dd/d17/d1d/d26/d41/d49 0 2024-04-05T03:02:37.202 INFO:tasks.workunit.client.1.smithi203.stdout:2/346: write d5/d10/d1a/d24/f37 [441199,40065] 0 2024-04-05T03:02:37.204 INFO:tasks.workunit.client.1.smithi203.stdout:0/272: readlink dd/lf 0 2024-04-05T03:02:37.206 INFO:tasks.workunit.client.1.smithi203.stdout:2/347: dread f3 [0,4194304] 0 2024-04-05T03:02:37.207 INFO:tasks.workunit.client.1.smithi203.stdout:0/273: chown dd/d17/d29/c31 146014028 1 2024-04-05T03:02:37.207 INFO:tasks.workunit.client.1.smithi203.stdout:0/274: readlink dd/lf 0 2024-04-05T03:02:37.208 INFO:tasks.workunit.client.0.smithi153.stdout:4/99: dwrite da/db/fd [4194304,4194304] 0 2024-04-05T03:02:37.208 INFO:tasks.workunit.client.1.smithi203.stdout:0/275: write dd/d17/d1d/d22/f3b [444758,6885] 0 2024-04-05T03:02:37.210 INFO:tasks.workunit.client.1.smithi203.stdout:2/348: mkdir d5/d10/d23/d72 0 2024-04-05T03:02:37.210 INFO:tasks.workunit.client.1.smithi203.stdout:0/276: stat dd/d17/d1d/d26/d41 0 2024-04-05T03:02:37.213 INFO:tasks.workunit.client.0.smithi153.stdout:4/100: dread f8 [0,4194304] 0 2024-04-05T03:02:37.213 INFO:tasks.workunit.client.1.smithi203.stdout:2/349: write d5/d10/d2f/f64 [4551281,45943] 0 2024-04-05T03:02:37.214 INFO:tasks.workunit.client.1.smithi203.stdout:0/277: mknod dd/d17/d1d/d26/d41/d49/c4a 0 2024-04-05T03:02:37.216 INFO:tasks.workunit.client.1.smithi203.stdout:2/350: mkdir d5/d10/d35/d73 0 2024-04-05T03:02:37.217 INFO:tasks.workunit.client.0.smithi153.stdout:4/101: creat da/f1d x:0 0 0 2024-04-05T03:02:37.217 INFO:tasks.workunit.client.1.smithi203.stdout:0/278: creat dd/d17/d1d/d26/d41/f4b x:0 0 0 2024-04-05T03:02:37.218 INFO:tasks.workunit.client.0.smithi153.stdout:4/102: write da/ff [733585,99902] 0 2024-04-05T03:02:37.219 INFO:tasks.workunit.client.1.smithi203.stdout:2/351: unlink d5/d10/d1a/d24/c51 0 2024-04-05T03:02:37.222 INFO:tasks.workunit.client.1.smithi203.stdout:2/352: symlink d5/d10/d35/l74 0 2024-04-05T03:02:37.224 INFO:tasks.workunit.client.1.smithi203.stdout:2/353: chown d5/d10/d35/d3e/f56 7751660 1 2024-04-05T03:02:37.225 INFO:tasks.workunit.client.1.smithi203.stdout:2/354: chown d5/d10/d1a/d57 382686 1 2024-04-05T03:02:37.226 INFO:tasks.workunit.client.1.smithi203.stdout:2/355: write d5/d63/f6e [58975,120767] 0 2024-04-05T03:02:37.228 INFO:tasks.workunit.client.1.smithi203.stdout:2/356: creat d5/d63/f75 x:0 0 0 2024-04-05T03:02:37.230 INFO:tasks.workunit.client.1.smithi203.stdout:2/357: mknod d5/d10/d23/d29/c76 0 2024-04-05T03:02:37.233 INFO:tasks.workunit.client.1.smithi203.stdout:0/279: dwrite dd/d17/d1d/d26/d28/f3f [0,4194304] 0 2024-04-05T03:02:37.234 INFO:tasks.workunit.client.1.smithi203.stdout:0/280: readlink dd/l13 0 2024-04-05T03:02:37.237 INFO:tasks.workunit.client.1.smithi203.stdout:2/358: rename d5/d10/d35/d2d/f2e to d5/d10/d23/d29/d67/f77 0 2024-04-05T03:02:37.238 INFO:tasks.workunit.client.1.smithi203.stdout:0/281: mknod dd/d17/d1d/d26/d41/c4c 0 2024-04-05T03:02:37.242 INFO:tasks.workunit.client.1.smithi203.stdout:2/359: rename d5/d10/d2f/f64 to d5/d10/d23/f78 0 2024-04-05T03:02:37.243 INFO:tasks.workunit.client.1.smithi203.stdout:0/282: symlink dd/d17/d1d/d26/d41/d49/l4d 0 2024-04-05T03:02:37.246 INFO:tasks.workunit.client.1.smithi203.stdout:0/283: creat dd/d17/d1d/d26/d28/f4e x:0 0 0 2024-04-05T03:02:37.246 INFO:tasks.workunit.client.1.smithi203.stdout:2/360: dread d5/d10/d2f/f59 [0,4194304] 0 2024-04-05T03:02:37.249 INFO:tasks.workunit.client.1.smithi203.stdout:2/361: rename d5/d10/d35/d3e to d5/d10/d35/d36/d41/d79 0 2024-04-05T03:02:37.254 INFO:tasks.workunit.client.1.smithi203.stdout:2/362: getdents d5/d10/d35/d2d 0 2024-04-05T03:02:37.256 INFO:tasks.workunit.client.1.smithi203.stdout:2/363: mkdir d5/d10/d23/d7a 0 2024-04-05T03:02:37.260 INFO:tasks.workunit.client.1.smithi203.stdout:2/364: write d5/d10/d23/d29/f53 [4814193,3516] 0 2024-04-05T03:02:37.267 INFO:tasks.workunit.client.1.smithi203.stdout:2/365: dwrite d5/d10/d35/d36/f58 [0,4194304] 0 2024-04-05T03:02:37.269 INFO:tasks.workunit.client.1.smithi203.stdout:2/366: chown d5/d10/d35/d2d/f38 141 1 2024-04-05T03:02:37.271 INFO:tasks.workunit.client.1.smithi203.stdout:2/367: rename d5/c42 to d5/d10/d1a/d57/d70/c7b 0 2024-04-05T03:02:37.276 INFO:tasks.workunit.client.1.smithi203.stdout:2/368: rename d5/d10/d23 to d5/d10/d35/d2d/d7c 0 2024-04-05T03:02:37.285 INFO:tasks.workunit.client.1.smithi203.stdout:2/369: truncate d5/d10/d35/d2d/d7c/f78 711255 0 2024-04-05T03:02:37.285 INFO:tasks.workunit.client.1.smithi203.stdout:2/370: fdatasync d5/d10/f1d 0 2024-04-05T03:02:37.287 INFO:tasks.workunit.client.1.smithi203.stdout:2/371: mkdir d5/d66/d7d 0 2024-04-05T03:02:37.288 INFO:tasks.workunit.client.1.smithi203.stdout:2/372: mknod d5/d66/d7d/c7e 0 2024-04-05T03:02:37.290 INFO:tasks.workunit.client.1.smithi203.stdout:2/373: mkdir d5/d10/d7f 0 2024-04-05T03:02:37.294 INFO:tasks.workunit.client.1.smithi203.stdout:2/374: dread d5/d10/d1a/f1b [0,4194304] 0 2024-04-05T03:02:37.294 INFO:tasks.workunit.client.1.smithi203.stdout:2/375: chown d5/d10/d1a/f5a 0 1 2024-04-05T03:02:37.302 INFO:tasks.workunit.client.1.smithi203.stdout:5/287: sync 2024-04-05T03:02:37.304 INFO:tasks.workunit.client.1.smithi203.stdout:2/376: write f3 [1261718,59306] 0 2024-04-05T03:02:37.311 INFO:tasks.workunit.client.1.smithi203.stdout:5/288: dwrite d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:37.311 INFO:tasks.workunit.client.1.smithi203.stdout:2/377: truncate d5/f9 263209 0 2024-04-05T03:02:37.312 INFO:tasks.workunit.client.1.smithi203.stdout:2/378: write d5/d10/d1a/d24/f37 [1158841,88328] 0 2024-04-05T03:02:37.315 INFO:tasks.workunit.client.1.smithi203.stdout:2/379: mknod d5/d10/d1a/d24/d71/c80 0 2024-04-05T03:02:37.319 INFO:tasks.workunit.client.1.smithi203.stdout:2/380: write d5/d10/d35/d36/d41/d79/f56 [1490718,110567] 0 2024-04-05T03:02:37.322 INFO:tasks.workunit.client.1.smithi203.stdout:2/381: rename d5/l69 to d5/d10/d35/d73/l81 0 2024-04-05T03:02:37.324 INFO:tasks.workunit.client.1.smithi203.stdout:5/289: dwrite d3/d4/d8/f44 [0,4194304] 0 2024-04-05T03:02:37.324 INFO:tasks.workunit.client.1.smithi203.stdout:5/290: readlink d3/la 0 2024-04-05T03:02:37.325 INFO:tasks.workunit.client.1.smithi203.stdout:2/382: mkdir d5/d10/d35/d2d/d7c/d29/d82 0 2024-04-05T03:02:37.334 INFO:tasks.workunit.client.1.smithi203.stdout:2/383: dread d5/d10/d1a/d24/f2b [0,4194304] 0 2024-04-05T03:02:37.343 INFO:tasks.workunit.client.1.smithi203.stdout:5/291: dread d3/d4/d8/f1c [0,4194304] 0 2024-04-05T03:02:37.343 INFO:tasks.workunit.client.1.smithi203.stdout:2/384: truncate d5/f43 2264861 0 2024-04-05T03:02:37.344 INFO:tasks.workunit.client.1.smithi203.stdout:5/292: read d3/d1f/f39 [2816386,5057] 0 2024-04-05T03:02:37.345 INFO:tasks.workunit.client.1.smithi203.stdout:5/293: chown d3/c9 198725486 1 2024-04-05T03:02:37.347 INFO:tasks.workunit.client.1.smithi203.stdout:5/294: mkdir d3/d5f 0 2024-04-05T03:02:37.349 INFO:tasks.workunit.client.1.smithi203.stdout:5/295: mkdir d3/d1f/d59/d60 0 2024-04-05T03:02:37.350 INFO:tasks.workunit.client.1.smithi203.stdout:5/296: fdatasync f2 0 2024-04-05T03:02:37.350 INFO:tasks.workunit.client.1.smithi203.stdout:2/385: dwrite d5/d10/d35/d36/f6c [0,4194304] 0 2024-04-05T03:02:37.355 INFO:tasks.workunit.client.1.smithi203.stdout:5/297: rename d3/la to d3/d4/dc/l61 0 2024-04-05T03:02:37.357 INFO:tasks.workunit.client.1.smithi203.stdout:5/298: write d3/d4/fb [12390186,57523] 0 2024-04-05T03:02:37.358 INFO:tasks.workunit.client.1.smithi203.stdout:2/386: rename d5/d10/d2f/c4e to d5/d10/d1a/d24/c83 0 2024-04-05T03:02:37.360 INFO:tasks.workunit.client.1.smithi203.stdout:2/387: dread d5/d10/d1a/d24/f27 [0,4194304] 0 2024-04-05T03:02:37.361 INFO:tasks.workunit.client.1.smithi203.stdout:5/299: creat d3/d1f/f62 x:0 0 0 2024-04-05T03:02:37.363 INFO:tasks.workunit.client.1.smithi203.stdout:2/388: mknod d5/d10/d35/c84 0 2024-04-05T03:02:37.366 INFO:tasks.workunit.client.1.smithi203.stdout:2/389: rmdir d5/d10/d35/d36/d41/d79 39 2024-04-05T03:02:37.370 INFO:tasks.workunit.client.1.smithi203.stdout:2/390: dread d5/d10/d1a/f4b [0,4194304] 0 2024-04-05T03:02:37.372 INFO:tasks.workunit.client.1.smithi203.stdout:2/391: mknod d5/d10/d44/c85 0 2024-04-05T03:02:37.374 INFO:tasks.workunit.client.1.smithi203.stdout:2/392: mkdir d5/d63/d86 0 2024-04-05T03:02:37.377 INFO:tasks.workunit.client.1.smithi203.stdout:5/300: dread d3/f18 [0,4194304] 0 2024-04-05T03:02:37.378 INFO:tasks.workunit.client.1.smithi203.stdout:2/393: mkdir d5/d10/d1a/d24/d87 0 2024-04-05T03:02:37.380 INFO:tasks.workunit.client.1.smithi203.stdout:2/394: write d5/d10/d1a/f4b [981759,35938] 0 2024-04-05T03:02:37.382 INFO:tasks.workunit.client.1.smithi203.stdout:2/395: fsync d5/d10/d1a/d24/f2b 0 2024-04-05T03:02:37.393 INFO:tasks.workunit.client.1.smithi203.stdout:5/301: dread d3/d4/d8/d12/f43 [0,4194304] 0 2024-04-05T03:02:37.398 INFO:tasks.workunit.client.1.smithi203.stdout:2/396: dread d5/d10/d1a/d24/f3f [0,4194304] 0 2024-04-05T03:02:37.399 INFO:tasks.workunit.client.1.smithi203.stdout:5/302: read d3/d1f/f39 [2010527,7177] 0 2024-04-05T03:02:37.405 INFO:tasks.workunit.client.1.smithi203.stdout:5/303: rename d3/d4/d8/d12/f4e to d3/d4/d8/f63 0 2024-04-05T03:02:37.406 INFO:tasks.workunit.client.1.smithi203.stdout:5/304: fsync d3/d4/d8/f2e 0 2024-04-05T03:02:37.412 INFO:tasks.workunit.client.1.smithi203.stdout:5/305: rename d3/d4/d8 to d3/d1f/d59/d64 0 2024-04-05T03:02:37.415 INFO:tasks.workunit.client.1.smithi203.stdout:5/306: symlink d3/d1f/d49/l65 0 2024-04-05T03:02:37.428 INFO:tasks.workunit.client.1.smithi203.stdout:5/307: dread d3/d1f/d59/d64/f2e [0,4194304] 0 2024-04-05T03:02:37.438 INFO:tasks.workunit.client.1.smithi203.stdout:5/308: dwrite d3/f18 [4194304,4194304] 0 2024-04-05T03:02:37.439 INFO:tasks.workunit.client.1.smithi203.stdout:5/309: chown d3/d4/dc/l17 52682649 1 2024-04-05T03:02:37.445 INFO:tasks.workunit.client.1.smithi203.stdout:5/310: truncate d3/f18 1831944 0 2024-04-05T03:02:37.452 INFO:tasks.workunit.client.1.smithi203.stdout:5/311: dwrite d3/d1f/d59/d64/d12/f43 [0,4194304] 0 2024-04-05T03:02:37.484 INFO:tasks.workunit.client.1.smithi203.stdout:8/305: sync 2024-04-05T03:02:37.487 INFO:tasks.workunit.client.1.smithi203.stdout:8/306: dread d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:37.491 INFO:tasks.workunit.client.1.smithi203.stdout:8/307: mkdir d1/d4d 0 2024-04-05T03:02:37.495 INFO:tasks.workunit.client.1.smithi203.stdout:8/308: mknod d1/d12/d2f/c4e 0 2024-04-05T03:02:37.502 INFO:tasks.workunit.client.1.smithi203.stdout:8/309: truncate d1/d12/d1a/f1e 1394871 0 2024-04-05T03:02:37.504 INFO:tasks.workunit.client.1.smithi203.stdout:8/310: symlink d1/d6/d2e/l4f 0 2024-04-05T03:02:37.505 INFO:tasks.workunit.client.1.smithi203.stdout:8/311: dread d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:37.511 INFO:tasks.workunit.client.1.smithi203.stdout:8/312: dwrite d1/d6/f45 [0,4194304] 0 2024-04-05T03:02:37.513 INFO:tasks.workunit.client.1.smithi203.stdout:8/313: chown d1/d6/d2e/c38 5200 1 2024-04-05T03:02:37.514 INFO:tasks.workunit.client.1.smithi203.stdout:8/314: write d1/d12/d2f/d1f/f23 [866289,13983] 0 2024-04-05T03:02:37.516 INFO:tasks.workunit.client.1.smithi203.stdout:8/315: write d1/d6/f45 [88211,100590] 0 2024-04-05T03:02:37.516 INFO:tasks.workunit.client.1.smithi203.stdout:8/316: readlink d1/d12/l32 0 2024-04-05T03:02:37.523 INFO:tasks.workunit.client.1.smithi203.stdout:8/317: dwrite d1/d6/f41 [0,4194304] 0 2024-04-05T03:02:37.526 INFO:tasks.workunit.client.1.smithi203.stdout:8/318: creat d1/d12/d2f/d1f/f50 x:0 0 0 2024-04-05T03:02:37.532 INFO:tasks.workunit.client.1.smithi203.stdout:8/319: rename d1/d12/d2f/d1f/l2b to d1/d12/l51 0 2024-04-05T03:02:37.533 INFO:tasks.workunit.client.1.smithi203.stdout:8/320: chown d1/d12/d1a/f46 17197571 1 2024-04-05T03:02:37.547 INFO:tasks.workunit.client.1.smithi203.stdout:8/321: truncate d1/d12/d17/d10/f15 1648558 0 2024-04-05T03:02:37.548 INFO:tasks.workunit.client.1.smithi203.stdout:8/322: readlink d1/l44 0 2024-04-05T03:02:37.549 INFO:tasks.workunit.client.1.smithi203.stdout:8/323: rmdir d1/d12/d1a 39 2024-04-05T03:02:37.549 INFO:tasks.workunit.client.1.smithi203.stdout:8/324: readlink d1/d12/d2f/l4c 0 2024-04-05T03:02:37.556 INFO:tasks.workunit.client.1.smithi203.stdout:8/325: write d1/d12/d17/d10/f15 [1781155,11608] 0 2024-04-05T03:02:37.565 INFO:tasks.workunit.client.1.smithi203.stdout:4/167: sync 2024-04-05T03:02:37.565 INFO:tasks.workunit.client.1.smithi203.stdout:7/200: sync 2024-04-05T03:02:37.566 INFO:tasks.workunit.client.1.smithi203.stdout:4/168: readlink da/l25 0 2024-04-05T03:02:37.567 INFO:tasks.workunit.client.1.smithi203.stdout:8/326: dwrite d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:37.575 INFO:tasks.workunit.client.1.smithi203.stdout:7/201: dwrite d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:37.579 INFO:tasks.workunit.client.1.smithi203.stdout:7/202: creat d8/f46 x:0 0 0 2024-04-05T03:02:37.581 INFO:tasks.workunit.client.1.smithi203.stdout:7/203: rename d8/d32/d16/d1a to d8/d32/d16/d1a/d3d/d47 22 2024-04-05T03:02:37.584 INFO:tasks.workunit.client.1.smithi203.stdout:7/204: symlink d8/d32/d16/d1a/d21/l48 0 2024-04-05T03:02:37.584 INFO:tasks.workunit.client.1.smithi203.stdout:7/205: chown d8/d32/d16/d1a/f1f 12 1 2024-04-05T03:02:37.590 INFO:tasks.workunit.client.1.smithi203.stdout:7/206: mknod d8/d2a/d3f/c49 0 2024-04-05T03:02:37.591 INFO:tasks.workunit.client.1.smithi203.stdout:7/207: truncate d8/da/d2f/f45 1024941 0 2024-04-05T03:02:37.597 INFO:tasks.workunit.client.1.smithi203.stdout:7/208: dwrite d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:37.600 INFO:tasks.workunit.client.1.smithi203.stdout:7/209: mknod d8/d32/d16/d1a/d21/c4a 0 2024-04-05T03:02:37.603 INFO:tasks.workunit.client.1.smithi203.stdout:7/210: write d8/f33 [4374448,52668] 0 2024-04-05T03:02:37.603 INFO:tasks.workunit.client.1.smithi203.stdout:7/211: chown d8/da/d2f/f45 0 1 2024-04-05T03:02:37.604 INFO:tasks.workunit.client.1.smithi203.stdout:7/212: truncate d8/f31 313764 0 2024-04-05T03:02:37.607 INFO:tasks.workunit.client.1.smithi203.stdout:7/213: symlink d8/d2a/d3f/l4b 0 2024-04-05T03:02:37.607 INFO:tasks.workunit.client.1.smithi203.stdout:7/214: fdatasync d8/f37 0 2024-04-05T03:02:37.608 INFO:tasks.workunit.client.1.smithi203.stdout:7/215: chown d8/da/d2f 23 1 2024-04-05T03:02:37.609 INFO:tasks.workunit.client.1.smithi203.stdout:7/216: write d8/d2a/f42 [471284,43267] 0 2024-04-05T03:02:37.611 INFO:tasks.workunit.client.1.smithi203.stdout:7/217: creat d8/f4c x:0 0 0 2024-04-05T03:02:37.612 INFO:tasks.workunit.client.1.smithi203.stdout:7/218: chown d8/da/d2f 1777904608 1 2024-04-05T03:02:37.614 INFO:tasks.workunit.client.1.smithi203.stdout:7/219: write d8/d32/d16/d1a/d21/f27 [3990285,79890] 0 2024-04-05T03:02:37.627 INFO:tasks.workunit.client.1.smithi203.stdout:7/220: dread d8/f17 [0,4194304] 0 2024-04-05T03:02:37.627 INFO:tasks.workunit.client.1.smithi203.stdout:7/221: readlink d8/d32/d16/d1a/d21/l26 0 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:6/277: sync 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:5/312: sync 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:1/226: sync 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:9/200: sync 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:0/284: sync 2024-04-05T03:02:37.655 INFO:tasks.workunit.client.1.smithi203.stdout:3/196: sync 2024-04-05T03:02:37.656 INFO:tasks.workunit.client.1.smithi203.stdout:2/397: sync 2024-04-05T03:02:37.657 INFO:tasks.workunit.client.1.smithi203.stdout:0/285: stat dd/d17/d1d/f37 0 2024-04-05T03:02:37.659 INFO:tasks.workunit.client.1.smithi203.stdout:2/398: creat d5/d10/d2f/f88 x:0 0 0 2024-04-05T03:02:37.660 INFO:tasks.workunit.client.1.smithi203.stdout:5/313: symlink d3/d1f/d59/l66 0 2024-04-05T03:02:37.663 INFO:tasks.workunit.client.1.smithi203.stdout:2/399: fsync d5/d10/d35/d2d/f33 0 2024-04-05T03:02:37.664 INFO:tasks.workunit.client.1.smithi203.stdout:6/278: truncate d0/f3 4784295 0 2024-04-05T03:02:37.666 INFO:tasks.workunit.client.1.smithi203.stdout:5/314: mkdir d3/d4/dc/d67 0 2024-04-05T03:02:37.668 INFO:tasks.workunit.client.1.smithi203.stdout:2/400: chown d5/d10/d35/d2d/d7c/f78 212574 1 2024-04-05T03:02:37.669 INFO:tasks.workunit.client.1.smithi203.stdout:0/286: creat dd/d17/d1d/d26/f4f x:0 0 0 2024-04-05T03:02:37.671 INFO:tasks.workunit.client.1.smithi203.stdout:1/227: dwrite d7/d24/f34 [0,4194304] 0 2024-04-05T03:02:37.672 INFO:tasks.workunit.client.1.smithi203.stdout:5/315: fdatasync d3/d1f/d59/d64/f4f 0 2024-04-05T03:02:37.673 INFO:tasks.workunit.client.1.smithi203.stdout:0/287: chown dd/d17/l1a 63556 1 2024-04-05T03:02:37.679 INFO:tasks.workunit.client.1.smithi203.stdout:3/197: dread d0/fd [0,4194304] 0 2024-04-05T03:02:37.680 INFO:tasks.workunit.client.1.smithi203.stdout:9/201: dread f7 [0,4194304] 0 2024-04-05T03:02:37.682 INFO:tasks.workunit.client.1.smithi203.stdout:6/279: write d0/da/d18/d38/d2a/d44/f4a [757702,105022] 0 2024-04-05T03:02:37.685 INFO:tasks.workunit.client.1.smithi203.stdout:0/288: symlink dd/d17/d1d/d26/d41/l50 0 2024-04-05T03:02:37.685 INFO:tasks.workunit.client.1.smithi203.stdout:1/228: symlink d7/d24/l4c 0 2024-04-05T03:02:37.688 INFO:tasks.workunit.client.1.smithi203.stdout:2/401: rename d5/d10/d2f/f59 to d5/d10/d1a/d24/d87/f89 0 2024-04-05T03:02:37.691 INFO:tasks.workunit.client.1.smithi203.stdout:5/316: truncate d3/d1f/d59/d64/f1c 3719804 0 2024-04-05T03:02:37.695 INFO:tasks.workunit.client.1.smithi203.stdout:0/289: dwrite dd/f16 [0,4194304] 0 2024-04-05T03:02:37.695 INFO:tasks.workunit.client.1.smithi203.stdout:9/202: link f2 d13/f3c 0 2024-04-05T03:02:37.698 INFO:tasks.workunit.client.1.smithi203.stdout:6/280: rename d0/d23/l35 to d0/da/d54/d4c/l55 0 2024-04-05T03:02:37.698 INFO:tasks.workunit.client.1.smithi203.stdout:3/198: dwrite d0/dc/d1a/d1f/f2d [4194304,4194304] 0 2024-04-05T03:02:37.699 INFO:tasks.workunit.client.1.smithi203.stdout:5/317: mknod d3/c68 0 2024-04-05T03:02:37.699 INFO:tasks.workunit.client.1.smithi203.stdout:0/290: stat dd/d17/d1d/d26/f4f 0 2024-04-05T03:02:37.701 INFO:tasks.workunit.client.1.smithi203.stdout:9/203: write d13/f20 [3090243,51798] 0 2024-04-05T03:02:37.701 INFO:tasks.workunit.client.1.smithi203.stdout:0/291: fsync dd/d17/f48 0 2024-04-05T03:02:37.701 INFO:tasks.workunit.client.1.smithi203.stdout:2/402: dread d5/d10/f5b [0,4194304] 0 2024-04-05T03:02:37.701 INFO:tasks.workunit.client.1.smithi203.stdout:6/281: write d0/da/f13 [1463545,13644] 0 2024-04-05T03:02:37.705 INFO:tasks.workunit.client.1.smithi203.stdout:3/199: mkdir d0/dc/d34 0 2024-04-05T03:02:37.705 INFO:tasks.workunit.client.1.smithi203.stdout:0/292: write dd/d17/f40 [1068273,95969] 0 2024-04-05T03:02:37.706 INFO:tasks.workunit.client.1.smithi203.stdout:3/200: chown d0 32590582 1 2024-04-05T03:02:37.710 INFO:tasks.workunit.client.1.smithi203.stdout:9/204: mknod d13/d1b/d26/c3d 0 2024-04-05T03:02:37.714 INFO:tasks.workunit.client.1.smithi203.stdout:9/205: dread d13/d1b/f21 [0,4194304] 0 2024-04-05T03:02:37.714 INFO:tasks.workunit.client.1.smithi203.stdout:9/206: stat d13/d1b/d26/l28 0 2024-04-05T03:02:37.715 INFO:tasks.workunit.client.1.smithi203.stdout:5/318: mknod d3/d5f/c69 0 2024-04-05T03:02:37.716 INFO:tasks.workunit.client.1.smithi203.stdout:6/282: truncate d0/da/d54/f4b 932746 0 2024-04-05T03:02:37.717 INFO:tasks.workunit.client.1.smithi203.stdout:2/403: symlink d5/d10/d35/d2d/d7c/l8a 0 2024-04-05T03:02:37.717 INFO:tasks.workunit.client.1.smithi203.stdout:5/319: truncate d3/d1f/f39 4790549 0 2024-04-05T03:02:37.717 INFO:tasks.workunit.client.1.smithi203.stdout:0/293: mkdir dd/d17/d1d/d22/d45/d51 0 2024-04-05T03:02:37.718 INFO:tasks.workunit.client.1.smithi203.stdout:3/201: mkdir d0/dc/d1a/d33/d35 0 2024-04-05T03:02:37.720 INFO:tasks.workunit.client.1.smithi203.stdout:5/320: rmdir d3 39 2024-04-05T03:02:37.725 INFO:tasks.workunit.client.1.smithi203.stdout:3/202: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:37.734 INFO:tasks.workunit.client.1.smithi203.stdout:0/294: dwrite dd/d17/d1d/f37 [0,4194304] 0 2024-04-05T03:02:37.737 INFO:tasks.workunit.client.1.smithi203.stdout:0/295: fsync dd/f14 0 2024-04-05T03:02:37.743 INFO:tasks.workunit.client.1.smithi203.stdout:6/283: dread d0/da/f10 [0,4194304] 0 2024-04-05T03:02:37.746 INFO:tasks.workunit.client.1.smithi203.stdout:9/207: creat d13/d1b/d26/d31/f3e x:0 0 0 2024-04-05T03:02:37.748 INFO:tasks.workunit.client.1.smithi203.stdout:6/284: dread d0/da/f10 [0,4194304] 0 2024-04-05T03:02:37.752 INFO:tasks.workunit.client.1.smithi203.stdout:5/321: symlink d3/d5f/l6a 0 2024-04-05T03:02:37.755 INFO:tasks.workunit.client.1.smithi203.stdout:3/203: mknod d0/c36 0 2024-04-05T03:02:37.756 INFO:tasks.workunit.client.1.smithi203.stdout:6/285: symlink d0/da/d18/d38/d2b/l56 0 2024-04-05T03:02:37.758 INFO:tasks.workunit.client.1.smithi203.stdout:6/286: truncate d0/f37 948742 0 2024-04-05T03:02:37.759 INFO:tasks.workunit.client.1.smithi203.stdout:2/404: getdents d5/d10/d35/d2d 0 2024-04-05T03:02:37.759 INFO:tasks.workunit.client.1.smithi203.stdout:3/204: symlink d0/dc/d1a/d1d/l37 0 2024-04-05T03:02:37.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/205: chown d0/dc/d1a/d1f/f32 119512 1 2024-04-05T03:02:37.761 INFO:tasks.workunit.client.1.smithi203.stdout:6/287: creat d0/da/d18/f57 x:0 0 0 2024-04-05T03:02:37.763 INFO:tasks.workunit.client.1.smithi203.stdout:2/405: creat d5/d10/d1a/d24/d71/f8b x:0 0 0 2024-04-05T03:02:37.763 INFO:tasks.workunit.client.1.smithi203.stdout:0/296: dread f6 [0,4194304] 0 2024-04-05T03:02:37.765 INFO:tasks.workunit.client.1.smithi203.stdout:6/288: symlink d0/l58 0 2024-04-05T03:02:37.767 INFO:tasks.workunit.client.1.smithi203.stdout:0/297: mknod dd/d17/d1d/d22/c52 0 2024-04-05T03:02:37.768 INFO:tasks.workunit.client.1.smithi203.stdout:9/208: rmdir d13 39 2024-04-05T03:02:37.773 INFO:tasks.workunit.client.1.smithi203.stdout:0/298: mkdir dd/d17/d1d/d22/d53 0 2024-04-05T03:02:37.773 INFO:tasks.workunit.client.1.smithi203.stdout:9/209: fsync f10 0 2024-04-05T03:02:37.774 INFO:tasks.workunit.client.1.smithi203.stdout:2/406: mknod d5/d10/d35/d2d/d7c/d29/d82/c8c 0 2024-04-05T03:02:37.775 INFO:tasks.workunit.client.1.smithi203.stdout:6/289: link d0/da/d54/f45 d0/da/d54/d53/f59 0 2024-04-05T03:02:37.776 INFO:tasks.workunit.client.1.smithi203.stdout:0/299: rmdir dd/d17/d1d/d22/d45 39 2024-04-05T03:02:37.776 INFO:tasks.workunit.client.1.smithi203.stdout:0/300: fsync f7 0 2024-04-05T03:02:37.782 INFO:tasks.workunit.client.1.smithi203.stdout:9/210: creat d13/d1b/d33/f3f x:0 0 0 2024-04-05T03:02:37.785 INFO:tasks.workunit.client.1.smithi203.stdout:9/211: write f12 [1918727,96338] 0 2024-04-05T03:02:37.786 INFO:tasks.workunit.client.1.smithi203.stdout:0/301: dwrite dd/f14 [0,4194304] 0 2024-04-05T03:02:37.789 INFO:tasks.workunit.client.1.smithi203.stdout:2/407: dwrite d5/d10/d35/d36/d41/d79/f56 [0,4194304] 0 2024-04-05T03:02:37.790 INFO:tasks.workunit.client.1.smithi203.stdout:0/302: rmdir dd/d17/d1d/d26 39 2024-04-05T03:02:37.795 INFO:tasks.workunit.client.1.smithi203.stdout:9/212: mknod d13/d1b/d26/d31/d38/c40 0 2024-04-05T03:02:37.796 INFO:tasks.workunit.client.1.smithi203.stdout:0/303: dread dd/d17/d1d/d22/f32 [0,4194304] 0 2024-04-05T03:02:37.796 INFO:tasks.workunit.client.1.smithi203.stdout:2/408: creat d5/f8d x:0 0 0 2024-04-05T03:02:37.797 INFO:tasks.workunit.client.1.smithi203.stdout:2/409: chown d5/d10/d35/d36/d41 550371 1 2024-04-05T03:02:37.804 INFO:tasks.workunit.client.1.smithi203.stdout:3/206: truncate d0/fd 128326 0 2024-04-05T03:02:37.806 INFO:tasks.workunit.client.1.smithi203.stdout:9/213: creat d13/d1b/d26/d31/f41 x:0 0 0 2024-04-05T03:02:37.807 INFO:tasks.workunit.client.1.smithi203.stdout:0/304: mknod dd/c54 0 2024-04-05T03:02:37.810 INFO:tasks.workunit.client.1.smithi203.stdout:2/410: dwrite d5/d10/d35/d2d/f50 [0,4194304] 0 2024-04-05T03:02:37.811 INFO:tasks.workunit.client.1.smithi203.stdout:3/207: dread d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:37.811 INFO:tasks.workunit.client.1.smithi203.stdout:0/305: stat dd/d17/d1d/d26/d34/f35 0 2024-04-05T03:02:37.812 INFO:tasks.workunit.client.1.smithi203.stdout:6/290: write d0/f1a [1733480,78454] 0 2024-04-05T03:02:37.813 INFO:tasks.workunit.client.1.smithi203.stdout:6/291: chown d0/da 1 1 2024-04-05T03:02:37.814 INFO:tasks.workunit.client.1.smithi203.stdout:2/411: creat d5/d63/d86/f8e x:0 0 0 2024-04-05T03:02:37.816 INFO:tasks.workunit.client.1.smithi203.stdout:6/292: chown d0/l2 37763 1 2024-04-05T03:02:37.817 INFO:tasks.workunit.client.1.smithi203.stdout:9/214: dread d13/d1b/f21 [0,4194304] 0 2024-04-05T03:02:37.817 INFO:tasks.workunit.client.1.smithi203.stdout:4/169: sync 2024-04-05T03:02:37.817 INFO:tasks.workunit.client.1.smithi203.stdout:2/412: unlink d5/fa 0 2024-04-05T03:02:37.817 INFO:tasks.workunit.client.1.smithi203.stdout:6/293: stat d0/da/d18/d38/l20 0 2024-04-05T03:02:37.818 INFO:tasks.workunit.client.1.smithi203.stdout:0/306: mkdir dd/d17/d1d/d22/d53/d55 0 2024-04-05T03:02:37.819 INFO:tasks.workunit.client.1.smithi203.stdout:6/294: chown d0/da/fd 4114 1 2024-04-05T03:02:37.820 INFO:tasks.workunit.client.1.smithi203.stdout:9/215: mkdir d13/d1b/d33/d42 0 2024-04-05T03:02:37.820 INFO:tasks.workunit.client.1.smithi203.stdout:4/170: mknod da/d29/d2e/c33 0 2024-04-05T03:02:37.821 INFO:tasks.workunit.client.1.smithi203.stdout:4/171: chown da/l16 14 1 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: Deploying cephadm binary to smithi153 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: mgrmap e22: smithi203.iakodz(active, since 1.02579s) 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: Deploying cephadm binary to smithi203 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: pgmap v3: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:37 smithi153 ceph-mon[29381]: [05/Apr/2024:03:02:37] ENGINE Bus STARTING 2024-04-05T03:02:37.823 INFO:tasks.workunit.client.1.smithi203.stdout:2/413: mkdir d5/d10/d35/d2d/d7c/d72/d8f 0 2024-04-05T03:02:37.825 INFO:tasks.workunit.client.1.smithi203.stdout:9/216: rmdir d13/d1b/d26/d31/d38 39 2024-04-05T03:02:37.825 INFO:tasks.workunit.client.1.smithi203.stdout:9/217: chown f7 20586 1 2024-04-05T03:02:37.826 INFO:tasks.workunit.client.1.smithi203.stdout:9/218: write f8 [2673286,120133] 0 2024-04-05T03:02:37.827 INFO:tasks.workunit.client.1.smithi203.stdout:0/307: mkdir dd/d17/d1d/d26/d34/d56 0 2024-04-05T03:02:37.828 INFO:tasks.workunit.client.1.smithi203.stdout:2/414: mknod d5/d10/d35/d2d/d7c/d29/d67/c90 0 2024-04-05T03:02:37.830 INFO:tasks.workunit.client.1.smithi203.stdout:9/219: stat d13/d1b/l2f 0 2024-04-05T03:02:37.831 INFO:tasks.workunit.client.1.smithi203.stdout:3/208: write d0/dc/d1a/d1d/f2f [1569034,36649] 0 2024-04-05T03:02:37.832 INFO:tasks.workunit.client.1.smithi203.stdout:2/415: symlink d5/d10/d1a/l91 0 2024-04-05T03:02:37.832 INFO:tasks.workunit.client.1.smithi203.stdout:2/416: readlink l2 0 2024-04-05T03:02:37.832 INFO:tasks.workunit.client.1.smithi203.stdout:0/308: dread f6 [0,4194304] 0 2024-04-05T03:02:37.835 INFO:tasks.workunit.client.1.smithi203.stdout:2/417: write d5/d10/d35/d36/f3b [580679,32958] 0 2024-04-05T03:02:37.835 INFO:tasks.workunit.client.1.smithi203.stdout:0/309: fdatasync dd/d17/d1d/d26/d28/f3f 0 2024-04-05T03:02:37.837 INFO:tasks.workunit.client.1.smithi203.stdout:2/418: chown d5/d10/d1a/f65 1289925 1 2024-04-05T03:02:37.838 INFO:tasks.workunit.client.1.smithi203.stdout:2/419: readlink d5/d10/d35/d36/l55 0 2024-04-05T03:02:37.839 INFO:tasks.workunit.client.1.smithi203.stdout:3/209: unlink d0/dc/d1a/d1f/f2d 0 2024-04-05T03:02:37.841 INFO:tasks.workunit.client.1.smithi203.stdout:3/210: write d0/dc/d1a/d1d/f29 [1610543,90762] 0 2024-04-05T03:02:37.844 INFO:tasks.workunit.client.1.smithi203.stdout:4/172: dread f5 [4194304,4194304] 0 2024-04-05T03:02:37.848 INFO:tasks.workunit.client.1.smithi203.stdout:0/310: rename f8 to dd/d17/d29/f57 0 2024-04-05T03:02:37.852 INFO:tasks.workunit.client.1.smithi203.stdout:0/311: dread dd/d17/d1d/f1e [0,4194304] 0 2024-04-05T03:02:37.854 INFO:tasks.workunit.client.1.smithi203.stdout:2/420: creat d5/d10/d35/d2d/d7c/d29/d82/f92 x:0 0 0 2024-04-05T03:02:37.855 INFO:tasks.workunit.client.1.smithi203.stdout:2/421: write f3 [5121925,114408] 0 2024-04-05T03:02:37.861 INFO:tasks.workunit.client.1.smithi203.stdout:9/220: dwrite f2 [0,4194304] 0 2024-04-05T03:02:37.862 INFO:tasks.workunit.client.1.smithi203.stdout:9/221: fdatasync fd 0 2024-04-05T03:02:37.863 INFO:tasks.workunit.client.1.smithi203.stdout:9/222: fsync d13/f2a 0 2024-04-05T03:02:37.872 INFO:tasks.workunit.client.1.smithi203.stdout:0/312: rename dd/d17/d1d/l1f to dd/d17/d1d/d22/d45/d51/l58 0 2024-04-05T03:02:37.873 INFO:tasks.workunit.client.1.smithi203.stdout:0/313: write dd/d17/d1d/d26/d28/f2b [527762,54291] 0 2024-04-05T03:02:37.874 INFO:tasks.workunit.client.1.smithi203.stdout:9/223: symlink d13/d1b/d33/l43 0 2024-04-05T03:02:37.876 INFO:tasks.workunit.client.1.smithi203.stdout:9/224: dread d13/f1d [0,4194304] 0 2024-04-05T03:02:37.876 INFO:tasks.workunit.client.1.smithi203.stdout:3/211: truncate d0/f5 893535 0 2024-04-05T03:02:37.877 INFO:tasks.workunit.client.1.smithi203.stdout:9/225: write fd [630787,105818] 0 2024-04-05T03:02:37.879 INFO:tasks.workunit.client.1.smithi203.stdout:4/173: getdents da 0 2024-04-05T03:02:37.879 INFO:tasks.workunit.client.1.smithi203.stdout:3/212: chown d0/dc/d1a/d1f/l26 0 1 2024-04-05T03:02:37.881 INFO:tasks.workunit.client.1.smithi203.stdout:4/174: write da/f1e [1205271,31369] 0 2024-04-05T03:02:37.883 INFO:tasks.workunit.client.1.smithi203.stdout:2/422: truncate d5/f5e 2207507 0 2024-04-05T03:02:37.884 INFO:tasks.workunit.client.1.smithi203.stdout:0/314: dwrite dd/d17/d1d/d26/f3d [0,4194304] 0 2024-04-05T03:02:37.885 INFO:tasks.workunit.client.1.smithi203.stdout:0/315: write dd/f27 [641443,52025] 0 2024-04-05T03:02:37.886 INFO:tasks.workunit.client.1.smithi203.stdout:4/175: creat da/d29/f34 x:0 0 0 2024-04-05T03:02:37.887 INFO:tasks.workunit.client.1.smithi203.stdout:2/423: creat d5/d63/f93 x:0 0 0 2024-04-05T03:02:37.889 INFO:tasks.workunit.client.1.smithi203.stdout:4/176: chown f9 30400751 1 2024-04-05T03:02:37.890 INFO:tasks.workunit.client.1.smithi203.stdout:3/213: getdents d0/dc/d1a 0 2024-04-05T03:02:37.890 INFO:tasks.workunit.client.1.smithi203.stdout:9/226: dwrite d13/f20 [0,4194304] 0 2024-04-05T03:02:37.890 INFO:tasks.workunit.client.1.smithi203.stdout:4/177: chown f5 554 1 2024-04-05T03:02:37.890 INFO:tasks.workunit.client.1.smithi203.stdout:9/227: stat d13/d1b/l2f 0 2024-04-05T03:02:37.893 INFO:tasks.workunit.client.1.smithi203.stdout:9/228: write d13/d1b/d26/d31/f41 [941684,6717] 0 2024-04-05T03:02:37.894 INFO:tasks.workunit.client.1.smithi203.stdout:2/424: creat d5/d66/d7d/f94 x:0 0 0 2024-04-05T03:02:37.896 INFO:tasks.workunit.client.1.smithi203.stdout:2/425: fdatasync d5/d63/f6e 0 2024-04-05T03:02:37.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/426: read d5/d10/d35/d36/f6c [266462,107821] 0 2024-04-05T03:02:37.898 INFO:tasks.workunit.client.1.smithi203.stdout:4/178: creat da/d29/d2e/f35 x:0 0 0 2024-04-05T03:02:37.899 INFO:tasks.workunit.client.1.smithi203.stdout:3/214: fsync d0/fd 0 2024-04-05T03:02:37.901 INFO:tasks.workunit.client.1.smithi203.stdout:4/179: chown da/c1c 1520418548 1 2024-04-05T03:02:37.901 INFO:tasks.workunit.client.1.smithi203.stdout:0/316: symlink dd/d17/d1d/d22/d53/d55/l59 0 2024-04-05T03:02:37.903 INFO:tasks.workunit.client.1.smithi203.stdout:4/180: chown da/d29/d2e 38 1 2024-04-05T03:02:37.904 INFO:tasks.workunit.client.1.smithi203.stdout:2/427: chown d5/d10/f20 927209 1 2024-04-05T03:02:37.904 INFO:tasks.workunit.client.1.smithi203.stdout:3/215: dread d0/fd [0,4194304] 0 2024-04-05T03:02:37.906 INFO:tasks.workunit.client.1.smithi203.stdout:0/317: unlink dd/l13 0 2024-04-05T03:02:37.906 INFO:tasks.workunit.client.1.smithi203.stdout:9/229: dwrite d13/d1b/d33/f3f [0,4194304] 0 2024-04-05T03:02:37.908 INFO:tasks.workunit.client.1.smithi203.stdout:0/318: write dd/d17/d1d/d26/f42 [34966,80768] 0 2024-04-05T03:02:37.908 INFO:tasks.workunit.client.1.smithi203.stdout:4/181: symlink da/l36 0 2024-04-05T03:02:37.911 INFO:tasks.workunit.client.1.smithi203.stdout:3/216: unlink d0/dc/c10 0 2024-04-05T03:02:37.916 INFO:tasks.workunit.client.1.smithi203.stdout:2/428: readlink d5/d10/d1a/l6a 0 2024-04-05T03:02:37.917 INFO:tasks.workunit.client.1.smithi203.stdout:0/319: symlink dd/d17/d1d/d22/d53/d55/l5a 0 2024-04-05T03:02:37.918 INFO:tasks.workunit.client.1.smithi203.stdout:2/429: write f3 [3495857,47310] 0 2024-04-05T03:02:37.920 INFO:tasks.workunit.client.1.smithi203.stdout:9/230: rename c5 to d13/d1b/d33/c44 0 2024-04-05T03:02:37.920 INFO:tasks.workunit.client.1.smithi203.stdout:2/430: dread - d5/d63/f75 zero size 2024-04-05T03:02:37.924 INFO:tasks.workunit.client.1.smithi203.stdout:3/217: dwrite d0/dc/d1a/d1d/f29 [0,4194304] 0 2024-04-05T03:02:37.925 INFO:tasks.workunit.client.1.smithi203.stdout:4/182: dread f9 [0,4194304] 0 2024-04-05T03:02:37.927 INFO:tasks.workunit.client.1.smithi203.stdout:0/320: getdents dd/d17/d1d/d26/d28 0 2024-04-05T03:02:37.930 INFO:tasks.workunit.client.1.smithi203.stdout:4/183: fsync f9 0 2024-04-05T03:02:37.930 INFO:tasks.workunit.client.1.smithi203.stdout:3/218: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:37.931 INFO:tasks.workunit.client.1.smithi203.stdout:0/321: fdatasync f2 0 2024-04-05T03:02:37.932 INFO:tasks.workunit.client.1.smithi203.stdout:9/231: dwrite f12 [0,4194304] 0 2024-04-05T03:02:37.933 INFO:tasks.workunit.client.1.smithi203.stdout:3/219: chown d0/f4 3 1 2024-04-05T03:02:37.934 INFO:tasks.workunit.client.1.smithi203.stdout:4/184: unlink f8 0 2024-04-05T03:02:37.935 INFO:tasks.workunit.client.1.smithi203.stdout:2/431: getdents d5/d10/d35/d2d/d7c 0 2024-04-05T03:02:37.937 INFO:tasks.workunit.client.1.smithi203.stdout:2/432: write d5/d10/d35/d2d/f50 [4581959,79895] 0 2024-04-05T03:02:37.937 INFO:tasks.workunit.client.1.smithi203.stdout:9/232: symlink d13/d1b/d26/l45 0 2024-04-05T03:02:37.937 INFO:tasks.workunit.client.1.smithi203.stdout:9/233: fsync d13/f25 0 2024-04-05T03:02:37.938 INFO:tasks.workunit.client.1.smithi203.stdout:3/220: mkdir d0/dc/d38 0 2024-04-05T03:02:37.940 INFO:tasks.workunit.client.1.smithi203.stdout:0/322: symlink dd/d17/d1d/d26/d28/l5b 0 2024-04-05T03:02:37.942 INFO:tasks.workunit.client.1.smithi203.stdout:0/323: readlink dd/d17/d1d/d26/d41/d49/l4d 0 2024-04-05T03:02:37.943 INFO:tasks.workunit.client.1.smithi203.stdout:2/433: creat d5/d10/d35/d73/f95 x:0 0 0 2024-04-05T03:02:37.943 INFO:tasks.workunit.client.1.smithi203.stdout:9/234: dread f12 [0,4194304] 0 2024-04-05T03:02:37.945 INFO:tasks.workunit.client.1.smithi203.stdout:3/221: symlink d0/dc/d1a/d1d/l39 0 2024-04-05T03:02:37.945 INFO:tasks.workunit.client.1.smithi203.stdout:3/222: read d0/fd [51771,52753] 0 2024-04-05T03:02:37.946 INFO:tasks.workunit.client.1.smithi203.stdout:3/223: fsync d0/f9 0 2024-04-05T03:02:37.947 INFO:tasks.workunit.client.1.smithi203.stdout:2/434: chown d5/d10/d1a/f1b 2 1 2024-04-05T03:02:37.948 INFO:tasks.workunit.client.1.smithi203.stdout:4/185: link da/f26 da/f37 0 2024-04-05T03:02:37.948 INFO:tasks.workunit.client.1.smithi203.stdout:2/435: fsync d5/d10/d35/d36/f3b 0 2024-04-05T03:02:37.949 INFO:tasks.workunit.client.1.smithi203.stdout:4/186: chown da/f21 508 1 2024-04-05T03:02:37.950 INFO:tasks.workunit.client.1.smithi203.stdout:3/224: fsync d0/dc/f16 0 2024-04-05T03:02:37.950 INFO:tasks.workunit.client.1.smithi203.stdout:4/187: rename da/d29 to da/d29/d2e/d38 22 2024-04-05T03:02:37.952 INFO:tasks.workunit.client.1.smithi203.stdout:2/436: stat d5/d10/d1a/d24/f37 0 2024-04-05T03:02:37.954 INFO:tasks.workunit.client.1.smithi203.stdout:8/327: sync 2024-04-05T03:02:37.955 INFO:tasks.workunit.client.1.smithi203.stdout:0/324: link dd/d17/d1d/d26/d41/d49/c4a dd/d17/d1d/d22/d53/c5c 0 2024-04-05T03:02:37.957 INFO:tasks.workunit.client.1.smithi203.stdout:4/188: dwrite da/d29/f2d [0,4194304] 0 2024-04-05T03:02:37.959 INFO:tasks.workunit.client.1.smithi203.stdout:2/437: mknod d5/d10/d1a/d57/d70/c96 0 2024-04-05T03:02:37.959 INFO:tasks.workunit.client.1.smithi203.stdout:2/438: chown d5/d10/d35/l74 2032698 1 2024-04-05T03:02:37.960 INFO:tasks.workunit.client.1.smithi203.stdout:8/328: mknod d1/d12/d1a/c52 0 2024-04-05T03:02:37.961 INFO:tasks.workunit.client.1.smithi203.stdout:0/325: rmdir dd/d17/d1d/d22/d45/d51 39 2024-04-05T03:02:37.964 INFO:tasks.workunit.client.1.smithi203.stdout:8/329: unlink d1/d12/d1a/c52 0 2024-04-05T03:02:37.967 INFO:tasks.workunit.client.1.smithi203.stdout:2/439: rename d5/d10/d35/d2d/d7c/d29/d82/f92 to d5/d10/d1a/d24/d71/f97 0 2024-04-05T03:02:37.968 INFO:tasks.workunit.client.1.smithi203.stdout:0/326: dwrite dd/d17/f48 [0,4194304] 0 2024-04-05T03:02:37.969 INFO:tasks.workunit.client.1.smithi203.stdout:0/327: readlink l4 0 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: Deploying cephadm binary to smithi153 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: mgrmap e22: smithi203.iakodz(active, since 1.02579s) 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: Deploying cephadm binary to smithi203 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: pgmap v3: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:37.971 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:37 smithi203 ceph-mon[39911]: [05/Apr/2024:03:02:37] ENGINE Bus STARTING 2024-04-05T03:02:37.973 INFO:tasks.workunit.client.1.smithi203.stdout:4/189: creat da/f39 x:0 0 0 2024-04-05T03:02:37.982 INFO:tasks.workunit.client.1.smithi203.stdout:8/330: write d1/d6/d2e/f30 [910908,117915] 0 2024-04-05T03:02:37.982 INFO:tasks.workunit.client.1.smithi203.stdout:0/328: write dd/d17/f48 [5087375,73513] 0 2024-04-05T03:02:37.984 INFO:tasks.workunit.client.1.smithi203.stdout:3/225: dwrite d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:37.986 INFO:tasks.workunit.client.1.smithi203.stdout:8/331: mkdir d1/d12/d17/d47/d53 0 2024-04-05T03:02:37.988 INFO:tasks.workunit.client.1.smithi203.stdout:8/332: write d1/d12/d17/d10/f49 [354577,42343] 0 2024-04-05T03:02:37.989 INFO:tasks.workunit.client.1.smithi203.stdout:3/226: rename d0/c15 to d0/dc/c3a 0 2024-04-05T03:02:37.991 INFO:tasks.workunit.client.1.smithi203.stdout:3/227: write d0/dc/d1a/d1d/f29 [4373955,65340] 0 2024-04-05T03:02:37.993 INFO:tasks.workunit.client.1.smithi203.stdout:2/440: write d5/f43 [3097160,52835] 0 2024-04-05T03:02:37.995 INFO:tasks.workunit.client.1.smithi203.stdout:8/333: mknod d1/d12/d17/d10/c54 0 2024-04-05T03:02:37.998 INFO:tasks.workunit.client.1.smithi203.stdout:0/329: link dd/c54 dd/d17/d1d/d22/c5d 0 2024-04-05T03:02:38.000 INFO:tasks.workunit.client.1.smithi203.stdout:3/228: creat d0/dc/d38/f3b x:0 0 0 2024-04-05T03:02:38.000 INFO:tasks.workunit.client.1.smithi203.stdout:8/334: fsync d1/d12/f31 0 2024-04-05T03:02:38.001 INFO:tasks.workunit.client.1.smithi203.stdout:8/335: chown d1/d4d 58350 1 2024-04-05T03:02:38.003 INFO:tasks.workunit.client.1.smithi203.stdout:2/441: getdents d5/d10/d35 0 2024-04-05T03:02:38.005 INFO:tasks.workunit.client.1.smithi203.stdout:3/229: truncate d0/ff 3404788 0 2024-04-05T03:02:38.006 INFO:tasks.workunit.client.1.smithi203.stdout:3/230: readlink d0/d21/l27 0 2024-04-05T03:02:38.007 INFO:tasks.workunit.client.1.smithi203.stdout:8/336: mkdir d1/d12/d2f/d55 0 2024-04-05T03:02:38.010 INFO:tasks.workunit.client.1.smithi203.stdout:4/190: rename da/d29/f34 to da/f3a 0 2024-04-05T03:02:38.013 INFO:tasks.workunit.client.1.smithi203.stdout:0/330: write dd/d17/d1d/f1e [1018488,83100] 0 2024-04-05T03:02:38.015 INFO:tasks.workunit.client.1.smithi203.stdout:8/337: link d1/d12/d1a/f42 d1/d6/f56 0 2024-04-05T03:02:38.019 INFO:tasks.workunit.client.1.smithi203.stdout:8/338: dread d1/d6/f45 [0,4194304] 0 2024-04-05T03:02:38.020 INFO:tasks.workunit.client.1.smithi203.stdout:0/331: mknod dd/d17/d1d/d26/c5e 0 2024-04-05T03:02:38.021 INFO:tasks.workunit.client.1.smithi203.stdout:0/332: chown dd/d17/d1d/d22/d53/d55/l59 0 1 2024-04-05T03:02:38.024 INFO:tasks.workunit.client.1.smithi203.stdout:4/191: symlink da/d29/d2e/l3b 0 2024-04-05T03:02:38.026 INFO:tasks.workunit.client.1.smithi203.stdout:3/231: write d0/ff [1683495,125502] 0 2024-04-05T03:02:38.027 INFO:tasks.workunit.client.1.smithi203.stdout:0/333: mkdir dd/d17/d1d/d22/d5f 0 2024-04-05T03:02:38.029 INFO:tasks.workunit.client.1.smithi203.stdout:0/334: fsync f6 0 2024-04-05T03:02:38.030 INFO:tasks.workunit.client.1.smithi203.stdout:8/339: dread d1/f34 [0,4194304] 0 2024-04-05T03:02:38.032 INFO:tasks.workunit.client.1.smithi203.stdout:4/192: dwrite da/d29/f2b [0,4194304] 0 2024-04-05T03:02:38.032 INFO:tasks.workunit.client.1.smithi203.stdout:0/335: symlink dd/d17/d1d/d26/d34/l60 0 2024-04-05T03:02:38.033 INFO:tasks.workunit.client.1.smithi203.stdout:4/193: chown da/d29 23024330 1 2024-04-05T03:02:38.035 INFO:tasks.workunit.client.1.smithi203.stdout:0/336: mkdir dd/d17/d1d/d22/d53/d61 0 2024-04-05T03:02:38.037 INFO:tasks.workunit.client.1.smithi203.stdout:0/337: write dd/d17/d1d/d26/f42 [1078938,51474] 0 2024-04-05T03:02:38.042 INFO:tasks.workunit.client.1.smithi203.stdout:0/338: mknod dd/d17/d1d/d22/d53/d61/c62 0 2024-04-05T03:02:38.042 INFO:tasks.workunit.client.1.smithi203.stdout:4/194: write da/f10 [1029712,45589] 0 2024-04-05T03:02:38.045 INFO:tasks.workunit.client.1.smithi203.stdout:0/339: write dd/d17/d1d/d26/d28/f3f [2140735,92471] 0 2024-04-05T03:02:38.045 INFO:tasks.workunit.client.1.smithi203.stdout:3/232: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:38.047 INFO:tasks.workunit.client.1.smithi203.stdout:0/340: truncate dd/d17/d1d/d22/f32 4569275 0 2024-04-05T03:02:38.048 INFO:tasks.workunit.client.1.smithi203.stdout:3/233: rename d0/fd to d0/dc/d1a/d33/d35/f3c 0 2024-04-05T03:02:38.049 INFO:tasks.workunit.client.1.smithi203.stdout:0/341: chown dd/c54 1837215223 1 2024-04-05T03:02:38.052 INFO:tasks.workunit.client.1.smithi203.stdout:3/234: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:38.052 INFO:tasks.workunit.client.1.smithi203.stdout:0/342: link f2 dd/d17/d1d/d22/d53/f63 0 2024-04-05T03:02:38.055 INFO:tasks.workunit.client.1.smithi203.stdout:3/235: write d0/dc/d1a/d1d/f2f [2815363,75506] 0 2024-04-05T03:02:38.059 INFO:tasks.workunit.client.1.smithi203.stdout:3/236: dread d0/dc/d1a/d33/d35/f3c [0,4194304] 0 2024-04-05T03:02:38.060 INFO:tasks.workunit.client.1.smithi203.stdout:0/343: dwrite dd/d17/f48 [0,4194304] 0 2024-04-05T03:02:38.060 INFO:tasks.workunit.client.1.smithi203.stdout:0/344: chown dd 85 1 2024-04-05T03:02:38.061 INFO:tasks.workunit.client.1.smithi203.stdout:0/345: dread - dd/f44 zero size 2024-04-05T03:02:38.064 INFO:tasks.workunit.client.1.smithi203.stdout:4/195: dread f5 [0,4194304] 0 2024-04-05T03:02:38.066 INFO:tasks.workunit.client.1.smithi203.stdout:3/237: getdents d0/dc 0 2024-04-05T03:02:38.068 INFO:tasks.workunit.client.1.smithi203.stdout:4/196: dread - da/f3a zero size 2024-04-05T03:02:38.070 INFO:tasks.workunit.client.1.smithi203.stdout:0/346: dwrite dd/d17/d1d/d26/d28/f4e [0,4194304] 0 2024-04-05T03:02:38.070 INFO:tasks.workunit.client.1.smithi203.stdout:3/238: fdatasync d0/dc/d1a/d33/d35/f3c 0 2024-04-05T03:02:38.073 INFO:tasks.workunit.client.1.smithi203.stdout:0/347: rmdir dd/d17/d1d 39 2024-04-05T03:02:38.077 INFO:tasks.workunit.client.1.smithi203.stdout:4/197: dread da/f10 [0,4194304] 0 2024-04-05T03:02:38.079 INFO:tasks.workunit.client.1.smithi203.stdout:0/348: truncate f7 2933920 0 2024-04-05T03:02:38.080 INFO:tasks.workunit.client.1.smithi203.stdout:0/349: fdatasync dd/f44 0 2024-04-05T03:02:38.081 INFO:tasks.workunit.client.1.smithi203.stdout:3/239: dwrite d0/dc/d38/f3b [0,4194304] 0 2024-04-05T03:02:38.082 INFO:tasks.workunit.client.1.smithi203.stdout:4/198: mkdir da/d3c 0 2024-04-05T03:02:38.083 INFO:tasks.workunit.client.1.smithi203.stdout:3/240: write d0/f23 [392753,122950] 0 2024-04-05T03:02:38.084 INFO:tasks.workunit.client.1.smithi203.stdout:0/350: chown dd/d17/d1d/d26/d28/f4e 0 1 2024-04-05T03:02:38.085 INFO:tasks.workunit.client.1.smithi203.stdout:4/199: fsync da/f3a 0 2024-04-05T03:02:38.088 INFO:tasks.workunit.client.1.smithi203.stdout:0/351: creat dd/d17/d1d/d26/d34/f64 x:0 0 0 2024-04-05T03:02:38.090 INFO:tasks.workunit.client.1.smithi203.stdout:0/352: write dd/d17/d1d/d26/f4f [10289,82388] 0 2024-04-05T03:02:38.095 INFO:tasks.workunit.client.1.smithi203.stdout:4/200: dwrite da/f1e [0,4194304] 0 2024-04-05T03:02:38.097 INFO:tasks.workunit.client.1.smithi203.stdout:0/353: creat dd/d17/d1d/d22/f65 x:0 0 0 2024-04-05T03:02:38.098 INFO:tasks.workunit.client.1.smithi203.stdout:4/201: read da/f10 [863459,14174] 0 2024-04-05T03:02:38.102 INFO:tasks.workunit.client.1.smithi203.stdout:3/241: write d0/dc/d1a/d33/d35/f3c [254931,110331] 0 2024-04-05T03:02:38.105 INFO:tasks.workunit.client.1.smithi203.stdout:4/202: getdents da/d29 0 2024-04-05T03:02:38.105 INFO:tasks.workunit.client.1.smithi203.stdout:3/242: write d0/f5 [4169545,49631] 0 2024-04-05T03:02:38.107 INFO:tasks.workunit.client.1.smithi203.stdout:3/243: fdatasync d0/dc/f14 0 2024-04-05T03:02:38.107 INFO:tasks.workunit.client.1.smithi203.stdout:0/354: dwrite dd/d17/f40 [0,4194304] 0 2024-04-05T03:02:38.108 INFO:tasks.workunit.client.1.smithi203.stdout:4/203: mknod da/d29/c3d 0 2024-04-05T03:02:38.109 INFO:tasks.workunit.client.1.smithi203.stdout:3/244: chown d0/dc/d1a/d1d/f29 113770 1 2024-04-05T03:02:38.111 INFO:tasks.workunit.client.1.smithi203.stdout:0/355: getdents dd/d17/d1d/d22/d5f 0 2024-04-05T03:02:38.114 INFO:tasks.workunit.client.1.smithi203.stdout:0/356: symlink dd/d17/d1d/l66 0 2024-04-05T03:02:38.115 INFO:tasks.workunit.client.1.smithi203.stdout:7/222: sync 2024-04-05T03:02:38.116 INFO:tasks.workunit.client.1.smithi203.stdout:7/223: dread - d8/f37 zero size 2024-04-05T03:02:38.117 INFO:tasks.workunit.client.1.smithi203.stdout:4/204: dread da/f1b [0,4194304] 0 2024-04-05T03:02:38.119 INFO:tasks.workunit.client.1.smithi203.stdout:3/245: dwrite d0/dc/d1a/d1f/f32 [0,4194304] 0 2024-04-05T03:02:38.120 INFO:tasks.workunit.client.1.smithi203.stdout:3/246: readlink d0/dc/l1c 0 2024-04-05T03:02:38.121 INFO:tasks.workunit.client.1.smithi203.stdout:3/247: read d0/f23 [120365,48249] 0 2024-04-05T03:02:38.121 INFO:tasks.workunit.client.1.smithi203.stdout:7/224: symlink d8/da/d2f/l4d 0 2024-04-05T03:02:38.122 INFO:tasks.workunit.client.1.smithi203.stdout:3/248: write d0/f9 [4020403,114859] 0 2024-04-05T03:02:38.124 INFO:tasks.workunit.client.1.smithi203.stdout:0/357: creat dd/d17/d1d/d26/f67 x:0 0 0 2024-04-05T03:02:38.125 INFO:tasks.workunit.client.1.smithi203.stdout:7/225: write d8/f3e [524394,49841] 0 2024-04-05T03:02:38.128 INFO:tasks.workunit.client.1.smithi203.stdout:3/249: creat d0/dc/d1a/d33/d35/f3d x:0 0 0 2024-04-05T03:02:38.130 INFO:tasks.workunit.client.1.smithi203.stdout:0/358: dread dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:38.131 INFO:tasks.workunit.client.1.smithi203.stdout:7/226: link d8/d32/d16/d1a/d21/f22 d8/da/f4e 0 2024-04-05T03:02:38.132 INFO:tasks.workunit.client.1.smithi203.stdout:3/250: symlink d0/dc/d1a/d33/l3e 0 2024-04-05T03:02:38.134 INFO:tasks.workunit.client.1.smithi203.stdout:3/251: mkdir d0/dc/d3f 0 2024-04-05T03:02:38.136 INFO:tasks.workunit.client.1.smithi203.stdout:4/205: dread da/f22 [0,4194304] 0 2024-04-05T03:02:38.136 INFO:tasks.workunit.client.1.smithi203.stdout:7/227: dread d8/f20 [0,4194304] 0 2024-04-05T03:02:38.138 INFO:tasks.workunit.client.1.smithi203.stdout:0/359: dwrite dd/d17/d1d/d26/f42 [0,4194304] 0 2024-04-05T03:02:38.139 INFO:tasks.workunit.client.1.smithi203.stdout:4/206: creat da/d29/d2e/f3e x:0 0 0 2024-04-05T03:02:38.140 INFO:tasks.workunit.client.1.smithi203.stdout:7/228: rename d8 to d8/d32/d16/d1a/d4f 22 2024-04-05T03:02:38.142 INFO:tasks.workunit.client.1.smithi203.stdout:0/360: dread dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:38.143 INFO:tasks.workunit.client.1.smithi203.stdout:7/229: write d8/d2a/f42 [960264,54281] 0 2024-04-05T03:02:38.144 INFO:tasks.workunit.client.1.smithi203.stdout:4/207: rename da/ld to da/d3c/l3f 0 2024-04-05T03:02:38.145 INFO:tasks.workunit.client.1.smithi203.stdout:3/252: dwrite d0/dc/d1a/d33/d35/f3c [0,4194304] 0 2024-04-05T03:02:38.145 INFO:tasks.workunit.client.1.smithi203.stdout:4/208: chown da/d29/d2e 22730925 1 2024-04-05T03:02:38.146 INFO:tasks.workunit.client.1.smithi203.stdout:3/253: readlink d0/dc/d1a/d1f/l2e 0 2024-04-05T03:02:38.147 INFO:tasks.workunit.client.1.smithi203.stdout:7/230: creat d8/da/f50 x:0 0 0 2024-04-05T03:02:38.147 INFO:tasks.workunit.client.1.smithi203.stdout:0/361: mknod dd/d17/d1d/d22/d5f/c68 0 2024-04-05T03:02:38.151 INFO:tasks.workunit.client.1.smithi203.stdout:4/209: truncate f5 3269045 0 2024-04-05T03:02:38.152 INFO:tasks.workunit.client.1.smithi203.stdout:7/231: symlink d8/d32/d16/d1a/d30/l51 0 2024-04-05T03:02:38.154 INFO:tasks.workunit.client.1.smithi203.stdout:0/362: chown dd/d17/d1d/d26/d28/f2d 56 1 2024-04-05T03:02:38.154 INFO:tasks.workunit.client.1.smithi203.stdout:3/254: symlink d0/l40 0 2024-04-05T03:02:38.155 INFO:tasks.workunit.client.1.smithi203.stdout:4/210: write da/f22 [4557443,123792] 0 2024-04-05T03:02:38.158 INFO:tasks.workunit.client.1.smithi203.stdout:3/255: mkdir d0/d41 0 2024-04-05T03:02:38.159 INFO:tasks.workunit.client.1.smithi203.stdout:4/211: fdatasync da/f27 0 2024-04-05T03:02:38.161 INFO:tasks.workunit.client.1.smithi203.stdout:0/363: dwrite f6 [4194304,4194304] 0 2024-04-05T03:02:38.161 INFO:tasks.workunit.client.1.smithi203.stdout:3/256: creat d0/f42 x:0 0 0 2024-04-05T03:02:38.162 INFO:tasks.workunit.client.1.smithi203.stdout:0/364: chown dd 734 1 2024-04-05T03:02:38.163 INFO:tasks.workunit.client.1.smithi203.stdout:3/257: stat d0/dc/c3a 0 2024-04-05T03:02:38.164 INFO:tasks.workunit.client.1.smithi203.stdout:4/212: getdents da/d29 0 2024-04-05T03:02:38.165 INFO:tasks.workunit.client.1.smithi203.stdout:3/258: write d0/dc/f16 [1585087,22519] 0 2024-04-05T03:02:38.165 INFO:tasks.workunit.client.1.smithi203.stdout:1/229: sync 2024-04-05T03:02:38.165 INFO:tasks.workunit.client.1.smithi203.stdout:7/232: rename d8/da/d2f to d8/d2a/d3f/d52 0 2024-04-05T03:02:38.166 INFO:tasks.workunit.client.1.smithi203.stdout:4/213: read da/f10 [504984,35098] 0 2024-04-05T03:02:38.166 INFO:tasks.workunit.client.1.smithi203.stdout:0/365: mkdir dd/d17/d1d/d22/d45/d51/d69 0 2024-04-05T03:02:38.170 INFO:tasks.workunit.client.1.smithi203.stdout:3/259: mkdir d0/dc/d1a/d1d/d43 0 2024-04-05T03:02:38.176 INFO:tasks.workunit.client.1.smithi203.stdout:7/233: dread d8/d32/f18 [0,4194304] 0 2024-04-05T03:02:38.176 INFO:tasks.workunit.client.1.smithi203.stdout:3/260: symlink d0/dc/d3f/l44 0 2024-04-05T03:02:38.179 INFO:tasks.workunit.client.1.smithi203.stdout:1/230: dwrite d7/f2d [0,4194304] 0 2024-04-05T03:02:38.179 INFO:tasks.workunit.client.1.smithi203.stdout:4/214: dwrite f9 [4194304,4194304] 0 2024-04-05T03:02:38.181 INFO:tasks.workunit.client.1.smithi203.stdout:0/366: dread dd/f14 [0,4194304] 0 2024-04-05T03:02:38.182 INFO:tasks.workunit.client.1.smithi203.stdout:3/261: dread - d0/dc/d1a/d33/d35/f3d zero size 2024-04-05T03:02:38.184 INFO:tasks.workunit.client.1.smithi203.stdout:7/234: truncate d8/d32/f44 465376 0 2024-04-05T03:02:38.185 INFO:tasks.workunit.client.1.smithi203.stdout:1/231: creat d7/d24/f4d x:0 0 0 2024-04-05T03:02:38.187 INFO:tasks.workunit.client.1.smithi203.stdout:4/215: creat da/d29/f40 x:0 0 0 2024-04-05T03:02:38.188 INFO:tasks.workunit.client.1.smithi203.stdout:4/216: dread - da/f3a zero size 2024-04-05T03:02:38.191 INFO:tasks.workunit.client.1.smithi203.stdout:3/262: creat d0/d21/f45 x:0 0 0 2024-04-05T03:02:38.193 INFO:tasks.workunit.client.1.smithi203.stdout:4/217: dread da/f22 [0,4194304] 0 2024-04-05T03:02:38.196 INFO:tasks.workunit.client.1.smithi203.stdout:3/263: rename d0/c36 to d0/dc/d1a/d1f/c46 0 2024-04-05T03:02:38.198 INFO:tasks.workunit.client.1.smithi203.stdout:4/218: unlink da/d29/f31 0 2024-04-05T03:02:38.202 INFO:tasks.workunit.client.1.smithi203.stdout:4/219: fdatasync da/f27 0 2024-04-05T03:02:38.202 INFO:tasks.workunit.client.1.smithi203.stdout:3/264: dread d0/f23 [0,4194304] 0 2024-04-05T03:02:38.203 INFO:tasks.workunit.client.1.smithi203.stdout:1/232: fsync d7/f2d 0 2024-04-05T03:02:38.205 INFO:tasks.workunit.client.1.smithi203.stdout:0/367: write f2 [7481604,92824] 0 2024-04-05T03:02:38.208 INFO:tasks.workunit.client.1.smithi203.stdout:4/220: write da/f13 [2884134,129092] 0 2024-04-05T03:02:38.208 INFO:tasks.workunit.client.1.smithi203.stdout:0/368: chown dd/d17/d1d/d26/d28/f2b 140456186 1 2024-04-05T03:02:38.212 INFO:tasks.workunit.client.1.smithi203.stdout:3/265: dwrite d0/dc/d1a/d1f/f32 [4194304,4194304] 0 2024-04-05T03:02:38.216 INFO:tasks.workunit.client.1.smithi203.stdout:4/221: creat da/d3c/f41 x:0 0 0 2024-04-05T03:02:38.218 INFO:tasks.workunit.client.1.smithi203.stdout:0/369: creat dd/d17/d1d/d22/d53/d61/f6a x:0 0 0 2024-04-05T03:02:38.220 INFO:tasks.workunit.client.1.smithi203.stdout:3/266: creat d0/dc/d1a/d33/d35/f47 x:0 0 0 2024-04-05T03:02:38.221 INFO:tasks.workunit.client.1.smithi203.stdout:4/222: creat da/d29/f42 x:0 0 0 2024-04-05T03:02:38.223 INFO:tasks.workunit.client.1.smithi203.stdout:1/233: getdents d7 0 2024-04-05T03:02:38.225 INFO:tasks.workunit.client.1.smithi203.stdout:0/370: rmdir dd/d17/d1d/d26 39 2024-04-05T03:02:38.226 INFO:tasks.workunit.client.1.smithi203.stdout:3/267: mknod d0/dc/d3f/c48 0 2024-04-05T03:02:38.227 INFO:tasks.workunit.client.1.smithi203.stdout:4/223: creat da/d3c/f43 x:0 0 0 2024-04-05T03:02:38.230 INFO:tasks.workunit.client.1.smithi203.stdout:3/268: mkdir d0/dc/d1a/d33/d35/d49 0 2024-04-05T03:02:38.231 INFO:tasks.workunit.client.1.smithi203.stdout:3/269: write d0/dc/d1a/d33/d35/f47 [904192,35743] 0 2024-04-05T03:02:38.234 INFO:tasks.workunit.client.1.smithi203.stdout:1/234: link d7/d24/l4c d7/d14/d2b/l4e 0 2024-04-05T03:02:38.239 INFO:tasks.workunit.client.1.smithi203.stdout:3/270: link d0/dc/d1a/d33/d35/f3c d0/d41/f4a 0 2024-04-05T03:02:38.239 INFO:tasks.workunit.client.1.smithi203.stdout:0/371: link dd/d17/d1d/d22/c52 dd/d17/d1d/d22/c6b 0 2024-04-05T03:02:38.240 INFO:tasks.workunit.client.1.smithi203.stdout:3/271: fsync d0/dc/d1a/d33/d35/f47 0 2024-04-05T03:02:38.242 INFO:tasks.workunit.client.1.smithi203.stdout:4/224: getdents da 0 2024-04-05T03:02:38.242 INFO:tasks.workunit.client.1.smithi203.stdout:1/235: getdents d7/d24/d33/d48/d4b 0 2024-04-05T03:02:38.243 INFO:tasks.workunit.client.1.smithi203.stdout:1/236: truncate d7/f23 123799 0 2024-04-05T03:02:38.243 INFO:tasks.workunit.client.1.smithi203.stdout:4/225: chown da/d29/d2e/l2f 252 1 2024-04-05T03:02:38.244 INFO:tasks.workunit.client.1.smithi203.stdout:0/372: fsync dd/d17/d1d/d26/d41/f4b 0 2024-04-05T03:02:38.244 INFO:tasks.workunit.client.1.smithi203.stdout:1/237: dread - d7/d14/d2b/f49 zero size 2024-04-05T03:02:38.244 INFO:tasks.workunit.client.1.smithi203.stdout:4/226: fdatasync da/d29/d2e/f3e 0 2024-04-05T03:02:38.244 INFO:tasks.workunit.client.1.smithi203.stdout:3/272: creat d0/dc/d1a/d1f/f4b x:0 0 0 2024-04-05T03:02:38.246 INFO:tasks.workunit.client.1.smithi203.stdout:4/227: write da/d3c/f43 [733498,127895] 0 2024-04-05T03:02:38.246 INFO:tasks.workunit.client.1.smithi203.stdout:1/238: creat d7/f4f x:0 0 0 2024-04-05T03:02:38.247 INFO:tasks.workunit.client.1.smithi203.stdout:0/373: symlink dd/d17/d1d/d22/d53/d55/l6c 0 2024-04-05T03:02:38.249 INFO:tasks.workunit.client.1.smithi203.stdout:0/374: dread dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:38.257 INFO:tasks.workunit.client.1.smithi203.stdout:4/228: write da/f22 [3687931,96081] 0 2024-04-05T03:02:38.257 INFO:tasks.workunit.client.1.smithi203.stdout:3/273: truncate d0/dc/d1a/d1d/f2f 2849470 0 2024-04-05T03:02:38.259 INFO:tasks.workunit.client.1.smithi203.stdout:1/239: mkdir d7/d24/d33/d48/d50 0 2024-04-05T03:02:38.260 INFO:tasks.workunit.client.1.smithi203.stdout:4/229: creat da/d3c/f44 x:0 0 0 2024-04-05T03:02:38.262 INFO:tasks.workunit.client.1.smithi203.stdout:3/274: creat d0/dc/d1a/d1d/d43/f4c x:0 0 0 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:8/340: sync 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:2/442: sync 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:6/295: sync 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:5/322: sync 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:9/235: sync 2024-04-05T03:02:38.263 INFO:tasks.workunit.client.1.smithi203.stdout:1/240: readlink d7/d14/l26 0 2024-04-05T03:02:38.264 INFO:tasks.workunit.client.1.smithi203.stdout:6/296: readlink d0/da/d18/d38/l20 0 2024-04-05T03:02:38.265 INFO:tasks.workunit.client.1.smithi203.stdout:5/323: write d3/f32 [961480,47119] 0 2024-04-05T03:02:38.266 INFO:tasks.workunit.client.1.smithi203.stdout:6/297: truncate d0/da/d54/f3f 856559 0 2024-04-05T03:02:38.266 INFO:tasks.workunit.client.1.smithi203.stdout:9/236: fdatasync d13/d1b/f21 0 2024-04-05T03:02:38.268 INFO:tasks.workunit.client.1.smithi203.stdout:6/298: write d0/da/d18/d38/d2a/d2c/f46 [883014,28663] 0 2024-04-05T03:02:38.268 INFO:tasks.workunit.client.1.smithi203.stdout:8/341: mknod d1/d12/d17/d10/c57 0 2024-04-05T03:02:38.269 INFO:tasks.workunit.client.1.smithi203.stdout:6/299: readlink d0/da/d18/d38/l20 0 2024-04-05T03:02:38.270 INFO:tasks.workunit.client.1.smithi203.stdout:5/324: mknod d3/d5f/c6b 0 2024-04-05T03:02:38.272 INFO:tasks.workunit.client.1.smithi203.stdout:2/443: creat d5/f98 x:0 0 0 2024-04-05T03:02:38.273 INFO:tasks.workunit.client.1.smithi203.stdout:1/241: mkdir d7/d24/d33/d48/d4b/d51 0 2024-04-05T03:02:38.273 INFO:tasks.workunit.client.1.smithi203.stdout:3/275: getdents d0/dc/d1a/d33 0 2024-04-05T03:02:38.274 INFO:tasks.workunit.client.1.smithi203.stdout:9/237: dwrite d13/d1b/d26/d31/f41 [0,4194304] 0 2024-04-05T03:02:38.276 INFO:tasks.workunit.client.1.smithi203.stdout:6/300: rename d0/f1a to d0/da/d18/d38/d2b/f5a 0 2024-04-05T03:02:38.279 INFO:tasks.workunit.client.1.smithi203.stdout:3/276: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:38.279 INFO:tasks.workunit.client.1.smithi203.stdout:1/242: chown d7/f2e 1211730 1 2024-04-05T03:02:38.288 INFO:tasks.workunit.client.1.smithi203.stdout:5/325: rename d3/d1f/f39 to d3/d4/dc/f6c 0 2024-04-05T03:02:38.289 INFO:tasks.workunit.client.1.smithi203.stdout:3/277: unlink d0/d21/l27 0 2024-04-05T03:02:38.292 INFO:tasks.workunit.client.1.smithi203.stdout:9/238: symlink d13/d1b/d33/d42/l46 0 2024-04-05T03:02:38.295 INFO:tasks.workunit.client.1.smithi203.stdout:9/239: fdatasync fa 0 2024-04-05T03:02:38.296 INFO:tasks.workunit.client.1.smithi203.stdout:3/278: dread d0/dc/d1a/d33/d35/f3c [0,4194304] 0 2024-04-05T03:02:38.297 INFO:tasks.workunit.client.1.smithi203.stdout:8/342: truncate d1/d12/d2f/d1f/f23 2453849 0 2024-04-05T03:02:38.299 INFO:tasks.workunit.client.1.smithi203.stdout:6/301: dread d0/f41 [0,4194304] 0 2024-04-05T03:02:38.302 INFO:tasks.workunit.client.1.smithi203.stdout:8/343: write d1/d12/d2f/f2a [2148743,18135] 0 2024-04-05T03:02:38.302 INFO:tasks.workunit.client.1.smithi203.stdout:2/444: dwrite d5/f8 [0,4194304] 0 2024-04-05T03:02:38.304 INFO:tasks.workunit.client.1.smithi203.stdout:5/326: chown d3/d1f/d59/d64/d12/f43 0 1 2024-04-05T03:02:38.306 INFO:tasks.workunit.client.1.smithi203.stdout:8/344: chown d1/d6/d2e/c38 2736998 1 2024-04-05T03:02:38.306 INFO:tasks.workunit.client.1.smithi203.stdout:3/279: rmdir d0/dc/d1a/d33/d35/d49 0 2024-04-05T03:02:38.308 INFO:tasks.workunit.client.1.smithi203.stdout:2/445: write d5/d10/d1a/d57/f6b [841559,118414] 0 2024-04-05T03:02:38.314 INFO:tasks.workunit.client.1.smithi203.stdout:9/240: getdents d13/d1b/d26/d31/d38 0 2024-04-05T03:02:38.315 INFO:tasks.workunit.client.1.smithi203.stdout:8/345: creat d1/f58 x:0 0 0 2024-04-05T03:02:38.316 INFO:tasks.workunit.client.1.smithi203.stdout:1/243: dread d7/fe [0,4194304] 0 2024-04-05T03:02:38.316 INFO:tasks.workunit.client.1.smithi203.stdout:3/280: dwrite d0/dc/d1a/d1d/f29 [0,4194304] 0 2024-04-05T03:02:38.317 INFO:tasks.workunit.client.1.smithi203.stdout:2/446: creat d5/d10/d7f/f99 x:0 0 0 2024-04-05T03:02:38.318 INFO:tasks.workunit.client.1.smithi203.stdout:1/244: chown d7/d14/d2b/c31 28510 1 2024-04-05T03:02:38.318 INFO:tasks.workunit.client.1.smithi203.stdout:9/241: unlink d13/f3c 0 2024-04-05T03:02:38.319 INFO:tasks.workunit.client.1.smithi203.stdout:1/245: fsync d7/d24/d33/f3f 0 2024-04-05T03:02:38.320 INFO:tasks.workunit.client.1.smithi203.stdout:9/242: chown d13/d1b/d26/d31 2705 1 2024-04-05T03:02:38.320 INFO:tasks.workunit.client.1.smithi203.stdout:6/302: truncate d0/da/d18/f30 634375 0 2024-04-05T03:02:38.323 INFO:tasks.workunit.client.1.smithi203.stdout:8/346: unlink d1/d6/d2e/c35 0 2024-04-05T03:02:38.323 INFO:tasks.workunit.client.1.smithi203.stdout:9/243: dread d13/f2a [0,4194304] 0 2024-04-05T03:02:38.325 INFO:tasks.workunit.client.1.smithi203.stdout:3/281: mknod d0/dc/d3f/c4d 0 2024-04-05T03:02:38.326 INFO:tasks.workunit.client.1.smithi203.stdout:1/246: fdatasync d7/f2a 0 2024-04-05T03:02:38.327 INFO:tasks.workunit.client.1.smithi203.stdout:6/303: creat d0/da/d54/d4c/f5b x:0 0 0 2024-04-05T03:02:38.329 INFO:tasks.workunit.client.1.smithi203.stdout:8/347: symlink d1/d12/d17/d47/d53/l59 0 2024-04-05T03:02:38.332 INFO:tasks.workunit.client.1.smithi203.stdout:6/304: unlink d0/da/d18/f25 0 2024-04-05T03:02:38.338 INFO:tasks.workunit.client.1.smithi203.stdout:3/282: dwrite d0/dc/d1a/d1f/f32 [0,4194304] 0 2024-04-05T03:02:38.340 INFO:tasks.workunit.client.1.smithi203.stdout:5/327: dread d3/d1f/d59/d64/f23 [0,4194304] 0 2024-04-05T03:02:38.343 INFO:tasks.workunit.client.1.smithi203.stdout:5/328: stat d3/d4/dc/c19 0 2024-04-05T03:02:38.344 INFO:tasks.workunit.client.1.smithi203.stdout:1/247: dwrite d7/f13 [0,4194304] 0 2024-04-05T03:02:38.344 INFO:tasks.workunit.client.1.smithi203.stdout:3/283: chown d0/dc/d1a/d1d/c1e 6512269 1 2024-04-05T03:02:38.347 INFO:tasks.workunit.client.1.smithi203.stdout:3/284: dread - d0/d21/f45 zero size 2024-04-05T03:02:38.347 INFO:tasks.workunit.client.1.smithi203.stdout:1/248: write d7/d14/f16 [1368781,41441] 0 2024-04-05T03:02:38.349 INFO:tasks.workunit.client.1.smithi203.stdout:5/329: read d3/d1f/d59/d64/f1c [2394968,11259] 0 2024-04-05T03:02:38.351 INFO:tasks.workunit.client.1.smithi203.stdout:6/305: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:38.355 INFO:tasks.workunit.client.1.smithi203.stdout:3/285: chown d0/dc/d1a/d1d/f2f 276 1 2024-04-05T03:02:38.356 INFO:tasks.workunit.client.1.smithi203.stdout:6/306: unlink d0/da/d18/l1c 0 2024-04-05T03:02:38.358 INFO:tasks.workunit.client.1.smithi203.stdout:5/330: rename d3/d1f/d59/d64/c30 to d3/d4/dc/c6d 0 2024-04-05T03:02:38.359 INFO:tasks.workunit.client.1.smithi203.stdout:1/249: dwrite d7/f2a [0,4194304] 0 2024-04-05T03:02:38.361 INFO:tasks.workunit.client.1.smithi203.stdout:6/307: unlink d0/da/d18/c1f 0 2024-04-05T03:02:38.363 INFO:tasks.workunit.client.1.smithi203.stdout:1/250: rmdir d7/d24/d33/d48 39 2024-04-05T03:02:38.366 INFO:tasks.workunit.client.1.smithi203.stdout:8/348: dread d1/d6/f56 [0,4194304] 0 2024-04-05T03:02:38.367 INFO:tasks.workunit.client.1.smithi203.stdout:5/331: getdents d3/d1f/d59/d64/d12 0 2024-04-05T03:02:38.369 INFO:tasks.workunit.client.1.smithi203.stdout:8/349: mknod d1/d6/c5a 0 2024-04-05T03:02:38.370 INFO:tasks.workunit.client.1.smithi203.stdout:6/308: dwrite d0/da/f10 [4194304,4194304] 0 2024-04-05T03:02:38.373 INFO:tasks.workunit.client.1.smithi203.stdout:8/350: write d1/d12/d1a/f2c [1499796,73772] 0 2024-04-05T03:02:38.375 INFO:tasks.workunit.client.1.smithi203.stdout:6/309: rmdir d0/da/d18/d38 39 2024-04-05T03:02:38.375 INFO:tasks.workunit.client.1.smithi203.stdout:5/332: getdents d3/d4 0 2024-04-05T03:02:38.377 INFO:tasks.workunit.client.1.smithi203.stdout:7/235: sync 2024-04-05T03:02:38.378 INFO:tasks.workunit.client.1.smithi203.stdout:8/351: rmdir d1/d4d 0 2024-04-05T03:02:38.379 INFO:tasks.workunit.client.1.smithi203.stdout:7/236: chown d8/d32/d16/d1a/d30/l51 24733665 1 2024-04-05T03:02:38.379 INFO:tasks.workunit.client.1.smithi203.stdout:5/333: chown d3/d1f/d59/d64/l55 1 1 2024-04-05T03:02:38.380 INFO:tasks.workunit.client.1.smithi203.stdout:8/352: write d1/d12/d1a/f2c [1072785,48131] 0 2024-04-05T03:02:38.381 INFO:tasks.workunit.client.1.smithi203.stdout:6/310: mknod d0/da/d18/d38/d2b/c5c 0 2024-04-05T03:02:38.381 INFO:tasks.workunit.client.1.smithi203.stdout:5/334: rmdir d3/d5f 39 2024-04-05T03:02:38.381 INFO:tasks.workunit.client.1.smithi203.stdout:7/237: write d8/d32/d16/d1a/d21/f27 [174617,34347] 0 2024-04-05T03:02:38.383 INFO:tasks.workunit.client.1.smithi203.stdout:8/353: mkdir d1/d12/d2f/d5b 0 2024-04-05T03:02:38.387 INFO:tasks.workunit.client.1.smithi203.stdout:7/238: dread d8/f31 [0,4194304] 0 2024-04-05T03:02:38.391 INFO:tasks.workunit.client.1.smithi203.stdout:6/311: dwrite d0/da/d18/d38/d2a/d2c/f4f [0,4194304] 0 2024-04-05T03:02:38.392 INFO:tasks.workunit.client.1.smithi203.stdout:5/335: dwrite d3/d1f/d59/d64/f2e [0,4194304] 0 2024-04-05T03:02:38.392 INFO:tasks.workunit.client.1.smithi203.stdout:7/239: truncate d8/d32/d16/d1a/d21/f23 4272006 0 2024-04-05T03:02:38.393 INFO:tasks.workunit.client.1.smithi203.stdout:7/240: chown d8/d32 31353 1 2024-04-05T03:02:38.393 INFO:tasks.workunit.client.1.smithi203.stdout:6/312: dread - d0/da/d18/d38/d2b/f2d zero size 2024-04-05T03:02:38.395 INFO:tasks.workunit.client.1.smithi203.stdout:7/241: read - d8/f4c zero size 2024-04-05T03:02:38.396 INFO:tasks.workunit.client.1.smithi203.stdout:6/313: write d0/da/f43 [310532,124036] 0 2024-04-05T03:02:38.404 INFO:tasks.workunit.client.1.smithi203.stdout:7/242: dwrite d8/d32/f44 [0,4194304] 0 2024-04-05T03:02:38.405 INFO:tasks.workunit.client.1.smithi203.stdout:5/336: truncate d3/d4/dc/f37 3650122 0 2024-04-05T03:02:38.406 INFO:tasks.workunit.client.1.smithi203.stdout:7/243: dread d8/d32/f18 [0,4194304] 0 2024-04-05T03:02:38.408 INFO:tasks.workunit.client.1.smithi203.stdout:6/314: link d0/da/d18/d38/d2a/d2c/l3b d0/da/d18/d38/d2a/l5d 0 2024-04-05T03:02:38.408 INFO:tasks.workunit.client.1.smithi203.stdout:7/244: write d8/d32/d16/d1a/f1f [892872,48411] 0 2024-04-05T03:02:38.411 INFO:tasks.workunit.client.1.smithi203.stdout:6/315: unlink d0/ff 0 2024-04-05T03:02:38.413 INFO:tasks.workunit.client.1.smithi203.stdout:6/316: chown d0/da/d18/d38/c22 94717 1 2024-04-05T03:02:38.415 INFO:tasks.workunit.client.1.smithi203.stdout:7/245: dread d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:38.417 INFO:tasks.workunit.client.1.smithi203.stdout:7/246: readlink d8/d32/d16/d1a/d21/l48 0 2024-04-05T03:02:38.418 INFO:tasks.workunit.client.1.smithi203.stdout:6/317: chown d0/da/d18/d38/d2a/d2c/l3b 384876 1 2024-04-05T03:02:38.422 INFO:tasks.workunit.client.1.smithi203.stdout:6/318: symlink d0/da/d18/d38/l5e 0 2024-04-05T03:02:38.422 INFO:tasks.workunit.client.1.smithi203.stdout:5/337: dread d3/d4/fb [0,4194304] 0 2024-04-05T03:02:38.426 INFO:tasks.workunit.client.1.smithi203.stdout:5/338: mknod d3/d1f/d49/c6e 0 2024-04-05T03:02:38.431 INFO:tasks.workunit.client.1.smithi203.stdout:5/339: rename d3/d4/c53 to d3/d1f/d49/c6f 0 2024-04-05T03:02:38.439 INFO:tasks.workunit.client.1.smithi203.stdout:6/319: write d0/da/d54/f45 [575508,26272] 0 2024-04-05T03:02:38.440 INFO:tasks.workunit.client.1.smithi203.stdout:7/247: dwrite d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:38.441 INFO:tasks.workunit.client.1.smithi203.stdout:7/248: fdatasync d8/d2a/d3f/d52/f45 0 2024-04-05T03:02:38.442 INFO:tasks.workunit.client.1.smithi203.stdout:7/249: stat d8/f4c 0 2024-04-05T03:02:38.443 INFO:tasks.workunit.client.1.smithi203.stdout:7/250: chown d8/da/cc 3111442 1 2024-04-05T03:02:38.445 INFO:tasks.workunit.client.1.smithi203.stdout:7/251: fsync d8/d2a/d3f/d52/f2e 0 2024-04-05T03:02:38.446 INFO:tasks.workunit.client.1.smithi203.stdout:7/252: chown d8/d32/d16/d1a/d30 17496 1 2024-04-05T03:02:38.447 INFO:tasks.workunit.client.1.smithi203.stdout:7/253: write d8/f37 [64002,5236] 0 2024-04-05T03:02:38.449 INFO:tasks.workunit.client.1.smithi203.stdout:7/254: rename d8/d32/d16 to d8/d32/d16/d1a/d30/d53 22 2024-04-05T03:02:38.450 INFO:tasks.workunit.client.1.smithi203.stdout:7/255: readlink d8/d2a/d3f/d52/l3c 0 2024-04-05T03:02:38.450 INFO:tasks.workunit.client.1.smithi203.stdout:6/320: write d0/da/d18/f26 [2706511,65965] 0 2024-04-05T03:02:38.453 INFO:tasks.workunit.client.1.smithi203.stdout:5/340: creat d3/d4/dc/d67/f70 x:0 0 0 2024-04-05T03:02:38.453 INFO:tasks.workunit.client.1.smithi203.stdout:6/321: creat d0/d23/f5f x:0 0 0 2024-04-05T03:02:38.456 INFO:tasks.workunit.client.1.smithi203.stdout:7/256: link d8/f31 d8/d32/d16/d1a/d3d/f54 0 2024-04-05T03:02:38.459 INFO:tasks.workunit.client.1.smithi203.stdout:7/257: unlink d8/f37 0 2024-04-05T03:02:38.461 INFO:tasks.workunit.client.1.smithi203.stdout:7/258: read - d8/d32/d16/f39 zero size 2024-04-05T03:02:38.463 INFO:tasks.workunit.client.1.smithi203.stdout:6/322: rename d0/da/d54/l51 to d0/da/d54/d53/l60 0 2024-04-05T03:02:38.465 INFO:tasks.workunit.client.1.smithi203.stdout:6/323: creat d0/da/f61 x:0 0 0 2024-04-05T03:02:38.466 INFO:tasks.workunit.client.1.smithi203.stdout:7/259: mkdir d8/d32/d16/d1a/d55 0 2024-04-05T03:02:38.468 INFO:tasks.workunit.client.1.smithi203.stdout:5/341: dread d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:38.469 INFO:tasks.workunit.client.1.smithi203.stdout:6/324: write d0/da/d54/f4b [1883822,93426] 0 2024-04-05T03:02:38.471 INFO:tasks.workunit.client.1.smithi203.stdout:6/325: mknod d0/da/d18/d38/d2b/c62 0 2024-04-05T03:02:38.473 INFO:tasks.workunit.client.1.smithi203.stdout:7/260: getdents d8/d2a/d3f/d52 0 2024-04-05T03:02:38.473 INFO:tasks.workunit.client.1.smithi203.stdout:7/261: chown d8/da/f14 398042812 1 2024-04-05T03:02:38.474 INFO:tasks.workunit.client.1.smithi203.stdout:5/342: creat d3/d4/f71 x:0 0 0 2024-04-05T03:02:38.475 INFO:tasks.workunit.client.1.smithi203.stdout:6/326: write d0/da/d54/d4c/f5b [688607,31019] 0 2024-04-05T03:02:38.476 INFO:tasks.workunit.client.1.smithi203.stdout:4/230: sync 2024-04-05T03:02:38.476 INFO:tasks.workunit.client.1.smithi203.stdout:0/375: sync 2024-04-05T03:02:38.477 INFO:tasks.workunit.client.1.smithi203.stdout:5/343: readlink d3/d4/dc/l38 0 2024-04-05T03:02:38.478 INFO:tasks.workunit.client.1.smithi203.stdout:6/327: creat d0/da/d54/d53/f63 x:0 0 0 2024-04-05T03:02:38.479 INFO:tasks.workunit.client.1.smithi203.stdout:4/231: creat da/d29/d2e/f45 x:0 0 0 2024-04-05T03:02:38.479 INFO:tasks.workunit.client.1.smithi203.stdout:5/344: write d3/f36 [2409494,9840] 0 2024-04-05T03:02:38.480 INFO:tasks.workunit.client.1.smithi203.stdout:6/328: write d0/f9 [3113056,123941] 0 2024-04-05T03:02:38.481 INFO:tasks.workunit.client.1.smithi203.stdout:7/262: getdents d8/d32 0 2024-04-05T03:02:38.481 INFO:tasks.workunit.client.1.smithi203.stdout:0/376: chown dd/d17/d1d/d22/c6b 2 1 2024-04-05T03:02:38.485 INFO:tasks.workunit.client.1.smithi203.stdout:7/263: truncate d8/d32/f3a 5036578 0 2024-04-05T03:02:38.486 INFO:tasks.workunit.client.1.smithi203.stdout:6/329: truncate d0/da/fd 1901101 0 2024-04-05T03:02:38.488 INFO:tasks.workunit.client.1.smithi203.stdout:4/232: dread da/fc [0,4194304] 0 2024-04-05T03:02:38.489 INFO:tasks.workunit.client.1.smithi203.stdout:7/264: chown d8/d32/d16/d1a/d30/l51 93161 1 2024-04-05T03:02:38.489 INFO:tasks.workunit.client.1.smithi203.stdout:6/330: truncate d0/da/f61 752101 0 2024-04-05T03:02:38.489 INFO:tasks.workunit.client.1.smithi203.stdout:7/265: read - d8/f46 zero size 2024-04-05T03:02:38.491 INFO:tasks.workunit.client.1.smithi203.stdout:7/266: write d8/d2a/f42 [1811568,112248] 0 2024-04-05T03:02:38.491 INFO:tasks.workunit.client.1.smithi203.stdout:4/233: symlink da/d29/l46 0 2024-04-05T03:02:38.492 INFO:tasks.workunit.client.1.smithi203.stdout:6/331: mknod d0/da/c64 0 2024-04-05T03:02:38.493 INFO:tasks.workunit.client.1.smithi203.stdout:7/267: chown d8/d32/d16/d1a/f1f 62 1 2024-04-05T03:02:38.495 INFO:tasks.workunit.client.1.smithi203.stdout:4/234: creat da/d29/f47 x:0 0 0 2024-04-05T03:02:38.496 INFO:tasks.workunit.client.1.smithi203.stdout:7/268: creat d8/d32/f56 x:0 0 0 2024-04-05T03:02:38.497 INFO:tasks.workunit.client.1.smithi203.stdout:4/235: creat da/d29/d2e/f48 x:0 0 0 2024-04-05T03:02:38.499 INFO:tasks.workunit.client.1.smithi203.stdout:0/377: truncate dd/d17/d1d/d26/d28/f4e 1533045 0 2024-04-05T03:02:38.502 INFO:tasks.workunit.client.1.smithi203.stdout:4/236: chown da/d29/d2e/f48 6670892 1 2024-04-05T03:02:38.502 INFO:tasks.workunit.client.1.smithi203.stdout:6/332: dwrite d0/da/f43 [0,4194304] 0 2024-04-05T03:02:38.506 INFO:tasks.workunit.client.1.smithi203.stdout:4/237: dread - da/d29/d2e/f3e zero size 2024-04-05T03:02:38.508 INFO:tasks.workunit.client.1.smithi203.stdout:5/345: dwrite d3/d1f/d59/d64/f3d [0,4194304] 0 2024-04-05T03:02:38.510 INFO:tasks.workunit.client.1.smithi203.stdout:7/269: dwrite d8/d32/d16/f39 [0,4194304] 0 2024-04-05T03:02:38.515 INFO:tasks.workunit.client.1.smithi203.stdout:4/238: creat da/d3c/f49 x:0 0 0 2024-04-05T03:02:38.516 INFO:tasks.workunit.client.1.smithi203.stdout:7/270: rmdir d8 39 2024-04-05T03:02:38.519 INFO:tasks.workunit.client.1.smithi203.stdout:5/346: creat d3/d1f/d59/d60/f72 x:0 0 0 2024-04-05T03:02:38.520 INFO:tasks.workunit.client.1.smithi203.stdout:4/239: creat da/f4a x:0 0 0 2024-04-05T03:02:38.524 INFO:tasks.workunit.client.1.smithi203.stdout:7/271: write d8/f33 [5639231,117734] 0 2024-04-05T03:02:38.525 INFO:tasks.workunit.client.1.smithi203.stdout:7/272: readlink d8/d32/d16/d35/l41 0 2024-04-05T03:02:38.526 INFO:tasks.workunit.client.1.smithi203.stdout:7/273: chown d8/da/f4e 4 1 2024-04-05T03:02:38.529 INFO:tasks.workunit.client.1.smithi203.stdout:7/274: rename d8/d32/d16/d1a/d21/c4a to d8/d2a/c57 0 2024-04-05T03:02:38.532 INFO:tasks.workunit.client.1.smithi203.stdout:7/275: symlink d8/d32/d16/d1a/d3d/l58 0 2024-04-05T03:02:38.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/276: fdatasync d8/d32/f3a 0 2024-04-05T03:02:38.535 INFO:tasks.workunit.client.1.smithi203.stdout:7/277: truncate d8/d32/f56 296894 0 2024-04-05T03:02:38.536 INFO:tasks.workunit.client.1.smithi203.stdout:4/240: chown da/d29/c30 59341 1 2024-04-05T03:02:38.538 INFO:tasks.workunit.client.1.smithi203.stdout:7/278: mkdir d8/d32/d16/d1a/d3d/d59 0 2024-04-05T03:02:38.540 INFO:tasks.workunit.client.1.smithi203.stdout:4/241: rename da/d29/c30 to da/d29/c4b 0 2024-04-05T03:02:38.541 INFO:tasks.workunit.client.1.smithi203.stdout:5/347: dwrite d3/d1f/f33 [0,4194304] 0 2024-04-05T03:02:38.543 INFO:tasks.workunit.client.1.smithi203.stdout:7/279: unlink d8/d32/d16/f39 0 2024-04-05T03:02:38.544 INFO:tasks.workunit.client.1.smithi203.stdout:7/280: write d8/f3e [1092542,22232] 0 2024-04-05T03:02:38.546 INFO:tasks.workunit.client.1.smithi203.stdout:5/348: readlink d3/d1f/l5a 0 2024-04-05T03:02:38.550 INFO:tasks.workunit.client.1.smithi203.stdout:4/242: mkdir da/d3c/d4c 0 2024-04-05T03:02:38.552 INFO:tasks.workunit.client.1.smithi203.stdout:7/281: mkdir d8/d32/d16/d1a/d5a 0 2024-04-05T03:02:38.555 INFO:tasks.workunit.client.1.smithi203.stdout:4/243: rename da/l23 to da/d3c/l4d 0 2024-04-05T03:02:38.561 INFO:tasks.workunit.client.1.smithi203.stdout:7/282: dread d8/d2a/f42 [0,4194304] 0 2024-04-05T03:02:38.564 INFO:tasks.workunit.client.1.smithi203.stdout:5/349: dwrite d3/d1f/d59/d64/d12/f46 [0,4194304] 0 2024-04-05T03:02:38.564 INFO:tasks.workunit.client.1.smithi203.stdout:7/283: write d8/d32/d16/d1a/d21/f22 [2817561,29925] 0 2024-04-05T03:02:38.570 INFO:tasks.workunit.client.1.smithi203.stdout:7/284: rename d8/d2a/f36 to d8/d2a/d3f/d52/f5b 0 2024-04-05T03:02:38.571 INFO:tasks.workunit.client.1.smithi203.stdout:7/285: dread - d8/d2a/f40 zero size 2024-04-05T03:02:38.572 INFO:tasks.workunit.client.1.smithi203.stdout:4/244: rmdir da/d29/d2e 39 2024-04-05T03:02:38.573 INFO:tasks.workunit.client.1.smithi203.stdout:7/286: write d8/f3e [1839551,36281] 0 2024-04-05T03:02:38.577 INFO:tasks.workunit.client.1.smithi203.stdout:4/245: write da/f3a [1047587,54788] 0 2024-04-05T03:02:38.580 INFO:tasks.workunit.client.1.smithi203.stdout:5/350: dwrite d3/d1f/f3b [4194304,4194304] 0 2024-04-05T03:02:38.581 INFO:tasks.workunit.client.1.smithi203.stdout:4/246: symlink da/d3c/l4e 0 2024-04-05T03:02:38.583 INFO:tasks.workunit.client.1.smithi203.stdout:4/247: chown da/d29/f2b 15 1 2024-04-05T03:02:38.585 INFO:tasks.workunit.client.1.smithi203.stdout:7/287: dwrite d8/d32/d16/d1a/d30/f38 [0,4194304] 0 2024-04-05T03:02:38.587 INFO:tasks.workunit.client.1.smithi203.stdout:4/248: truncate da/fc 8760886 0 2024-04-05T03:02:38.596 INFO:tasks.workunit.client.1.smithi203.stdout:7/288: creat d8/d32/d16/d1a/f5c x:0 0 0 2024-04-05T03:02:38.598 INFO:tasks.workunit.client.1.smithi203.stdout:5/351: dwrite d3/f7 [4194304,4194304] 0 2024-04-05T03:02:38.604 INFO:tasks.workunit.client.1.smithi203.stdout:5/352: chown d3/d1f/d59/d64/f1c 0 1 2024-04-05T03:02:38.606 INFO:tasks.workunit.client.1.smithi203.stdout:4/249: dwrite da/f39 [0,4194304] 0 2024-04-05T03:02:38.610 INFO:tasks.workunit.client.1.smithi203.stdout:5/353: write d3/f4d [123052,73565] 0 2024-04-05T03:02:38.610 INFO:tasks.workunit.client.1.smithi203.stdout:2/447: sync 2024-04-05T03:02:38.611 INFO:tasks.workunit.client.1.smithi203.stdout:2/448: stat d5/d63 0 2024-04-05T03:02:38.612 INFO:tasks.workunit.client.1.smithi203.stdout:7/289: dwrite d8/d2a/d3f/d52/f45 [0,4194304] 0 2024-04-05T03:02:38.612 INFO:tasks.workunit.client.1.smithi203.stdout:4/250: rmdir da/d29/d2e 39 2024-04-05T03:02:38.615 INFO:tasks.workunit.client.1.smithi203.stdout:7/290: mkdir d8/d32/d5d 0 2024-04-05T03:02:38.616 INFO:tasks.workunit.client.1.smithi203.stdout:4/251: fdatasync da/f18 0 2024-04-05T03:02:38.621 INFO:tasks.workunit.client.1.smithi203.stdout:4/252: symlink da/d3c/d4c/l4f 0 2024-04-05T03:02:38.622 INFO:tasks.workunit.client.1.smithi203.stdout:2/449: dread d5/d10/d1a/d24/d87/f89 [0,4194304] 0 2024-04-05T03:02:38.625 INFO:tasks.workunit.client.1.smithi203.stdout:4/253: unlink da/d3c/f41 0 2024-04-05T03:02:38.628 INFO:tasks.workunit.client.1.smithi203.stdout:7/291: dwrite d8/da/f50 [0,4194304] 0 2024-04-05T03:02:38.630 INFO:tasks.workunit.client.1.smithi203.stdout:2/450: rename d5/d10/d1a/d24/f2b to d5/d10/d35/d2d/d7c/d7a/f9a 0 2024-04-05T03:02:38.630 INFO:tasks.workunit.client.1.smithi203.stdout:7/292: write d8/d2a/d3f/d52/f45 [2646125,47513] 0 2024-04-05T03:02:38.632 INFO:tasks.workunit.client.1.smithi203.stdout:4/254: creat da/d29/f50 x:0 0 0 2024-04-05T03:02:38.634 INFO:tasks.workunit.client.1.smithi203.stdout:2/451: mknod d5/d10/d1a/d57/d70/c9b 0 2024-04-05T03:02:38.636 INFO:tasks.workunit.client.1.smithi203.stdout:7/293: mkdir d8/d2a/d3f/d5e 0 2024-04-05T03:02:38.636 INFO:tasks.workunit.client.1.smithi203.stdout:5/354: truncate d3/d4/dc/f6c 3123586 0 2024-04-05T03:02:38.638 INFO:tasks.workunit.client.1.smithi203.stdout:2/452: creat d5/d63/f9c x:0 0 0 2024-04-05T03:02:38.640 INFO:tasks.workunit.client.1.smithi203.stdout:7/294: symlink d8/d2a/l5f 0 2024-04-05T03:02:38.642 INFO:tasks.workunit.client.1.smithi203.stdout:2/453: creat d5/d10/d2f/f9d x:0 0 0 2024-04-05T03:02:38.642 INFO:tasks.workunit.client.1.smithi203.stdout:4/255: dwrite da/f4a [0,4194304] 0 2024-04-05T03:02:38.642 INFO:tasks.workunit.client.1.smithi203.stdout:5/355: mknod d3/d1f/d59/d64/c73 0 2024-04-05T03:02:38.647 INFO:tasks.workunit.client.1.smithi203.stdout:2/454: fdatasync d5/d10/d1a/f65 0 2024-04-05T03:02:38.650 INFO:tasks.workunit.client.1.smithi203.stdout:5/356: creat d3/d1f/d59/d60/f74 x:0 0 0 2024-04-05T03:02:38.652 INFO:tasks.workunit.client.1.smithi203.stdout:5/357: fsync d3/d1f/d59/d64/d12/f46 0 2024-04-05T03:02:38.653 INFO:tasks.workunit.client.1.smithi203.stdout:2/455: mkdir d5/d10/d35/d2d/d7c/d29/d67/d9e 0 2024-04-05T03:02:38.659 INFO:tasks.workunit.client.1.smithi203.stdout:2/456: getdents d5/d10/d44 0 2024-04-05T03:02:38.661 INFO:tasks.workunit.client.1.smithi203.stdout:2/457: rmdir d5/d10/d1a/d57/d70 39 2024-04-05T03:02:38.664 INFO:tasks.workunit.client.1.smithi203.stdout:2/458: creat d5/d10/f9f x:0 0 0 2024-04-05T03:02:38.665 INFO:tasks.workunit.client.1.smithi203.stdout:4/256: dwrite da/f26 [0,4194304] 0 2024-04-05T03:02:38.669 INFO:tasks.workunit.client.1.smithi203.stdout:4/257: write da/f22 [5526180,70658] 0 2024-04-05T03:02:38.671 INFO:tasks.workunit.client.1.smithi203.stdout:2/459: creat d5/d10/d35/d2d/d7c/d72/fa0 x:0 0 0 2024-04-05T03:02:38.673 INFO:tasks.workunit.client.1.smithi203.stdout:7/295: dread d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:38.675 INFO:tasks.workunit.client.1.smithi203.stdout:2/460: symlink d5/d63/d86/la1 0 2024-04-05T03:02:38.676 INFO:tasks.workunit.client.1.smithi203.stdout:2/461: dread - d5/d10/d2f/d5d/f68 zero size 2024-04-05T03:02:38.677 INFO:tasks.workunit.client.1.smithi203.stdout:7/296: mknod d8/d32/d16/d1a/d5a/c60 0 2024-04-05T03:02:38.680 INFO:tasks.workunit.client.1.smithi203.stdout:2/462: creat d5/d10/d1a/d24/fa2 x:0 0 0 2024-04-05T03:02:38.680 INFO:tasks.workunit.client.1.smithi203.stdout:2/463: chown d5/d10/f5b 776 1 2024-04-05T03:02:38.682 INFO:tasks.workunit.client.1.smithi203.stdout:7/297: creat d8/d32/d16/d1a/d3d/d59/f61 x:0 0 0 2024-04-05T03:02:38.682 INFO:tasks.workunit.client.1.smithi203.stdout:2/464: write d5/d63/f9c [107516,28480] 0 2024-04-05T03:02:38.683 INFO:tasks.workunit.client.1.smithi203.stdout:4/258: truncate da/d3c/f43 472 0 2024-04-05T03:02:38.684 INFO:tasks.workunit.client.1.smithi203.stdout:7/298: chown d8/d2a/c57 185781737 1 2024-04-05T03:02:38.685 INFO:tasks.workunit.client.1.smithi203.stdout:2/465: stat d5/d10/d35/d2d/d7c/d29/d67/c90 0 2024-04-05T03:02:38.685 INFO:tasks.workunit.client.1.smithi203.stdout:4/259: mknod da/d3c/c51 0 2024-04-05T03:02:38.687 INFO:tasks.workunit.client.1.smithi203.stdout:4/260: creat da/d3c/d4c/f52 x:0 0 0 2024-04-05T03:02:38.688 INFO:tasks.workunit.client.1.smithi203.stdout:7/299: truncate d8/d32/f18 1115820 0 2024-04-05T03:02:38.689 INFO:tasks.workunit.client.1.smithi203.stdout:7/300: fsync d8/d32/d16/d1a/f1f 0 2024-04-05T03:02:38.690 INFO:tasks.workunit.client.1.smithi203.stdout:4/261: mkdir da/d29/d2e/d53 0 2024-04-05T03:02:38.690 INFO:tasks.workunit.client.1.smithi203.stdout:7/301: read d8/fe [439811,20222] 0 2024-04-05T03:02:38.691 INFO:tasks.workunit.client.1.smithi203.stdout:2/466: getdents d5/d66/d7d 0 2024-04-05T03:02:38.691 INFO:tasks.workunit.client.1.smithi203.stdout:4/262: stat da/d3c/d4c/f52 0 2024-04-05T03:02:38.693 INFO:tasks.workunit.client.1.smithi203.stdout:2/467: chown d5/d10/d35/d2d/d7c/f78 24 1 2024-04-05T03:02:38.694 INFO:tasks.workunit.client.1.smithi203.stdout:2/468: chown d5/d10/d7f/f99 1209 1 2024-04-05T03:02:38.694 INFO:tasks.workunit.client.1.smithi203.stdout:7/302: creat d8/d32/d16/d1a/d21/f62 x:0 0 0 2024-04-05T03:02:38.695 INFO:tasks.workunit.client.1.smithi203.stdout:7/303: dread - d8/f4c zero size 2024-04-05T03:02:38.696 INFO:tasks.workunit.client.1.smithi203.stdout:4/263: symlink da/d29/l54 0 2024-04-05T03:02:38.697 INFO:tasks.workunit.client.1.smithi203.stdout:7/304: dread - d8/f4c zero size 2024-04-05T03:02:38.697 INFO:tasks.workunit.client.1.smithi203.stdout:7/305: fdatasync d8/da/f50 0 2024-04-05T03:02:38.698 INFO:tasks.workunit.client.1.smithi203.stdout:2/469: rename f3 to d5/d63/fa3 0 2024-04-05T03:02:38.699 INFO:tasks.workunit.client.1.smithi203.stdout:2/470: fsync d5/d10/d35/d36/f54 0 2024-04-05T03:02:38.701 INFO:tasks.workunit.client.1.smithi203.stdout:2/471: chown d5/d10/d35/d2d/l3d 49 1 2024-04-05T03:02:38.701 INFO:tasks.workunit.client.1.smithi203.stdout:2/472: readlink l2 0 2024-04-05T03:02:38.704 INFO:tasks.workunit.client.1.smithi203.stdout:7/306: dwrite d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:38.705 INFO:tasks.workunit.client.1.smithi203.stdout:7/307: fsync d8/d32/d16/d1a/f1f 0 2024-04-05T03:02:38.706 INFO:tasks.workunit.client.1.smithi203.stdout:2/473: mknod d5/d10/d35/d2d/d7c/d72/d8f/ca4 0 2024-04-05T03:02:38.708 INFO:tasks.workunit.client.1.smithi203.stdout:2/474: truncate d5/d10/d35/d36/f3b 1297273 0 2024-04-05T03:02:38.709 INFO:tasks.workunit.client.1.smithi203.stdout:2/475: write d5/f43 [4143711,35111] 0 2024-04-05T03:02:38.714 INFO:tasks.workunit.client.1.smithi203.stdout:7/308: rmdir d8/d32/d16/d1a 39 2024-04-05T03:02:38.719 INFO:tasks.workunit.client.1.smithi203.stdout:7/309: getdents d8/d2a 0 2024-04-05T03:02:38.720 INFO:tasks.workunit.client.1.smithi203.stdout:7/310: stat d8/d32/d16/d1a 0 2024-04-05T03:02:38.721 INFO:tasks.workunit.client.1.smithi203.stdout:7/311: truncate d8/d32/f44 4399206 0 2024-04-05T03:02:38.722 INFO:tasks.workunit.client.1.smithi203.stdout:4/264: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:38.723 INFO:tasks.workunit.client.1.smithi203.stdout:7/312: mkdir d8/d2a/d3f/d5e/d63 0 2024-04-05T03:02:38.723 INFO:tasks.workunit.client.1.smithi203.stdout:4/265: chown da/fc 104 1 2024-04-05T03:02:38.724 INFO:tasks.workunit.client.1.smithi203.stdout:7/313: chown d8/c9 6 1 2024-04-05T03:02:38.726 INFO:tasks.workunit.client.1.smithi203.stdout:7/314: write d8/d32/d16/d1a/d21/f62 [1044937,25700] 0 2024-04-05T03:02:38.729 INFO:tasks.workunit.client.1.smithi203.stdout:4/266: write da/f11 [2647561,28770] 0 2024-04-05T03:02:38.735 INFO:tasks.workunit.client.1.smithi203.stdout:7/315: dwrite d8/f31 [0,4194304] 0 2024-04-05T03:02:38.737 INFO:tasks.workunit.client.1.smithi203.stdout:7/316: mkdir d8/d32/d16/d35/d64 0 2024-04-05T03:02:38.754 INFO:tasks.workunit.client.1.smithi203.stdout:7/317: dread d8/da/fd [0,4194304] 0 2024-04-05T03:02:38.755 INFO:tasks.workunit.client.1.smithi203.stdout:7/318: truncate d8/da/f50 4500460 0 2024-04-05T03:02:38.759 INFO:tasks.workunit.client.1.smithi203.stdout:7/319: write d8/fe [192471,113567] 0 2024-04-05T03:02:38.761 INFO:tasks.workunit.client.1.smithi203.stdout:7/320: mkdir d8/d2a/d3f/d52/d65 0 2024-04-05T03:02:38.764 INFO:tasks.workunit.client.1.smithi203.stdout:3/286: sync 2024-04-05T03:02:38.764 INFO:tasks.workunit.client.1.smithi203.stdout:8/354: sync 2024-04-05T03:02:38.765 INFO:tasks.workunit.client.1.smithi203.stdout:0/378: sync 2024-04-05T03:02:38.765 INFO:tasks.workunit.client.1.smithi203.stdout:1/251: sync 2024-04-05T03:02:38.765 INFO:tasks.workunit.client.1.smithi203.stdout:9/244: sync 2024-04-05T03:02:38.767 INFO:tasks.workunit.client.1.smithi203.stdout:9/245: stat d13/d1b/d33/c44 0 2024-04-05T03:02:38.767 INFO:tasks.workunit.client.1.smithi203.stdout:8/355: mkdir d1/d6/d2e/d5c 0 2024-04-05T03:02:38.768 INFO:tasks.workunit.client.1.smithi203.stdout:7/321: dwrite d8/f31 [0,4194304] 0 2024-04-05T03:02:38.769 INFO:tasks.workunit.client.1.smithi203.stdout:8/356: chown d1/d12/d1a 55396 1 2024-04-05T03:02:38.769 INFO:tasks.workunit.client.1.smithi203.stdout:0/379: mkdir dd/d17/d1d/d22/d5f/d6d 0 2024-04-05T03:02:38.770 INFO:tasks.workunit.client.1.smithi203.stdout:7/322: read d8/da/f4e [2282947,63854] 0 2024-04-05T03:02:38.771 INFO:tasks.workunit.client.1.smithi203.stdout:3/287: rename d0/dc/d1a/d33/l3e to d0/dc/d1a/d33/d35/l4e 0 2024-04-05T03:02:38.772 INFO:tasks.workunit.client.1.smithi203.stdout:8/357: truncate d1/d12/d17/d10/f49 2070491 0 2024-04-05T03:02:38.774 INFO:tasks.workunit.client.1.smithi203.stdout:9/246: link d13/d1b/d33/f34 d13/d1b/f47 0 2024-04-05T03:02:38.850 INFO:tasks.workunit.client.1.smithi203.stdout:9/247: write d13/d1b/d26/d31/f41 [703626,56953] 0 2024-04-05T03:02:38.850 INFO:tasks.workunit.client.1.smithi203.stdout:7/323: truncate d8/d2a/d3f/d52/f2e 3393704 0 2024-04-05T03:02:38.852 INFO:tasks.workunit.client.1.smithi203.stdout:8/358: rename d1/d12/l4a to d1/d12/d2f/d55/l5d 0 2024-04-05T03:02:38.854 INFO:tasks.workunit.client.1.smithi203.stdout:8/359: write d1/d12/d17/d10/f40 [187927,6237] 0 2024-04-05T03:02:38.854 INFO:tasks.workunit.client.1.smithi203.stdout:9/248: rmdir d13/d1b/d33 39 2024-04-05T03:02:38.854 INFO:tasks.workunit.client.1.smithi203.stdout:7/324: rmdir d8/d32/d16/d1a/d3d/d59 39 2024-04-05T03:02:38.857 INFO:tasks.workunit.client.1.smithi203.stdout:1/252: dread f2 [4194304,4194304] 0 2024-04-05T03:02:38.858 INFO:tasks.workunit.client.1.smithi203.stdout:8/360: symlink d1/d12/d2f/d5b/l5e 0 2024-04-05T03:02:38.859 INFO:tasks.workunit.client.1.smithi203.stdout:1/253: read d7/d24/d33/f3f [703660,69655] 0 2024-04-05T03:02:38.859 INFO:tasks.workunit.client.1.smithi203.stdout:9/249: mknod d13/d1b/d26/d31/c48 0 2024-04-05T03:02:38.859 INFO:tasks.workunit.client.1.smithi203.stdout:1/254: write d7/fe [4365011,84479] 0 2024-04-05T03:02:38.860 INFO:tasks.workunit.client.1.smithi203.stdout:8/361: mkdir d1/d4b/d5f 0 2024-04-05T03:02:38.861 INFO:tasks.workunit.client.1.smithi203.stdout:3/288: dread d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:38.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/289: readlink d0/l40 0 2024-04-05T03:02:38.865 INFO:tasks.workunit.client.1.smithi203.stdout:8/362: write d1/d12/d1a/f42 [5140594,101243] 0 2024-04-05T03:02:38.865 INFO:tasks.workunit.client.1.smithi203.stdout:3/290: unlink d0/f4 0 2024-04-05T03:02:38.867 INFO:tasks.workunit.client.1.smithi203.stdout:3/291: chown d0/dc/d38 171689 1 2024-04-05T03:02:38.870 INFO:tasks.workunit.client.1.smithi203.stdout:3/292: chown d0/dc/l1c 2298 1 2024-04-05T03:02:38.870 INFO:tasks.workunit.client.1.smithi203.stdout:1/255: dwrite d7/d24/f35 [0,4194304] 0 2024-04-05T03:02:38.871 INFO:tasks.workunit.client.1.smithi203.stdout:9/250: dwrite d13/d1b/d33/f3f [0,4194304] 0 2024-04-05T03:02:38.873 INFO:tasks.workunit.client.1.smithi203.stdout:9/251: fdatasync fa 0 2024-04-05T03:02:38.875 INFO:tasks.workunit.client.1.smithi203.stdout:1/256: write d7/d24/d33/f36 [2092116,117473] 0 2024-04-05T03:02:38.875 INFO:tasks.workunit.client.1.smithi203.stdout:9/252: stat d13/f20 0 2024-04-05T03:02:38.876 INFO:tasks.workunit.client.1.smithi203.stdout:9/253: stat d13/d1b/d26/d31/l37 0 2024-04-05T03:02:38.880 INFO:tasks.workunit.client.1.smithi203.stdout:9/254: creat d13/d1b/d33/d42/f49 x:0 0 0 2024-04-05T03:02:38.880 INFO:tasks.workunit.client.1.smithi203.stdout:1/257: dread d7/f2a [0,4194304] 0 2024-04-05T03:02:38.889 INFO:tasks.workunit.client.1.smithi203.stdout:1/258: dwrite d7/f13 [0,4194304] 0 2024-04-05T03:02:38.891 INFO:tasks.workunit.client.1.smithi203.stdout:7/325: dread d8/d32/f3a [0,4194304] 0 2024-04-05T03:02:38.891 INFO:tasks.workunit.client.1.smithi203.stdout:9/255: dwrite d13/f2a [0,4194304] 0 2024-04-05T03:02:38.894 INFO:tasks.workunit.client.1.smithi203.stdout:1/259: chown c1 18 1 2024-04-05T03:02:38.894 INFO:tasks.workunit.client.1.smithi203.stdout:9/256: rename d13/f25 to d13/f4a 0 2024-04-05T03:02:38.895 INFO:tasks.workunit.client.1.smithi203.stdout:8/363: dread d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:38.898 INFO:tasks.workunit.client.1.smithi203.stdout:7/326: mknod d8/d2a/d3f/d52/c66 0 2024-04-05T03:02:38.902 INFO:tasks.workunit.client.1.smithi203.stdout:1/260: rename d7/d24/f4d to d7/d14/f52 0 2024-04-05T03:02:38.904 INFO:tasks.workunit.client.1.smithi203.stdout:7/327: symlink d8/d32/d16/d35/l67 0 2024-04-05T03:02:38.907 INFO:tasks.workunit.client.1.smithi203.stdout:1/261: chown d7/c22 255329025 1 2024-04-05T03:02:38.907 INFO:tasks.workunit.client.1.smithi203.stdout:8/364: rename d1/d6/d2e/l3b to d1/d12/d1a/l60 0 2024-04-05T03:02:38.909 INFO:tasks.workunit.client.1.smithi203.stdout:9/257: dread f7 [0,4194304] 0 2024-04-05T03:02:38.910 INFO:tasks.workunit.client.1.smithi203.stdout:7/328: mkdir d8/d32/d16/d1a/d68 0 2024-04-05T03:02:38.910 INFO:tasks.workunit.client.1.smithi203.stdout:1/262: rename d7/d24 to d7/d24/d33/d48/d53 22 2024-04-05T03:02:38.911 INFO:tasks.workunit.client.1.smithi203.stdout:8/365: creat d1/d12/d2f/f61 x:0 0 0 2024-04-05T03:02:38.911 INFO:tasks.workunit.client.1.smithi203.stdout:3/293: dwrite d0/dc/f14 [0,4194304] 0 2024-04-05T03:02:38.913 INFO:tasks.workunit.client.1.smithi203.stdout:5/358: sync 2024-04-05T03:02:38.913 INFO:tasks.workunit.client.1.smithi203.stdout:6/333: sync 2024-04-05T03:02:38.914 INFO:tasks.workunit.client.1.smithi203.stdout:8/366: write d1/d12/d2f/f2a [2738295,27147] 0 2024-04-05T03:02:38.916 INFO:tasks.workunit.client.1.smithi203.stdout:5/359: chown d3/d1f/f62 0 1 2024-04-05T03:02:38.916 INFO:tasks.workunit.client.1.smithi203.stdout:7/329: unlink d8/d32/d16/d35/l67 0 2024-04-05T03:02:38.920 INFO:tasks.workunit.client.1.smithi203.stdout:7/330: fsync d8/d32/f43 0 2024-04-05T03:02:38.921 INFO:tasks.workunit.client.1.smithi203.stdout:8/367: chown d1/d6/d2e/l4f 11193 1 2024-04-05T03:02:38.922 INFO:tasks.workunit.client.1.smithi203.stdout:5/360: creat d3/d5f/f75 x:0 0 0 2024-04-05T03:02:38.923 INFO:tasks.workunit.client.1.smithi203.stdout:1/263: link d7/l9 d7/d24/l54 0 2024-04-05T03:02:38.924 INFO:tasks.workunit.client.1.smithi203.stdout:9/258: dwrite f8 [0,4194304] 0 2024-04-05T03:02:38.926 INFO:tasks.workunit.client.1.smithi203.stdout:3/294: dread d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:38.926 INFO:tasks.workunit.client.1.smithi203.stdout:8/368: mknod d1/d12/d17/c62 0 2024-04-05T03:02:38.927 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:38 smithi153 ceph-mon[29381]: [05/Apr/2024:03:02:37] ENGINE Serving on http://172.21.15.203:8765 2024-04-05T03:02:38.927 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:38 smithi153 ceph-mon[29381]: [05/Apr/2024:03:02:37] ENGINE Serving on https://172.21.15.203:7150 2024-04-05T03:02:38.927 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:38 smithi153 ceph-mon[29381]: [05/Apr/2024:03:02:37] ENGINE Bus STARTED 2024-04-05T03:02:38.927 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:38 smithi153 ceph-mon[29381]: mgrmap e23: smithi203.iakodz(active, since 2s) 2024-04-05T03:02:38.927 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:38 smithi153 ceph-mon[29381]: pgmap v4: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:38.929 INFO:tasks.workunit.client.1.smithi203.stdout:6/334: dread d0/da/d18/f1d [0,4194304] 0 2024-04-05T03:02:38.930 INFO:tasks.workunit.client.1.smithi203.stdout:9/259: mknod d13/d1b/d26/d31/d38/c4b 0 2024-04-05T03:02:38.930 INFO:tasks.workunit.client.1.smithi203.stdout:3/295: chown d0/dc/d1a/d1d/f29 4303252 1 2024-04-05T03:02:38.931 INFO:tasks.workunit.client.1.smithi203.stdout:6/335: readlink d0/da/d18/l31 0 2024-04-05T03:02:38.934 INFO:tasks.workunit.client.1.smithi203.stdout:8/369: creat d1/f63 x:0 0 0 2024-04-05T03:02:38.935 INFO:tasks.workunit.client.1.smithi203.stdout:7/331: dwrite d8/d32/f43 [0,4194304] 0 2024-04-05T03:02:38.937 INFO:tasks.workunit.client.1.smithi203.stdout:1/264: dread d7/d24/f35 [0,4194304] 0 2024-04-05T03:02:38.937 INFO:tasks.workunit.client.1.smithi203.stdout:8/370: dread - d1/f58 zero size 2024-04-05T03:02:38.937 INFO:tasks.workunit.client.1.smithi203.stdout:9/260: creat d13/d1b/d26/f4c x:0 0 0 2024-04-05T03:02:38.940 INFO:tasks.workunit.client.1.smithi203.stdout:3/296: mknod d0/dc/d1a/d33/c4f 0 2024-04-05T03:02:38.941 INFO:tasks.workunit.client.1.smithi203.stdout:6/336: read d0/f37 [934664,116646] 0 2024-04-05T03:02:38.942 INFO:tasks.workunit.client.1.smithi203.stdout:9/261: write fb [4254659,85497] 0 2024-04-05T03:02:38.943 INFO:tasks.workunit.client.1.smithi203.stdout:5/361: dwrite d3/d1f/d59/d64/f63 [4194304,4194304] 0 2024-04-05T03:02:38.947 INFO:tasks.workunit.client.1.smithi203.stdout:7/332: write d8/d32/d16/d1a/f1f [1021371,99540] 0 2024-04-05T03:02:38.947 INFO:tasks.workunit.client.1.smithi203.stdout:8/371: dread d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:38.947 INFO:tasks.workunit.client.1.smithi203.stdout:3/297: dread - d0/dc/d1a/d33/d35/f3d zero size 2024-04-05T03:02:38.949 INFO:tasks.workunit.client.1.smithi203.stdout:5/362: fsync d3/d1f/d59/d64/f26 0 2024-04-05T03:02:38.949 INFO:tasks.workunit.client.1.smithi203.stdout:6/337: unlink d0/da/d18/f57 0 2024-04-05T03:02:38.953 INFO:tasks.workunit.client.1.smithi203.stdout:8/372: stat d1/d12/d17/d10 0 2024-04-05T03:02:38.956 INFO:tasks.workunit.client.1.smithi203.stdout:9/262: write d13/d1b/d33/d42/f49 [45481,53044] 0 2024-04-05T03:02:38.957 INFO:tasks.workunit.client.1.smithi203.stdout:1/265: rename d7/d14/l20 to d7/l55 0 2024-04-05T03:02:38.957 INFO:tasks.workunit.client.1.smithi203.stdout:7/333: read d8/da/f1d [483956,359] 0 2024-04-05T03:02:38.958 INFO:tasks.workunit.client.1.smithi203.stdout:8/373: dread d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:38.962 INFO:tasks.workunit.client.1.smithi203.stdout:7/334: write d8/f17 [1254567,23189] 0 2024-04-05T03:02:38.964 INFO:tasks.workunit.client.1.smithi203.stdout:6/338: getdents d0/da/d18/d38/d2b 0 2024-04-05T03:02:38.967 INFO:tasks.workunit.client.1.smithi203.stdout:8/374: fsync d1/f63 0 2024-04-05T03:02:38.971 INFO:tasks.workunit.client.1.smithi203.stdout:3/298: dwrite d0/dc/d1a/d1d/f29 [0,4194304] 0 2024-04-05T03:02:38.973 INFO:tasks.workunit.client.1.smithi203.stdout:8/375: write d1/d12/f28 [3467848,126940] 0 2024-04-05T03:02:38.976 INFO:tasks.workunit.client.1.smithi203.stdout:8/376: readlink d1/l48 0 2024-04-05T03:02:38.976 INFO:tasks.workunit.client.1.smithi203.stdout:7/335: link d8/d32/d16/d1a/d30/f38 d8/d32/d5d/f69 0 2024-04-05T03:02:38.980 INFO:tasks.workunit.client.1.smithi203.stdout:9/263: write d13/d1b/d33/f34 [796436,107583] 0 2024-04-05T03:02:38.982 INFO:tasks.workunit.client.1.smithi203.stdout:3/299: rename d0/dc/d1a/d1d/l37 to d0/dc/d1a/d1d/d43/l50 0 2024-04-05T03:02:38.983 INFO:tasks.workunit.client.1.smithi203.stdout:8/377: symlink d1/d6/l64 0 2024-04-05T03:02:38.983 INFO:tasks.workunit.client.1.smithi203.stdout:7/336: rmdir d8/d32/d16/d1a/d3d/d59 39 2024-04-05T03:02:38.984 INFO:tasks.workunit.client.1.smithi203.stdout:9/264: fsync d13/d1b/f47 0 2024-04-05T03:02:38.984 INFO:tasks.workunit.client.1.smithi203.stdout:5/363: dread d3/f32 [0,4194304] 0 2024-04-05T03:02:38.986 INFO:tasks.workunit.client.1.smithi203.stdout:6/339: dwrite d0/da/d18/d38/d2a/d2c/f46 [0,4194304] 0 2024-04-05T03:02:38.989 INFO:tasks.workunit.client.1.smithi203.stdout:9/265: truncate d13/f2a 4862452 0 2024-04-05T03:02:38.991 INFO:tasks.workunit.client.1.smithi203.stdout:9/266: chown d13/d1b/d26/d31/f41 56916154 1 2024-04-05T03:02:38.991 INFO:tasks.workunit.client.1.smithi203.stdout:9/267: stat cc 0 2024-04-05T03:02:39.001 INFO:tasks.workunit.client.1.smithi203.stdout:9/268: symlink d13/d1b/l4d 0 2024-04-05T03:02:39.003 INFO:tasks.workunit.client.1.smithi203.stdout:7/337: dwrite d8/d32/f3a [0,4194304] 0 2024-04-05T03:02:39.003 INFO:tasks.workunit.client.1.smithi203.stdout:6/340: dread d0/da/fd [0,4194304] 0 2024-04-05T03:02:39.007 INFO:tasks.workunit.client.1.smithi203.stdout:6/341: write d0/da/d18/f26 [3034056,37983] 0 2024-04-05T03:02:39.009 INFO:tasks.workunit.client.1.smithi203.stdout:8/378: truncate d1/d12/d1a/f1e 1146660 0 2024-04-05T03:02:39.010 INFO:tasks.workunit.client.1.smithi203.stdout:3/300: truncate d0/dc/d1a/d1d/f29 2235901 0 2024-04-05T03:02:39.011 INFO:tasks.workunit.client.1.smithi203.stdout:9/269: symlink d13/d1b/d26/d31/l4e 0 2024-04-05T03:02:39.012 INFO:tasks.workunit.client.1.smithi203.stdout:5/364: dwrite d3/f18 [0,4194304] 0 2024-04-05T03:02:39.018 INFO:tasks.workunit.client.1.smithi203.stdout:3/301: fsync d0/d41/f4a 0 2024-04-05T03:02:39.018 INFO:tasks.workunit.client.1.smithi203.stdout:6/342: rmdir d0/da/d18/d38/d2b 39 2024-04-05T03:02:39.018 INFO:tasks.workunit.client.1.smithi203.stdout:5/365: write d3/d1f/d59/d64/f10 [3813513,8856] 0 2024-04-05T03:02:39.023 INFO:tasks.workunit.client.1.smithi203.stdout:9/270: rename d13/d1b/d26/d31/l4e to d13/d1b/d26/d31/l4f 0 2024-04-05T03:02:39.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/379: rmdir d1/d12/d2f/d5b 39 2024-04-05T03:02:39.028 INFO:tasks.workunit.client.1.smithi203.stdout:3/302: dwrite d0/f42 [0,4194304] 0 2024-04-05T03:02:39.029 INFO:tasks.workunit.client.1.smithi203.stdout:9/271: mknod d13/d1b/c50 0 2024-04-05T03:02:39.031 INFO:tasks.workunit.client.1.smithi203.stdout:6/343: rename d0/da/d18/d38/d2a/l3c to d0/da/d54/l65 0 2024-04-05T03:02:39.031 INFO:tasks.workunit.client.1.smithi203.stdout:5/366: dread d3/d1f/d59/d64/f63 [4194304,4194304] 0 2024-04-05T03:02:39.031 INFO:tasks.workunit.client.1.smithi203.stdout:9/272: write fb [1772729,38687] 0 2024-04-05T03:02:39.036 INFO:tasks.workunit.client.1.smithi203.stdout:3/303: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:39.038 INFO:tasks.workunit.client.1.smithi203.stdout:7/338: truncate d8/f31 3264923 0 2024-04-05T03:02:39.040 INFO:tasks.workunit.client.1.smithi203.stdout:5/367: getdents d3/d1f/d59/d64/d12 0 2024-04-05T03:02:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:38 smithi203 ceph-mon[39911]: [05/Apr/2024:03:02:37] ENGINE Serving on http://172.21.15.203:8765 2024-04-05T03:02:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:38 smithi203 ceph-mon[39911]: [05/Apr/2024:03:02:37] ENGINE Serving on https://172.21.15.203:7150 2024-04-05T03:02:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:38 smithi203 ceph-mon[39911]: [05/Apr/2024:03:02:37] ENGINE Bus STARTED 2024-04-05T03:02:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:38 smithi203 ceph-mon[39911]: mgrmap e23: smithi203.iakodz(active, since 2s) 2024-04-05T03:02:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:38 smithi203 ceph-mon[39911]: pgmap v4: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:39.047 INFO:tasks.workunit.client.1.smithi203.stdout:7/339: write d8/d32/d16/d1a/d21/f23 [3437437,123137] 0 2024-04-05T03:02:39.047 INFO:tasks.workunit.client.1.smithi203.stdout:3/304: getdents d0/dc/d1a/d33/d35 0 2024-04-05T03:02:39.048 INFO:tasks.workunit.client.1.smithi203.stdout:9/273: dwrite d13/d1b/d33/f3f [0,4194304] 0 2024-04-05T03:02:39.048 INFO:tasks.workunit.client.1.smithi203.stdout:5/368: fsync d3/d1f/d59/d64/f4f 0 2024-04-05T03:02:39.051 INFO:tasks.workunit.client.1.smithi203.stdout:6/344: dwrite d0/da/d18/d38/d2a/d2c/f3a [0,4194304] 0 2024-04-05T03:02:39.051 INFO:tasks.workunit.client.1.smithi203.stdout:3/305: write d0/dc/d38/f3b [2566115,73597] 0 2024-04-05T03:02:39.058 INFO:tasks.workunit.client.1.smithi203.stdout:5/369: write d3/d4/dc/d67/f70 [1021641,17789] 0 2024-04-05T03:02:39.058 INFO:tasks.workunit.client.1.smithi203.stdout:8/380: write d1/d12/d1a/f1e [87905,57513] 0 2024-04-05T03:02:39.060 INFO:tasks.workunit.client.1.smithi203.stdout:6/345: rmdir d0/da/d54/d53 39 2024-04-05T03:02:39.060 INFO:tasks.workunit.client.1.smithi203.stdout:7/340: read d8/d32/d16/d1a/d21/f23 [2743633,50194] 0 2024-04-05T03:02:39.064 INFO:tasks.workunit.client.1.smithi203.stdout:9/274: dread f12 [0,4194304] 0 2024-04-05T03:02:39.066 INFO:tasks.workunit.client.1.smithi203.stdout:7/341: fsync d8/d32/f43 0 2024-04-05T03:02:39.070 INFO:tasks.workunit.client.1.smithi203.stdout:8/381: mknod d1/d4b/d5f/c65 0 2024-04-05T03:02:39.070 INFO:tasks.workunit.client.1.smithi203.stdout:9/275: chown d13/d1b/l4d 88756 1 2024-04-05T03:02:39.073 INFO:tasks.workunit.client.1.smithi203.stdout:6/346: truncate d0/da/d54/d4c/f5b 1531782 0 2024-04-05T03:02:39.073 INFO:tasks.workunit.client.1.smithi203.stdout:2/476: sync 2024-04-05T03:02:39.074 INFO:tasks.workunit.client.1.smithi203.stdout:3/306: dwrite d0/d21/f45 [0,4194304] 0 2024-04-05T03:02:39.074 INFO:tasks.workunit.client.1.smithi203.stdout:5/370: rename d3/d4/f1a to d3/d1f/d59/d60/f76 0 2024-04-05T03:02:39.075 INFO:tasks.workunit.client.1.smithi203.stdout:8/382: mknod d1/c66 0 2024-04-05T03:02:39.082 INFO:tasks.workunit.client.1.smithi203.stdout:8/383: rename d1/d6/c25 to d1/d4b/c67 0 2024-04-05T03:02:39.084 INFO:tasks.workunit.client.1.smithi203.stdout:7/342: dread d8/da/fd [0,4194304] 0 2024-04-05T03:02:39.088 INFO:tasks.workunit.client.1.smithi203.stdout:3/307: dwrite d0/dc/d1a/d33/d35/f3d [0,4194304] 0 2024-04-05T03:02:39.089 INFO:tasks.workunit.client.1.smithi203.stdout:3/308: chown d0/f9 28303 1 2024-04-05T03:02:39.089 INFO:tasks.workunit.client.1.smithi203.stdout:8/384: mkdir d1/d4b/d68 0 2024-04-05T03:02:39.090 INFO:tasks.workunit.client.1.smithi203.stdout:8/385: fdatasync d1/f63 0 2024-04-05T03:02:39.091 INFO:tasks.workunit.client.1.smithi203.stdout:5/371: rmdir d3/d4 39 2024-04-05T03:02:39.093 INFO:tasks.workunit.client.1.smithi203.stdout:7/343: chown d8/c1c 0 1 2024-04-05T03:02:39.095 INFO:tasks.workunit.client.1.smithi203.stdout:3/309: creat d0/dc/d1a/d1d/d43/f51 x:0 0 0 2024-04-05T03:02:39.097 INFO:tasks.workunit.client.1.smithi203.stdout:6/347: getdents d0/da/d54/d53 0 2024-04-05T03:02:39.097 INFO:tasks.workunit.client.1.smithi203.stdout:5/372: mknod d3/d1f/d59/d64/d12/c77 0 2024-04-05T03:02:39.099 INFO:tasks.workunit.client.1.smithi203.stdout:7/344: write d8/d32/f56 [1285431,61850] 0 2024-04-05T03:02:39.100 INFO:tasks.workunit.client.1.smithi203.stdout:3/310: rmdir d0/dc 39 2024-04-05T03:02:39.103 INFO:tasks.workunit.client.1.smithi203.stdout:9/276: dwrite f2 [0,4194304] 0 2024-04-05T03:02:39.104 INFO:tasks.workunit.client.1.smithi203.stdout:7/345: write d8/d32/f3a [2065674,128864] 0 2024-04-05T03:02:39.105 INFO:tasks.workunit.client.1.smithi203.stdout:6/348: symlink d0/da/d18/d38/d2a/d2c/l66 0 2024-04-05T03:02:39.108 INFO:tasks.workunit.client.1.smithi203.stdout:5/373: creat d3/d1f/d59/d64/f78 x:0 0 0 2024-04-05T03:02:39.109 INFO:tasks.workunit.client.1.smithi203.stdout:9/277: mkdir d13/d1b/d26/d51 0 2024-04-05T03:02:39.109 INFO:tasks.workunit.client.1.smithi203.stdout:9/278: stat d13/d1b 0 2024-04-05T03:02:39.110 INFO:tasks.workunit.client.1.smithi203.stdout:9/279: fsync f10 0 2024-04-05T03:02:39.111 INFO:tasks.workunit.client.1.smithi203.stdout:8/386: truncate d1/d6/d2e/f30 1239354 0 2024-04-05T03:02:39.115 INFO:tasks.workunit.client.1.smithi203.stdout:9/280: chown d13/d1b/d33/f3f 8 1 2024-04-05T03:02:39.115 INFO:tasks.workunit.client.1.smithi203.stdout:8/387: chown d1/d12/d2f/d1f 103390038 1 2024-04-05T03:02:39.116 INFO:tasks.workunit.client.1.smithi203.stdout:7/346: dwrite d8/d2a/d3f/d52/f45 [4194304,4194304] 0 2024-04-05T03:02:39.117 INFO:tasks.workunit.client.1.smithi203.stdout:3/311: rename d0/dc/d1a/d1f/f4b to d0/f52 0 2024-04-05T03:02:39.118 INFO:tasks.workunit.client.1.smithi203.stdout:3/312: chown d0/dc/d1a/d1d/d43 171366760 1 2024-04-05T03:02:39.119 INFO:tasks.workunit.client.1.smithi203.stdout:5/374: creat d3/d4/dc/f79 x:0 0 0 2024-04-05T03:02:39.120 INFO:tasks.workunit.client.1.smithi203.stdout:9/281: rmdir d13/d1b/d33/d42 39 2024-04-05T03:02:39.120 INFO:tasks.workunit.client.1.smithi203.stdout:3/313: chown d0/dc/d1a/d33 386531 1 2024-04-05T03:02:39.122 INFO:tasks.workunit.client.1.smithi203.stdout:9/282: chown d13/d1b/c50 165840851 1 2024-04-05T03:02:39.127 INFO:tasks.workunit.client.1.smithi203.stdout:8/388: truncate d1/d12/d2f/d1f/f23 3010791 0 2024-04-05T03:02:39.128 INFO:tasks.workunit.client.1.smithi203.stdout:3/314: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:39.129 INFO:tasks.workunit.client.1.smithi203.stdout:6/349: write d0/da/d18/d38/f47 [306954,4089] 0 2024-04-05T03:02:39.129 INFO:tasks.workunit.client.1.smithi203.stdout:9/283: creat d13/d1b/d26/d51/f52 x:0 0 0 2024-04-05T03:02:39.131 INFO:tasks.workunit.client.1.smithi203.stdout:7/347: dwrite d8/da/f1d [0,4194304] 0 2024-04-05T03:02:39.132 INFO:tasks.workunit.client.1.smithi203.stdout:3/315: chown d0/l17 1669421552 1 2024-04-05T03:02:39.133 INFO:tasks.workunit.client.1.smithi203.stdout:8/389: mkdir d1/d12/d17/d47/d53/d69 0 2024-04-05T03:02:39.135 INFO:tasks.workunit.client.1.smithi203.stdout:8/390: dread - d1/d12/d2f/d1f/f50 zero size 2024-04-05T03:02:39.137 INFO:tasks.workunit.client.1.smithi203.stdout:6/350: rename d0/d23 to d0/da/d67 0 2024-04-05T03:02:39.138 INFO:tasks.workunit.client.1.smithi203.stdout:9/284: creat d13/d1b/d26/d51/f53 x:0 0 0 2024-04-05T03:02:39.139 INFO:tasks.workunit.client.1.smithi203.stdout:7/348: mknod d8/d32/d16/d35/c6a 0 2024-04-05T03:02:39.139 INFO:tasks.workunit.client.1.smithi203.stdout:8/391: truncate d1/d12/f31 191559 0 2024-04-05T03:02:39.142 INFO:tasks.workunit.client.1.smithi203.stdout:9/285: rmdir d13/d1b/d33 39 2024-04-05T03:02:39.145 INFO:tasks.workunit.client.1.smithi203.stdout:9/286: stat d13/d1b/d26/d51/f53 0 2024-04-05T03:02:39.148 INFO:tasks.workunit.client.1.smithi203.stdout:8/392: write d1/d12/d17/d10/f40 [218654,55301] 0 2024-04-05T03:02:39.149 INFO:tasks.workunit.client.1.smithi203.stdout:6/351: creat d0/da/d18/d38/f68 x:0 0 0 2024-04-05T03:02:39.149 INFO:tasks.workunit.client.1.smithi203.stdout:7/349: creat d8/d32/d16/d1a/d3d/f6b x:0 0 0 2024-04-05T03:02:39.152 INFO:tasks.workunit.client.1.smithi203.stdout:3/316: dwrite d0/dc/d38/f3b [0,4194304] 0 2024-04-05T03:02:39.152 INFO:tasks.workunit.client.1.smithi203.stdout:6/352: fdatasync d0/da/f43 0 2024-04-05T03:02:39.156 INFO:tasks.workunit.client.1.smithi203.stdout:5/375: dread d3/d4/dc/f6c [0,4194304] 0 2024-04-05T03:02:39.157 INFO:tasks.workunit.client.1.smithi203.stdout:7/350: chown d8/d2a/d3f/d52/f5b 991544 1 2024-04-05T03:02:39.160 INFO:tasks.workunit.client.1.smithi203.stdout:3/317: mkdir d0/dc/d1a/d1d/d53 0 2024-04-05T03:02:39.162 INFO:tasks.workunit.client.1.smithi203.stdout:8/393: link d1/l44 d1/d6/d2e/d5c/l6a 0 2024-04-05T03:02:39.162 INFO:tasks.workunit.client.1.smithi203.stdout:9/287: dwrite d13/d1b/d26/f4c [0,4194304] 0 2024-04-05T03:02:39.162 INFO:tasks.workunit.client.1.smithi203.stdout:5/376: fsync d3/d4/dc/f37 0 2024-04-05T03:02:39.165 INFO:tasks.workunit.client.1.smithi203.stdout:9/288: fdatasync d13/f20 0 2024-04-05T03:02:39.166 INFO:tasks.workunit.client.1.smithi203.stdout:9/289: fsync fb 0 2024-04-05T03:02:39.168 INFO:tasks.workunit.client.1.smithi203.stdout:5/377: stat d3/d4/dc/l48 0 2024-04-05T03:02:39.171 INFO:tasks.workunit.client.1.smithi203.stdout:7/351: rmdir d8/d32/d16 39 2024-04-05T03:02:39.174 INFO:tasks.workunit.client.1.smithi203.stdout:6/353: write d0/f37 [25654,27264] 0 2024-04-05T03:02:39.175 INFO:tasks.workunit.client.1.smithi203.stdout:6/354: read - d0/da/d18/d38/d2a/d2c/f4e zero size 2024-04-05T03:02:39.178 INFO:tasks.workunit.client.1.smithi203.stdout:3/318: getdents d0/dc/d3f 0 2024-04-05T03:02:39.179 INFO:tasks.workunit.client.1.smithi203.stdout:9/290: dread d13/f24 [0,4194304] 0 2024-04-05T03:02:39.179 INFO:tasks.workunit.client.1.smithi203.stdout:7/352: mkdir d8/d6c 0 2024-04-05T03:02:39.180 INFO:tasks.workunit.client.1.smithi203.stdout:8/394: getdents d1/d6/d2e 0 2024-04-05T03:02:39.181 INFO:tasks.workunit.client.1.smithi203.stdout:3/319: creat d0/dc/f54 x:0 0 0 2024-04-05T03:02:39.184 INFO:tasks.workunit.client.1.smithi203.stdout:5/378: getdents d3/d1f/d49 0 2024-04-05T03:02:39.188 INFO:tasks.workunit.client.1.smithi203.stdout:6/355: truncate d0/da/f13 253859 0 2024-04-05T03:02:39.189 INFO:tasks.workunit.client.1.smithi203.stdout:9/291: fsync d13/d1b/d33/d42/f49 0 2024-04-05T03:02:39.190 INFO:tasks.workunit.client.1.smithi203.stdout:8/395: dwrite d1/f63 [0,4194304] 0 2024-04-05T03:02:39.192 INFO:tasks.workunit.client.1.smithi203.stdout:9/292: readlink d13/d1b/l2f 0 2024-04-05T03:02:39.193 INFO:tasks.workunit.client.1.smithi203.stdout:3/320: dread d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:39.194 INFO:tasks.workunit.client.1.smithi203.stdout:9/293: chown d13/d1b/f47 45500456 1 2024-04-05T03:02:39.195 INFO:tasks.workunit.client.1.smithi203.stdout:4/267: sync 2024-04-05T03:02:39.195 INFO:tasks.workunit.client.1.smithi203.stdout:1/266: sync 2024-04-05T03:02:39.195 INFO:tasks.workunit.client.1.smithi203.stdout:0/380: sync 2024-04-05T03:02:39.197 INFO:tasks.workunit.client.1.smithi203.stdout:7/353: dread d8/f31 [0,4194304] 0 2024-04-05T03:02:39.199 INFO:tasks.workunit.client.1.smithi203.stdout:8/396: stat d1/d12/d1a/l60 0 2024-04-05T03:02:39.208 INFO:tasks.workunit.client.1.smithi203.stdout:5/379: dwrite d3/d4/dc/f6c [0,4194304] 0 2024-04-05T03:02:39.210 INFO:tasks.workunit.client.1.smithi203.stdout:9/294: unlink d13/d1b/d33/f34 0 2024-04-05T03:02:39.210 INFO:tasks.workunit.client.1.smithi203.stdout:8/397: write d1/d12/d1a/f1e [633365,126818] 0 2024-04-05T03:02:39.211 INFO:tasks.workunit.client.1.smithi203.stdout:7/354: read d8/fe [109274,1456] 0 2024-04-05T03:02:39.211 INFO:tasks.workunit.client.1.smithi203.stdout:4/268: write da/f13 [742814,22705] 0 2024-04-05T03:02:39.212 INFO:tasks.workunit.client.1.smithi203.stdout:6/356: truncate d0/da/d18/d38/d2a/d2c/f4e 639929 0 2024-04-05T03:02:39.214 INFO:tasks.workunit.client.1.smithi203.stdout:3/321: rename d0/c1 to d0/c55 0 2024-04-05T03:02:39.214 INFO:tasks.workunit.client.1.smithi203.stdout:8/398: write d1/d12/d1a/f42 [1220408,82222] 0 2024-04-05T03:02:39.217 INFO:tasks.workunit.client.1.smithi203.stdout:8/399: chown d1/d12/d2f/d55 94772 1 2024-04-05T03:02:39.221 INFO:tasks.workunit.client.1.smithi203.stdout:7/355: rmdir d8/d2a/d3f/d5e 39 2024-04-05T03:02:39.222 INFO:tasks.workunit.client.1.smithi203.stdout:3/322: chown d0/dc/d1a/d1f/c31 13475584 1 2024-04-05T03:02:39.224 INFO:tasks.workunit.client.1.smithi203.stdout:1/267: dwrite d7/d14/f1e [4194304,4194304] 0 2024-04-05T03:02:39.227 INFO:tasks.workunit.client.1.smithi203.stdout:4/269: truncate f5 849527 0 2024-04-05T03:02:39.227 INFO:tasks.workunit.client.1.smithi203.stdout:8/400: write d1/f63 [799610,42289] 0 2024-04-05T03:02:39.230 INFO:tasks.workunit.client.1.smithi203.stdout:5/380: write d3/d4/dc/f37 [252296,48300] 0 2024-04-05T03:02:39.237 INFO:tasks.workunit.client.1.smithi203.stdout:8/401: readlink d1/d12/l51 0 2024-04-05T03:02:39.237 INFO:tasks.workunit.client.1.smithi203.stdout:9/295: dwrite d13/d1b/d33/d42/f49 [0,4194304] 0 2024-04-05T03:02:39.238 INFO:tasks.workunit.client.1.smithi203.stdout:3/323: mknod d0/dc/d1a/d33/c56 0 2024-04-05T03:02:39.238 INFO:tasks.workunit.client.1.smithi203.stdout:0/381: dread dd/d17/d1d/d26/d34/f35 [0,4194304] 0 2024-04-05T03:02:39.238 INFO:tasks.workunit.client.1.smithi203.stdout:1/268: read d7/d24/f34 [1841342,22552] 0 2024-04-05T03:02:39.241 INFO:tasks.workunit.client.1.smithi203.stdout:8/402: readlink d1/d12/d2f/l4c 0 2024-04-05T03:02:39.244 INFO:tasks.workunit.client.1.smithi203.stdout:6/357: dwrite d0/da/d54/f45 [0,4194304] 0 2024-04-05T03:02:39.247 INFO:tasks.workunit.client.1.smithi203.stdout:4/270: dread da/f21 [0,4194304] 0 2024-04-05T03:02:39.250 INFO:tasks.workunit.client.1.smithi203.stdout:3/324: write d0/dc/f16 [1387434,97600] 0 2024-04-05T03:02:39.251 INFO:tasks.workunit.client.1.smithi203.stdout:5/381: getdents d3/d1f 0 2024-04-05T03:02:39.252 INFO:tasks.workunit.client.1.smithi203.stdout:3/325: read - d0/dc/f54 zero size 2024-04-05T03:02:39.253 INFO:tasks.workunit.client.1.smithi203.stdout:9/296: dwrite f2 [4194304,4194304] 0 2024-04-05T03:02:39.254 INFO:tasks.workunit.client.1.smithi203.stdout:1/269: truncate d7/d24/d33/f3f 508279 0 2024-04-05T03:02:39.254 INFO:tasks.workunit.client.1.smithi203.stdout:3/326: chown d0/dc/d1a/d33/d35/f3d 43684403 1 2024-04-05T03:02:39.261 INFO:tasks.workunit.client.1.smithi203.stdout:3/327: rename d0/f9 to d0/dc/d1a/f57 0 2024-04-05T03:02:39.264 INFO:tasks.workunit.client.1.smithi203.stdout:8/403: getdents d1/d12/d1a 0 2024-04-05T03:02:39.264 INFO:tasks.workunit.client.1.smithi203.stdout:9/297: write d13/d1b/d26/d31/f3e [512497,45105] 0 2024-04-05T03:02:39.266 INFO:tasks.workunit.client.1.smithi203.stdout:5/382: creat d3/f7a x:0 0 0 2024-04-05T03:02:39.269 INFO:tasks.workunit.client.1.smithi203.stdout:1/270: creat d7/d24/d33/f56 x:0 0 0 2024-04-05T03:02:39.270 INFO:tasks.workunit.client.1.smithi203.stdout:9/298: rename fa to d13/d1b/d26/f54 0 2024-04-05T03:02:39.276 INFO:tasks.workunit.client.1.smithi203.stdout:1/271: write d7/f3d [894691,85482] 0 2024-04-05T03:02:39.281 INFO:tasks.workunit.client.1.smithi203.stdout:9/299: rmdir d13/d1b/d26/d31/d38 39 2024-04-05T03:02:39.281 INFO:tasks.workunit.client.1.smithi203.stdout:8/404: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:39.282 INFO:tasks.workunit.client.1.smithi203.stdout:5/383: dwrite d3/d1f/d59/d64/f78 [0,4194304] 0 2024-04-05T03:02:39.287 INFO:tasks.workunit.client.1.smithi203.stdout:1/272: rename l5 to d7/d14/l57 0 2024-04-05T03:02:39.292 INFO:tasks.workunit.client.1.smithi203.stdout:4/271: dwrite da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:39.293 INFO:tasks.workunit.client.1.smithi203.stdout:6/358: dread d0/da/d67/f28 [0,4194304] 0 2024-04-05T03:02:39.295 INFO:tasks.workunit.client.1.smithi203.stdout:3/328: dread d0/dc/d1a/d1f/f32 [0,4194304] 0 2024-04-05T03:02:39.296 INFO:tasks.workunit.client.1.smithi203.stdout:8/405: creat d1/d6/d2e/f6b x:0 0 0 2024-04-05T03:02:39.298 INFO:tasks.workunit.client.1.smithi203.stdout:9/300: rename d13/l18 to d13/d1b/d33/d42/l55 0 2024-04-05T03:02:39.300 INFO:tasks.workunit.client.1.smithi203.stdout:1/273: dread f2 [4194304,4194304] 0 2024-04-05T03:02:39.301 INFO:tasks.workunit.client.1.smithi203.stdout:9/301: dread - d13/d1b/d26/d51/f53 zero size 2024-04-05T03:02:39.301 INFO:tasks.workunit.client.1.smithi203.stdout:4/272: getdents da/d29/d2e/d53 0 2024-04-05T03:02:39.302 INFO:tasks.workunit.client.1.smithi203.stdout:5/384: dwrite d3/f7 [8388608,4194304] 0 2024-04-05T03:02:39.303 INFO:tasks.workunit.client.1.smithi203.stdout:3/329: creat d0/dc/d3f/f58 x:0 0 0 2024-04-05T03:02:39.306 INFO:tasks.workunit.client.1.smithi203.stdout:8/406: creat d1/d12/f6c x:0 0 0 2024-04-05T03:02:39.310 INFO:tasks.workunit.client.1.smithi203.stdout:1/274: dread d7/f13 [0,4194304] 0 2024-04-05T03:02:39.310 INFO:tasks.workunit.client.1.smithi203.stdout:3/330: creat d0/dc/d3f/f59 x:0 0 0 2024-04-05T03:02:39.311 INFO:tasks.workunit.client.1.smithi203.stdout:5/385: mknod d3/d4/c7b 0 2024-04-05T03:02:39.313 INFO:tasks.workunit.client.1.smithi203.stdout:5/386: write d3/f7a [880000,109100] 0 2024-04-05T03:02:39.313 INFO:tasks.workunit.client.1.smithi203.stdout:6/359: rename d0/da/c64 to d0/da/d18/d38/d2a/c69 0 2024-04-05T03:02:39.314 INFO:tasks.workunit.client.1.smithi203.stdout:3/331: write d0/dc/d1a/d1d/d43/f51 [380677,15326] 0 2024-04-05T03:02:39.317 INFO:tasks.workunit.client.1.smithi203.stdout:6/360: write d0/da/f10 [5096843,46394] 0 2024-04-05T03:02:39.317 INFO:tasks.workunit.client.1.smithi203.stdout:8/407: dwrite d1/d12/f6c [0,4194304] 0 2024-04-05T03:02:39.318 INFO:tasks.workunit.client.1.smithi203.stdout:5/387: symlink d3/d4/l7c 0 2024-04-05T03:02:39.321 INFO:tasks.workunit.client.1.smithi203.stdout:6/361: creat d0/da/d54/d53/f6a x:0 0 0 2024-04-05T03:02:39.321 INFO:tasks.workunit.client.1.smithi203.stdout:5/388: write d3/d1f/d59/d64/f3d [3854887,91282] 0 2024-04-05T03:02:39.323 INFO:tasks.workunit.client.1.smithi203.stdout:5/389: fdatasync d3/d1f/f3b 0 2024-04-05T03:02:39.324 INFO:tasks.workunit.client.1.smithi203.stdout:5/390: chown d3/d1f/f3b 776142 1 2024-04-05T03:02:39.328 INFO:tasks.workunit.client.1.smithi203.stdout:3/332: mkdir d0/dc/d1a/d1d/d53/d5a 0 2024-04-05T03:02:39.329 INFO:tasks.workunit.client.1.smithi203.stdout:3/333: write d0/d21/f45 [4605460,26993] 0 2024-04-05T03:02:39.330 INFO:tasks.workunit.client.1.smithi203.stdout:3/334: chown d0/dc/d1a/d1d/d53/d5a 627823 1 2024-04-05T03:02:39.331 INFO:tasks.workunit.client.1.smithi203.stdout:3/335: stat d0/dc/d1a/d33/d35/f3c 0 2024-04-05T03:02:39.335 INFO:tasks.workunit.client.1.smithi203.stdout:3/336: creat d0/dc/d3f/f5b x:0 0 0 2024-04-05T03:02:39.336 INFO:tasks.workunit.client.1.smithi203.stdout:1/275: truncate d7/f27 8088544 0 2024-04-05T03:02:39.337 INFO:tasks.workunit.client.1.smithi203.stdout:5/391: symlink d3/d4/l7d 0 2024-04-05T03:02:39.337 INFO:tasks.workunit.client.1.smithi203.stdout:3/337: write d0/dc/f54 [825985,56077] 0 2024-04-05T03:02:39.338 INFO:tasks.workunit.client.1.smithi203.stdout:1/276: mkdir d7/d14/d2b/d58 0 2024-04-05T03:02:39.339 INFO:tasks.workunit.client.1.smithi203.stdout:3/338: creat d0/dc/d1a/d1d/f5c x:0 0 0 2024-04-05T03:02:39.340 INFO:tasks.workunit.client.1.smithi203.stdout:3/339: stat d0/dc/d1a/d1f/f32 0 2024-04-05T03:02:39.341 INFO:tasks.workunit.client.1.smithi203.stdout:5/392: rename d3/d1f/d59/d64/f26 to d3/d4/dc/d67/f7e 0 2024-04-05T03:02:39.341 INFO:tasks.workunit.client.1.smithi203.stdout:3/340: fsync d0/dc/d38/f3b 0 2024-04-05T03:02:39.341 INFO:tasks.workunit.client.1.smithi203.stdout:3/341: readlink d0/l1b 0 2024-04-05T03:02:39.341 INFO:tasks.workunit.client.1.smithi203.stdout:1/277: fdatasync d7/fe 0 2024-04-05T03:02:39.346 INFO:tasks.workunit.client.1.smithi203.stdout:5/393: link d3/d1f/f1e d3/d4/dc/d67/f7f 0 2024-04-05T03:02:39.353 INFO:tasks.workunit.client.1.smithi203.stdout:1/278: dwrite d7/d14/d2b/f49 [0,4194304] 0 2024-04-05T03:02:39.354 INFO:tasks.workunit.client.1.smithi203.stdout:8/408: dwrite d1/d12/d17/d10/f49 [0,4194304] 0 2024-04-05T03:02:39.354 INFO:tasks.workunit.client.1.smithi203.stdout:2/477: sync 2024-04-05T03:02:39.355 INFO:tasks.workunit.client.1.smithi203.stdout:5/394: symlink d3/d1f/d59/d64/l80 0 2024-04-05T03:02:39.357 INFO:tasks.workunit.client.1.smithi203.stdout:6/362: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:39.359 INFO:tasks.workunit.client.1.smithi203.stdout:5/395: stat d3/d1f/d59/d64/d12 0 2024-04-05T03:02:39.361 INFO:tasks.workunit.client.1.smithi203.stdout:8/409: mkdir d1/d12/d1a/d6d 0 2024-04-05T03:02:39.362 INFO:tasks.workunit.client.1.smithi203.stdout:2/478: write d5/d10/d35/d2d/f38 [584167,80692] 0 2024-04-05T03:02:39.364 INFO:tasks.workunit.client.1.smithi203.stdout:1/279: rename d7/d24/d33/d48/d50 to d7/d24/d33/d48/d59 0 2024-04-05T03:02:39.365 INFO:tasks.workunit.client.1.smithi203.stdout:5/396: chown d3/d4/dc/c2a 13407023 1 2024-04-05T03:02:39.367 INFO:tasks.workunit.client.1.smithi203.stdout:8/410: creat d1/d12/d17/d10/f6e x:0 0 0 2024-04-05T03:02:39.368 INFO:tasks.workunit.client.1.smithi203.stdout:5/397: fsync d3/d4/dc/d67/f7e 0 2024-04-05T03:02:39.368 INFO:tasks.workunit.client.1.smithi203.stdout:1/280: chown d7/f2e 906476392 1 2024-04-05T03:02:39.370 INFO:tasks.workunit.client.1.smithi203.stdout:8/411: symlink d1/d4b/d5f/l6f 0 2024-04-05T03:02:39.450 INFO:tasks.workunit.client.1.smithi203.stdout:6/363: dwrite d0/da/d54/d4c/f5b [0,4194304] 0 2024-04-05T03:02:39.457 INFO:tasks.workunit.client.1.smithi203.stdout:8/412: unlink d1/d12/d17/d47/d53/l59 0 2024-04-05T03:02:39.458 INFO:tasks.workunit.client.1.smithi203.stdout:1/281: dwrite d7/d14/f52 [0,4194304] 0 2024-04-05T03:02:39.460 INFO:tasks.workunit.client.1.smithi203.stdout:6/364: dwrite d0/da/d54/d53/f63 [0,4194304] 0 2024-04-05T03:02:39.466 INFO:tasks.workunit.client.1.smithi203.stdout:5/398: link d3/d4/dc/l2b d3/l81 0 2024-04-05T03:02:39.471 INFO:tasks.workunit.client.1.smithi203.stdout:8/413: dwrite d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:39.482 INFO:tasks.workunit.client.1.smithi203.stdout:8/414: dwrite d1/d12/d2f/f61 [0,4194304] 0 2024-04-05T03:02:39.483 INFO:tasks.workunit.client.1.smithi203.stdout:1/282: dread d7/fe [0,4194304] 0 2024-04-05T03:02:39.487 INFO:tasks.workunit.client.1.smithi203.stdout:8/415: chown d1/d12/f31 65309 1 2024-04-05T03:02:39.487 INFO:tasks.workunit.client.1.smithi203.stdout:1/283: chown d7/d24/l32 525 1 2024-04-05T03:02:39.488 INFO:tasks.workunit.client.1.smithi203.stdout:1/284: dread - d7/f4f zero size 2024-04-05T03:02:39.490 INFO:tasks.workunit.client.1.smithi203.stdout:1/285: fsync f2 0 2024-04-05T03:02:39.491 INFO:tasks.workunit.client.1.smithi203.stdout:8/416: mknod d1/d4b/d68/c70 0 2024-04-05T03:02:39.493 INFO:tasks.workunit.client.1.smithi203.stdout:1/286: chown d7/fa 1298825 1 2024-04-05T03:02:39.494 INFO:tasks.workunit.client.1.smithi203.stdout:6/365: dread d0/da/d18/d38/d2b/f5a [0,4194304] 0 2024-04-05T03:02:39.495 INFO:tasks.workunit.client.1.smithi203.stdout:8/417: symlink d1/d12/d2f/d55/l71 0 2024-04-05T03:02:39.496 INFO:tasks.workunit.client.1.smithi203.stdout:1/287: rename d7/f2e to d7/d14/d2b/f5a 0 2024-04-05T03:02:39.498 INFO:tasks.workunit.client.1.smithi203.stdout:1/288: write d7/d24/d33/f45 [591715,34896] 0 2024-04-05T03:02:39.499 INFO:tasks.workunit.client.1.smithi203.stdout:6/366: mknod d0/c6b 0 2024-04-05T03:02:39.502 INFO:tasks.workunit.client.1.smithi203.stdout:5/399: dread d3/d1f/d59/d64/f63 [0,4194304] 0 2024-04-05T03:02:39.504 INFO:tasks.workunit.client.1.smithi203.stdout:6/367: read d0/da/d18/d38/d2a/d2c/f4f [2361831,66571] 0 2024-04-05T03:02:39.509 INFO:tasks.workunit.client.1.smithi203.stdout:1/289: rename f2 to d7/d24/d33/f5b 0 2024-04-05T03:02:39.513 INFO:tasks.workunit.client.1.smithi203.stdout:8/418: link d1/c13 d1/d12/d2f/d5b/c72 0 2024-04-05T03:02:39.515 INFO:tasks.workunit.client.1.smithi203.stdout:5/400: creat d3/f82 x:0 0 0 2024-04-05T03:02:39.517 INFO:tasks.workunit.client.1.smithi203.stdout:1/290: creat d7/d24/d33/d48/d4b/f5c x:0 0 0 2024-04-05T03:02:39.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/368: symlink d0/da/d18/d38/d2a/l6c 0 2024-04-05T03:02:39.523 INFO:tasks.workunit.client.1.smithi203.stdout:8/419: rename d1/ld to d1/d12/d17/d10/l73 0 2024-04-05T03:02:39.525 INFO:tasks.workunit.client.1.smithi203.stdout:1/291: symlink d7/d24/d33/l5d 0 2024-04-05T03:02:39.530 INFO:tasks.workunit.client.1.smithi203.stdout:1/292: chown d7/c43 67 1 2024-04-05T03:02:39.530 INFO:tasks.workunit.client.1.smithi203.stdout:5/401: getdents d3/d5f 0 2024-04-05T03:02:39.531 INFO:tasks.workunit.client.1.smithi203.stdout:8/420: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:39.534 INFO:tasks.workunit.client.1.smithi203.stdout:1/293: mknod d7/c5e 0 2024-04-05T03:02:39.537 INFO:tasks.workunit.client.1.smithi203.stdout:5/402: mknod d3/d1f/d59/d64/d12/c83 0 2024-04-05T03:02:39.540 INFO:tasks.workunit.client.1.smithi203.stdout:1/294: dread d7/d24/d33/f45 [0,4194304] 0 2024-04-05T03:02:39.543 INFO:tasks.workunit.client.1.smithi203.stdout:8/421: dread d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:39.544 INFO:tasks.workunit.client.1.smithi203.stdout:1/295: mknod d7/d14/c5f 0 2024-04-05T03:02:39.548 INFO:tasks.workunit.client.1.smithi203.stdout:5/403: dwrite f2 [4194304,4194304] 0 2024-04-05T03:02:39.549 INFO:tasks.workunit.client.1.smithi203.stdout:1/296: write d7/d24/d33/f45 [361533,125782] 0 2024-04-05T03:02:39.550 INFO:tasks.workunit.client.1.smithi203.stdout:6/369: dwrite d0/da/f13 [0,4194304] 0 2024-04-05T03:02:39.560 INFO:tasks.workunit.client.1.smithi203.stdout:5/404: creat d3/d1f/d49/f84 x:0 0 0 2024-04-05T03:02:39.560 INFO:tasks.workunit.client.1.smithi203.stdout:6/370: chown d0/c6b 1 1 2024-04-05T03:02:39.560 INFO:tasks.workunit.client.1.smithi203.stdout:8/422: dwrite d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:39.563 INFO:tasks.workunit.client.1.smithi203.stdout:8/423: chown d1/d12/d1a/c27 145678 1 2024-04-05T03:02:39.563 INFO:tasks.workunit.client.1.smithi203.stdout:1/297: creat d7/d14/d2b/d58/f60 x:0 0 0 2024-04-05T03:02:39.566 INFO:tasks.workunit.client.1.smithi203.stdout:1/298: fsync d7/d24/d33/f56 0 2024-04-05T03:02:39.570 INFO:tasks.workunit.client.1.smithi203.stdout:8/424: dread d1/d12/d2f/f61 [0,4194304] 0 2024-04-05T03:02:39.571 INFO:tasks.workunit.client.1.smithi203.stdout:6/371: creat d0/da/d18/d38/f6d x:0 0 0 2024-04-05T03:02:39.572 INFO:tasks.workunit.client.1.smithi203.stdout:8/425: write d1/f63 [3989594,128253] 0 2024-04-05T03:02:39.574 INFO:tasks.workunit.client.1.smithi203.stdout:1/299: dwrite d7/fe [4194304,4194304] 0 2024-04-05T03:02:39.576 INFO:tasks.workunit.client.1.smithi203.stdout:1/300: chown d7/c5e 251465109 1 2024-04-05T03:02:39.578 INFO:tasks.workunit.client.1.smithi203.stdout:1/301: mknod d7/d24/c61 0 2024-04-05T03:02:39.581 INFO:tasks.workunit.client.1.smithi203.stdout:8/426: dwrite d1/d6/f41 [0,4194304] 0 2024-04-05T03:02:39.582 INFO:tasks.workunit.client.1.smithi203.stdout:1/302: unlink d7/d24/c28 0 2024-04-05T03:02:39.582 INFO:tasks.workunit.client.1.smithi203.stdout:7/356: sync 2024-04-05T03:02:39.582 INFO:tasks.workunit.client.1.smithi203.stdout:4/273: sync 2024-04-05T03:02:39.583 INFO:tasks.workunit.client.1.smithi203.stdout:0/382: sync 2024-04-05T03:02:39.583 INFO:tasks.workunit.client.1.smithi203.stdout:8/427: write d1/d12/d2f/f2a [1241114,72338] 0 2024-04-05T03:02:39.585 INFO:tasks.workunit.client.1.smithi203.stdout:7/357: write d8/da/f50 [3974205,66165] 0 2024-04-05T03:02:39.586 INFO:tasks.workunit.client.1.smithi203.stdout:1/303: write d7/d24/f35 [5336046,107182] 0 2024-04-05T03:02:39.587 INFO:tasks.workunit.client.1.smithi203.stdout:5/405: write d3/d1f/d59/d64/f4f [50435,40210] 0 2024-04-05T03:02:39.589 INFO:tasks.workunit.client.1.smithi203.stdout:4/274: symlink da/d3c/d4c/l55 0 2024-04-05T03:02:39.591 INFO:tasks.workunit.client.1.smithi203.stdout:7/358: chown d8/d32/d5d 115659620 1 2024-04-05T03:02:39.592 INFO:tasks.workunit.client.1.smithi203.stdout:5/406: read - d3/f82 zero size 2024-04-05T03:02:39.597 INFO:tasks.workunit.client.1.smithi203.stdout:1/304: rmdir d7/d24/d33 39 2024-04-05T03:02:39.599 INFO:tasks.workunit.client.1.smithi203.stdout:6/372: fsync d0/da/d18/d38/d2a/d2c/f4e 0 2024-04-05T03:02:39.601 INFO:tasks.workunit.client.1.smithi203.stdout:0/383: dwrite dd/f16 [0,4194304] 0 2024-04-05T03:02:39.602 INFO:tasks.workunit.client.1.smithi203.stdout:6/373: chown d0/da/d18/l31 760653 1 2024-04-05T03:02:39.603 INFO:tasks.workunit.client.1.smithi203.stdout:1/305: chown d7/d24/d33/d48/d4b/d51 63 1 2024-04-05T03:02:39.603 INFO:tasks.workunit.client.1.smithi203.stdout:8/428: dwrite d1/d6/d2e/f6b [0,4194304] 0 2024-04-05T03:02:39.605 INFO:tasks.workunit.client.1.smithi203.stdout:8/429: fsync d1/f34 0 2024-04-05T03:02:39.605 INFO:tasks.workunit.client.1.smithi203.stdout:8/430: stat d1/c66 0 2024-04-05T03:02:39.606 INFO:tasks.workunit.client.1.smithi203.stdout:6/374: chown d0/da/d18/d38/d2b/c5c 2 1 2024-04-05T03:02:39.609 INFO:tasks.workunit.client.1.smithi203.stdout:6/375: chown d0/da/d54/d53/f63 172 1 2024-04-05T03:02:39.610 INFO:tasks.workunit.client.1.smithi203.stdout:7/359: rename d8/d32/d16/d1a/d55 to d8/d32/d5d/d6d 0 2024-04-05T03:02:39.611 INFO:tasks.workunit.client.1.smithi203.stdout:6/376: chown d0/da/d18/d38/d2a/d2c/f4e 6398643 1 2024-04-05T03:02:39.611 INFO:tasks.workunit.client.1.smithi203.stdout:3/342: sync 2024-04-05T03:02:39.611 INFO:tasks.workunit.client.1.smithi203.stdout:9/302: sync 2024-04-05T03:02:39.611 INFO:tasks.workunit.client.1.smithi203.stdout:2/479: sync 2024-04-05T03:02:39.612 INFO:tasks.workunit.client.1.smithi203.stdout:0/384: mknod dd/d17/d1d/d22/d53/d55/c6e 0 2024-04-05T03:02:39.613 INFO:tasks.workunit.client.1.smithi203.stdout:1/306: symlink d7/d24/d33/d48/l62 0 2024-04-05T03:02:39.615 INFO:tasks.workunit.client.1.smithi203.stdout:6/377: chown d0/da/d18/d38/d2b/c62 25910 1 2024-04-05T03:02:39.616 INFO:tasks.workunit.client.1.smithi203.stdout:9/303: symlink d13/l56 0 2024-04-05T03:02:39.617 INFO:tasks.workunit.client.1.smithi203.stdout:8/431: dwrite d1/f63 [0,4194304] 0 2024-04-05T03:02:39.617 INFO:tasks.workunit.client.1.smithi203.stdout:4/275: rmdir da/d3c 39 2024-04-05T03:02:39.618 INFO:tasks.workunit.client.1.smithi203.stdout:2/480: truncate d5/d10/d1a/d57/f6b 1770276 0 2024-04-05T03:02:39.618 INFO:tasks.workunit.client.1.smithi203.stdout:3/343: unlink d0/dc/d1a/d1d/l2a 0 2024-04-05T03:02:39.620 INFO:tasks.workunit.client.1.smithi203.stdout:0/385: symlink dd/d17/l6f 0 2024-04-05T03:02:39.625 INFO:tasks.workunit.client.1.smithi203.stdout:1/307: mknod d7/d24/d33/c63 0 2024-04-05T03:02:39.632 INFO:tasks.workunit.client.1.smithi203.stdout:2/481: dwrite d5/d10/d35/d2d/d7c/d7a/f9a [0,4194304] 0 2024-04-05T03:02:39.635 INFO:tasks.workunit.client.1.smithi203.stdout:7/360: getdents d8/d32/d16/d1a/d5a 0 2024-04-05T03:02:39.637 INFO:tasks.workunit.client.1.smithi203.stdout:6/378: write d0/da/fd [1659375,44588] 0 2024-04-05T03:02:39.638 INFO:tasks.workunit.client.1.smithi203.stdout:5/407: dread d3/d4/dc/f54 [0,4194304] 0 2024-04-05T03:02:39.640 INFO:tasks.workunit.client.1.smithi203.stdout:6/379: chown d0/da/d18/d38/d2a/d2c/f4e 1 1 2024-04-05T03:02:39.641 INFO:tasks.workunit.client.1.smithi203.stdout:9/304: rename d13/d1b/d33/d42/l46 to d13/d1b/l57 0 2024-04-05T03:02:39.643 INFO:tasks.workunit.client.1.smithi203.stdout:7/361: fdatasync d8/d2a/f42 0 2024-04-05T03:02:39.645 INFO:tasks.workunit.client.1.smithi203.stdout:4/276: creat da/f56 x:0 0 0 2024-04-05T03:02:39.648 INFO:tasks.workunit.client.1.smithi203.stdout:1/308: creat d7/d24/d33/f64 x:0 0 0 2024-04-05T03:02:39.649 INFO:tasks.workunit.client.1.smithi203.stdout:8/432: dread d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:39.651 INFO:tasks.workunit.client.1.smithi203.stdout:1/309: fsync d7/d14/f16 0 2024-04-05T03:02:39.651 INFO:tasks.workunit.client.1.smithi203.stdout:3/344: truncate d0/dc/f14 3600795 0 2024-04-05T03:02:39.651 INFO:tasks.workunit.client.1.smithi203.stdout:9/305: dwrite d13/d1b/d33/d42/f49 [0,4194304] 0 2024-04-05T03:02:39.651 INFO:tasks.workunit.client.1.smithi203.stdout:6/380: symlink d0/da/d18/l6e 0 2024-04-05T03:02:39.653 INFO:tasks.workunit.client.1.smithi203.stdout:8/433: write d1/d12/d1a/f42 [4945910,103546] 0 2024-04-05T03:02:39.655 INFO:tasks.workunit.client.1.smithi203.stdout:1/310: write d7/fe [2598442,37009] 0 2024-04-05T03:02:39.657 INFO:tasks.workunit.client.1.smithi203.stdout:8/434: readlink d1/d6/d2e/l4f 0 2024-04-05T03:02:39.658 INFO:tasks.workunit.client.1.smithi203.stdout:5/408: dwrite d3/f32 [0,4194304] 0 2024-04-05T03:02:39.658 INFO:tasks.workunit.client.1.smithi203.stdout:1/311: dread - d7/d24/f3e zero size 2024-04-05T03:02:39.660 INFO:tasks.workunit.client.1.smithi203.stdout:3/345: mkdir d0/dc/d38/d5d 0 2024-04-05T03:02:39.661 INFO:tasks.workunit.client.1.smithi203.stdout:7/362: mknod d8/d32/d16/d1a/d3d/d59/c6e 0 2024-04-05T03:02:39.662 INFO:tasks.workunit.client.1.smithi203.stdout:5/409: read d3/f7a [473458,53506] 0 2024-04-05T03:02:39.663 INFO:tasks.workunit.client.1.smithi203.stdout:9/306: creat d13/d1b/d33/d42/f58 x:0 0 0 2024-04-05T03:02:39.665 INFO:tasks.workunit.client.1.smithi203.stdout:9/307: write d13/d1b/d33/f3f [3471207,7860] 0 2024-04-05T03:02:39.666 INFO:tasks.workunit.client.1.smithi203.stdout:2/482: dwrite d5/d10/d35/d36/f6c [4194304,4194304] 0 2024-04-05T03:02:39.668 INFO:tasks.workunit.client.1.smithi203.stdout:4/277: mkdir da/d29/d2e/d57 0 2024-04-05T03:02:39.668 INFO:tasks.workunit.client.1.smithi203.stdout:1/312: symlink d7/d14/l65 0 2024-04-05T03:02:39.713 INFO:tasks.workunit.client.1.smithi203.stdout:6/381: symlink d0/da/d54/l6f 0 2024-04-05T03:02:39.713 INFO:tasks.workunit.client.1.smithi203.stdout:5/410: rename d3/d1f/f1e to d3/d4/dc/f85 0 2024-04-05T03:02:39.714 INFO:tasks.workunit.client.1.smithi203.stdout:3/346: creat d0/dc/d1a/d1d/d43/f5e x:0 0 0 2024-04-05T03:02:39.715 INFO:tasks.workunit.client.1.smithi203.stdout:1/313: truncate d7/d24/f35 5775226 0 2024-04-05T03:02:39.716 INFO:tasks.workunit.client.1.smithi203.stdout:2/483: creat d5/d10/d35/d2d/d7c/d72/d8f/fa5 x:0 0 0 2024-04-05T03:02:39.719 INFO:tasks.workunit.client.1.smithi203.stdout:3/347: write d0/dc/d1a/d33/d35/f3d [3019533,2785] 0 2024-04-05T03:02:39.722 INFO:tasks.workunit.client.1.smithi203.stdout:9/308: mkdir d13/d1b/d26/d31/d38/d59 0 2024-04-05T03:02:39.723 INFO:tasks.workunit.client.1.smithi203.stdout:4/278: dwrite da/d29/f42 [0,4194304] 0 2024-04-05T03:02:39.723 INFO:tasks.workunit.client.1.smithi203.stdout:1/314: write d7/d14/f29 [283647,107666] 0 2024-04-05T03:02:39.725 INFO:tasks.workunit.client.1.smithi203.stdout:8/435: getdents d1 0 2024-04-05T03:02:39.729 INFO:tasks.workunit.client.1.smithi203.stdout:3/348: creat d0/dc/d38/f5f x:0 0 0 2024-04-05T03:02:39.729 INFO:tasks.workunit.client.1.smithi203.stdout:9/309: fsync f7 0 2024-04-05T03:02:39.732 INFO:tasks.workunit.client.1.smithi203.stdout:4/279: mknod da/d29/d2e/c58 0 2024-04-05T03:02:39.733 INFO:tasks.workunit.client.1.smithi203.stdout:1/315: unlink d7/d14/l65 0 2024-04-05T03:02:39.734 INFO:tasks.workunit.client.1.smithi203.stdout:2/484: creat d5/d10/d1a/fa6 x:0 0 0 2024-04-05T03:02:39.735 INFO:tasks.workunit.client.1.smithi203.stdout:8/436: mkdir d1/d4b/d5f/d74 0 2024-04-05T03:02:39.737 INFO:tasks.workunit.client.1.smithi203.stdout:1/316: fsync d7/f4f 0 2024-04-05T03:02:39.740 INFO:tasks.workunit.client.1.smithi203.stdout:2/485: creat d5/d10/d1a/d24/fa7 x:0 0 0 2024-04-05T03:02:39.742 INFO:tasks.workunit.client.1.smithi203.stdout:1/317: rmdir d7/d14/d2b 39 2024-04-05T03:02:39.742 INFO:tasks.workunit.client.1.smithi203.stdout:5/411: dwrite d3/d1f/d59/d64/f78 [0,4194304] 0 2024-04-05T03:02:39.742 INFO:tasks.workunit.client.1.smithi203.stdout:8/437: chown d1/l44 39216 1 2024-04-05T03:02:39.743 INFO:tasks.workunit.client.1.smithi203.stdout:3/349: dwrite d0/dc/d1a/d1d/d43/f51 [0,4194304] 0 2024-04-05T03:02:39.743 INFO:tasks.workunit.client.1.smithi203.stdout:2/486: readlink d5/d10/d35/d36/l55 0 2024-04-05T03:02:39.745 INFO:tasks.workunit.client.1.smithi203.stdout:4/280: rename da/f21 to da/d29/d2e/f59 0 2024-04-05T03:02:39.748 INFO:tasks.workunit.client.1.smithi203.stdout:8/438: mkdir d1/d4b/d68/d75 0 2024-04-05T03:02:39.751 INFO:tasks.workunit.client.1.smithi203.stdout:9/310: link d13/c36 d13/d1b/d33/c5a 0 2024-04-05T03:02:39.752 INFO:tasks.workunit.client.1.smithi203.stdout:1/318: readlink d7/d14/d2b/l4e 0 2024-04-05T03:02:39.752 INFO:tasks.workunit.client.1.smithi203.stdout:1/319: chown d7 1327 1 2024-04-05T03:02:39.753 INFO:tasks.workunit.client.1.smithi203.stdout:8/439: unlink d1/d12/d17/c3d 0 2024-04-05T03:02:39.758 INFO:tasks.workunit.client.1.smithi203.stdout:2/487: dwrite d5/d66/d7d/f94 [0,4194304] 0 2024-04-05T03:02:39.761 INFO:tasks.workunit.client.1.smithi203.stdout:8/440: mknod d1/d12/d2f/d1f/c76 0 2024-04-05T03:02:39.765 INFO:tasks.workunit.client.1.smithi203.stdout:5/412: dwrite d3/d1f/f3b [4194304,4194304] 0 2024-04-05T03:02:39.766 INFO:tasks.workunit.client.1.smithi203.stdout:9/311: dwrite fd [0,4194304] 0 2024-04-05T03:02:39.768 INFO:tasks.workunit.client.1.smithi203.stdout:2/488: unlink d5/d10/d35/d2d/d7c/d72/d8f/ca4 0 2024-04-05T03:02:39.768 INFO:tasks.workunit.client.1.smithi203.stdout:1/320: dread d7/d24/d33/f5b [4194304,4194304] 0 2024-04-05T03:02:39.769 INFO:tasks.workunit.client.1.smithi203.stdout:4/281: creat da/f5a x:0 0 0 2024-04-05T03:02:39.771 INFO:tasks.workunit.client.1.smithi203.stdout:9/312: mknod d13/d1b/d26/d51/c5b 0 2024-04-05T03:02:39.773 INFO:tasks.workunit.client.1.smithi203.stdout:9/313: chown d13/f20 2884 1 2024-04-05T03:02:39.774 INFO:tasks.workunit.client.1.smithi203.stdout:3/350: truncate d0/f42 1332582 0 2024-04-05T03:02:39.776 INFO:tasks.workunit.client.1.smithi203.stdout:8/441: dwrite d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:39.776 INFO:tasks.workunit.client.1.smithi203.stdout:3/351: write d0/dc/d38/f5f [821664,34404] 0 2024-04-05T03:02:39.782 INFO:tasks.workunit.client.1.smithi203.stdout:8/442: truncate d1/d12/d1a/f2c 2292458 0 2024-04-05T03:02:39.783 INFO:tasks.workunit.client.1.smithi203.stdout:5/413: rename d3/d1f/d59/d64/d12/l51 to d3/d1f/d59/l86 0 2024-04-05T03:02:39.783 INFO:tasks.workunit.client.1.smithi203.stdout:9/314: mkdir d13/d5c 0 2024-04-05T03:02:39.787 INFO:tasks.workunit.client.1.smithi203.stdout:5/414: readlink d3/l81 0 2024-04-05T03:02:39.788 INFO:tasks.workunit.client.1.smithi203.stdout:4/282: dwrite da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:39.790 INFO:tasks.workunit.client.1.smithi203.stdout:1/321: dwrite d7/d24/d33/f5b [0,4194304] 0 2024-04-05T03:02:39.791 INFO:tasks.workunit.client.1.smithi203.stdout:4/283: readlink da/d3c/d4c/l4f 0 2024-04-05T03:02:39.796 INFO:tasks.workunit.client.1.smithi203.stdout:2/489: rename d5/f43 to d5/d10/d1a/d57/d70/fa8 0 2024-04-05T03:02:39.797 INFO:tasks.workunit.client.1.smithi203.stdout:3/352: link d0/dc/d1a/d1f/l2e d0/dc/d1a/d33/l60 0 2024-04-05T03:02:39.798 INFO:tasks.workunit.client.1.smithi203.stdout:3/353: chown d0/dc/d1a/d33 1153 1 2024-04-05T03:02:39.799 INFO:tasks.workunit.client.1.smithi203.stdout:1/322: readlink d7/d24/l4c 0 2024-04-05T03:02:39.800 INFO:tasks.workunit.client.1.smithi203.stdout:2/490: readlink d5/d63/d86/la1 0 2024-04-05T03:02:39.800 INFO:tasks.workunit.client.1.smithi203.stdout:9/315: link d13/l56 d13/d1b/d26/d31/d38/d59/l5d 0 2024-04-05T03:02:39.801 INFO:tasks.workunit.client.1.smithi203.stdout:8/443: dread d1/d12/d2f/f61 [0,4194304] 0 2024-04-05T03:02:39.803 INFO:tasks.workunit.client.1.smithi203.stdout:5/415: creat d3/d4/f87 x:0 0 0 2024-04-05T03:02:39.806 INFO:tasks.workunit.client.1.smithi203.stdout:4/284: dwrite da/d29/f40 [0,4194304] 0 2024-04-05T03:02:39.807 INFO:tasks.workunit.client.1.smithi203.stdout:3/354: rename d0/dc/d1a/f57 to d0/dc/d34/f61 0 2024-04-05T03:02:39.810 INFO:tasks.workunit.client.1.smithi203.stdout:9/316: creat d13/d1b/d26/d31/d38/f5e x:0 0 0 2024-04-05T03:02:39.811 INFO:tasks.workunit.client.1.smithi203.stdout:3/355: read d0/dc/f54 [45949,72704] 0 2024-04-05T03:02:39.813 INFO:tasks.workunit.client.1.smithi203.stdout:9/317: readlink d13/l2c 0 2024-04-05T03:02:39.814 INFO:tasks.workunit.client.1.smithi203.stdout:1/323: unlink d7/c8 0 2024-04-05T03:02:39.814 INFO:tasks.workunit.client.1.smithi203.stdout:9/318: chown d13/d1b/d26/f54 1 1 2024-04-05T03:02:39.815 INFO:tasks.workunit.client.1.smithi203.stdout:8/444: dwrite d1/d12/d1a/f42 [0,4194304] 0 2024-04-05T03:02:39.817 INFO:tasks.workunit.client.1.smithi203.stdout:4/285: readlink da/l16 0 2024-04-05T03:02:39.818 INFO:tasks.workunit.client.1.smithi203.stdout:8/445: write d1/d12/d1a/f1e [5013124,121867] 0 2024-04-05T03:02:39.819 INFO:tasks.workunit.client.1.smithi203.stdout:4/286: chown da/d29/f42 0 1 2024-04-05T03:02:39.820 INFO:tasks.workunit.client.1.smithi203.stdout:1/324: rmdir d7/d24 39 2024-04-05T03:02:39.827 INFO:tasks.workunit.client.1.smithi203.stdout:8/446: dwrite d1/d12/d2f/d1f/f50 [0,4194304] 0 2024-04-05T03:02:39.835 INFO:tasks.workunit.client.1.smithi203.stdout:8/447: rmdir d1/d4b 39 2024-04-05T03:02:39.837 INFO:tasks.workunit.client.1.smithi203.stdout:8/448: chown d1/d6/d2e/f6b 7 1 2024-04-05T03:02:39.838 INFO:tasks.workunit.client.1.smithi203.stdout:2/491: dread d5/d10/d35/d36/d41/d79/f56 [0,4194304] 0 2024-04-05T03:02:39.841 INFO:tasks.workunit.client.1.smithi203.stdout:9/319: getdents d13/d1b 0 2024-04-05T03:02:39.844 INFO:tasks.workunit.client.1.smithi203.stdout:3/356: unlink d0/f42 0 2024-04-05T03:02:39.847 INFO:tasks.workunit.client.1.smithi203.stdout:2/492: stat d5/d10/d1a/c45 0 2024-04-05T03:02:39.848 INFO:tasks.workunit.client.1.smithi203.stdout:9/320: dwrite d13/d1b/d26/d51/f53 [0,4194304] 0 2024-04-05T03:02:39.849 INFO:tasks.workunit.client.1.smithi203.stdout:1/325: creat d7/d24/f66 x:0 0 0 2024-04-05T03:02:39.852 INFO:tasks.workunit.client.1.smithi203.stdout:4/287: dread da/f37 [0,4194304] 0 2024-04-05T03:02:39.852 INFO:tasks.workunit.client.1.smithi203.stdout:3/357: write d0/dc/d1a/d1d/f29 [513616,48489] 0 2024-04-05T03:02:39.853 INFO:tasks.workunit.client.1.smithi203.stdout:2/493: read d5/d10/d35/d2d/d7c/d29/f53 [2059749,20441] 0 2024-04-05T03:02:39.857 INFO:tasks.workunit.client.1.smithi203.stdout:9/321: mkdir d13/d1b/d26/d31/d38/d59/d5f 0 2024-04-05T03:02:39.858 INFO:tasks.workunit.client.1.smithi203.stdout:1/326: unlink d7/c1f 0 2024-04-05T03:02:39.858 INFO:tasks.workunit.client.1.smithi203.stdout:5/416: dread d3/d4/fb [8388608,4194304] 0 2024-04-05T03:02:39.861 INFO:tasks.workunit.client.1.smithi203.stdout:2/494: chown d5/d10/d1a/c3c 124882 1 2024-04-05T03:02:39.863 INFO:tasks.workunit.client.1.smithi203.stdout:0/386: sync 2024-04-05T03:02:39.866 INFO:tasks.workunit.client.1.smithi203.stdout:4/288: dwrite da/d29/d2e/f3e [0,4194304] 0 2024-04-05T03:02:39.869 INFO:tasks.workunit.client.1.smithi203.stdout:3/358: getdents d0/dc/d1a/d1d 0 2024-04-05T03:02:39.869 INFO:tasks.workunit.client.1.smithi203.stdout:8/449: write d1/d12/d2f/d1f/f23 [1137149,121596] 0 2024-04-05T03:02:39.870 INFO:tasks.workunit.client.1.smithi203.stdout:2/495: creat d5/d66/fa9 x:0 0 0 2024-04-05T03:02:39.870 INFO:tasks.workunit.client.1.smithi203.stdout:1/327: creat d7/d24/d33/d48/d4b/d51/f67 x:0 0 0 2024-04-05T03:02:39.871 INFO:tasks.workunit.client.1.smithi203.stdout:8/450: chown d1/d6/d2e/l4f 47953 1 2024-04-05T03:02:39.871 INFO:tasks.workunit.client.1.smithi203.stdout:3/359: chown d0/dc/d1a/d1d/d43/f4c 16 1 2024-04-05T03:02:39.872 INFO:tasks.workunit.client.1.smithi203.stdout:9/322: creat d13/d1b/d26/d31/f60 x:0 0 0 2024-04-05T03:02:39.874 INFO:tasks.workunit.client.1.smithi203.stdout:9/323: write d13/d1b/d26/d51/f52 [486473,22876] 0 2024-04-05T03:02:39.874 INFO:tasks.workunit.client.1.smithi203.stdout:9/324: chown f12 0 1 2024-04-05T03:02:39.875 INFO:tasks.workunit.client.1.smithi203.stdout:0/387: mknod dd/d17/d1d/d26/d34/d56/c70 0 2024-04-05T03:02:39.875 INFO:tasks.workunit.client.1.smithi203.stdout:2/496: fsync d5/f9 0 2024-04-05T03:02:39.876 INFO:tasks.workunit.client.1.smithi203.stdout:0/388: chown dd/d17/d1d/d26/c5e 691969 1 2024-04-05T03:02:39.876 INFO:tasks.workunit.client.1.smithi203.stdout:4/289: getdents da/d29/d2e/d53 0 2024-04-05T03:02:39.877 INFO:tasks.workunit.client.1.smithi203.stdout:0/389: readlink dd/d17/d1d/d26/l43 0 2024-04-05T03:02:39.878 INFO:tasks.workunit.client.1.smithi203.stdout:4/290: chown da/f39 1693 1 2024-04-05T03:02:39.880 INFO:tasks.workunit.client.1.smithi203.stdout:1/328: symlink d7/l68 0 2024-04-05T03:02:39.881 INFO:tasks.workunit.client.1.smithi203.stdout:2/497: creat d5/d10/d1a/d24/d87/faa x:0 0 0 2024-04-05T03:02:39.882 INFO:tasks.workunit.client.1.smithi203.stdout:2/498: chown d5/d10/d35/d36/f54 1047201 1 2024-04-05T03:02:39.883 INFO:tasks.workunit.client.1.smithi203.stdout:2/499: fdatasync d5/d10/d2f/f9d 0 2024-04-05T03:02:39.883 INFO:tasks.workunit.client.1.smithi203.stdout:4/291: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:39.885 INFO:tasks.workunit.client.1.smithi203.stdout:0/390: rename l4 to dd/d17/d1d/d22/d45/d51/l71 0 2024-04-05T03:02:39.885 INFO:tasks.workunit.client.1.smithi203.stdout:2/500: write d5/d10/d7f/f99 [924613,80740] 0 2024-04-05T03:02:39.886 INFO:tasks.workunit.client.1.smithi203.stdout:0/391: readlink dd/lf 0 2024-04-05T03:02:39.886 INFO:tasks.workunit.client.1.smithi203.stdout:7/363: sync 2024-04-05T03:02:39.886 INFO:tasks.workunit.client.1.smithi203.stdout:6/382: sync 2024-04-05T03:02:39.889 INFO:tasks.workunit.client.1.smithi203.stdout:1/329: fsync d7/d24/d33/f3f 0 2024-04-05T03:02:39.891 INFO:tasks.workunit.client.1.smithi203.stdout:8/451: getdents d1/d6/d2e 0 2024-04-05T03:02:39.894 INFO:tasks.workunit.client.1.smithi203.stdout:4/292: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:39.894 INFO:tasks.workunit.client.1.smithi203.stdout:0/392: creat dd/d17/d1d/d26/d41/f72 x:0 0 0 2024-04-05T03:02:39.895 INFO:tasks.workunit.client.1.smithi203.stdout:1/330: dread - d7/d24/d33/f64 zero size 2024-04-05T03:02:39.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/501: rename d5/d10/d1a/d57/d70 to d5/d10/d35/d36/d41/d79/dab 0 2024-04-05T03:02:39.899 INFO:tasks.workunit.client.1.smithi203.stdout:7/364: link d8/d32/d16/d35/c6a d8/d32/d16/d1a/d3d/d59/c6f 0 2024-04-05T03:02:39.900 INFO:tasks.workunit.client.1.smithi203.stdout:5/417: dread d3/d1f/d59/d64/f23 [0,4194304] 0 2024-04-05T03:02:39.903 INFO:tasks.workunit.client.1.smithi203.stdout:6/383: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:39.904 INFO:tasks.workunit.client.1.smithi203.stdout:9/325: dread fb [0,4194304] 0 2024-04-05T03:02:39.904 INFO:tasks.workunit.client.1.smithi203.stdout:1/331: write d7/d14/d2b/d58/f60 [976068,85757] 0 2024-04-05T03:02:39.905 INFO:tasks.workunit.client.1.smithi203.stdout:2/502: fdatasync d5/d10/d35/d2d/d7c/d72/fa0 0 2024-04-05T03:02:39.905 INFO:tasks.workunit.client.1.smithi203.stdout:3/360: dwrite d0/dc/f54 [0,4194304] 0 2024-04-05T03:02:39.909 INFO:tasks.workunit.client.1.smithi203.stdout:8/452: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:39.915 INFO:tasks.workunit.client.1.smithi203.stdout:7/365: creat d8/d32/f70 x:0 0 0 2024-04-05T03:02:39.915 INFO:tasks.workunit.client.1.smithi203.stdout:9/326: dread d13/f24 [0,4194304] 0 2024-04-05T03:02:39.917 INFO:tasks.workunit.client.1.smithi203.stdout:4/293: dwrite da/f1e [0,4194304] 0 2024-04-05T03:02:39.920 INFO:tasks.workunit.client.1.smithi203.stdout:6/384: unlink d0/da/d54/c3e 0 2024-04-05T03:02:39.920 INFO:tasks.workunit.client.1.smithi203.stdout:1/332: mknod d7/d24/d33/d48/d4b/c69 0 2024-04-05T03:02:39.921 INFO:tasks.workunit.client.1.smithi203.stdout:3/361: creat d0/dc/d38/d5d/f62 x:0 0 0 2024-04-05T03:02:39.922 INFO:tasks.workunit.client.1.smithi203.stdout:9/327: symlink d13/l61 0 2024-04-05T03:02:39.922 INFO:tasks.workunit.client.1.smithi203.stdout:5/418: dwrite f2 [0,4194304] 0 2024-04-05T03:02:39.924 INFO:tasks.workunit.client.1.smithi203.stdout:7/366: unlink d8/d32/d16/d1a/f1f 0 2024-04-05T03:02:39.926 INFO:tasks.workunit.client.1.smithi203.stdout:2/503: creat d5/d10/d1a/d57/fac x:0 0 0 2024-04-05T03:02:39.928 INFO:tasks.workunit.client.1.smithi203.stdout:6/385: readlink d0/l29 0 2024-04-05T03:02:39.929 INFO:tasks.workunit.client.1.smithi203.stdout:1/333: symlink d7/d14/d2b/d58/l6a 0 2024-04-05T03:02:39.932 INFO:tasks.workunit.client.1.smithi203.stdout:3/362: rename d0/dc/d1a/d33/c56 to d0/dc/d3f/c63 0 2024-04-05T03:02:39.934 INFO:tasks.workunit.client.1.smithi203.stdout:7/367: mkdir d8/da/d71 0 2024-04-05T03:02:39.935 INFO:tasks.workunit.client.1.smithi203.stdout:8/453: mkdir d1/d4b/d68/d75/d77 0 2024-04-05T03:02:39.938 INFO:tasks.workunit.client.1.smithi203.stdout:2/504: dwrite d5/d10/d35/d73/f95 [0,4194304] 0 2024-04-05T03:02:39.941 INFO:tasks.workunit.client.1.smithi203.stdout:4/294: creat da/d29/f5b x:0 0 0 2024-04-05T03:02:39.942 INFO:tasks.workunit.client.1.smithi203.stdout:9/328: creat d13/d1b/d26/d31/f62 x:0 0 0 2024-04-05T03:02:39.944 INFO:tasks.workunit.client.1.smithi203.stdout:8/454: creat d1/d12/d1a/f78 x:0 0 0 2024-04-05T03:02:39.950 INFO:tasks.workunit.client.1.smithi203.stdout:1/334: dwrite d7/d24/d33/f36 [4194304,4194304] 0 2024-04-05T03:02:39.953 INFO:tasks.workunit.client.1.smithi203.stdout:4/295: creat da/d3c/d4c/f5c x:0 0 0 2024-04-05T03:02:39.954 INFO:tasks.workunit.client.1.smithi203.stdout:9/329: mkdir d13/d1b/d33/d63 0 2024-04-05T03:02:39.957 INFO:tasks.workunit.client.1.smithi203.stdout:6/386: dwrite d0/da/d18/d38/d2a/d2c/f4f [0,4194304] 0 2024-04-05T03:02:39.958 INFO:tasks.workunit.client.1.smithi203.stdout:3/363: dwrite d0/dc/d1a/d1d/d43/f4c [0,4194304] 0 2024-04-05T03:02:39.964 INFO:tasks.workunit.client.1.smithi203.stdout:3/364: rename d0/dc to d0/dc/d64 22 2024-04-05T03:02:39.965 INFO:tasks.workunit.client.1.smithi203.stdout:1/335: mkdir d7/d24/d33/d48/d4b/d6b 0 2024-04-05T03:02:39.968 INFO:tasks.workunit.client.1.smithi203.stdout:4/296: symlink da/d29/l5d 0 2024-04-05T03:02:39.968 INFO:tasks.workunit.client.1.smithi203.stdout:8/455: dwrite d1/d12/d1a/f78 [0,4194304] 0 2024-04-05T03:02:39.971 INFO:tasks.workunit.client.1.smithi203.stdout:3/365: mknod d0/dc/d1a/d33/c65 0 2024-04-05T03:02:39.972 INFO:tasks.workunit.client.1.smithi203.stdout:9/330: mkdir d13/d1b/d33/d64 0 2024-04-05T03:02:39.973 INFO:tasks.workunit.client.1.smithi203.stdout:9/331: chown d13/d1b/d26/d31/d38/d59 48982 1 2024-04-05T03:02:39.977 INFO:tasks.workunit.client.1.smithi203.stdout:8/456: unlink d1/d12/l51 0 2024-04-05T03:02:39.978 INFO:tasks.workunit.client.1.smithi203.stdout:4/297: mknod da/d3c/d4c/c5e 0 2024-04-05T03:02:39.979 INFO:tasks.workunit.client.1.smithi203.stdout:3/366: mkdir d0/dc/d38/d5d/d66 0 2024-04-05T03:02:39.980 INFO:tasks.workunit.client.1.smithi203.stdout:8/457: dread - d1/d12/d1a/f46 zero size 2024-04-05T03:02:39.980 INFO:tasks.workunit.client.1.smithi203.stdout:5/419: dread d3/d1f/d59/d64/f44 [0,4194304] 0 2024-04-05T03:02:39.982 INFO:tasks.workunit.client.1.smithi203.stdout:1/336: creat d7/d14/f6c x:0 0 0 2024-04-05T03:02:39.982 INFO:tasks.workunit.client.1.smithi203.stdout:7/368: dwrite d8/d32/d5d/f69 [0,4194304] 0 2024-04-05T03:02:39.987 INFO:tasks.workunit.client.1.smithi203.stdout:9/332: creat d13/d1b/d33/f65 x:0 0 0 2024-04-05T03:02:39.988 INFO:tasks.workunit.client.1.smithi203.stdout:5/420: symlink d3/d1f/d49/l88 0 2024-04-05T03:02:39.989 INFO:tasks.workunit.client.1.smithi203.stdout:9/333: write f7 [3683747,2254] 0 2024-04-05T03:02:39.991 INFO:tasks.workunit.client.1.smithi203.stdout:1/337: mknod d7/d24/d33/d48/c6d 0 2024-04-05T03:02:39.991 INFO:tasks.workunit.client.1.smithi203.stdout:3/367: dread d0/dc/d1a/d1d/d43/f4c [0,4194304] 0 2024-04-05T03:02:39.991 INFO:tasks.workunit.client.1.smithi203.stdout:8/458: dread d1/d6/f41 [0,4194304] 0 2024-04-05T03:02:39.992 INFO:tasks.workunit.client.1.smithi203.stdout:7/369: truncate d8/da/fd 3530821 0 2024-04-05T03:02:39.993 INFO:tasks.workunit.client.1.smithi203.stdout:2/505: dread d5/d10/d35/d36/f58 [0,4194304] 0 2024-04-05T03:02:39.993 INFO:tasks.workunit.client.1.smithi203.stdout:9/334: read d13/f1d [14806,12930] 0 2024-04-05T03:02:39.994 INFO:tasks.workunit.client.1.smithi203.stdout:5/421: rename d3/d4/dc/c19 to d3/d4/dc/c89 0 2024-04-05T03:02:39.994 INFO:tasks.workunit.client.1.smithi203.stdout:7/370: write d8/d2a/f40 [256859,6149] 0 2024-04-05T03:02:39.998 INFO:tasks.workunit.client.1.smithi203.stdout:9/335: chown l4 8537 1 2024-04-05T03:02:39.998 INFO:tasks.workunit.client.1.smithi203.stdout:2/506: creat d5/d10/d35/fad x:0 0 0 2024-04-05T03:02:39.999 INFO:tasks.workunit.client.1.smithi203.stdout:1/338: creat d7/d24/f6e x:0 0 0 2024-04-05T03:02:40.000 INFO:tasks.workunit.client.1.smithi203.stdout:6/387: dread d0/da/d18/d38/d2a/d2c/f3a [0,4194304] 0 2024-04-05T03:02:40.003 INFO:tasks.workunit.client.1.smithi203.stdout:7/371: rename d8/d32/d16/d1a/d68 to d8/da/d72 0 2024-04-05T03:02:40.004 INFO:tasks.workunit.client.1.smithi203.stdout:7/372: readlink d8/d32/d16/d1a/d30/l51 0 2024-04-05T03:02:40.005 INFO:tasks.workunit.client.1.smithi203.stdout:9/336: link d13/d1b/d33/d42/f58 d13/d1b/d33/d42/f66 0 2024-04-05T03:02:40.005 INFO:tasks.workunit.client.1.smithi203.stdout:9/337: chown d13/f20 27040260 1 2024-04-05T03:02:40.007 INFO:tasks.workunit.client.1.smithi203.stdout:6/388: fsync d0/da/d18/d38/d2a/d2c/f3a 0 2024-04-05T03:02:40.013 INFO:tasks.workunit.client.1.smithi203.stdout:9/338: write d13/d1b/d26/f54 [1671329,24404] 0 2024-04-05T03:02:40.017 INFO:tasks.workunit.client.1.smithi203.stdout:6/389: dwrite d0/da/d67/f5f [0,4194304] 0 2024-04-05T03:02:40.018 INFO:tasks.workunit.client.1.smithi203.stdout:8/459: dread d1/d12/d17/d10/f49 [0,4194304] 0 2024-04-05T03:02:40.022 INFO:tasks.workunit.client.1.smithi203.stdout:4/298: dwrite f5 [0,4194304] 0 2024-04-05T03:02:40.022 INFO:tasks.workunit.client.1.smithi203.stdout:9/339: mkdir d13/d1b/d33/d42/d67 0 2024-04-05T03:02:40.024 INFO:tasks.workunit.client.1.smithi203.stdout:8/460: write d1/d12/d17/d10/f40 [4973938,51067] 0 2024-04-05T03:02:40.028 INFO:tasks.workunit.client.1.smithi203.stdout:1/339: write d7/d14/f1b [2838205,86681] 0 2024-04-05T03:02:40.031 INFO:tasks.workunit.client.1.smithi203.stdout:8/461: chown d1/d6/c5a 3296904 1 2024-04-05T03:02:40.031 INFO:tasks.workunit.client.1.smithi203.stdout:6/390: mknod d0/da/d18/d38/d2a/c70 0 2024-04-05T03:02:40.031 INFO:tasks.workunit.client.1.smithi203.stdout:2/507: write d5/d10/d35/d2d/d7c/f25 [4248471,62510] 0 2024-04-05T03:02:40.032 INFO:tasks.workunit.client.1.smithi203.stdout:7/373: write d8/d32/d16/d1a/d21/f22 [1958650,48453] 0 2024-04-05T03:02:40.032 INFO:tasks.workunit.client.1.smithi203.stdout:3/368: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:40.034 INFO:tasks.workunit.client.1.smithi203.stdout:1/340: write d7/d24/f6e [605368,47337] 0 2024-04-05T03:02:40.034 INFO:tasks.workunit.client.1.smithi203.stdout:4/299: write da/f18 [199493,63919] 0 2024-04-05T03:02:40.038 INFO:tasks.workunit.client.1.smithi203.stdout:9/340: link d13/f4a d13/d1b/d26/d31/d38/f68 0 2024-04-05T03:02:40.043 INFO:tasks.workunit.client.1.smithi203.stdout:4/300: creat da/d3c/f5f x:0 0 0 2024-04-05T03:02:40.044 INFO:tasks.workunit.client.1.smithi203.stdout:6/391: write d0/da/d67/f28 [65778,87569] 0 2024-04-05T03:02:40.044 INFO:tasks.workunit.client.1.smithi203.stdout:2/508: write d5/d10/d1a/f4b [202807,86516] 0 2024-04-05T03:02:40.044 INFO:tasks.workunit.client.1.smithi203.stdout:3/369: symlink d0/dc/d1a/d1d/d53/d5a/l67 0 2024-04-05T03:02:40.045 INFO:tasks.workunit.client.1.smithi203.stdout:7/374: getdents d8/d32/d16/d35 0 2024-04-05T03:02:40.050 INFO:tasks.workunit.client.1.smithi203.stdout:4/301: unlink da/d3c/d4c/f52 0 2024-04-05T03:02:40.052 INFO:tasks.workunit.client.1.smithi203.stdout:2/509: symlink d5/d10/d2f/d5d/lae 0 2024-04-05T03:02:40.052 INFO:tasks.workunit.client.1.smithi203.stdout:1/341: dwrite d7/f4f [0,4194304] 0 2024-04-05T03:02:40.052 INFO:tasks.workunit.client.1.smithi203.stdout:8/462: dwrite d1/f34 [0,4194304] 0 2024-04-05T03:02:40.054 INFO:tasks.workunit.client.1.smithi203.stdout:9/341: creat d13/d1b/d33/f69 x:0 0 0 2024-04-05T03:02:40.054 INFO:tasks.workunit.client.1.smithi203.stdout:6/392: dread d0/da/d54/f3f [0,4194304] 0 2024-04-05T03:02:40.057 INFO:tasks.workunit.client.1.smithi203.stdout:7/375: mkdir d8/d2a/d3f/d73 0 2024-04-05T03:02:40.057 INFO:tasks.workunit.client.1.smithi203.stdout:6/393: fsync d0/da/d18/f1d 0 2024-04-05T03:02:40.059 INFO:tasks.workunit.client.1.smithi203.stdout:6/394: fsync d0/da/f61 0 2024-04-05T03:02:40.060 INFO:tasks.workunit.client.1.smithi203.stdout:6/395: chown d0/da/d54/d4c 795924169 1 2024-04-05T03:02:40.061 INFO:tasks.workunit.client.1.smithi203.stdout:6/396: fsync d0/da/d67/f5f 0 2024-04-05T03:02:40.061 INFO:tasks.workunit.client.1.smithi203.stdout:1/342: symlink d7/d24/d33/d48/d4b/l6f 0 2024-04-05T03:02:40.063 INFO:tasks.workunit.client.1.smithi203.stdout:1/343: truncate d7/d24/d33/f56 549301 0 2024-04-05T03:02:40.064 INFO:tasks.workunit.client.1.smithi203.stdout:4/302: mknod da/c60 0 2024-04-05T03:02:40.066 INFO:tasks.workunit.client.1.smithi203.stdout:6/397: dread d0/da/d18/f1d [0,4194304] 0 2024-04-05T03:02:40.068 INFO:tasks.workunit.client.1.smithi203.stdout:7/376: symlink d8/d32/d16/d35/l74 0 2024-04-05T03:02:40.071 INFO:tasks.workunit.client.1.smithi203.stdout:8/463: mknod d1/d4b/d68/d75/d77/c79 0 2024-04-05T03:02:40.071 INFO:tasks.workunit.client.1.smithi203.stdout:3/370: truncate d0/f5 2269332 0 2024-04-05T03:02:40.073 INFO:tasks.workunit.client.1.smithi203.stdout:4/303: rename da/f18 to da/d29/d2e/f61 0 2024-04-05T03:02:40.074 INFO:tasks.workunit.client.1.smithi203.stdout:6/398: dwrite d0/da/d54/d53/f59 [4194304,4194304] 0 2024-04-05T03:02:40.076 INFO:tasks.workunit.client.1.smithi203.stdout:1/344: symlink d7/l70 0 2024-04-05T03:02:40.076 INFO:tasks.workunit.client.1.smithi203.stdout:6/399: write d0/da/f43 [2909875,37491] 0 2024-04-05T03:02:40.077 INFO:tasks.workunit.client.1.smithi203.stdout:1/345: truncate d7/d24/d33/f64 1035804 0 2024-04-05T03:02:40.078 INFO:tasks.workunit.client.1.smithi203.stdout:8/464: fsync d1/d12/f28 0 2024-04-05T03:02:40.078 INFO:tasks.workunit.client.1.smithi203.stdout:1/346: read - d7/d24/f3e zero size 2024-04-05T03:02:40.082 INFO:tasks.workunit.client.1.smithi203.stdout:8/465: dread d1/d12/d17/d10/f49 [0,4194304] 0 2024-04-05T03:02:40.083 INFO:tasks.workunit.client.1.smithi203.stdout:3/371: mknod d0/dc/d1a/d1d/d43/c68 0 2024-04-05T03:02:40.087 INFO:tasks.workunit.client.1.smithi203.stdout:2/510: getdents d5/d10/d1a/d24/d87 0 2024-04-05T03:02:40.090 INFO:tasks.workunit.client.1.smithi203.stdout:9/342: link d13/d1b/c50 d13/d1b/d33/c6a 0 2024-04-05T03:02:40.091 INFO:tasks.workunit.client.1.smithi203.stdout:8/466: fsync d1/d12/d17/d10/f49 0 2024-04-05T03:02:40.091 INFO:tasks.workunit.client.1.smithi203.stdout:4/304: mknod da/d29/c62 0 2024-04-05T03:02:40.092 INFO:tasks.workunit.client.1.smithi203.stdout:7/377: creat d8/d2a/d3f/d5e/d63/f75 x:0 0 0 2024-04-05T03:02:40.093 INFO:tasks.workunit.client.1.smithi203.stdout:8/467: read d1/d12/d1a/f42 [3233300,11616] 0 2024-04-05T03:02:40.094 INFO:tasks.workunit.client.1.smithi203.stdout:3/372: mkdir d0/d69 0 2024-04-05T03:02:40.095 INFO:tasks.workunit.client.1.smithi203.stdout:4/305: chown da/l24 1 1 2024-04-05T03:02:40.096 INFO:tasks.workunit.client.1.smithi203.stdout:6/400: dwrite d0/f3 [4194304,4194304] 0 2024-04-05T03:02:40.099 INFO:tasks.workunit.client.1.smithi203.stdout:3/373: chown d0/dc/d3f/c63 36 1 2024-04-05T03:02:40.103 INFO:tasks.workunit.client.1.smithi203.stdout:6/401: mkdir d0/da/d67/d71 0 2024-04-05T03:02:40.106 INFO:tasks.workunit.client.1.smithi203.stdout:3/374: creat d0/dc/d1a/d1d/d43/f6a x:0 0 0 2024-04-05T03:02:40.107 INFO:tasks.workunit.client.1.smithi203.stdout:8/468: mknod d1/d12/d2f/d55/c7a 0 2024-04-05T03:02:40.109 INFO:tasks.workunit.client.1.smithi203.stdout:2/511: dwrite d5/d10/d35/d36/f54 [0,4194304] 0 2024-04-05T03:02:40.111 INFO:tasks.workunit.client.1.smithi203.stdout:3/375: write d0/dc/d1a/d1d/f29 [252687,10141] 0 2024-04-05T03:02:40.113 INFO:tasks.workunit.client.1.smithi203.stdout:4/306: dwrite da/f5a [0,4194304] 0 2024-04-05T03:02:40.113 INFO:tasks.workunit.client.1.smithi203.stdout:1/347: dread d7/d24/d33/f36 [0,4194304] 0 2024-04-05T03:02:40.120 INFO:tasks.workunit.client.1.smithi203.stdout:3/376: fdatasync d0/dc/d1a/d1d/d43/f51 0 2024-04-05T03:02:40.120 INFO:tasks.workunit.client.1.smithi203.stdout:4/307: rename da/d29/d2e/f59 to da/d29/d2e/f63 0 2024-04-05T03:02:40.121 INFO:tasks.workunit.client.1.smithi203.stdout:6/402: link d0/l58 d0/da/d67/d71/l72 0 2024-04-05T03:02:40.123 INFO:tasks.workunit.client.1.smithi203.stdout:1/348: unlink d7/d24/d33/f3f 0 2024-04-05T03:02:40.124 INFO:tasks.workunit.client.1.smithi203.stdout:2/512: dwrite d5/d10/d35/d36/f6c [4194304,4194304] 0 2024-04-05T03:02:40.125 INFO:tasks.workunit.client.1.smithi203.stdout:7/378: dread d8/d32/d16/d1a/d3d/f54 [0,4194304] 0 2024-04-05T03:02:40.126 INFO:tasks.workunit.client.1.smithi203.stdout:4/308: mknod da/d3c/d4c/c64 0 2024-04-05T03:02:40.127 INFO:tasks.workunit.client.1.smithi203.stdout:4/309: write da/f5a [2850662,30293] 0 2024-04-05T03:02:40.128 INFO:tasks.workunit.client.1.smithi203.stdout:2/513: symlink d5/d63/d86/laf 0 2024-04-05T03:02:40.129 INFO:tasks.workunit.client.1.smithi203.stdout:2/514: chown d5/d10/d2f/d5d 112612 1 2024-04-05T03:02:40.133 INFO:tasks.workunit.client.1.smithi203.stdout:4/310: unlink da/d3c/d4c/c5e 0 2024-04-05T03:02:40.134 INFO:tasks.workunit.client.1.smithi203.stdout:2/515: write d5/d10/d35/d36/d41/d79/f56 [3011778,88797] 0 2024-04-05T03:02:40.137 INFO:tasks.workunit.client.1.smithi203.stdout:8/469: getdents d1/d12/d2f/d1f 0 2024-04-05T03:02:40.138 INFO:tasks.workunit.client.1.smithi203.stdout:0/393: sync 2024-04-05T03:02:40.139 INFO:tasks.workunit.client.1.smithi203.stdout:4/311: rename da/d29/l5d to da/d29/d2e/d57/l65 0 2024-04-05T03:02:40.140 INFO:tasks.workunit.client.1.smithi203.stdout:2/516: creat d5/d63/d86/fb0 x:0 0 0 2024-04-05T03:02:40.141 INFO:tasks.workunit.client.1.smithi203.stdout:4/312: write da/f56 [517984,43665] 0 2024-04-05T03:02:40.141 INFO:tasks.workunit.client.1.smithi203.stdout:0/394: truncate dd/d17/d1d/d22/f65 1012333 0 2024-04-05T03:02:40.141 INFO:tasks.workunit.client.1.smithi203.stdout:2/517: chown d5/f8d 415 1 2024-04-05T03:02:40.145 INFO:tasks.workunit.client.1.smithi203.stdout:7/379: creat d8/d32/d16/d1a/d3d/f76 x:0 0 0 2024-04-05T03:02:40.148 INFO:tasks.workunit.client.1.smithi203.stdout:0/395: write dd/d17/d1d/d22/f3b [974650,101538] 0 2024-04-05T03:02:40.148 INFO:tasks.workunit.client.1.smithi203.stdout:6/403: link d0/da/d54/l65 d0/da/d18/d38/d2a/d2c/l73 0 2024-04-05T03:02:40.150 INFO:tasks.workunit.client.1.smithi203.stdout:2/518: mkdir d5/d10/d35/d2d/d7c/d29/db1 0 2024-04-05T03:02:40.150 INFO:tasks.workunit.client.1.smithi203.stdout:2/519: dread - d5/d66/fa9 zero size 2024-04-05T03:02:40.152 INFO:tasks.workunit.client.1.smithi203.stdout:1/349: truncate d7/d24/f35 1683196 0 2024-04-05T03:02:40.153 INFO:tasks.workunit.client.0.smithi153.stdout:0/31: sync 2024-04-05T03:02:40.153 INFO:tasks.workunit.client.0.smithi153.stdout:0/32: stat l1 0 2024-04-05T03:02:40.154 INFO:tasks.workunit.client.1.smithi203.stdout:4/313: link da/f5a da/d29/d2e/d57/f66 0 2024-04-05T03:02:40.155 INFO:tasks.workunit.client.1.smithi203.stdout:4/314: truncate da/d29/d2e/f63 299408 0 2024-04-05T03:02:40.156 INFO:tasks.workunit.client.1.smithi203.stdout:0/396: unlink dd/d17/d1d/d26/d41/c4c 0 2024-04-05T03:02:40.156 INFO:tasks.workunit.client.1.smithi203.stdout:8/470: write d1/d12/f31 [658362,69000] 0 2024-04-05T03:02:40.157 INFO:tasks.workunit.client.1.smithi203.stdout:8/471: dread - d1/d12/d1a/f46 zero size 2024-04-05T03:02:40.160 INFO:tasks.workunit.client.0.smithi153.stdout:0/33: dwrite f5 [0,4194304] 0 2024-04-05T03:02:40.163 INFO:tasks.workunit.client.0.smithi153.stdout:0/34: creat f7 x:0 0 0 2024-04-05T03:02:40.163 INFO:tasks.workunit.client.1.smithi203.stdout:7/380: write d8/d32/d16/d1a/d21/f23 [948074,54389] 0 2024-04-05T03:02:40.165 INFO:tasks.workunit.client.0.smithi153.stdout:0/35: mknod c8 0 2024-04-05T03:02:40.165 INFO:tasks.workunit.client.1.smithi203.stdout:4/315: symlink da/d3c/d4c/l67 0 2024-04-05T03:02:40.166 INFO:tasks.workunit.client.1.smithi203.stdout:2/520: symlink d5/d10/d35/d2d/d7c/d29/db1/lb2 0 2024-04-05T03:02:40.169 INFO:tasks.workunit.client.1.smithi203.stdout:7/381: mkdir d8/d2a/d3f/d5e/d77 0 2024-04-05T03:02:40.169 INFO:tasks.workunit.client.1.smithi203.stdout:1/350: fsync d7/f27 0 2024-04-05T03:02:40.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/15: sync 2024-04-05T03:02:40.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/16: readlink - no filename 2024-04-05T03:02:40.171 INFO:tasks.workunit.client.0.smithi153.stdout:2/27: sync 2024-04-05T03:02:40.171 INFO:tasks.workunit.client.0.smithi153.stdout:7/78: sync 2024-04-05T03:02:40.171 INFO:tasks.workunit.client.0.smithi153.stdout:5/50: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.0.smithi153.stdout:6/49: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.0.smithi153.stdout:9/15: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.0.smithi153.stdout:1/18: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.0.smithi153.stdout:4/103: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.0.smithi153.stdout:3/29: sync 2024-04-05T03:02:40.172 INFO:tasks.workunit.client.1.smithi203.stdout:8/472: symlink d1/d12/d17/d47/d53/d69/l7b 0 2024-04-05T03:02:40.173 INFO:tasks.workunit.client.0.smithi153.stdout:9/16: stat d1 0 2024-04-05T03:02:40.173 INFO:tasks.workunit.client.0.smithi153.stdout:6/50: dread - d4/f7 zero size 2024-04-05T03:02:40.173 INFO:tasks.workunit.client.1.smithi203.stdout:6/404: getdents d0/da/d18/d38/d2a/d2c 0 2024-04-05T03:02:40.173 INFO:tasks.workunit.client.1.smithi203.stdout:8/473: chown d1/d6/f45 1 1 2024-04-05T03:02:40.174 INFO:tasks.workunit.client.1.smithi203.stdout:8/474: stat d1/d12/d17/c3e 0 2024-04-05T03:02:40.174 INFO:tasks.workunit.client.0.smithi153.stdout:1/19: write f2 [265251,37463] 0 2024-04-05T03:02:40.175 INFO:tasks.workunit.client.0.smithi153.stdout:8/17: write f0 [3897650,11233] 0 2024-04-05T03:02:40.175 INFO:tasks.workunit.client.0.smithi153.stdout:2/28: creat d9/ff x:0 0 0 2024-04-05T03:02:40.175 INFO:tasks.workunit.client.1.smithi203.stdout:4/316: creat da/d29/d2e/d53/f68 x:0 0 0 2024-04-05T03:02:40.176 INFO:tasks.workunit.client.1.smithi203.stdout:1/351: symlink d7/d14/d2b/l71 0 2024-04-05T03:02:40.176 INFO:tasks.workunit.client.1.smithi203.stdout:1/352: dread - d7/d24/f66 zero size 2024-04-05T03:02:40.176 INFO:tasks.workunit.client.1.smithi203.stdout:0/397: link dd/d17/d1d/d26/d28/l5b dd/d17/d1d/d26/d34/d56/l73 0 2024-04-05T03:02:40.178 INFO:tasks.workunit.client.0.smithi153.stdout:3/30: link c9 cc 0 2024-04-05T03:02:40.178 INFO:tasks.workunit.client.1.smithi203.stdout:6/405: symlink d0/da/l74 0 2024-04-05T03:02:40.179 INFO:tasks.workunit.client.1.smithi203.stdout:8/475: creat d1/d6/f7c x:0 0 0 2024-04-05T03:02:40.181 INFO:tasks.workunit.client.1.smithi203.stdout:8/476: write d1/d12/d17/d10/f49 [2825126,21337] 0 2024-04-05T03:02:40.182 INFO:tasks.workunit.client.1.smithi203.stdout:4/317: symlink da/d29/d2e/d53/l69 0 2024-04-05T03:02:40.184 INFO:tasks.workunit.client.1.smithi203.stdout:2/521: creat d5/fb3 x:0 0 0 2024-04-05T03:02:40.186 INFO:tasks.workunit.client.1.smithi203.stdout:8/477: dread d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:40.187 INFO:tasks.workunit.client.1.smithi203.stdout:7/382: truncate d8/d2a/f42 242615 0 2024-04-05T03:02:40.187 INFO:tasks.workunit.client.0.smithi153.stdout:8/18: chown d1 58163289 1 2024-04-05T03:02:40.188 INFO:tasks.workunit.client.0.smithi153.stdout:5/51: write d0/de/f10 [151996,59737] 0 2024-04-05T03:02:40.188 INFO:tasks.workunit.client.1.smithi203.stdout:1/353: creat d7/d14/d2b/f72 x:0 0 0 2024-04-05T03:02:40.190 INFO:tasks.workunit.client.0.smithi153.stdout:2/29: truncate d9/da/fe 704200 0 2024-04-05T03:02:40.190 INFO:tasks.workunit.client.0.smithi153.stdout:7/79: rename d2/da/df to d2/da/d1f 0 2024-04-05T03:02:40.191 INFO:tasks.workunit.client.0.smithi153.stdout:3/31: fsync f3 0 2024-04-05T03:02:40.192 INFO:tasks.workunit.client.0.smithi153.stdout:1/20: dread f3 [0,4194304] 0 2024-04-05T03:02:40.194 INFO:tasks.workunit.client.1.smithi203.stdout:4/318: write f9 [887576,7514] 0 2024-04-05T03:02:40.195 INFO:tasks.workunit.client.1.smithi203.stdout:2/522: symlink d5/d10/d44/lb4 0 2024-04-05T03:02:40.195 INFO:tasks.workunit.client.0.smithi153.stdout:9/17: mkdir d1/d7 0 2024-04-05T03:02:40.195 INFO:tasks.workunit.client.1.smithi203.stdout:2/523: chown d5 171638 1 2024-04-05T03:02:40.196 INFO:tasks.workunit.client.1.smithi203.stdout:6/406: dread d0/da/d18/f30 [0,4194304] 0 2024-04-05T03:02:40.196 INFO:tasks.workunit.client.1.smithi203.stdout:2/524: dread - d5/d10/f9f zero size 2024-04-05T03:02:40.198 INFO:tasks.workunit.client.1.smithi203.stdout:6/407: write d0/da/d54/d4c/f5b [3142063,104728] 0 2024-04-05T03:02:40.198 INFO:tasks.workunit.client.0.smithi153.stdout:6/51: creat d4/dc/df/f12 x:0 0 0 2024-04-05T03:02:40.198 INFO:tasks.workunit.client.0.smithi153.stdout:6/52: dread - d4/fb zero size 2024-04-05T03:02:40.199 INFO:tasks.workunit.client.1.smithi203.stdout:8/478: creat d1/d4b/d5f/f7d x:0 0 0 2024-04-05T03:02:40.199 INFO:tasks.workunit.client.0.smithi153.stdout:6/53: dread - d4/dc/df/f12 zero size 2024-04-05T03:02:40.200 INFO:tasks.workunit.client.0.smithi153.stdout:5/52: unlink d0/d9/fc 0 2024-04-05T03:02:40.201 INFO:tasks.workunit.client.1.smithi203.stdout:0/398: symlink dd/d17/d1d/d22/l74 0 2024-04-05T03:02:40.201 INFO:tasks.workunit.client.0.smithi153.stdout:5/53: fdatasync d0/f3 0 2024-04-05T03:02:40.202 INFO:tasks.workunit.client.1.smithi203.stdout:8/479: truncate d1/d12/d1a/f46 977267 0 2024-04-05T03:02:40.204 INFO:tasks.workunit.client.0.smithi153.stdout:3/32: unlink lb 0 2024-04-05T03:02:40.204 INFO:tasks.workunit.client.0.smithi153.stdout:3/33: readlink l7 0 2024-04-05T03:02:40.205 INFO:tasks.workunit.client.0.smithi153.stdout:3/34: fsync f3 0 2024-04-05T03:02:40.205 INFO:tasks.workunit.client.1.smithi203.stdout:0/399: truncate dd/d17/d1d/d22/f3b 1502233 0 2024-04-05T03:02:40.205 INFO:tasks.workunit.client.1.smithi203.stdout:4/319: write da/f3a [917368,61819] 0 2024-04-05T03:02:40.206 INFO:tasks.workunit.client.0.smithi153.stdout:1/21: creat f4 x:0 0 0 2024-04-05T03:02:40.208 INFO:tasks.workunit.client.1.smithi203.stdout:6/408: dwrite d0/da/d54/d53/f6a [0,4194304] 0 2024-04-05T03:02:40.208 INFO:tasks.workunit.client.1.smithi203.stdout:2/525: mknod d5/d10/d2f/d5d/cb5 0 2024-04-05T03:02:40.208 INFO:tasks.workunit.client.0.smithi153.stdout:5/54: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:40.209 INFO:tasks.workunit.client.1.smithi203.stdout:6/409: write d0/da/d18/d38/d2a/d2c/f4e [197352,25339] 0 2024-04-05T03:02:40.209 INFO:tasks.workunit.client.1.smithi203.stdout:2/526: dread - d5/f8d zero size 2024-04-05T03:02:40.210 INFO:tasks.workunit.client.1.smithi203.stdout:2/527: dread - d5/f8d zero size 2024-04-05T03:02:40.212 INFO:tasks.workunit.client.0.smithi153.stdout:6/54: creat d4/dc/df/f13 x:0 0 0 2024-04-05T03:02:40.217 INFO:tasks.workunit.client.1.smithi203.stdout:7/383: dread d8/d2a/f42 [0,4194304] 0 2024-04-05T03:02:40.217 INFO:tasks.workunit.client.1.smithi203.stdout:1/354: link d7/d24/d33/d48/c6d d7/d24/d33/d48/d4b/c73 0 2024-04-05T03:02:40.219 INFO:tasks.workunit.client.1.smithi203.stdout:7/384: chown d8/da/f1d 22108 1 2024-04-05T03:02:40.220 INFO:tasks.workunit.client.1.smithi203.stdout:1/355: write d7/f4f [1024760,103919] 0 2024-04-05T03:02:40.220 INFO:tasks.workunit.client.1.smithi203.stdout:6/410: symlink d0/da/l75 0 2024-04-05T03:02:40.220 INFO:tasks.workunit.client.0.smithi153.stdout:1/22: mkdir d5 0 2024-04-05T03:02:40.220 INFO:tasks.workunit.client.0.smithi153.stdout:1/23: stat c1 0 2024-04-05T03:02:40.221 INFO:tasks.workunit.client.1.smithi203.stdout:1/356: chown d7 15488374 1 2024-04-05T03:02:40.222 INFO:tasks.workunit.client.0.smithi153.stdout:1/24: chown f2 5617 1 2024-04-05T03:02:40.223 INFO:tasks.workunit.client.1.smithi203.stdout:7/385: write d8/d32/d16/d1a/d30/f38 [1099257,79892] 0 2024-04-05T03:02:40.223 INFO:tasks.workunit.client.1.smithi203.stdout:2/528: rename d5/d10/d1a/d24 to d5/d10/d35/d36/db6 0 2024-04-05T03:02:40.225 INFO:tasks.workunit.client.0.smithi153.stdout:5/55: dwrite d0/f5 [0,4194304] 0 2024-04-05T03:02:40.226 INFO:tasks.workunit.client.0.smithi153.stdout:5/56: write d0/de/f10 [771456,53189] 0 2024-04-05T03:02:40.227 INFO:tasks.workunit.client.0.smithi153.stdout:6/55: creat d4/dc/f14 x:0 0 0 2024-04-05T03:02:40.229 INFO:tasks.workunit.client.1.smithi203.stdout:0/400: mkdir dd/d17/d1d/d75 0 2024-04-05T03:02:40.230 INFO:tasks.workunit.client.0.smithi153.stdout:4/104: getdents da 0 2024-04-05T03:02:40.231 INFO:tasks.workunit.client.0.smithi153.stdout:6/56: write d4/dc/f14 [910854,102126] 0 2024-04-05T03:02:40.231 INFO:tasks.workunit.client.0.smithi153.stdout:3/35: chown c9 693212 1 2024-04-05T03:02:40.231 INFO:tasks.workunit.client.0.smithi153.stdout:4/105: chown da/de 1098425 1 2024-04-05T03:02:40.234 INFO:tasks.workunit.client.0.smithi153.stdout:4/106: stat da/db 0 2024-04-05T03:02:40.234 INFO:tasks.workunit.client.1.smithi203.stdout:6/411: write d0/da/d54/f4b [1496788,8674] 0 2024-04-05T03:02:40.235 INFO:tasks.workunit.client.0.smithi153.stdout:1/25: rename c1 to d5/c6 0 2024-04-05T03:02:40.236 INFO:tasks.workunit.client.1.smithi203.stdout:2/529: fsync d5/d10/d1a/f1b 0 2024-04-05T03:02:40.237 INFO:tasks.workunit.client.0.smithi153.stdout:2/30: getdents d9/da 0 2024-04-05T03:02:40.237 INFO:tasks.workunit.client.1.smithi203.stdout:6/412: dread d0/da/d18/f1d [0,4194304] 0 2024-04-05T03:02:40.239 INFO:tasks.workunit.client.0.smithi153.stdout:1/26: truncate f4 321027 0 2024-04-05T03:02:40.239 INFO:tasks.workunit.client.1.smithi203.stdout:5/422: sync 2024-04-05T03:02:40.239 INFO:tasks.workunit.client.1.smithi203.stdout:2/530: read d5/d10/d35/d2d/d7c/f25 [4810153,82916] 0 2024-04-05T03:02:40.240 INFO:tasks.workunit.client.1.smithi203.stdout:6/413: dread d0/da/d54/f3f [0,4194304] 0 2024-04-05T03:02:40.240 INFO:tasks.workunit.client.1.smithi203.stdout:7/386: unlink d8/d32/d16/d1a/d30/l51 0 2024-04-05T03:02:40.241 INFO:tasks.workunit.client.1.smithi203.stdout:2/531: dread - d5/d10/d35/d36/db6/d71/f8b zero size 2024-04-05T03:02:40.242 INFO:tasks.workunit.client.0.smithi153.stdout:5/57: dwrite d0/fd [0,4194304] 0 2024-04-05T03:02:40.242 INFO:tasks.workunit.client.1.smithi203.stdout:7/387: read d8/d32/f56 [732879,45189] 0 2024-04-05T03:02:40.243 INFO:tasks.workunit.client.1.smithi203.stdout:1/357: mkdir d7/d24/d33/d48/d4b/d6b/d74 0 2024-04-05T03:02:40.245 INFO:tasks.workunit.client.1.smithi203.stdout:2/532: chown d5/d10/d35/d36/db6/d71/f97 13772963 1 2024-04-05T03:02:40.245 INFO:tasks.workunit.client.1.smithi203.stdout:1/358: stat d7/d24/d33/d48 0 2024-04-05T03:02:40.245 INFO:tasks.workunit.client.0.smithi153.stdout:3/36: dwrite f3 [0,4194304] 0 2024-04-05T03:02:40.245 INFO:tasks.workunit.client.0.smithi153.stdout:6/57: creat d4/f15 x:0 0 0 2024-04-05T03:02:40.246 INFO:tasks.workunit.client.0.smithi153.stdout:3/37: readlink l5 0 2024-04-05T03:02:40.246 INFO:tasks.workunit.client.0.smithi153.stdout:6/58: readlink d4/dc/l10 0 2024-04-05T03:02:40.246 INFO:tasks.workunit.client.1.smithi203.stdout:2/533: stat d5/d10/d2f/d5d/cb5 0 2024-04-05T03:02:40.246 INFO:tasks.workunit.client.0.smithi153.stdout:3/38: readlink l7 0 2024-04-05T03:02:40.247 INFO:tasks.workunit.client.1.smithi203.stdout:2/534: chown d5/d63 21818646 1 2024-04-05T03:02:40.247 INFO:tasks.workunit.client.0.smithi153.stdout:4/107: creat da/de/f1e x:0 0 0 2024-04-05T03:02:40.247 INFO:tasks.workunit.client.1.smithi203.stdout:1/359: write d7/d14/d2b/f49 [3898731,1833] 0 2024-04-05T03:02:40.248 INFO:tasks.workunit.client.1.smithi203.stdout:8/480: link d1/d12/d2f/d5b/c72 d1/d4b/d5f/c7e 0 2024-04-05T03:02:40.248 INFO:tasks.workunit.client.0.smithi153.stdout:2/31: mkdir d9/da/d10 0 2024-04-05T03:02:40.250 INFO:tasks.workunit.client.0.smithi153.stdout:2/32: write d9/da/fb [529699,39337] 0 2024-04-05T03:02:40.250 INFO:tasks.workunit.client.0.smithi153.stdout:1/27: creat d5/f7 x:0 0 0 2024-04-05T03:02:40.254 INFO:tasks.workunit.client.0.smithi153.stdout:6/59: mkdir d4/dc/df/d16 0 2024-04-05T03:02:40.258 INFO:tasks.workunit.client.1.smithi203.stdout:0/401: dread dd/d17/d1d/d26/d28/f4e [0,4194304] 0 2024-04-05T03:02:40.259 INFO:tasks.workunit.client.0.smithi153.stdout:1/28: mkdir d5/d8 0 2024-04-05T03:02:40.260 INFO:tasks.workunit.client.1.smithi203.stdout:8/481: readlink d1/d12/d2f/d55/l5d 0 2024-04-05T03:02:40.260 INFO:tasks.workunit.client.0.smithi153.stdout:5/58: unlink d0/d9/lb 0 2024-04-05T03:02:40.261 INFO:tasks.workunit.client.1.smithi203.stdout:7/388: dread d8/f17 [4194304,4194304] 0 2024-04-05T03:02:40.262 INFO:tasks.workunit.client.1.smithi203.stdout:1/360: link d7/d14/f1b d7/d24/d33/d48/d4b/d6b/f75 0 2024-04-05T03:02:40.263 INFO:tasks.workunit.client.0.smithi153.stdout:6/60: dwrite d4/fb [0,4194304] 0 2024-04-05T03:02:40.264 INFO:tasks.workunit.client.1.smithi203.stdout:2/535: truncate d5/d10/d35/d2d/f33 3909081 0 2024-04-05T03:02:40.266 INFO:tasks.workunit.client.1.smithi203.stdout:8/482: rename d1/d12/d17/d47/d53/d69/l7b to d1/d12/d17/l7f 0 2024-04-05T03:02:40.266 INFO:tasks.workunit.client.0.smithi153.stdout:1/29: creat d5/f9 x:0 0 0 2024-04-05T03:02:40.267 INFO:tasks.workunit.client.1.smithi203.stdout:2/536: chown d5/d10/d1a/c45 50 1 2024-04-05T03:02:40.268 INFO:tasks.workunit.client.1.smithi203.stdout:1/361: dread d7/d14/d2b/f5a [0,4194304] 0 2024-04-05T03:02:40.268 INFO:tasks.workunit.client.1.smithi203.stdout:7/389: creat d8/d2a/d3f/d73/f78 x:0 0 0 2024-04-05T03:02:40.270 INFO:tasks.workunit.client.1.smithi203.stdout:1/362: fdatasync d7/d14/d2b/f72 0 2024-04-05T03:02:40.270 INFO:tasks.workunit.client.1.smithi203.stdout:7/390: readlink d8/d2a/l5f 0 2024-04-05T03:02:40.272 INFO:tasks.workunit.client.0.smithi153.stdout:5/59: unlink d0/l4 0 2024-04-05T03:02:40.273 INFO:tasks.workunit.client.1.smithi203.stdout:2/537: creat d5/d10/d35/d2d/d7c/d29/db1/fb7 x:0 0 0 2024-04-05T03:02:40.273 INFO:tasks.workunit.client.0.smithi153.stdout:5/60: write d0/f14 [338078,89013] 0 2024-04-05T03:02:40.274 INFO:tasks.workunit.client.1.smithi203.stdout:0/402: link dd/d17/d1d/d22/d53/d55/l6c dd/d17/d1d/d26/d41/d49/l76 0 2024-04-05T03:02:40.276 INFO:tasks.workunit.client.1.smithi203.stdout:6/414: dread d0/da/f10 [4194304,4194304] 0 2024-04-05T03:02:40.276 INFO:tasks.workunit.client.1.smithi203.stdout:0/403: chown dd/d17/d1d/d26/d28/f2b 3 1 2024-04-05T03:02:40.278 INFO:tasks.workunit.client.1.smithi203.stdout:0/404: write f2 [5421201,70657] 0 2024-04-05T03:02:40.279 INFO:tasks.workunit.client.0.smithi153.stdout:6/61: mkdir d4/dc/df/d16/d17 0 2024-04-05T03:02:40.279 INFO:tasks.workunit.client.0.smithi153.stdout:5/61: symlink d0/de/l15 0 2024-04-05T03:02:40.280 INFO:tasks.workunit.client.1.smithi203.stdout:7/391: symlink d8/d2a/d3f/d5e/d77/l79 0 2024-04-05T03:02:40.280 INFO:tasks.workunit.client.1.smithi203.stdout:0/405: write dd/d17/d29/f3c [1756895,19279] 0 2024-04-05T03:02:40.282 INFO:tasks.workunit.client.0.smithi153.stdout:6/62: fdatasync d4/dc/df/f12 0 2024-04-05T03:02:40.283 INFO:tasks.workunit.client.0.smithi153.stdout:1/30: dwrite f3 [0,4194304] 0 2024-04-05T03:02:40.283 INFO:tasks.workunit.client.1.smithi203.stdout:6/415: dread d0/da/d54/f45 [4194304,4194304] 0 2024-04-05T03:02:40.284 INFO:tasks.workunit.client.0.smithi153.stdout:1/31: stat d5/f9 0 2024-04-05T03:02:40.285 INFO:tasks.workunit.client.0.smithi153.stdout:6/63: write d4/f15 [418940,95573] 0 2024-04-05T03:02:40.286 INFO:tasks.workunit.client.1.smithi203.stdout:7/392: mknod d8/da/d72/c7a 0 2024-04-05T03:02:40.287 INFO:tasks.workunit.client.0.smithi153.stdout:5/62: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:40.288 INFO:tasks.workunit.client.0.smithi153.stdout:1/32: mknod d5/ca 0 2024-04-05T03:02:40.289 INFO:tasks.workunit.client.1.smithi203.stdout:1/363: rename d7/f13 to d7/d14/f76 0 2024-04-05T03:02:40.291 INFO:tasks.workunit.client.0.smithi153.stdout:6/64: mknod d4/dc/c18 0 2024-04-05T03:02:40.292 INFO:tasks.workunit.client.1.smithi203.stdout:0/406: dwrite dd/d17/d1d/d22/f65 [0,4194304] 0 2024-04-05T03:02:40.293 INFO:tasks.workunit.client.1.smithi203.stdout:6/416: unlink d0/da/d18/d38/f68 0 2024-04-05T03:02:40.294 INFO:tasks.workunit.client.0.smithi153.stdout:5/63: unlink d0/d9/l13 0 2024-04-05T03:02:40.295 INFO:tasks.workunit.client.1.smithi203.stdout:6/417: fsync d0/da/d54/d53/f63 0 2024-04-05T03:02:40.296 INFO:tasks.workunit.client.1.smithi203.stdout:1/364: read d7/fa [2710745,54621] 0 2024-04-05T03:02:40.298 INFO:tasks.workunit.client.0.smithi153.stdout:1/33: mkdir d5/db 0 2024-04-05T03:02:40.299 INFO:tasks.workunit.client.0.smithi153.stdout:6/65: symlink d4/dc/l19 0 2024-04-05T03:02:40.299 INFO:tasks.workunit.client.1.smithi203.stdout:0/407: write dd/d17/d1d/d26/f67 [242653,3203] 0 2024-04-05T03:02:40.300 INFO:tasks.workunit.client.1.smithi203.stdout:7/393: link d8/d32/f44 d8/d2a/d3f/d73/f7b 0 2024-04-05T03:02:40.302 INFO:tasks.workunit.client.1.smithi203.stdout:1/365: chown d7/f1c 11741907 1 2024-04-05T03:02:40.303 INFO:tasks.workunit.client.0.smithi153.stdout:5/64: creat d0/f16 x:0 0 0 2024-04-05T03:02:40.303 INFO:tasks.workunit.client.0.smithi153.stdout:1/34: creat d5/db/fc x:0 0 0 2024-04-05T03:02:40.304 INFO:tasks.workunit.client.1.smithi203.stdout:1/366: write d7/d24/f3e [1042304,18838] 0 2024-04-05T03:02:40.304 INFO:tasks.workunit.client.1.smithi203.stdout:8/483: dread d1/f63 [0,4194304] 0 2024-04-05T03:02:40.309 INFO:tasks.workunit.client.0.smithi153.stdout:1/35: mkdir d5/dd 0 2024-04-05T03:02:40.310 INFO:tasks.workunit.client.0.smithi153.stdout:5/65: creat d0/de/f17 x:0 0 0 2024-04-05T03:02:40.312 INFO:tasks.workunit.client.0.smithi153.stdout:5/66: creat d0/d9/f18 x:0 0 0 2024-04-05T03:02:40.312 INFO:tasks.workunit.client.0.smithi153.stdout:5/67: dread - d0/f8 zero size 2024-04-05T03:02:40.313 INFO:tasks.workunit.client.0.smithi153.stdout:5/68: write d0/f3 [3560854,76480] 0 2024-04-05T03:02:40.314 INFO:tasks.workunit.client.0.smithi153.stdout:5/69: write d0/f14 [704113,108046] 0 2024-04-05T03:02:40.317 INFO:tasks.workunit.client.0.smithi153.stdout:5/70: unlink d0/f3 0 2024-04-05T03:02:40.321 INFO:tasks.workunit.client.0.smithi153.stdout:5/71: dread d0/de/f10 [0,4194304] 0 2024-04-05T03:02:40.324 INFO:tasks.workunit.client.0.smithi153.stdout:5/72: dread d0/f14 [0,4194304] 0 2024-04-05T03:02:40.325 INFO:tasks.workunit.client.0.smithi153.stdout:5/73: read d0/de/f10 [147380,27950] 0 2024-04-05T03:02:40.332 INFO:tasks.workunit.client.0.smithi153.stdout:5/74: dwrite d0/f8 [0,4194304] 0 2024-04-05T03:02:40.340 INFO:tasks.workunit.client.0.smithi153.stdout:9/18: getdents d1 0 2024-04-05T03:02:40.340 INFO:tasks.workunit.client.0.smithi153.stdout:0/36: dwrite f5 [4194304,4194304] 0 2024-04-05T03:02:40.341 INFO:tasks.workunit.client.1.smithi203.stdout:6/418: rename d0/da/d67/l36 to d0/da/d67/l76 0 2024-04-05T03:02:40.342 INFO:tasks.workunit.client.0.smithi153.stdout:0/37: chown f7 12331117 1 2024-04-05T03:02:40.343 INFO:tasks.workunit.client.1.smithi203.stdout:6/419: chown d0/da/d54/f4b 502850 1 2024-04-05T03:02:40.344 INFO:tasks.workunit.client.0.smithi153.stdout:9/19: fdatasync d1/f4 0 2024-04-05T03:02:40.346 INFO:tasks.workunit.client.1.smithi203.stdout:2/538: rename d5/d10/d35/d36/d41/d79 to d5/d10/db8 0 2024-04-05T03:02:40.346 INFO:tasks.workunit.client.0.smithi153.stdout:0/38: mkdir d9 0 2024-04-05T03:02:40.348 INFO:tasks.workunit.client.1.smithi203.stdout:0/408: fsync dd/d17/d1d/d26/d34/f35 0 2024-04-05T03:02:40.351 INFO:tasks.workunit.client.0.smithi153.stdout:0/39: unlink f5 0 2024-04-05T03:02:40.351 INFO:tasks.workunit.client.0.smithi153.stdout:0/40: readlink l6 0 2024-04-05T03:02:40.352 INFO:tasks.workunit.client.1.smithi203.stdout:1/367: rmdir d7/d24/d33/d48 39 2024-04-05T03:02:40.354 INFO:tasks.workunit.client.1.smithi203.stdout:6/420: creat d0/da/d54/d4c/f77 x:0 0 0 2024-04-05T03:02:40.355 INFO:tasks.workunit.client.0.smithi153.stdout:9/20: link d1/c3 d1/c8 0 2024-04-05T03:02:40.355 INFO:tasks.workunit.client.1.smithi203.stdout:6/421: truncate d0/da/f61 1193145 0 2024-04-05T03:02:40.355 INFO:tasks.workunit.client.1.smithi203.stdout:2/539: creat d5/d10/d35/d2d/d7c/d72/d8f/fb9 x:0 0 0 2024-04-05T03:02:40.356 INFO:tasks.workunit.client.0.smithi153.stdout:0/41: mknod d9/ca 0 2024-04-05T03:02:40.357 INFO:tasks.workunit.client.1.smithi203.stdout:6/422: dread - d0/da/d18/d38/f6d zero size 2024-04-05T03:02:40.357 INFO:tasks.workunit.client.0.smithi153.stdout:0/42: chown f7 14999 1 2024-04-05T03:02:40.357 INFO:tasks.workunit.client.1.smithi203.stdout:6/423: stat d0/da/d67 0 2024-04-05T03:02:40.357 INFO:tasks.workunit.client.0.smithi153.stdout:0/43: write f0 [634263,12800] 0 2024-04-05T03:02:40.358 INFO:tasks.workunit.client.1.smithi203.stdout:6/424: readlink d0/da/d54/l6f 0 2024-04-05T03:02:40.358 INFO:tasks.workunit.client.0.smithi153.stdout:0/44: dread - f3 zero size 2024-04-05T03:02:40.359 INFO:tasks.workunit.client.1.smithi203.stdout:0/409: dread dd/d17/d1d/d26/d28/f2b [0,4194304] 0 2024-04-05T03:02:40.361 INFO:tasks.workunit.client.1.smithi203.stdout:6/425: write d0/da/f61 [588339,33363] 0 2024-04-05T03:02:40.363 INFO:tasks.workunit.client.1.smithi203.stdout:7/394: link l7 d8/d2a/d3f/d5e/l7c 0 2024-04-05T03:02:40.364 INFO:tasks.workunit.client.0.smithi153.stdout:0/45: dread f0 [0,4194304] 0 2024-04-05T03:02:40.366 INFO:tasks.workunit.client.1.smithi203.stdout:2/540: dwrite d5/d10/d2f/d5d/f68 [0,4194304] 0 2024-04-05T03:02:40.367 INFO:tasks.workunit.client.0.smithi153.stdout:0/46: mkdir d9/db 0 2024-04-05T03:02:40.367 INFO:tasks.workunit.client.0.smithi153.stdout:0/47: fdatasync f3 0 2024-04-05T03:02:40.368 INFO:tasks.workunit.client.1.smithi203.stdout:2/541: fdatasync d5/d10/d1a/d57/fac 0 2024-04-05T03:02:40.368 INFO:tasks.workunit.client.0.smithi153.stdout:3/39: truncate f3 1548666 0 2024-04-05T03:02:40.369 INFO:tasks.workunit.client.0.smithi153.stdout:3/40: readlink l6 0 2024-04-05T03:02:40.371 INFO:tasks.workunit.client.0.smithi153.stdout:0/48: symlink d9/lc 0 2024-04-05T03:02:40.371 INFO:tasks.workunit.client.0.smithi153.stdout:0/49: chown f2 1090369 1 2024-04-05T03:02:40.374 INFO:tasks.workunit.client.1.smithi203.stdout:6/426: readlink d0/da/d18/d38/d2a/d2c/l73 0 2024-04-05T03:02:40.374 INFO:tasks.workunit.client.0.smithi153.stdout:2/33: truncate d9/da/fc 1200356 0 2024-04-05T03:02:40.377 INFO:tasks.workunit.client.1.smithi203.stdout:1/368: dread d7/fe [0,4194304] 0 2024-04-05T03:02:40.377 INFO:tasks.workunit.client.1.smithi203.stdout:9/343: sync 2024-04-05T03:02:40.377 INFO:tasks.workunit.client.1.smithi203.stdout:3/377: sync 2024-04-05T03:02:40.377 INFO:tasks.workunit.client.1.smithi203.stdout:7/395: rename d8/d2a/c57 to d8/da/d72/c7d 0 2024-04-05T03:02:40.379 INFO:tasks.workunit.client.1.smithi203.stdout:8/484: dwrite d1/d6/f56 [0,4194304] 0 2024-04-05T03:02:40.380 INFO:tasks.workunit.client.0.smithi153.stdout:0/50: creat d9/db/fd x:0 0 0 2024-04-05T03:02:40.381 INFO:tasks.workunit.client.0.smithi153.stdout:2/34: dread d9/da/fb [0,4194304] 0 2024-04-05T03:02:40.381 INFO:tasks.workunit.client.1.smithi203.stdout:1/369: chown d7/d24/d33/l41 0 1 2024-04-05T03:02:40.383 INFO:tasks.workunit.client.0.smithi153.stdout:0/51: fdatasync f0 0 2024-04-05T03:02:40.384 INFO:tasks.workunit.client.0.smithi153.stdout:0/52: truncate d9/db/fd 949334 0 2024-04-05T03:02:40.385 INFO:tasks.workunit.client.1.smithi203.stdout:6/427: mkdir d0/da/d18/d38/d2a/d44/d78 0 2024-04-05T03:02:40.385 INFO:tasks.workunit.client.0.smithi153.stdout:2/35: creat d9/f11 x:0 0 0 2024-04-05T03:02:40.387 INFO:tasks.workunit.client.1.smithi203.stdout:7/396: creat d8/d2a/f7e x:0 0 0 2024-04-05T03:02:40.389 INFO:tasks.workunit.client.0.smithi153.stdout:0/53: rename c8 to d9/ce 0 2024-04-05T03:02:40.389 INFO:tasks.workunit.client.1.smithi203.stdout:1/370: fdatasync d7/d24/d33/f36 0 2024-04-05T03:02:40.390 INFO:tasks.workunit.client.0.smithi153.stdout:0/54: truncate f7 726751 0 2024-04-05T03:02:40.392 INFO:tasks.workunit.client.0.smithi153.stdout:6/66: truncate d4/fb 2865002 0 2024-04-05T03:02:40.393 INFO:tasks.workunit.client.1.smithi203.stdout:0/410: creat dd/d17/d1d/d22/f77 x:0 0 0 2024-04-05T03:02:40.397 INFO:tasks.workunit.client.1.smithi203.stdout:5/423: sync 2024-04-05T03:02:40.397 INFO:tasks.workunit.client.1.smithi203.stdout:4/320: sync 2024-04-05T03:02:40.398 INFO:tasks.workunit.client.0.smithi153.stdout:2/36: rename l1 to d9/l12 0 2024-04-05T03:02:40.398 INFO:tasks.workunit.client.0.smithi153.stdout:0/55: mkdir d9/df 0 2024-04-05T03:02:40.399 INFO:tasks.workunit.client.0.smithi153.stdout:0/56: write f2 [1146404,76319] 0 2024-04-05T03:02:40.399 INFO:tasks.workunit.client.0.smithi153.stdout:5/75: rmdir d0 39 2024-04-05T03:02:40.403 INFO:tasks.workunit.client.1.smithi203.stdout:9/344: dwrite d13/f20 [0,4194304] 0 2024-04-05T03:02:40.406 INFO:tasks.workunit.client.0.smithi153.stdout:0/57: mknod d9/c10 0 2024-04-05T03:02:40.406 INFO:tasks.workunit.client.0.smithi153.stdout:0/58: fsync f7 0 2024-04-05T03:02:40.409 INFO:tasks.workunit.client.1.smithi203.stdout:3/378: dwrite d0/dc/d1a/d1d/d43/f6a [0,4194304] 0 2024-04-05T03:02:40.409 INFO:tasks.workunit.client.1.smithi203.stdout:2/542: dread d5/f8 [0,4194304] 0 2024-04-05T03:02:40.413 INFO:tasks.workunit.client.1.smithi203.stdout:8/485: dwrite d1/d12/d2f/d1f/f50 [4194304,4194304] 0 2024-04-05T03:02:40.413 INFO:tasks.workunit.client.1.smithi203.stdout:9/345: readlink d13/d1b/l57 0 2024-04-05T03:02:40.413 INFO:tasks.workunit.client.0.smithi153.stdout:6/67: rename d4/cd to d4/dc/df/d16/d17/c1a 0 2024-04-05T03:02:40.415 INFO:tasks.workunit.client.1.smithi203.stdout:6/428: truncate d0/da/d67/f5f 1293447 0 2024-04-05T03:02:40.417 INFO:tasks.workunit.client.0.smithi153.stdout:9/21: getdents d1 0 2024-04-05T03:02:40.418 INFO:tasks.workunit.client.1.smithi203.stdout:4/321: mkdir da/d29/d6a 0 2024-04-05T03:02:40.419 INFO:tasks.workunit.client.1.smithi203.stdout:1/371: dwrite d7/f1c [0,4194304] 0 2024-04-05T03:02:40.420 INFO:tasks.workunit.client.1.smithi203.stdout:3/379: read d0/dc/d34/f61 [2467484,47508] 0 2024-04-05T03:02:40.422 INFO:tasks.workunit.client.0.smithi153.stdout:0/59: creat d9/db/f11 x:0 0 0 2024-04-05T03:02:40.424 INFO:tasks.workunit.client.1.smithi203.stdout:4/322: truncate da/d3c/f43 5142244 0 2024-04-05T03:02:40.424 INFO:tasks.workunit.client.1.smithi203.stdout:8/486: symlink d1/d12/d1a/l80 0 2024-04-05T03:02:40.425 INFO:tasks.workunit.client.1.smithi203.stdout:8/487: chown d1/d12/d2f/f61 8245 1 2024-04-05T03:02:40.425 INFO:tasks.workunit.client.1.smithi203.stdout:2/543: fdatasync d5/d10/d35/d36/db6/d87/f89 0 2024-04-05T03:02:40.427 INFO:tasks.workunit.client.1.smithi203.stdout:9/346: mkdir d13/d1b/d6b 0 2024-04-05T03:02:40.429 INFO:tasks.workunit.client.1.smithi203.stdout:6/429: creat d0/da/d67/d71/f79 x:0 0 0 2024-04-05T03:02:40.430 INFO:tasks.workunit.client.0.smithi153.stdout:6/68: write f3 [3829588,54229] 0 2024-04-05T03:02:40.430 INFO:tasks.workunit.client.0.smithi153.stdout:0/60: dread f2 [0,4194304] 0 2024-04-05T03:02:40.431 INFO:tasks.workunit.client.0.smithi153.stdout:9/22: mkdir d1/d9 0 2024-04-05T03:02:40.431 INFO:tasks.workunit.client.0.smithi153.stdout:2/37: link c7 d9/da/d10/c13 0 2024-04-05T03:02:40.432 INFO:tasks.workunit.client.0.smithi153.stdout:0/61: readlink d9/lc 0 2024-04-05T03:02:40.433 INFO:tasks.workunit.client.0.smithi153.stdout:5/76: creat d0/d9/f19 x:0 0 0 2024-04-05T03:02:40.434 INFO:tasks.workunit.client.0.smithi153.stdout:0/62: chown f3 74255351 1 2024-04-05T03:02:40.436 INFO:tasks.workunit.client.0.smithi153.stdout:3/41: dwrite f3 [0,4194304] 0 2024-04-05T03:02:40.437 INFO:tasks.workunit.client.1.smithi203.stdout:3/380: read d0/dc/d1a/d1f/f32 [422581,112712] 0 2024-04-05T03:02:40.437 INFO:tasks.workunit.client.0.smithi153.stdout:3/42: read f3 [4165803,1805] 0 2024-04-05T03:02:40.438 INFO:tasks.workunit.client.0.smithi153.stdout:6/69: mknod d4/dc/df/d16/c1b 0 2024-04-05T03:02:40.440 INFO:tasks.workunit.client.0.smithi153.stdout:9/23: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:40.441 INFO:tasks.workunit.client.1.smithi203.stdout:5/424: dwrite d3/d1f/d59/d64/f63 [0,4194304] 0 2024-04-05T03:02:40.442 INFO:tasks.workunit.client.1.smithi203.stdout:0/411: dwrite dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:40.445 INFO:tasks.workunit.client.0.smithi153.stdout:6/70: dwrite f3 [4194304,4194304] 0 2024-04-05T03:02:40.446 INFO:tasks.workunit.client.1.smithi203.stdout:2/544: fsync d5/d10/f1d 0 2024-04-05T03:02:40.446 INFO:tasks.workunit.client.1.smithi203.stdout:1/372: creat d7/f77 x:0 0 0 2024-04-05T03:02:40.447 INFO:tasks.workunit.client.0.smithi153.stdout:6/71: truncate d4/f6 4305690 0 2024-04-05T03:02:40.449 INFO:tasks.workunit.client.1.smithi203.stdout:9/347: truncate d13/f2a 2843791 0 2024-04-05T03:02:40.449 INFO:tasks.workunit.client.1.smithi203.stdout:6/430: dread d0/da/d54/d53/f6a [0,4194304] 0 2024-04-05T03:02:40.450 INFO:tasks.workunit.client.0.smithi153.stdout:3/43: creat fd x:0 0 0 2024-04-05T03:02:40.453 INFO:tasks.workunit.client.0.smithi153.stdout:2/38: symlink d9/da/d10/l14 0 2024-04-05T03:02:40.463 INFO:tasks.workunit.client.0.smithi153.stdout:0/63: mkdir d9/df/d12 0 2024-04-05T03:02:40.464 INFO:tasks.workunit.client.0.smithi153.stdout:3/44: symlink le 0 2024-04-05T03:02:40.464 INFO:tasks.workunit.client.1.smithi203.stdout:0/412: chown dd/d17/d1d/d22/f65 3185 1 2024-04-05T03:02:40.465 INFO:tasks.workunit.client.1.smithi203.stdout:8/488: dwrite d1/d12/d1a/f2c [0,4194304] 0 2024-04-05T03:02:40.465 INFO:tasks.workunit.client.1.smithi203.stdout:8/489: stat d1/d6/l29 0 2024-04-05T03:02:40.465 INFO:tasks.workunit.client.1.smithi203.stdout:8/490: dread - d1/f58 zero size 2024-04-05T03:02:40.467 INFO:tasks.workunit.client.1.smithi203.stdout:7/397: dread d8/da/f1d [0,4194304] 0 2024-04-05T03:02:40.468 INFO:tasks.workunit.client.1.smithi203.stdout:6/431: dread d0/da/d18/d38/f47 [0,4194304] 0 2024-04-05T03:02:40.471 INFO:tasks.workunit.client.1.smithi203.stdout:7/398: dread - d8/d2a/d3f/d73/f78 zero size 2024-04-05T03:02:40.478 INFO:tasks.workunit.client.1.smithi203.stdout:0/413: dwrite dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:40.480 INFO:tasks.workunit.client.1.smithi203.stdout:6/432: chown d0/da/d18/l6e 31864 1 2024-04-05T03:02:40.480 INFO:tasks.workunit.client.1.smithi203.stdout:9/348: truncate d13/d1b/d26/d31/f62 73350 0 2024-04-05T03:02:40.480 INFO:tasks.workunit.client.1.smithi203.stdout:3/381: rename d0/dc/d1a/d33/l60 to d0/dc/d38/l6b 0 2024-04-05T03:02:40.482 INFO:tasks.workunit.client.1.smithi203.stdout:8/491: mknod d1/c81 0 2024-04-05T03:02:40.483 INFO:tasks.workunit.client.1.smithi203.stdout:6/433: write d0/da/d54/f4b [2470251,72301] 0 2024-04-05T03:02:40.483 INFO:tasks.workunit.client.1.smithi203.stdout:1/373: creat d7/f78 x:0 0 0 2024-04-05T03:02:40.483 INFO:tasks.workunit.client.1.smithi203.stdout:8/492: write d1/f58 [654249,121859] 0 2024-04-05T03:02:40.484 INFO:tasks.workunit.client.0.smithi153.stdout:3/45: dwrite fd [0,4194304] 0 2024-04-05T03:02:40.488 INFO:tasks.workunit.client.1.smithi203.stdout:7/399: rmdir d8/d2a/d3f/d52 39 2024-04-05T03:02:40.490 INFO:tasks.workunit.client.1.smithi203.stdout:3/382: creat d0/d21/f6c x:0 0 0 2024-04-05T03:02:40.493 INFO:tasks.workunit.client.1.smithi203.stdout:6/434: creat d0/da/d67/f7a x:0 0 0 2024-04-05T03:02:40.497 INFO:tasks.workunit.client.1.smithi203.stdout:8/493: creat d1/d12/d17/d47/d53/f82 x:0 0 0 2024-04-05T03:02:40.498 INFO:tasks.workunit.client.1.smithi203.stdout:3/383: dread d0/dc/f54 [0,4194304] 0 2024-04-05T03:02:40.500 INFO:tasks.workunit.client.1.smithi203.stdout:5/425: link d3/d1f/d59/d64/c73 d3/d4/dc/d67/c8a 0 2024-04-05T03:02:40.501 INFO:tasks.workunit.client.1.smithi203.stdout:7/400: dread d8/d32/d5d/f69 [0,4194304] 0 2024-04-05T03:02:40.503 INFO:tasks.workunit.client.1.smithi203.stdout:1/374: rmdir d7/d24/d33/d48 39 2024-04-05T03:02:40.503 INFO:tasks.workunit.client.1.smithi203.stdout:5/426: chown d3/f4d 48562 1 2024-04-05T03:02:40.503 INFO:tasks.workunit.client.1.smithi203.stdout:6/435: mknod d0/da/d18/d38/d2a/d2c/c7b 0 2024-04-05T03:02:40.505 INFO:tasks.workunit.client.1.smithi203.stdout:0/414: rename dd/d17/d1d/d22/f32 to dd/d17/d1d/d22/f78 0 2024-04-05T03:02:40.506 INFO:tasks.workunit.client.1.smithi203.stdout:7/401: rename d8 to d8/d7f 22 2024-04-05T03:02:40.508 INFO:tasks.workunit.client.1.smithi203.stdout:2/545: dread d5/d66/d7d/f94 [0,4194304] 0 2024-04-05T03:02:40.509 INFO:tasks.workunit.client.1.smithi203.stdout:7/402: truncate d8/d2a/f40 334184 0 2024-04-05T03:02:40.509 INFO:tasks.workunit.client.1.smithi203.stdout:6/436: creat d0/da/d18/d38/d2b/f7c x:0 0 0 2024-04-05T03:02:40.513 INFO:tasks.workunit.client.1.smithi203.stdout:5/427: symlink d3/d1f/d59/d64/l8b 0 2024-04-05T03:02:40.516 INFO:tasks.workunit.client.1.smithi203.stdout:3/384: getdents d0/dc/d38/d5d 0 2024-04-05T03:02:40.521 INFO:tasks.workunit.client.1.smithi203.stdout:5/428: creat d3/f8c x:0 0 0 2024-04-05T03:02:40.525 INFO:tasks.workunit.client.0.smithi153.stdout:6/72: truncate d4/f6 939543 0 2024-04-05T03:02:40.525 INFO:tasks.workunit.client.1.smithi203.stdout:3/385: read d0/dc/d1a/d33/d35/f3c [1230359,44957] 0 2024-04-05T03:02:40.526 INFO:tasks.workunit.client.0.smithi153.stdout:6/73: fsync d4/dc/df/f13 0 2024-04-05T03:02:40.527 INFO:tasks.workunit.client.1.smithi203.stdout:9/349: write f8 [3233089,63204] 0 2024-04-05T03:02:40.528 INFO:tasks.workunit.client.1.smithi203.stdout:5/429: chown d3/d1f/d59/d64/d12/f43 824210 1 2024-04-05T03:02:40.529 INFO:tasks.workunit.client.0.smithi153.stdout:1/36: sync 2024-04-05T03:02:40.529 INFO:tasks.workunit.client.0.smithi153.stdout:4/108: sync 2024-04-05T03:02:40.529 INFO:tasks.workunit.client.0.smithi153.stdout:7/80: sync 2024-04-05T03:02:40.529 INFO:tasks.workunit.client.0.smithi153.stdout:8/19: sync 2024-04-05T03:02:40.529 INFO:tasks.workunit.client.0.smithi153.stdout:0/64: dwrite f2 [0,4194304] 0 2024-04-05T03:02:40.531 INFO:tasks.workunit.client.1.smithi203.stdout:2/546: dwrite d5/d63/f93 [0,4194304] 0 2024-04-05T03:02:40.531 INFO:tasks.workunit.client.1.smithi203.stdout:1/375: dwrite d7/d24/d33/d48/d4b/d51/f67 [0,4194304] 0 2024-04-05T03:02:40.531 INFO:tasks.workunit.client.0.smithi153.stdout:0/65: fdatasync f3 0 2024-04-05T03:02:40.532 INFO:tasks.workunit.client.1.smithi203.stdout:6/437: dwrite d0/da/d54/f4b [0,4194304] 0 2024-04-05T03:02:40.532 INFO:tasks.workunit.client.0.smithi153.stdout:4/109: chown f4 28 1 2024-04-05T03:02:40.534 INFO:tasks.workunit.client.0.smithi153.stdout:7/81: write d2/da/d11/d12/f19 [669179,29566] 0 2024-04-05T03:02:40.541 INFO:tasks.workunit.client.1.smithi203.stdout:6/438: fdatasync d0/da/d54/d4c/f77 0 2024-04-05T03:02:40.541 INFO:tasks.workunit.client.0.smithi153.stdout:0/66: dread d9/db/fd [0,4194304] 0 2024-04-05T03:02:40.542 INFO:tasks.workunit.client.1.smithi203.stdout:5/430: mknod d3/d5f/c8d 0 2024-04-05T03:02:40.543 INFO:tasks.workunit.client.0.smithi153.stdout:0/67: chown l6 9332173 1 2024-04-05T03:02:40.543 INFO:tasks.workunit.client.0.smithi153.stdout:1/37: dwrite d5/f7 [0,4194304] 0 2024-04-05T03:02:40.544 INFO:tasks.workunit.client.0.smithi153.stdout:0/68: chown d9/lc 87301206 1 2024-04-05T03:02:40.544 INFO:tasks.workunit.client.0.smithi153.stdout:1/38: truncate d5/f9 493663 0 2024-04-05T03:02:40.545 INFO:tasks.workunit.client.1.smithi203.stdout:3/386: symlink d0/dc/d1a/d33/d35/l6d 0 2024-04-05T03:02:40.545 INFO:tasks.workunit.client.1.smithi203.stdout:2/547: mknod d5/d10/d2f/cba 0 2024-04-05T03:02:40.546 INFO:tasks.workunit.client.0.smithi153.stdout:1/39: read - d5/db/fc zero size 2024-04-05T03:02:40.547 INFO:tasks.workunit.client.1.smithi203.stdout:6/439: readlink d0/da/d54/l65 0 2024-04-05T03:02:40.547 INFO:tasks.workunit.client.1.smithi203.stdout:0/415: dwrite dd/d17/d1d/d26/d28/f2d [4194304,4194304] 0 2024-04-05T03:02:40.550 INFO:tasks.workunit.client.0.smithi153.stdout:0/69: dread f2 [0,4194304] 0 2024-04-05T03:02:40.553 INFO:tasks.workunit.client.1.smithi203.stdout:7/403: dwrite d8/d2a/d3f/d73/f7b [4194304,4194304] 0 2024-04-05T03:02:40.554 INFO:tasks.workunit.client.1.smithi203.stdout:2/548: write d5/d10/d35/d2d/d7c/d29/db1/fb7 [232343,110376] 0 2024-04-05T03:02:40.554 INFO:tasks.workunit.client.1.smithi203.stdout:1/376: dread d7/d14/f29 [0,4194304] 0 2024-04-05T03:02:40.554 INFO:tasks.workunit.client.1.smithi203.stdout:3/387: creat d0/dc/d34/f6e x:0 0 0 2024-04-05T03:02:40.557 INFO:tasks.workunit.client.1.smithi203.stdout:6/440: creat d0/da/d18/d38/d2b/f7d x:0 0 0 2024-04-05T03:02:40.558 INFO:tasks.workunit.client.1.smithi203.stdout:5/431: creat d3/d4/f8e x:0 0 0 2024-04-05T03:02:40.561 INFO:tasks.workunit.client.1.smithi203.stdout:3/388: chown d0/dc/d1a/d1f 377557 1 2024-04-05T03:02:40.561 INFO:tasks.workunit.client.0.smithi153.stdout:0/70: dwrite f0 [0,4194304] 0 2024-04-05T03:02:40.563 INFO:tasks.workunit.client.0.smithi153.stdout:0/71: chown f2 2 1 2024-04-05T03:02:40.565 INFO:tasks.workunit.client.1.smithi203.stdout:1/377: chown d7/d14/l26 52188 1 2024-04-05T03:02:40.565 INFO:tasks.workunit.client.1.smithi203.stdout:7/404: unlink d8/d32/d16/d1a/d3d/f54 0 2024-04-05T03:02:40.565 INFO:tasks.workunit.client.1.smithi203.stdout:6/441: symlink d0/da/l7e 0 2024-04-05T03:02:40.565 INFO:tasks.workunit.client.1.smithi203.stdout:0/416: dwrite dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:40.566 INFO:tasks.workunit.client.1.smithi203.stdout:6/442: dread - d0/da/d18/d38/d2b/f2d zero size 2024-04-05T03:02:40.569 INFO:tasks.workunit.client.1.smithi203.stdout:0/417: write dd/d17/d1d/d26/d34/f47 [987443,61577] 0 2024-04-05T03:02:40.571 INFO:tasks.workunit.client.1.smithi203.stdout:6/443: write d0/f9 [4869339,44384] 0 2024-04-05T03:02:40.572 INFO:tasks.workunit.client.1.smithi203.stdout:3/389: rename d0/l2c to d0/dc/d1a/d1f/l6f 0 2024-04-05T03:02:40.572 INFO:tasks.workunit.client.1.smithi203.stdout:1/378: chown d7/c22 800308 1 2024-04-05T03:02:40.573 INFO:tasks.workunit.client.0.smithi153.stdout:0/72: dwrite d9/db/f11 [0,4194304] 0 2024-04-05T03:02:40.574 INFO:tasks.workunit.client.1.smithi203.stdout:3/390: fdatasync d0/dc/d1a/d1d/f29 0 2024-04-05T03:02:40.575 INFO:tasks.workunit.client.1.smithi203.stdout:6/444: chown d0/da/d54/d53/f59 5183211 1 2024-04-05T03:02:40.577 INFO:tasks.workunit.client.1.smithi203.stdout:7/405: creat d8/d2a/d3f/d5e/d77/f80 x:0 0 0 2024-04-05T03:02:40.578 INFO:tasks.workunit.client.1.smithi203.stdout:7/406: readlink d8/d32/d16/d35/l41 0 2024-04-05T03:02:40.579 INFO:tasks.workunit.client.1.smithi203.stdout:5/432: mknod d3/d1f/d59/d64/c8f 0 2024-04-05T03:02:40.583 INFO:tasks.workunit.client.1.smithi203.stdout:2/549: rename d5/d63/d86/f8e to d5/d10/d35/d36/db6/d71/fbb 0 2024-04-05T03:02:40.589 INFO:tasks.workunit.client.1.smithi203.stdout:5/433: chown d3/d4/dc/d67/f70 50 1 2024-04-05T03:02:40.590 INFO:tasks.workunit.client.1.smithi203.stdout:1/379: dwrite d7/d14/d2b/d58/f60 [0,4194304] 0 2024-04-05T03:02:40.592 INFO:tasks.workunit.client.1.smithi203.stdout:1/380: write d7/d24/d33/f36 [500959,65800] 0 2024-04-05T03:02:40.592 INFO:tasks.workunit.client.1.smithi203.stdout:9/350: truncate d13/d1b/d33/d42/f49 3342808 0 2024-04-05T03:02:40.596 INFO:tasks.workunit.client.1.smithi203.stdout:3/391: truncate d0/dc/f16 2360650 0 2024-04-05T03:02:40.603 INFO:tasks.workunit.client.1.smithi203.stdout:5/434: symlink d3/d4/dc/l90 0 2024-04-05T03:02:40.604 INFO:tasks.workunit.client.1.smithi203.stdout:2/550: symlink d5/lbc 0 2024-04-05T03:02:40.605 INFO:tasks.workunit.client.1.smithi203.stdout:6/445: rmdir d0 39 2024-04-05T03:02:40.605 INFO:tasks.workunit.client.1.smithi203.stdout:3/392: chown d0/dc/d38/d5d/d66 215988 1 2024-04-05T03:02:40.606 INFO:tasks.workunit.client.1.smithi203.stdout:1/381: mkdir d7/d79 0 2024-04-05T03:02:40.608 INFO:tasks.workunit.client.1.smithi203.stdout:9/351: dwrite d13/f20 [4194304,4194304] 0 2024-04-05T03:02:40.611 INFO:tasks.workunit.client.1.smithi203.stdout:0/418: rename dd/d17/d1d/d22/f78 to dd/d17/f79 0 2024-04-05T03:02:40.611 INFO:tasks.workunit.client.1.smithi203.stdout:0/419: fdatasync dd/d17/d1d/d26/f42 0 2024-04-05T03:02:40.613 INFO:tasks.workunit.client.1.smithi203.stdout:2/551: mkdir d5/d10/d35/d2d/d7c/d7a/dbd 0 2024-04-05T03:02:40.614 INFO:tasks.workunit.client.1.smithi203.stdout:1/382: mkdir d7/d14/d2b/d58/d7a 0 2024-04-05T03:02:40.617 INFO:tasks.workunit.client.1.smithi203.stdout:3/393: mkdir d0/dc/d1a/d33/d35/d70 0 2024-04-05T03:02:40.619 INFO:tasks.workunit.client.1.smithi203.stdout:3/394: write d0/dc/d3f/f59 [271113,116930] 0 2024-04-05T03:02:40.621 INFO:tasks.workunit.client.1.smithi203.stdout:2/552: mkdir d5/d10/db8/dbe 0 2024-04-05T03:02:40.621 INFO:tasks.workunit.client.1.smithi203.stdout:7/407: dread d8/da/f50 [0,4194304] 0 2024-04-05T03:02:40.621 INFO:tasks.workunit.client.1.smithi203.stdout:9/352: write d13/f24 [788232,95082] 0 2024-04-05T03:02:40.621 INFO:tasks.workunit.client.1.smithi203.stdout:7/408: chown d8 327909484 1 2024-04-05T03:02:40.622 INFO:tasks.workunit.client.1.smithi203.stdout:6/446: mkdir d0/da/d54/d4c/d7f 0 2024-04-05T03:02:40.624 INFO:tasks.workunit.client.1.smithi203.stdout:9/353: write d13/d1b/d26/d51/f53 [1159481,118062] 0 2024-04-05T03:02:40.627 INFO:tasks.workunit.client.1.smithi203.stdout:6/447: unlink d0/da/cb 0 2024-04-05T03:02:40.634 INFO:tasks.workunit.client.1.smithi203.stdout:3/395: dwrite d0/dc/d1a/d1f/f32 [4194304,4194304] 0 2024-04-05T03:02:40.634 INFO:tasks.workunit.client.1.smithi203.stdout:2/553: symlink d5/d10/lbf 0 2024-04-05T03:02:40.635 INFO:tasks.workunit.client.1.smithi203.stdout:6/448: symlink d0/da/d54/d53/l80 0 2024-04-05T03:02:40.636 INFO:tasks.workunit.client.1.smithi203.stdout:1/383: rmdir d7/d24/d33/d48/d59 0 2024-04-05T03:02:40.638 INFO:tasks.workunit.client.1.smithi203.stdout:2/554: fdatasync d5/d10/f1d 0 2024-04-05T03:02:40.639 INFO:tasks.workunit.client.1.smithi203.stdout:9/354: truncate d13/d1b/d26/d51/f52 1395014 0 2024-04-05T03:02:40.641 INFO:tasks.workunit.client.1.smithi203.stdout:2/555: fdatasync d5/d10/d35/d2d/f38 0 2024-04-05T03:02:40.643 INFO:tasks.workunit.client.1.smithi203.stdout:1/384: fsync d7/d24/f34 0 2024-04-05T03:02:40.647 INFO:tasks.workunit.client.1.smithi203.stdout:1/385: readlink d7/d24/l4c 0 2024-04-05T03:02:40.651 INFO:tasks.workunit.client.1.smithi203.stdout:1/386: chown d7/d24/d33/f45 514 1 2024-04-05T03:02:40.653 INFO:tasks.workunit.client.1.smithi203.stdout:6/449: dwrite d0/da/d54/d53/f59 [0,4194304] 0 2024-04-05T03:02:40.658 INFO:tasks.workunit.client.1.smithi203.stdout:6/450: chown d0/da/d18/d38/c22 437558737 1 2024-04-05T03:02:40.658 INFO:tasks.workunit.client.1.smithi203.stdout:1/387: dread d7/d24/d33/f56 [0,4194304] 0 2024-04-05T03:02:40.658 INFO:tasks.workunit.client.1.smithi203.stdout:7/409: dread d8/d2a/d3f/d52/f45 [0,4194304] 0 2024-04-05T03:02:40.659 INFO:tasks.workunit.client.1.smithi203.stdout:1/388: read d7/f4f [3961274,49997] 0 2024-04-05T03:02:40.661 INFO:tasks.workunit.client.1.smithi203.stdout:3/396: write d0/f5 [55613,77655] 0 2024-04-05T03:02:40.663 INFO:tasks.workunit.client.1.smithi203.stdout:9/355: write d13/d1b/f47 [1028883,26423] 0 2024-04-05T03:02:40.663 INFO:tasks.workunit.client.1.smithi203.stdout:7/410: stat d8/d32/d16/d1a/c34 0 2024-04-05T03:02:40.665 INFO:tasks.workunit.client.1.smithi203.stdout:1/389: truncate d7/d14/d2b/f5a 1257298 0 2024-04-05T03:02:40.666 INFO:tasks.workunit.client.1.smithi203.stdout:1/390: dread - d7/f77 zero size 2024-04-05T03:02:40.670 INFO:tasks.workunit.client.1.smithi203.stdout:1/391: write d7/d14/f16 [2298822,46497] 0 2024-04-05T03:02:40.670 INFO:tasks.workunit.client.1.smithi203.stdout:7/411: fdatasync d8/d2a/f42 0 2024-04-05T03:02:40.671 INFO:tasks.workunit.client.1.smithi203.stdout:7/412: chown d8/da/cc 36751 1 2024-04-05T03:02:40.672 INFO:tasks.workunit.client.1.smithi203.stdout:3/397: getdents d0/dc/d38/d5d 0 2024-04-05T03:02:40.673 INFO:tasks.workunit.client.0.smithi153.stdout:9/24: rmdir d1/d7 0 2024-04-05T03:02:40.673 INFO:tasks.workunit.client.1.smithi203.stdout:3/398: fsync d0/d21/f6c 0 2024-04-05T03:02:40.674 INFO:tasks.workunit.client.0.smithi153.stdout:5/77: creat d0/de/f1a x:0 0 0 2024-04-05T03:02:40.675 INFO:tasks.workunit.client.1.smithi203.stdout:1/392: truncate d7/f27 8359503 0 2024-04-05T03:02:40.675 INFO:tasks.workunit.client.0.smithi153.stdout:3/46: write f3 [4605967,26296] 0 2024-04-05T03:02:40.678 INFO:tasks.workunit.client.1.smithi203.stdout:1/393: dread d7/d24/d33/f64 [0,4194304] 0 2024-04-05T03:02:40.680 INFO:tasks.workunit.client.0.smithi153.stdout:6/74: unlink f3 0 2024-04-05T03:02:40.680 INFO:tasks.workunit.client.1.smithi203.stdout:7/413: rename d8/d32/f3a to d8/d32/d16/d1a/d30/f81 0 2024-04-05T03:02:40.681 INFO:tasks.workunit.client.0.smithi153.stdout:3/47: dread f3 [0,4194304] 0 2024-04-05T03:02:40.681 INFO:tasks.workunit.client.1.smithi203.stdout:3/399: symlink d0/dc/d1a/d1d/l71 0 2024-04-05T03:02:40.682 INFO:tasks.workunit.client.1.smithi203.stdout:3/400: readlink d0/l17 0 2024-04-05T03:02:40.682 INFO:tasks.workunit.client.0.smithi153.stdout:3/48: dread f3 [4194304,4194304] 0 2024-04-05T03:02:40.683 INFO:tasks.workunit.client.1.smithi203.stdout:2/556: dread d5/d10/d1a/f1b [0,4194304] 0 2024-04-05T03:02:40.683 INFO:tasks.workunit.client.0.smithi153.stdout:4/110: symlink da/de/l1f 0 2024-04-05T03:02:40.683 INFO:tasks.workunit.client.0.smithi153.stdout:3/49: chown l6 1370 1 2024-04-05T03:02:40.683 INFO:tasks.workunit.client.1.smithi203.stdout:6/451: truncate d0/da/d54/d53/f59 2268855 0 2024-04-05T03:02:40.686 INFO:tasks.workunit.client.0.smithi153.stdout:1/40: creat d5/db/fe x:0 0 0 2024-04-05T03:02:40.686 INFO:tasks.workunit.client.0.smithi153.stdout:1/41: readlink l0 0 2024-04-05T03:02:40.687 INFO:tasks.workunit.client.1.smithi203.stdout:6/452: stat d0/da/d67/f28 0 2024-04-05T03:02:40.688 INFO:tasks.workunit.client.1.smithi203.stdout:7/414: mkdir d8/d2a/d3f/d52/d65/d82 0 2024-04-05T03:02:40.690 INFO:tasks.workunit.client.1.smithi203.stdout:3/401: creat d0/dc/d1a/d1d/f72 x:0 0 0 2024-04-05T03:02:40.692 INFO:tasks.workunit.client.0.smithi153.stdout:0/73: creat d9/db/f13 x:0 0 0 2024-04-05T03:02:40.693 INFO:tasks.workunit.client.1.smithi203.stdout:2/557: rename d5/d10/d35/d2d/f38 to d5/d10/d7f/fc0 0 2024-04-05T03:02:40.693 INFO:tasks.workunit.client.1.smithi203.stdout:6/453: mknod d0/da/d18/c81 0 2024-04-05T03:02:40.694 INFO:tasks.workunit.client.1.smithi203.stdout:7/415: write d8/d32/d16/d1a/d21/f62 [957845,33288] 0 2024-04-05T03:02:40.694 INFO:tasks.workunit.client.1.smithi203.stdout:9/356: dwrite d13/d1b/d33/d42/f49 [0,4194304] 0 2024-04-05T03:02:40.695 INFO:tasks.workunit.client.0.smithi153.stdout:5/78: mknod d0/de/c1b 0 2024-04-05T03:02:40.697 INFO:tasks.workunit.client.1.smithi203.stdout:1/394: dread d7/d24/f35 [0,4194304] 0 2024-04-05T03:02:40.699 INFO:tasks.workunit.client.1.smithi203.stdout:3/402: creat d0/dc/d1a/d1d/d53/f73 x:0 0 0 2024-04-05T03:02:40.700 INFO:tasks.workunit.client.1.smithi203.stdout:2/558: truncate d5/d10/d35/d36/db6/f37 1573023 0 2024-04-05T03:02:40.703 INFO:tasks.workunit.client.1.smithi203.stdout:9/357: symlink d13/d1b/d33/d64/l6c 0 2024-04-05T03:02:40.707 INFO:tasks.workunit.client.1.smithi203.stdout:1/395: getdents d7/d24/d33/d48/d4b/d6b/d74 0 2024-04-05T03:02:40.708 INFO:tasks.workunit.client.1.smithi203.stdout:2/559: creat d5/fc1 x:0 0 0 2024-04-05T03:02:40.714 INFO:tasks.workunit.client.1.smithi203.stdout:6/454: dread d0/da/d54/d53/f59 [0,4194304] 0 2024-04-05T03:02:40.715 INFO:tasks.workunit.client.0.smithi153.stdout:9/25: creat d1/d9/fa x:0 0 0 2024-04-05T03:02:40.716 INFO:tasks.workunit.client.1.smithi203.stdout:7/416: dread d8/d2a/d3f/d52/f5b [0,4194304] 0 2024-04-05T03:02:40.717 INFO:tasks.workunit.client.1.smithi203.stdout:9/358: link d13/d1b/f47 d13/d1b/d33/d64/f6d 0 2024-04-05T03:02:40.717 INFO:tasks.workunit.client.0.smithi153.stdout:0/74: unlink l1 0 2024-04-05T03:02:40.719 INFO:tasks.workunit.client.1.smithi203.stdout:7/417: chown d8/d32/f56 311105089 1 2024-04-05T03:02:40.720 INFO:tasks.workunit.client.1.smithi203.stdout:6/455: chown d0/da/d18/d38/d2b/f5a 309031229 1 2024-04-05T03:02:40.720 INFO:tasks.workunit.client.1.smithi203.stdout:1/396: rmdir d7/d24/d33/d48/d4b/d6b/d74 0 2024-04-05T03:02:40.720 INFO:tasks.workunit.client.1.smithi203.stdout:2/560: dwrite d5/d10/d35/d36/db6/fa7 [0,4194304] 0 2024-04-05T03:02:40.721 INFO:tasks.workunit.client.0.smithi153.stdout:6/75: chown d4/f6 188 1 2024-04-05T03:02:40.721 INFO:tasks.workunit.client.1.smithi203.stdout:6/456: chown d0/f3 4987 1 2024-04-05T03:02:40.722 INFO:tasks.workunit.client.0.smithi153.stdout:8/20: rmdir d1 0 2024-04-05T03:02:40.723 INFO:tasks.workunit.client.1.smithi203.stdout:9/359: creat d13/d1b/d33/d63/f6e x:0 0 0 2024-04-05T03:02:40.723 INFO:tasks.workunit.client.1.smithi203.stdout:9/360: fdatasync fb 0 2024-04-05T03:02:40.723 INFO:tasks.workunit.client.1.smithi203.stdout:3/403: truncate d0/dc/d1a/d33/d35/f3d 3972706 0 2024-04-05T03:02:40.724 INFO:tasks.workunit.client.0.smithi153.stdout:4/111: unlink da/db/l1b 0 2024-04-05T03:02:40.727 INFO:tasks.workunit.client.1.smithi203.stdout:4/323: sync 2024-04-05T03:02:40.729 INFO:tasks.workunit.client.1.smithi203.stdout:6/457: fdatasync d0/da/d18/d38/d2a/d44/f4a 0 2024-04-05T03:02:40.730 INFO:tasks.workunit.client.0.smithi153.stdout:6/76: rename d4/dc/df/d16/c1b to d4/dc/df/c1c 0 2024-04-05T03:02:40.731 INFO:tasks.workunit.client.1.smithi203.stdout:7/418: mknod d8/d32/d16/d1a/d21/c83 0 2024-04-05T03:02:40.731 INFO:tasks.workunit.client.1.smithi203.stdout:3/404: dread - d0/dc/d34/f6e zero size 2024-04-05T03:02:40.732 INFO:tasks.workunit.client.0.smithi153.stdout:6/77: read d4/dc/f14 [204046,1821] 0 2024-04-05T03:02:40.733 INFO:tasks.workunit.client.0.smithi153.stdout:4/112: mkdir da/db/d20 0 2024-04-05T03:02:40.733 INFO:tasks.workunit.client.0.smithi153.stdout:6/78: dread - d4/f7 zero size 2024-04-05T03:02:40.733 INFO:tasks.workunit.client.1.smithi203.stdout:9/361: creat d13/d1b/d26/f6f x:0 0 0 2024-04-05T03:02:40.736 INFO:tasks.workunit.client.1.smithi203.stdout:6/458: chown d0/da/d67/d71/f79 35 1 2024-04-05T03:02:40.737 INFO:tasks.workunit.client.1.smithi203.stdout:7/419: creat d8/d2a/f84 x:0 0 0 2024-04-05T03:02:40.738 INFO:tasks.workunit.client.1.smithi203.stdout:9/362: chown d13/f24 1238 1 2024-04-05T03:02:40.738 INFO:tasks.workunit.client.1.smithi203.stdout:1/397: dwrite d7/d24/d33/f36 [4194304,4194304] 0 2024-04-05T03:02:40.741 INFO:tasks.workunit.client.1.smithi203.stdout:6/459: chown d0/da/d18/d38/d2a/d2c/f46 6131575 1 2024-04-05T03:02:40.741 INFO:tasks.workunit.client.1.smithi203.stdout:7/420: dread - d8/d2a/f84 zero size 2024-04-05T03:02:40.741 INFO:tasks.workunit.client.1.smithi203.stdout:4/324: link da/f13 da/d29/d6a/f6b 0 2024-04-05T03:02:40.741 INFO:tasks.workunit.client.1.smithi203.stdout:3/405: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:40.742 INFO:tasks.workunit.client.1.smithi203.stdout:9/363: mknod d13/d1b/d33/d42/c70 0 2024-04-05T03:02:40.743 INFO:tasks.workunit.client.1.smithi203.stdout:7/421: write d8/f4c [491452,121800] 0 2024-04-05T03:02:40.745 INFO:tasks.workunit.client.1.smithi203.stdout:9/364: chown f8 0 1 2024-04-05T03:02:40.747 INFO:tasks.workunit.client.1.smithi203.stdout:9/365: dread - d13/d1b/d33/f65 zero size 2024-04-05T03:02:40.749 INFO:tasks.workunit.client.1.smithi203.stdout:1/398: mknod d7/d14/c7b 0 2024-04-05T03:02:40.750 INFO:tasks.workunit.client.1.smithi203.stdout:2/561: write d5/f5e [2543724,85483] 0 2024-04-05T03:02:40.751 INFO:tasks.workunit.client.1.smithi203.stdout:7/422: creat d8/d32/d16/d1a/d5a/f85 x:0 0 0 2024-04-05T03:02:40.751 INFO:tasks.workunit.client.1.smithi203.stdout:3/406: symlink d0/dc/d38/d5d/d66/l74 0 2024-04-05T03:02:40.751 INFO:tasks.workunit.client.1.smithi203.stdout:3/407: stat d0 0 2024-04-05T03:02:40.752 INFO:tasks.workunit.client.1.smithi203.stdout:4/325: write da/f1b [4979610,91100] 0 2024-04-05T03:02:40.753 INFO:tasks.workunit.client.1.smithi203.stdout:9/366: creat d13/d1b/d26/d31/d38/d59/d5f/f71 x:0 0 0 2024-04-05T03:02:40.754 INFO:tasks.workunit.client.1.smithi203.stdout:7/423: rmdir d8 39 2024-04-05T03:02:40.758 INFO:tasks.workunit.client.1.smithi203.stdout:3/408: read d0/ff [2629667,28933] 0 2024-04-05T03:02:40.758 INFO:tasks.workunit.client.1.smithi203.stdout:1/399: creat d7/d24/d33/d48/d4b/d51/f7c x:0 0 0 2024-04-05T03:02:40.758 INFO:tasks.workunit.client.1.smithi203.stdout:3/409: write d0/dc/d34/f6e [77476,107891] 0 2024-04-05T03:02:40.759 INFO:tasks.workunit.client.1.smithi203.stdout:3/410: dread - d0/dc/d3f/f58 zero size 2024-04-05T03:02:40.766 INFO:tasks.workunit.client.1.smithi203.stdout:3/411: creat d0/dc/d1a/d1d/d53/f75 x:0 0 0 2024-04-05T03:02:40.768 INFO:tasks.workunit.client.1.smithi203.stdout:1/400: rename d7/f23 to d7/d79/f7d 0 2024-04-05T03:02:40.771 INFO:tasks.workunit.client.1.smithi203.stdout:1/401: read d7/d24/d33/d48/d4b/d51/f67 [3410118,82940] 0 2024-04-05T03:02:40.774 INFO:tasks.workunit.client.1.smithi203.stdout:4/326: dread da/d29/f40 [0,4194304] 0 2024-04-05T03:02:40.775 INFO:tasks.workunit.client.1.smithi203.stdout:9/367: dread d13/d1b/d26/d31/f41 [0,4194304] 0 2024-04-05T03:02:40.781 INFO:tasks.workunit.client.1.smithi203.stdout:4/327: dwrite da/d29/d2e/f35 [0,4194304] 0 2024-04-05T03:02:40.800 INFO:tasks.workunit.client.1.smithi203.stdout:3/412: rename d0/dc/d1a/d33 to d0/dc/d1a/d76 0 2024-04-05T03:02:40.801 INFO:tasks.workunit.client.1.smithi203.stdout:2/562: getdents d5/d10/d35/d2d/d7c/d29/d67 0 2024-04-05T03:02:40.808 INFO:tasks.workunit.client.1.smithi203.stdout:4/328: fdatasync da/f11 0 2024-04-05T03:02:40.810 INFO:tasks.workunit.client.1.smithi203.stdout:7/424: write d8/f20 [62596,19631] 0 2024-04-05T03:02:40.811 INFO:tasks.workunit.client.1.smithi203.stdout:9/368: creat d13/d1b/d6b/f72 x:0 0 0 2024-04-05T03:02:40.815 INFO:tasks.workunit.client.0.smithi153.stdout:1/42: rmdir d5/db 39 2024-04-05T03:02:40.816 INFO:tasks.workunit.client.1.smithi203.stdout:2/563: mknod d5/d10/d35/d2d/d7c/d29/d67/d9e/cc2 0 2024-04-05T03:02:40.817 INFO:tasks.workunit.client.1.smithi203.stdout:9/369: mknod d13/c73 0 2024-04-05T03:02:40.819 INFO:tasks.workunit.client.0.smithi153.stdout:1/43: link f3 d5/d8/ff 0 2024-04-05T03:02:40.820 INFO:tasks.workunit.client.0.smithi153.stdout:1/44: write f4 [295844,32271] 0 2024-04-05T03:02:40.821 INFO:tasks.workunit.client.0.smithi153.stdout:1/45: write f2 [1024917,118282] 0 2024-04-05T03:02:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:40 smithi153 ceph-mon[29381]: pgmap v5: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:40 smithi153 ceph-mon[29381]: mgrmap e24: smithi203.iakodz(active, since 4s) 2024-04-05T03:02:40.822 INFO:tasks.workunit.client.1.smithi203.stdout:2/564: mknod d5/d10/db8/cc3 0 2024-04-05T03:02:40.823 INFO:tasks.workunit.client.0.smithi153.stdout:7/82: truncate d2/da/d11/d12/f19 250683 0 2024-04-05T03:02:40.824 INFO:tasks.workunit.client.0.smithi153.stdout:7/83: fsync d2/f5 0 2024-04-05T03:02:40.827 INFO:tasks.workunit.client.0.smithi153.stdout:1/46: mknod d5/c10 0 2024-04-05T03:02:40.827 INFO:tasks.workunit.client.1.smithi203.stdout:4/329: dwrite da/d29/f47 [0,4194304] 0 2024-04-05T03:02:40.827 INFO:tasks.workunit.client.0.smithi153.stdout:1/47: chown f4 374 1 2024-04-05T03:02:40.828 INFO:tasks.workunit.client.0.smithi153.stdout:7/84: fsync d2/f6 0 2024-04-05T03:02:40.831 INFO:tasks.workunit.client.0.smithi153.stdout:1/48: read f4 [234646,82935] 0 2024-04-05T03:02:40.831 INFO:tasks.workunit.client.1.smithi203.stdout:7/425: dwrite d8/d32/d16/d1a/d3d/f6b [0,4194304] 0 2024-04-05T03:02:40.831 INFO:tasks.workunit.client.1.smithi203.stdout:9/370: link d13/d1b/c30 d13/d1b/d26/d31/d38/c74 0 2024-04-05T03:02:40.832 INFO:tasks.workunit.client.1.smithi203.stdout:2/565: creat d5/d10/d35/d2d/d7c/d29/d82/fc4 x:0 0 0 2024-04-05T03:02:40.833 INFO:tasks.workunit.client.0.smithi153.stdout:3/50: truncate f3 3281616 0 2024-04-05T03:02:40.837 INFO:tasks.workunit.client.1.smithi203.stdout:1/402: dwrite d7/d24/d33/d48/d4b/d6b/f75 [0,4194304] 0 2024-04-05T03:02:40.838 INFO:tasks.workunit.client.1.smithi203.stdout:9/371: truncate d13/d1b/d26/d31/d38/f68 4799031 0 2024-04-05T03:02:40.839 INFO:tasks.workunit.client.0.smithi153.stdout:8/21: truncate f0 776000 0 2024-04-05T03:02:40.841 INFO:tasks.workunit.client.0.smithi153.stdout:7/85: dwrite d2/da/d11/d12/f1e [0,4194304] 0 2024-04-05T03:02:40.842 INFO:tasks.workunit.client.1.smithi203.stdout:9/372: write f7 [1644334,54574] 0 2024-04-05T03:02:40.844 INFO:tasks.workunit.client.0.smithi153.stdout:3/51: rename l6 to lf 0 2024-04-05T03:02:40.844 INFO:tasks.workunit.client.0.smithi153.stdout:7/86: dread - d2/da/d11/f13 zero size 2024-04-05T03:02:40.845 INFO:tasks.workunit.client.1.smithi203.stdout:4/330: creat da/d29/d6a/f6c x:0 0 0 2024-04-05T03:02:40.846 INFO:tasks.workunit.client.0.smithi153.stdout:2/39: write d9/da/fc [1154663,130060] 0 2024-04-05T03:02:40.847 INFO:tasks.workunit.client.1.smithi203.stdout:7/426: dread d8/d32/d16/d1a/d30/f38 [0,4194304] 0 2024-04-05T03:02:40.847 INFO:tasks.workunit.client.0.smithi153.stdout:1/49: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:40.849 INFO:tasks.workunit.client.0.smithi153.stdout:1/50: readlink l0 0 2024-04-05T03:02:40.853 INFO:tasks.workunit.client.1.smithi203.stdout:2/566: rename d5/d10/d44/c52 to d5/d10/d35/d36/d41/cc5 0 2024-04-05T03:02:40.854 INFO:tasks.workunit.client.1.smithi203.stdout:2/567: read - d5/d63/f75 zero size 2024-04-05T03:02:40.855 INFO:tasks.workunit.client.1.smithi203.stdout:4/331: creat da/d3c/f6d x:0 0 0 2024-04-05T03:02:40.856 INFO:tasks.workunit.client.1.smithi203.stdout:4/332: chown da/d3c/d4c 11 1 2024-04-05T03:02:40.858 INFO:tasks.workunit.client.0.smithi153.stdout:6/79: dwrite d4/fb [0,4194304] 0 2024-04-05T03:02:40.859 INFO:tasks.workunit.client.1.smithi203.stdout:9/373: link d13/d1b/d26/f4c d13/d1b/d33/d64/f75 0 2024-04-05T03:02:40.861 INFO:tasks.workunit.client.1.smithi203.stdout:7/427: symlink d8/d32/l86 0 2024-04-05T03:02:40.861 INFO:tasks.workunit.client.0.smithi153.stdout:3/52: rename cc to c10 0 2024-04-05T03:02:40.862 INFO:tasks.workunit.client.1.smithi203.stdout:2/568: symlink d5/lc6 0 2024-04-05T03:02:40.863 INFO:tasks.workunit.client.1.smithi203.stdout:7/428: readlink d8/d32/d16/d35/l74 0 2024-04-05T03:02:40.863 INFO:tasks.workunit.client.0.smithi153.stdout:2/40: link d9/da/fb d9/da/f15 0 2024-04-05T03:02:40.865 INFO:tasks.workunit.client.1.smithi203.stdout:2/569: truncate d5/d10/d1a/f65 614808 0 2024-04-05T03:02:40.866 INFO:tasks.workunit.client.1.smithi203.stdout:9/374: mknod d13/d1b/d26/d31/c76 0 2024-04-05T03:02:40.867 INFO:tasks.workunit.client.1.smithi203.stdout:9/375: chown d13/c73 25691 1 2024-04-05T03:02:40.868 INFO:tasks.workunit.client.0.smithi153.stdout:6/80: symlink d4/l1d 0 2024-04-05T03:02:40.870 INFO:tasks.workunit.client.0.smithi153.stdout:3/53: chown c1 656448 1 2024-04-05T03:02:40.871 INFO:tasks.workunit.client.1.smithi203.stdout:9/376: mkdir d13/d1b/d26/d31/d38/d77 0 2024-04-05T03:02:40.872 INFO:tasks.workunit.client.0.smithi153.stdout:2/41: mknod d9/da/d10/c16 0 2024-04-05T03:02:40.875 INFO:tasks.workunit.client.1.smithi203.stdout:2/570: getdents d5/d10/d1a/d57 0 2024-04-05T03:02:40.875 INFO:tasks.workunit.client.0.smithi153.stdout:6/81: mknod d4/dc/df/d16/c1e 0 2024-04-05T03:02:40.876 INFO:tasks.workunit.client.1.smithi203.stdout:1/403: dread d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:40.878 INFO:tasks.workunit.client.1.smithi203.stdout:2/571: chown d5/d10/d35/d36/db6/d87/f89 10209 1 2024-04-05T03:02:40.878 INFO:tasks.workunit.client.1.smithi203.stdout:1/404: chown d7/d14/c7b 622 1 2024-04-05T03:02:40.878 INFO:tasks.workunit.client.0.smithi153.stdout:3/54: dwrite fd [4194304,4194304] 0 2024-04-05T03:02:40.881 INFO:tasks.workunit.client.1.smithi203.stdout:1/405: write d7/fa [921782,39429] 0 2024-04-05T03:02:40.882 INFO:tasks.workunit.client.0.smithi153.stdout:2/42: write d9/da/fb [455118,130164] 0 2024-04-05T03:02:40.883 INFO:tasks.workunit.client.1.smithi203.stdout:1/406: chown d7/d14/l26 31 1 2024-04-05T03:02:40.884 INFO:tasks.workunit.client.0.smithi153.stdout:7/87: fdatasync d2/da/d11/d12/f1e 0 2024-04-05T03:02:40.885 INFO:tasks.workunit.client.1.smithi203.stdout:2/572: rename d5/d10/d2f/d5d/c60 to d5/d10/d35/d2d/d7c/d29/db1/cc7 0 2024-04-05T03:02:40.885 INFO:tasks.workunit.client.0.smithi153.stdout:2/43: dread - d9/f11 zero size 2024-04-05T03:02:40.885 INFO:tasks.workunit.client.0.smithi153.stdout:7/88: fsync d2/f3 0 2024-04-05T03:02:40.887 INFO:tasks.workunit.client.0.smithi153.stdout:7/89: fdatasync d2/f5 0 2024-04-05T03:02:40.892 INFO:tasks.workunit.client.1.smithi203.stdout:2/573: symlink d5/d10/d35/d2d/d7c/d72/d8f/lc8 0 2024-04-05T03:02:40.893 INFO:tasks.workunit.client.0.smithi153.stdout:1/51: truncate f3 425354 0 2024-04-05T03:02:40.894 INFO:tasks.workunit.client.1.smithi203.stdout:4/333: dread da/f13 [0,4194304] 0 2024-04-05T03:02:40.895 INFO:tasks.workunit.client.1.smithi203.stdout:1/407: dwrite d7/d24/d33/d48/d4b/d6b/f75 [0,4194304] 0 2024-04-05T03:02:40.895 INFO:tasks.workunit.client.1.smithi203.stdout:9/377: getdents d13/d1b/d33/d64 0 2024-04-05T03:02:40.896 INFO:tasks.workunit.client.1.smithi203.stdout:4/334: write da/d29/f47 [4538169,23474] 0 2024-04-05T03:02:40.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/574: symlink d5/d10/d2f/d5d/lc9 0 2024-04-05T03:02:40.900 INFO:tasks.workunit.client.1.smithi203.stdout:4/335: dread - da/d3c/f49 zero size 2024-04-05T03:02:40.901 INFO:tasks.workunit.client.1.smithi203.stdout:2/575: chown d5/d10/d44/lb4 3 1 2024-04-05T03:02:40.903 INFO:tasks.workunit.client.1.smithi203.stdout:1/408: truncate d7/f4f 2762907 0 2024-04-05T03:02:40.904 INFO:tasks.workunit.client.1.smithi203.stdout:7/429: truncate d8/d2a/d3f/d52/f45 5521785 0 2024-04-05T03:02:40.904 INFO:tasks.workunit.client.1.smithi203.stdout:9/378: dread d13/f20 [4194304,4194304] 0 2024-04-05T03:02:40.906 INFO:tasks.workunit.client.1.smithi203.stdout:9/379: fdatasync d13/d1b/d26/d31/d38/f5e 0 2024-04-05T03:02:40.906 INFO:tasks.workunit.client.1.smithi203.stdout:7/430: dread - d8/d32/d16/d1a/d5a/f85 zero size 2024-04-05T03:02:40.906 INFO:tasks.workunit.client.0.smithi153.stdout:3/55: fsync fd 0 2024-04-05T03:02:40.907 INFO:tasks.workunit.client.0.smithi153.stdout:2/44: getdents d9 0 2024-04-05T03:02:40.907 INFO:tasks.workunit.client.1.smithi203.stdout:7/431: dread - d8/d32/f70 zero size 2024-04-05T03:02:40.910 INFO:tasks.workunit.client.0.smithi153.stdout:2/45: write d9/da/fb [676393,51992] 0 2024-04-05T03:02:40.910 INFO:tasks.workunit.client.0.smithi153.stdout:3/56: symlink l11 0 2024-04-05T03:02:40.911 INFO:tasks.workunit.client.1.smithi203.stdout:9/380: chown d13/d1b/c1f 189115 1 2024-04-05T03:02:40.913 INFO:tasks.workunit.client.1.smithi203.stdout:7/432: creat d8/d2a/d3f/d5e/d77/f87 x:0 0 0 2024-04-05T03:02:40.914 INFO:tasks.workunit.client.1.smithi203.stdout:9/381: fsync d13/d1b/d26/d31/f41 0 2024-04-05T03:02:40.916 INFO:tasks.workunit.client.0.smithi153.stdout:2/46: truncate d9/f11 745565 0 2024-04-05T03:02:40.918 INFO:tasks.workunit.client.1.smithi203.stdout:2/576: rmdir d5/d10/d35/d2d/d7c/d72/d8f 39 2024-04-05T03:02:40.919 INFO:tasks.workunit.client.0.smithi153.stdout:8/22: dwrite f0 [0,4194304] 0 2024-04-05T03:02:40.920 INFO:tasks.workunit.client.1.smithi203.stdout:7/433: symlink d8/d2a/d3f/d5e/d63/l88 0 2024-04-05T03:02:40.920 INFO:tasks.workunit.client.0.smithi153.stdout:8/23: chown f0 1 1 2024-04-05T03:02:40.920 INFO:tasks.workunit.client.1.smithi203.stdout:2/577: truncate d5/d10/d35/d2d/d7c/d72/fa0 943372 0 2024-04-05T03:02:40.921 INFO:tasks.workunit.client.1.smithi203.stdout:4/336: dread da/f26 [0,4194304] 0 2024-04-05T03:02:40.921 INFO:tasks.workunit.client.0.smithi153.stdout:2/47: read d9/da/f15 [202117,34586] 0 2024-04-05T03:02:40.921 INFO:tasks.workunit.client.1.smithi203.stdout:1/409: link d7/d14/l26 d7/d24/l7e 0 2024-04-05T03:02:40.927 INFO:tasks.workunit.client.0.smithi153.stdout:7/90: dwrite d2/f1d [0,4194304] 0 2024-04-05T03:02:40.927 INFO:tasks.workunit.client.0.smithi153.stdout:1/52: chown d5/d8/ff 42634032 1 2024-04-05T03:02:40.929 INFO:tasks.workunit.client.0.smithi153.stdout:3/57: mknod c12 0 2024-04-05T03:02:40.930 INFO:tasks.workunit.client.1.smithi203.stdout:4/337: dwrite da/f11 [0,4194304] 0 2024-04-05T03:02:40.930 INFO:tasks.workunit.client.0.smithi153.stdout:8/24: creat f3 x:0 0 0 2024-04-05T03:02:40.931 INFO:tasks.workunit.client.1.smithi203.stdout:7/434: creat d8/d32/d16/d1a/d30/f89 x:0 0 0 2024-04-05T03:02:40.932 INFO:tasks.workunit.client.0.smithi153.stdout:2/48: symlink d9/da/d10/l17 0 2024-04-05T03:02:40.932 INFO:tasks.workunit.client.0.smithi153.stdout:8/25: dread - f3 zero size 2024-04-05T03:02:40.932 INFO:tasks.workunit.client.1.smithi203.stdout:1/410: dwrite d7/d14/d2b/f72 [0,4194304] 0 2024-04-05T03:02:40.933 INFO:tasks.workunit.client.0.smithi153.stdout:7/91: write d2/f9 [52584,89585] 0 2024-04-05T03:02:40.937 INFO:tasks.workunit.client.1.smithi203.stdout:2/578: mknod d5/d10/d35/d2d/d7c/d29/d67/d9e/cca 0 2024-04-05T03:02:40.937 INFO:tasks.workunit.client.0.smithi153.stdout:8/26: creat f4 x:0 0 0 2024-04-05T03:02:40.938 INFO:tasks.workunit.client.0.smithi153.stdout:2/49: write d9/da/fb [691341,121433] 0 2024-04-05T03:02:40.941 INFO:tasks.workunit.client.1.smithi203.stdout:4/338: fdatasync da/d29/d2e/f63 0 2024-04-05T03:02:40.943 INFO:tasks.workunit.client.1.smithi203.stdout:9/382: rename d13/d1b/d26/l39 to d13/d1b/d26/d31/l78 0 2024-04-05T03:02:40.944 INFO:tasks.workunit.client.0.smithi153.stdout:7/92: creat d2/da/d1f/f20 x:0 0 0 2024-04-05T03:02:40.945 INFO:tasks.workunit.client.0.smithi153.stdout:7/93: stat d2/da/d11 0 2024-04-05T03:02:40.946 INFO:tasks.workunit.client.1.smithi203.stdout:2/579: creat d5/d66/fcb x:0 0 0 2024-04-05T03:02:40.946 INFO:tasks.workunit.client.1.smithi203.stdout:4/339: mknod da/d29/d2e/d53/c6e 0 2024-04-05T03:02:40.948 INFO:tasks.workunit.client.0.smithi153.stdout:2/50: creat d9/da/f18 x:0 0 0 2024-04-05T03:02:40.950 INFO:tasks.workunit.client.0.smithi153.stdout:1/53: getdents d5 0 2024-04-05T03:02:40.950 INFO:tasks.workunit.client.1.smithi203.stdout:4/340: write da/d29/d6a/f6b [2864457,107870] 0 2024-04-05T03:02:40.954 INFO:tasks.workunit.client.1.smithi203.stdout:2/580: symlink d5/d10/d35/d2d/d7c/d29/d82/lcc 0 2024-04-05T03:02:40.955 INFO:tasks.workunit.client.1.smithi203.stdout:2/581: write d5/d10/d35/d36/db6/d87/faa [840590,98757] 0 2024-04-05T03:02:40.956 INFO:tasks.workunit.client.0.smithi153.stdout:8/27: link f3 f5 0 2024-04-05T03:02:40.956 INFO:tasks.workunit.client.1.smithi203.stdout:7/435: read d8/d2a/d3f/d52/f45 [5089236,102519] 0 2024-04-05T03:02:40.957 INFO:tasks.workunit.client.0.smithi153.stdout:8/28: fsync f0 0 2024-04-05T03:02:40.958 INFO:tasks.workunit.client.0.smithi153.stdout:2/51: dwrite d9/ff [0,4194304] 0 2024-04-05T03:02:40.961 INFO:tasks.workunit.client.1.smithi203.stdout:7/436: truncate d8/f31 1420629 0 2024-04-05T03:02:40.962 INFO:tasks.workunit.client.0.smithi153.stdout:1/54: link f2 d5/db/f11 0 2024-04-05T03:02:40.962 INFO:tasks.workunit.client.0.smithi153.stdout:8/29: creat f6 x:0 0 0 2024-04-05T03:02:40.962 INFO:tasks.workunit.client.0.smithi153.stdout:1/55: chown f2 21529 1 2024-04-05T03:02:40.962 INFO:tasks.workunit.client.0.smithi153.stdout:2/52: dread d9/f11 [0,4194304] 0 2024-04-05T03:02:40.963 INFO:tasks.workunit.client.1.smithi203.stdout:2/582: symlink d5/d10/d35/d2d/d7c/d7a/dbd/lcd 0 2024-04-05T03:02:40.963 INFO:tasks.workunit.client.0.smithi153.stdout:2/53: write d9/da/fe [836860,76084] 0 2024-04-05T03:02:40.964 INFO:tasks.workunit.client.0.smithi153.stdout:1/56: mkdir d5/d12 0 2024-04-05T03:02:40.964 INFO:tasks.workunit.client.1.smithi203.stdout:7/437: mkdir d8/d32/d8a 0 2024-04-05T03:02:40.965 INFO:tasks.workunit.client.0.smithi153.stdout:2/54: chown d9/da/d10/l14 51248 1 2024-04-05T03:02:40.966 INFO:tasks.workunit.client.1.smithi203.stdout:2/583: mknod d5/d10/d7f/cce 0 2024-04-05T03:02:40.967 INFO:tasks.workunit.client.0.smithi153.stdout:8/30: dread f0 [0,4194304] 0 2024-04-05T03:02:40.967 INFO:tasks.workunit.client.0.smithi153.stdout:8/31: rmdir - no directory 2024-04-05T03:02:40.967 INFO:tasks.workunit.client.1.smithi203.stdout:5/435: sync 2024-04-05T03:02:40.968 INFO:tasks.workunit.client.1.smithi203.stdout:8/494: sync 2024-04-05T03:02:40.968 INFO:tasks.workunit.client.0.smithi153.stdout:2/55: dread d9/da/fb [0,4194304] 0 2024-04-05T03:02:40.968 INFO:tasks.workunit.client.0.smithi153.stdout:8/32: chown f4 15388 1 2024-04-05T03:02:40.969 INFO:tasks.workunit.client.0.smithi153.stdout:8/33: dread - f4 zero size 2024-04-05T03:02:40.969 INFO:tasks.workunit.client.0.smithi153.stdout:8/34: rmdir - no directory 2024-04-05T03:02:40.970 INFO:tasks.workunit.client.1.smithi203.stdout:7/438: creat d8/d2a/d3f/d52/f8b x:0 0 0 2024-04-05T03:02:40.970 INFO:tasks.workunit.client.1.smithi203.stdout:8/495: write d1/d12/d1a/f46 [1385037,69950] 0 2024-04-05T03:02:40.972 INFO:tasks.workunit.client.1.smithi203.stdout:2/584: symlink d5/d10/d35/d2d/d7c/d7a/lcf 0 2024-04-05T03:02:40.972 INFO:tasks.workunit.client.0.smithi153.stdout:5/79: sync 2024-04-05T03:02:40.974 INFO:tasks.workunit.client.1.smithi203.stdout:1/411: write d7/d14/f1e [9172690,56477] 0 2024-04-05T03:02:40.974 INFO:tasks.workunit.client.0.smithi153.stdout:8/35: mkdir d7 0 2024-04-05T03:02:40.975 INFO:tasks.workunit.client.1.smithi203.stdout:2/585: write d5/d66/fa9 [873004,24280] 0 2024-04-05T03:02:40.975 INFO:tasks.workunit.client.0.smithi153.stdout:8/36: dread - f5 zero size 2024-04-05T03:02:40.975 INFO:tasks.workunit.client.0.smithi153.stdout:3/58: write f3 [187859,39065] 0 2024-04-05T03:02:40.975 INFO:tasks.workunit.client.0.smithi153.stdout:2/56: dread d9/da/fe [0,4194304] 0 2024-04-05T03:02:40.975 INFO:tasks.workunit.client.1.smithi203.stdout:7/439: creat d8/d2a/f8c x:0 0 0 2024-04-05T03:02:40.977 INFO:tasks.workunit.client.1.smithi203.stdout:8/496: mkdir d1/d12/d2f/d1f/d83 0 2024-04-05T03:02:40.978 INFO:tasks.workunit.client.0.smithi153.stdout:2/57: write d9/da/f18 [939001,117523] 0 2024-04-05T03:02:40.979 INFO:tasks.workunit.client.1.smithi203.stdout:1/412: creat d7/d24/d33/d48/d4b/d51/f7f x:0 0 0 2024-04-05T03:02:40.980 INFO:tasks.workunit.client.0.smithi153.stdout:3/59: dread f3 [0,4194304] 0 2024-04-05T03:02:40.981 INFO:tasks.workunit.client.0.smithi153.stdout:3/60: read fd [4465721,84637] 0 2024-04-05T03:02:40.981 INFO:tasks.workunit.client.0.smithi153.stdout:3/61: rmdir - no directory 2024-04-05T03:02:40.983 INFO:tasks.workunit.client.1.smithi203.stdout:8/497: getdents d1/d4b 0 2024-04-05T03:02:40.985 INFO:tasks.workunit.client.0.smithi153.stdout:2/58: mkdir d9/da/d19 0 2024-04-05T03:02:40.986 INFO:tasks.workunit.client.1.smithi203.stdout:8/498: write d1/f34 [2278166,110229] 0 2024-04-05T03:02:40.987 INFO:tasks.workunit.client.1.smithi203.stdout:7/440: dread d8/d32/f18 [0,4194304] 0 2024-04-05T03:02:40.988 INFO:tasks.workunit.client.0.smithi153.stdout:5/80: creat d0/f1c x:0 0 0 2024-04-05T03:02:40.989 INFO:tasks.workunit.client.1.smithi203.stdout:2/586: dwrite d5/d10/d35/d2d/d7c/d29/db1/fb7 [0,4194304] 0 2024-04-05T03:02:40.989 INFO:tasks.workunit.client.0.smithi153.stdout:7/94: dwrite d2/da/d11/d12/f19 [0,4194304] 0 2024-04-05T03:02:40.991 INFO:tasks.workunit.client.1.smithi203.stdout:1/413: rename d7/d24/f38 to d7/f80 0 2024-04-05T03:02:40.992 INFO:tasks.workunit.client.1.smithi203.stdout:5/436: dread d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:40.993 INFO:tasks.workunit.client.0.smithi153.stdout:3/62: symlink l13 0 2024-04-05T03:02:40.993 INFO:tasks.workunit.client.0.smithi153.stdout:8/37: link f4 d7/f8 0 2024-04-05T03:02:40.993 INFO:tasks.workunit.client.1.smithi203.stdout:8/499: mknod d1/d12/d1a/c84 0 2024-04-05T03:02:40.995 INFO:tasks.workunit.client.1.smithi203.stdout:5/437: write d3/f7 [7290466,104371] 0 2024-04-05T03:02:40.999 INFO:tasks.workunit.client.1.smithi203.stdout:5/438: unlink d3/d5f/c8d 0 2024-04-05T03:02:41.001 INFO:tasks.workunit.client.0.smithi153.stdout:5/81: chown d0/lf 56 1 2024-04-05T03:02:41.001 INFO:tasks.workunit.client.1.smithi203.stdout:5/439: write d3/f7 [1326126,82221] 0 2024-04-05T03:02:41.002 INFO:tasks.workunit.client.1.smithi203.stdout:2/587: getdents d5/d10/db8/dab 0 2024-04-05T03:02:41.004 INFO:tasks.workunit.client.1.smithi203.stdout:7/441: dwrite d8/d32/d16/d1a/d21/f62 [0,4194304] 0 2024-04-05T03:02:41.004 INFO:tasks.workunit.client.0.smithi153.stdout:2/59: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:41.006 INFO:tasks.workunit.client.0.smithi153.stdout:8/38: mknod d7/c9 0 2024-04-05T03:02:41.006 INFO:tasks.workunit.client.1.smithi203.stdout:7/442: fsync d8/f3e 0 2024-04-05T03:02:41.007 INFO:tasks.workunit.client.0.smithi153.stdout:8/39: dread - f4 zero size 2024-04-05T03:02:41.008 INFO:tasks.workunit.client.0.smithi153.stdout:5/82: mknod d0/de/c1d 0 2024-04-05T03:02:41.009 INFO:tasks.workunit.client.0.smithi153.stdout:7/95: dwrite d2/f1d [0,4194304] 0 2024-04-05T03:02:41.011 INFO:tasks.workunit.client.0.smithi153.stdout:1/57: creat d5/dd/f13 x:0 0 0 2024-04-05T03:02:41.011 INFO:tasks.workunit.client.1.smithi203.stdout:2/588: unlink d5/d10/d35/d2d/d7c/d29/d67/d9e/cc2 0 2024-04-05T03:02:41.012 INFO:tasks.workunit.client.0.smithi153.stdout:1/58: dread - d5/dd/f13 zero size 2024-04-05T03:02:41.015 INFO:tasks.workunit.client.0.smithi153.stdout:7/96: rmdir d2/da/d1f 39 2024-04-05T03:02:41.020 INFO:tasks.workunit.client.0.smithi153.stdout:8/40: dwrite f6 [0,4194304] 0 2024-04-05T03:02:41.021 INFO:tasks.workunit.client.1.smithi203.stdout:2/589: creat d5/d10/d35/d73/fd0 x:0 0 0 2024-04-05T03:02:41.023 INFO:tasks.workunit.client.0.smithi153.stdout:1/59: write d5/f9 [2006931,97011] 0 2024-04-05T03:02:41.023 INFO:tasks.workunit.client.1.smithi203.stdout:5/440: creat d3/d1f/d59/d64/f91 x:0 0 0 2024-04-05T03:02:41.024 INFO:tasks.workunit.client.1.smithi203.stdout:7/443: dwrite d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:41.027 INFO:tasks.workunit.client.0.smithi153.stdout:7/97: write d2/da/d1f/f10 [578786,70220] 0 2024-04-05T03:02:41.027 INFO:tasks.workunit.client.1.smithi203.stdout:2/590: unlink d5/d10/d7f/f99 0 2024-04-05T03:02:41.028 INFO:tasks.workunit.client.1.smithi203.stdout:5/441: write d3/d1f/d49/f57 [129097,67277] 0 2024-04-05T03:02:41.029 INFO:tasks.workunit.client.1.smithi203.stdout:2/591: chown d5/d10/d1a/f4b 6 1 2024-04-05T03:02:41.031 INFO:tasks.workunit.client.0.smithi153.stdout:8/41: creat d7/fa x:0 0 0 2024-04-05T03:02:41.033 INFO:tasks.workunit.client.0.smithi153.stdout:8/42: chown d7/c9 50 1 2024-04-05T03:02:41.033 INFO:tasks.workunit.client.1.smithi203.stdout:7/444: truncate d8/da/f14 104987 0 2024-04-05T03:02:41.036 INFO:tasks.workunit.client.0.smithi153.stdout:8/43: truncate d7/fa 801178 0 2024-04-05T03:02:41.040 INFO:tasks.workunit.client.1.smithi203.stdout:5/442: mkdir d3/d5f/d92 0 2024-04-05T03:02:41.041 INFO:tasks.workunit.client.0.smithi153.stdout:7/98: dwrite d2/da/d1f/f10 [0,4194304] 0 2024-04-05T03:02:41.043 INFO:tasks.workunit.client.1.smithi203.stdout:1/414: fsync d7/f80 0 2024-04-05T03:02:41.043 INFO:tasks.workunit.client.1.smithi203.stdout:2/592: dwrite d5/d10/d2f/f88 [0,4194304] 0 2024-04-05T03:02:41.045 INFO:tasks.workunit.client.0.smithi153.stdout:7/99: fdatasync d2/da/d11/d12/f19 0 2024-04-05T03:02:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:40 smithi203 ceph-mon[39911]: pgmap v5: 65 pgs: 65 active+clean; 758 MiB data, 3.8 GiB used, 533 GiB / 536 GiB avail 2024-04-05T03:02:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:40 smithi203 ceph-mon[39911]: mgrmap e24: smithi203.iakodz(active, since 4s) 2024-04-05T03:02:41.047 INFO:tasks.workunit.client.1.smithi203.stdout:2/593: write d5/d63/d86/fb0 [915771,121614] 0 2024-04-05T03:02:41.048 INFO:tasks.workunit.client.0.smithi153.stdout:8/44: dwrite f6 [0,4194304] 0 2024-04-05T03:02:41.049 INFO:tasks.workunit.client.0.smithi153.stdout:5/83: rmdir d0 39 2024-04-05T03:02:41.052 INFO:tasks.workunit.client.0.smithi153.stdout:8/45: symlink d7/lb 0 2024-04-05T03:02:41.053 INFO:tasks.workunit.client.1.smithi203.stdout:1/415: dread d7/d24/d33/d48/d4b/d6b/f75 [0,4194304] 0 2024-04-05T03:02:41.056 INFO:tasks.workunit.client.0.smithi153.stdout:8/46: creat d7/fc x:0 0 0 2024-04-05T03:02:41.057 INFO:tasks.workunit.client.1.smithi203.stdout:2/594: rename d5/d10/d35/d36/db6/d87/faa to d5/d10/d35/d36/fd1 0 2024-04-05T03:02:41.057 INFO:tasks.workunit.client.0.smithi153.stdout:8/47: fdatasync d7/fc 0 2024-04-05T03:02:41.058 INFO:tasks.workunit.client.0.smithi153.stdout:1/60: link d5/c10 d5/d8/c14 0 2024-04-05T03:02:41.058 INFO:tasks.workunit.client.0.smithi153.stdout:5/84: fdatasync d0/fd 0 2024-04-05T03:02:41.059 INFO:tasks.workunit.client.1.smithi203.stdout:1/416: readlink d7/l10 0 2024-04-05T03:02:41.061 INFO:tasks.workunit.client.0.smithi153.stdout:8/48: creat d7/fd x:0 0 0 2024-04-05T03:02:41.062 INFO:tasks.workunit.client.1.smithi203.stdout:7/445: getdents d8/d32/d16/d1a 0 2024-04-05T03:02:41.063 INFO:tasks.workunit.client.0.smithi153.stdout:5/85: creat d0/d9/f1e x:0 0 0 2024-04-05T03:02:41.063 INFO:tasks.workunit.client.1.smithi203.stdout:1/417: rmdir d7/d79 39 2024-04-05T03:02:41.064 INFO:tasks.workunit.client.0.smithi153.stdout:1/61: chown d5/d8/ff 4929 1 2024-04-05T03:02:41.064 INFO:tasks.workunit.client.1.smithi203.stdout:2/595: write d5/d10/d35/d36/db6/d71/f97 [462098,95636] 0 2024-04-05T03:02:41.065 INFO:tasks.workunit.client.0.smithi153.stdout:5/86: stat d0/de/f10 0 2024-04-05T03:02:41.068 INFO:tasks.workunit.client.0.smithi153.stdout:2/60: truncate d9/da/fc 2614859 0 2024-04-05T03:02:41.068 INFO:tasks.workunit.client.0.smithi153.stdout:5/87: truncate d0/d9/f19 557718 0 2024-04-05T03:02:41.070 INFO:tasks.workunit.client.0.smithi153.stdout:1/62: chown d5/c6 181014623 1 2024-04-05T03:02:41.071 INFO:tasks.workunit.client.0.smithi153.stdout:8/49: dwrite f0 [0,4194304] 0 2024-04-05T03:02:41.075 INFO:tasks.workunit.client.1.smithi203.stdout:7/446: dwrite d8/d2a/d3f/d5e/d77/f87 [0,4194304] 0 2024-04-05T03:02:41.076 INFO:tasks.workunit.client.0.smithi153.stdout:1/63: rmdir d5 39 2024-04-05T03:02:41.080 INFO:tasks.workunit.client.0.smithi153.stdout:8/50: creat d7/fe x:0 0 0 2024-04-05T03:02:41.081 INFO:tasks.workunit.client.0.smithi153.stdout:2/61: dwrite d9/da/f15 [0,4194304] 0 2024-04-05T03:02:41.081 INFO:tasks.workunit.client.1.smithi203.stdout:2/596: dwrite d5/d10/d35/d2d/d7c/d72/d8f/fb9 [0,4194304] 0 2024-04-05T03:02:41.082 INFO:tasks.workunit.client.0.smithi153.stdout:8/51: chown d7/fe 376 1 2024-04-05T03:02:41.082 INFO:tasks.workunit.client.0.smithi153.stdout:1/64: dread f3 [0,4194304] 0 2024-04-05T03:02:41.083 INFO:tasks.workunit.client.0.smithi153.stdout:8/52: chown d7/fd 489896 1 2024-04-05T03:02:41.084 INFO:tasks.workunit.client.1.smithi203.stdout:5/443: write d3/d4/dc/d67/f7f [3409223,114739] 0 2024-04-05T03:02:41.085 INFO:tasks.workunit.client.0.smithi153.stdout:8/53: chown d7/lb 115859 1 2024-04-05T03:02:41.091 INFO:tasks.workunit.client.0.smithi153.stdout:2/62: dwrite d9/da/f15 [0,4194304] 0 2024-04-05T03:02:41.092 INFO:tasks.workunit.client.1.smithi203.stdout:2/597: mknod d5/d10/d35/d36/db6/cd2 0 2024-04-05T03:02:41.093 INFO:tasks.workunit.client.1.smithi203.stdout:7/447: dwrite d8/d32/d16/d1a/d5a/f85 [0,4194304] 0 2024-04-05T03:02:41.098 INFO:tasks.workunit.client.1.smithi203.stdout:5/444: dwrite d3/d1f/d59/d60/f74 [0,4194304] 0 2024-04-05T03:02:41.098 INFO:tasks.workunit.client.1.smithi203.stdout:1/418: write d7/d24/d33/d48/d4b/d51/f67 [3446242,43231] 0 2024-04-05T03:02:41.101 INFO:tasks.workunit.client.1.smithi203.stdout:5/445: chown d3/d4/dc/l5c 194119912 1 2024-04-05T03:02:41.105 INFO:tasks.workunit.client.1.smithi203.stdout:7/448: rmdir d8/d32/d8a 0 2024-04-05T03:02:41.109 INFO:tasks.workunit.client.1.smithi203.stdout:1/419: truncate d7/f2d 3304541 0 2024-04-05T03:02:41.110 INFO:tasks.workunit.client.0.smithi153.stdout:1/65: getdents d5 0 2024-04-05T03:02:41.111 INFO:tasks.workunit.client.1.smithi203.stdout:1/420: chown d7/d24/c61 333 1 2024-04-05T03:02:41.113 INFO:tasks.workunit.client.0.smithi153.stdout:1/66: symlink d5/dd/l15 0 2024-04-05T03:02:41.114 INFO:tasks.workunit.client.1.smithi203.stdout:5/446: dwrite d3/d4/f71 [0,4194304] 0 2024-04-05T03:02:41.117 INFO:tasks.workunit.client.1.smithi203.stdout:1/421: mknod d7/d14/d2b/d58/d7a/c81 0 2024-04-05T03:02:41.118 INFO:tasks.workunit.client.1.smithi203.stdout:5/447: chown d3/c9 39 1 2024-04-05T03:02:41.120 INFO:tasks.workunit.client.1.smithi203.stdout:1/422: fdatasync d7/f1c 0 2024-04-05T03:02:41.121 INFO:tasks.workunit.client.0.smithi153.stdout:1/67: unlink d5/db/fc 0 2024-04-05T03:02:41.121 INFO:tasks.workunit.client.1.smithi203.stdout:7/449: dwrite d8/f17 [0,4194304] 0 2024-04-05T03:02:41.122 INFO:tasks.workunit.client.1.smithi203.stdout:5/448: fdatasync d3/d4/dc/d67/f7e 0 2024-04-05T03:02:41.126 INFO:tasks.workunit.client.0.smithi153.stdout:1/68: mkdir d5/d12/d16 0 2024-04-05T03:02:41.127 INFO:tasks.workunit.client.1.smithi203.stdout:7/450: creat d8/d32/d5d/d6d/f8d x:0 0 0 2024-04-05T03:02:41.130 INFO:tasks.workunit.client.1.smithi203.stdout:6/460: sync 2024-04-05T03:02:41.130 INFO:tasks.workunit.client.1.smithi203.stdout:0/420: sync 2024-04-05T03:02:41.130 INFO:tasks.workunit.client.0.smithi153.stdout:1/69: mknod d5/db/c17 0 2024-04-05T03:02:41.134 INFO:tasks.workunit.client.1.smithi203.stdout:2/598: dread d5/d10/d1a/f4b [0,4194304] 0 2024-04-05T03:02:41.135 INFO:tasks.workunit.client.1.smithi203.stdout:1/423: link d7/d14/c7b d7/c82 0 2024-04-05T03:02:41.135 INFO:tasks.workunit.client.1.smithi203.stdout:2/599: fdatasync d5/d10/d35/d36/db6/f37 0 2024-04-05T03:02:41.137 INFO:tasks.workunit.client.1.smithi203.stdout:1/424: read d7/d14/d2b/d58/f60 [2786362,52955] 0 2024-04-05T03:02:41.137 INFO:tasks.workunit.client.1.smithi203.stdout:7/451: creat d8/d32/d16/d35/d64/f8e x:0 0 0 2024-04-05T03:02:41.139 INFO:tasks.workunit.client.1.smithi203.stdout:0/421: mknod dd/d17/c7a 0 2024-04-05T03:02:41.140 INFO:tasks.workunit.client.1.smithi203.stdout:0/422: fsync dd/d17/d1d/d22/f77 0 2024-04-05T03:02:41.141 INFO:tasks.workunit.client.1.smithi203.stdout:7/452: creat d8/d32/d5d/d6d/f8f x:0 0 0 2024-04-05T03:02:41.143 INFO:tasks.workunit.client.1.smithi203.stdout:1/425: mkdir d7/d14/d83 0 2024-04-05T03:02:41.143 INFO:tasks.workunit.client.1.smithi203.stdout:2/600: dwrite d5/d10/d1a/d57/fac [0,4194304] 0 2024-04-05T03:02:41.144 INFO:tasks.workunit.client.1.smithi203.stdout:7/453: read - d8/d2a/d3f/d5e/d63/f75 zero size 2024-04-05T03:02:41.147 INFO:tasks.workunit.client.1.smithi203.stdout:2/601: unlink d5/d10/d35/d73/fd0 0 2024-04-05T03:02:41.148 INFO:tasks.workunit.client.0.smithi153.stdout:4/113: sync 2024-04-05T03:02:41.148 INFO:tasks.workunit.client.0.smithi153.stdout:9/26: sync 2024-04-05T03:02:41.148 INFO:tasks.workunit.client.0.smithi153.stdout:6/82: sync 2024-04-05T03:02:41.148 INFO:tasks.workunit.client.0.smithi153.stdout:0/75: sync 2024-04-05T03:02:41.150 INFO:tasks.workunit.client.1.smithi203.stdout:2/602: fsync d5/d10/d35/d2d/d7c/d72/d8f/fb9 0 2024-04-05T03:02:41.151 INFO:tasks.workunit.client.1.smithi203.stdout:2/603: readlink d5/lc6 0 2024-04-05T03:02:41.152 INFO:tasks.workunit.client.0.smithi153.stdout:8/54: write f4 [570013,53517] 0 2024-04-05T03:02:41.153 INFO:tasks.workunit.client.1.smithi203.stdout:7/454: dwrite d8/d2a/d3f/d73/f7b [0,4194304] 0 2024-04-05T03:02:41.154 INFO:tasks.workunit.client.0.smithi153.stdout:0/76: dread - d9/db/f13 zero size 2024-04-05T03:02:41.154 INFO:tasks.workunit.client.0.smithi153.stdout:9/27: mkdir d1/db 0 2024-04-05T03:02:41.155 INFO:tasks.workunit.client.0.smithi153.stdout:6/83: mknod d4/dc/c1f 0 2024-04-05T03:02:41.160 INFO:tasks.workunit.client.1.smithi203.stdout:7/455: dread d8/da/f1d [0,4194304] 0 2024-04-05T03:02:41.162 INFO:tasks.workunit.client.1.smithi203.stdout:7/456: read d8/d2a/d3f/d52/f45 [5272134,126245] 0 2024-04-05T03:02:41.162 INFO:tasks.workunit.client.0.smithi153.stdout:1/70: rename d5/d12 to d5/db/d18 0 2024-04-05T03:02:41.162 INFO:tasks.workunit.client.0.smithi153.stdout:0/77: write f7 [83494,82488] 0 2024-04-05T03:02:41.166 INFO:tasks.workunit.client.0.smithi153.stdout:6/84: dread - d4/dc/df/f12 zero size 2024-04-05T03:02:41.166 INFO:tasks.workunit.client.0.smithi153.stdout:4/114: creat da/db/d20/f21 x:0 0 0 2024-04-05T03:02:41.166 INFO:tasks.workunit.client.1.smithi203.stdout:5/449: truncate d3/d1f/d59/d64/f3d 1588689 0 2024-04-05T03:02:41.167 INFO:tasks.workunit.client.1.smithi203.stdout:6/461: write d0/da/d18/f30 [635053,71761] 0 2024-04-05T03:02:41.168 INFO:tasks.workunit.client.0.smithi153.stdout:6/85: chown d4/dc/df 1 1 2024-04-05T03:02:41.168 INFO:tasks.workunit.client.1.smithi203.stdout:7/457: unlink d8/d2a/d3f/d52/f8b 0 2024-04-05T03:02:41.169 INFO:tasks.workunit.client.0.smithi153.stdout:0/78: write f0 [4624136,114343] 0 2024-04-05T03:02:41.170 INFO:tasks.workunit.client.0.smithi153.stdout:4/115: chown da/db/d20 0 1 2024-04-05T03:02:41.171 INFO:tasks.workunit.client.0.smithi153.stdout:2/63: dwrite d9/da/fe [0,4194304] 0 2024-04-05T03:02:41.174 INFO:tasks.workunit.client.0.smithi153.stdout:6/86: creat d4/f20 x:0 0 0 2024-04-05T03:02:41.174 INFO:tasks.workunit.client.0.smithi153.stdout:8/55: dwrite d7/fc [0,4194304] 0 2024-04-05T03:02:41.174 INFO:tasks.workunit.client.1.smithi203.stdout:7/458: dread d8/da/f50 [0,4194304] 0 2024-04-05T03:02:41.175 INFO:tasks.workunit.client.0.smithi153.stdout:2/64: chown d9/da/d19 0 1 2024-04-05T03:02:41.175 INFO:tasks.workunit.client.1.smithi203.stdout:5/450: truncate d3/d1f/f33 5093365 0 2024-04-05T03:02:41.176 INFO:tasks.workunit.client.0.smithi153.stdout:4/116: unlink da/de/l1f 0 2024-04-05T03:02:41.177 INFO:tasks.workunit.client.1.smithi203.stdout:7/459: truncate d8/d32/d16/d1a/d3d/f76 291816 0 2024-04-05T03:02:41.178 INFO:tasks.workunit.client.1.smithi203.stdout:2/604: dread d5/d10/d7f/fc0 [0,4194304] 0 2024-04-05T03:02:41.180 INFO:tasks.workunit.client.0.smithi153.stdout:1/71: truncate f3 793958 0 2024-04-05T03:02:41.181 INFO:tasks.workunit.client.0.smithi153.stdout:0/79: dread f0 [0,4194304] 0 2024-04-05T03:02:41.182 INFO:tasks.workunit.client.1.smithi203.stdout:7/460: fsync d8/d2a/d3f/d52/f45 0 2024-04-05T03:02:41.183 INFO:tasks.workunit.client.0.smithi153.stdout:2/65: dread d9/f11 [0,4194304] 0 2024-04-05T03:02:41.183 INFO:tasks.workunit.client.1.smithi203.stdout:2/605: symlink d5/d10/d35/d2d/d7c/d7a/dbd/ld3 0 2024-04-05T03:02:41.184 INFO:tasks.workunit.client.1.smithi203.stdout:0/423: truncate dd/d17/d1d/d22/d53/d61/f6a 2455520 0 2024-04-05T03:02:41.186 INFO:tasks.workunit.client.0.smithi153.stdout:6/87: creat d4/dc/df/d16/f21 x:0 0 0 2024-04-05T03:02:41.187 INFO:tasks.workunit.client.0.smithi153.stdout:4/117: mknod da/db/c22 0 2024-04-05T03:02:41.187 INFO:tasks.workunit.client.0.smithi153.stdout:4/118: stat da/db 0 2024-04-05T03:02:41.189 INFO:tasks.workunit.client.1.smithi203.stdout:7/461: dread d8/f17 [0,4194304] 0 2024-04-05T03:02:41.190 INFO:tasks.workunit.client.0.smithi153.stdout:9/28: getdents d1 0 2024-04-05T03:02:41.191 INFO:tasks.workunit.client.0.smithi153.stdout:9/29: fdatasync d1/d9/fa 0 2024-04-05T03:02:41.196 INFO:tasks.workunit.client.0.smithi153.stdout:8/56: dread d7/f8 [0,4194304] 0 2024-04-05T03:02:41.196 INFO:tasks.workunit.client.0.smithi153.stdout:9/30: truncate d1/f4 5057512 0 2024-04-05T03:02:41.196 INFO:tasks.workunit.client.1.smithi203.stdout:6/462: truncate d0/da/d18/f30 702158 0 2024-04-05T03:02:41.196 INFO:tasks.workunit.client.1.smithi203.stdout:5/451: write d3/f7a [1752286,61965] 0 2024-04-05T03:02:41.198 INFO:tasks.workunit.client.0.smithi153.stdout:6/88: symlink d4/dc/df/l22 0 2024-04-05T03:02:41.198 INFO:tasks.workunit.client.0.smithi153.stdout:4/119: dread da/fc [0,4194304] 0 2024-04-05T03:02:41.198 INFO:tasks.workunit.client.1.smithi203.stdout:5/452: stat d3/d1f/d59/d64/f23 0 2024-04-05T03:02:41.200 INFO:tasks.workunit.client.1.smithi203.stdout:5/453: write d3/d1f/d59/d64/d12/f46 [2853746,58660] 0 2024-04-05T03:02:41.200 INFO:tasks.workunit.client.1.smithi203.stdout:6/463: mkdir d0/da/d54/d82 0 2024-04-05T03:02:41.202 INFO:tasks.workunit.client.0.smithi153.stdout:4/120: chown da/db/c22 9326 1 2024-04-05T03:02:41.202 INFO:tasks.workunit.client.0.smithi153.stdout:0/80: dwrite f3 [0,4194304] 0 2024-04-05T03:02:41.202 INFO:tasks.workunit.client.1.smithi203.stdout:6/464: write d0/da/d18/d38/f6d [930772,128296] 0 2024-04-05T03:02:41.203 INFO:tasks.workunit.client.0.smithi153.stdout:9/31: unlink d1/c6 0 2024-04-05T03:02:41.203 INFO:tasks.workunit.client.0.smithi153.stdout:0/81: readlink l4 0 2024-04-05T03:02:41.205 INFO:tasks.workunit.client.0.smithi153.stdout:4/121: write da/db/f1a [513888,90627] 0 2024-04-05T03:02:41.205 INFO:tasks.workunit.client.0.smithi153.stdout:4/122: readlink l6 0 2024-04-05T03:02:41.206 INFO:tasks.workunit.client.0.smithi153.stdout:8/57: creat d7/ff x:0 0 0 2024-04-05T03:02:41.207 INFO:tasks.workunit.client.0.smithi153.stdout:0/82: mkdir d9/d14 0 2024-04-05T03:02:41.215 INFO:tasks.workunit.client.1.smithi203.stdout:6/465: creat d0/da/d54/d82/f83 x:0 0 0 2024-04-05T03:02:41.216 INFO:tasks.workunit.client.1.smithi203.stdout:5/454: write d3/d1f/d59/d64/f3d [928216,83248] 0 2024-04-05T03:02:41.217 INFO:tasks.workunit.client.0.smithi153.stdout:1/72: mknod d5/d8/c19 0 2024-04-05T03:02:41.219 INFO:tasks.workunit.client.1.smithi203.stdout:5/455: stat d3/d1f/d59/d64/d12/c83 0 2024-04-05T03:02:41.220 INFO:tasks.workunit.client.1.smithi203.stdout:5/456: fsync d3/d4/f71 0 2024-04-05T03:02:41.220 INFO:tasks.workunit.client.0.smithi153.stdout:2/66: truncate d9/da/fe 3365059 0 2024-04-05T03:02:41.222 INFO:tasks.workunit.client.1.smithi203.stdout:7/462: dwrite d8/d32/f18 [0,4194304] 0 2024-04-05T03:02:41.223 INFO:tasks.workunit.client.1.smithi203.stdout:7/463: fsync d8/d2a/d3f/d73/f78 0 2024-04-05T03:02:41.226 INFO:tasks.workunit.client.0.smithi153.stdout:1/73: read d5/f7 [314046,107025] 0 2024-04-05T03:02:41.228 INFO:tasks.workunit.client.0.smithi153.stdout:2/67: creat d9/da/d10/f1a x:0 0 0 2024-04-05T03:02:41.230 INFO:tasks.workunit.client.0.smithi153.stdout:6/89: getdents d4/dc/df 0 2024-04-05T03:02:41.230 INFO:tasks.workunit.client.1.smithi203.stdout:2/606: dread d5/d10/d35/d36/f54 [0,4194304] 0 2024-04-05T03:02:41.232 INFO:tasks.workunit.client.0.smithi153.stdout:6/90: fsync d4/fb 0 2024-04-05T03:02:41.233 INFO:tasks.workunit.client.1.smithi203.stdout:7/464: write d8/da/f50 [4725961,1098] 0 2024-04-05T03:02:41.233 INFO:tasks.workunit.client.0.smithi153.stdout:4/123: dread da/f17 [0,4194304] 0 2024-04-05T03:02:41.234 INFO:tasks.workunit.client.1.smithi203.stdout:6/466: link d0/da/l7e d0/da/d18/d38/d2a/d44/l84 0 2024-04-05T03:02:41.237 INFO:tasks.workunit.client.1.smithi203.stdout:0/424: dwrite dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:41.237 INFO:tasks.workunit.client.1.smithi203.stdout:5/457: rename d3/d4/dc/f85 to d3/d1f/f93 0 2024-04-05T03:02:41.238 INFO:tasks.workunit.client.0.smithi153.stdout:2/68: symlink d9/da/d19/l1b 0 2024-04-05T03:02:41.240 INFO:tasks.workunit.client.1.smithi203.stdout:2/607: mknod d5/d10/d35/d36/db6/d71/cd4 0 2024-04-05T03:02:41.240 INFO:tasks.workunit.client.1.smithi203.stdout:5/458: write f2 [1769939,89159] 0 2024-04-05T03:02:41.243 INFO:tasks.workunit.client.1.smithi203.stdout:0/425: mknod dd/d17/d1d/d26/d28/c7b 0 2024-04-05T03:02:41.246 INFO:tasks.workunit.client.1.smithi203.stdout:0/426: write dd/d17/d29/f3c [1333448,42979] 0 2024-04-05T03:02:41.249 INFO:tasks.workunit.client.1.smithi203.stdout:5/459: mkdir d3/d4/d94 0 2024-04-05T03:02:41.249 INFO:tasks.workunit.client.1.smithi203.stdout:0/427: chown dd/d17/d29/c36 278 1 2024-04-05T03:02:41.249 INFO:tasks.workunit.client.1.smithi203.stdout:7/465: dwrite d8/d32/d16/d1a/d5a/f85 [0,4194304] 0 2024-04-05T03:02:41.251 INFO:tasks.workunit.client.1.smithi203.stdout:2/608: creat d5/d10/d35/d2d/d7c/d29/d67/fd5 x:0 0 0 2024-04-05T03:02:41.252 INFO:tasks.workunit.client.1.smithi203.stdout:5/460: creat d3/d4/dc/f95 x:0 0 0 2024-04-05T03:02:41.254 INFO:tasks.workunit.client.1.smithi203.stdout:0/428: rmdir dd/d17/d1d/d26/d28 39 2024-04-05T03:02:41.255 INFO:tasks.workunit.client.1.smithi203.stdout:7/466: mkdir d8/d32/d16/d1a/d30/d90 0 2024-04-05T03:02:41.258 INFO:tasks.workunit.client.1.smithi203.stdout:2/609: creat d5/d10/d35/d2d/d7c/d29/d67/d9e/fd6 x:0 0 0 2024-04-05T03:02:41.262 INFO:tasks.workunit.client.1.smithi203.stdout:2/610: creat d5/d10/d44/fd7 x:0 0 0 2024-04-05T03:02:41.264 INFO:tasks.workunit.client.1.smithi203.stdout:7/467: link d8/d2a/d3f/d52/c66 d8/d32/d5d/d6d/c91 0 2024-04-05T03:02:41.267 INFO:tasks.workunit.client.1.smithi203.stdout:2/611: symlink d5/d10/db8/ld8 0 2024-04-05T03:02:41.268 INFO:tasks.workunit.client.1.smithi203.stdout:0/429: link dd/d17/d1d/d22/d45/d51/l71 dd/d17/d1d/d26/d41/d49/l7c 0 2024-04-05T03:02:41.269 INFO:tasks.workunit.client.1.smithi203.stdout:6/467: dread d0/da/f43 [0,4194304] 0 2024-04-05T03:02:41.270 INFO:tasks.workunit.client.1.smithi203.stdout:6/468: readlink d0/l2 0 2024-04-05T03:02:41.270 INFO:tasks.workunit.client.1.smithi203.stdout:5/461: dread d3/d4/dc/f54 [0,4194304] 0 2024-04-05T03:02:41.271 INFO:tasks.workunit.client.1.smithi203.stdout:0/430: chown dd/d17/d1d/d22 28 1 2024-04-05T03:02:41.273 INFO:tasks.workunit.client.1.smithi203.stdout:0/431: write dd/f14 [3193326,66223] 0 2024-04-05T03:02:41.274 INFO:tasks.workunit.client.1.smithi203.stdout:7/468: dwrite d8/d32/d16/d1a/d3d/d59/f61 [0,4194304] 0 2024-04-05T03:02:41.275 INFO:tasks.workunit.client.1.smithi203.stdout:6/469: unlink d0/da/d18/d38/d2a/d2c/l73 0 2024-04-05T03:02:41.276 INFO:tasks.workunit.client.1.smithi203.stdout:2/612: dread d5/d63/f93 [0,4194304] 0 2024-04-05T03:02:41.277 INFO:tasks.workunit.client.1.smithi203.stdout:5/462: creat d3/d1f/d59/d64/d12/f96 x:0 0 0 2024-04-05T03:02:41.280 INFO:tasks.workunit.client.1.smithi203.stdout:6/470: mknod d0/da/d54/d53/c85 0 2024-04-05T03:02:41.280 INFO:tasks.workunit.client.1.smithi203.stdout:7/469: truncate d8/d2a/d3f/d5e/d77/f80 196713 0 2024-04-05T03:02:41.281 INFO:tasks.workunit.client.1.smithi203.stdout:2/613: write d5/d63/d86/fb0 [1937176,90912] 0 2024-04-05T03:02:41.283 INFO:tasks.workunit.client.1.smithi203.stdout:5/463: symlink d3/d1f/d59/d64/d12/l97 0 2024-04-05T03:02:41.283 INFO:tasks.workunit.client.1.smithi203.stdout:7/470: write d8/d32/d16/d1a/d21/f23 [4394237,37465] 0 2024-04-05T03:02:41.284 INFO:tasks.workunit.client.1.smithi203.stdout:2/614: stat d5/d10/d35/d2d/d7c/d29/db1/fb7 0 2024-04-05T03:02:41.286 INFO:tasks.workunit.client.1.smithi203.stdout:0/432: truncate dd/d17/d1d/d26/d34/f35 1916219 0 2024-04-05T03:02:41.287 INFO:tasks.workunit.client.1.smithi203.stdout:0/433: write dd/d17/d1d/d26/d41/f4b [376695,85846] 0 2024-04-05T03:02:41.290 INFO:tasks.workunit.client.1.smithi203.stdout:7/471: mkdir d8/d32/d16/d92 0 2024-04-05T03:02:41.291 INFO:tasks.workunit.client.1.smithi203.stdout:7/472: read d8/d32/f18 [983873,96478] 0 2024-04-05T03:02:41.292 INFO:tasks.workunit.client.1.smithi203.stdout:7/473: stat d8/d32/d16/d1a/d21/c83 0 2024-04-05T03:02:41.295 INFO:tasks.workunit.client.1.smithi203.stdout:6/471: rename d0/da/d18/d38/d2b to d0/da/d18/d38/d86 0 2024-04-05T03:02:41.296 INFO:tasks.workunit.client.1.smithi203.stdout:5/464: link d3/d1f/d49/c6f d3/d1f/d59/d60/c98 0 2024-04-05T03:02:41.298 INFO:tasks.workunit.client.1.smithi203.stdout:5/465: fsync d3/d1f/d59/d60/f72 0 2024-04-05T03:02:41.298 INFO:tasks.workunit.client.1.smithi203.stdout:7/474: rmdir d8/d2a 39 2024-04-05T03:02:41.300 INFO:tasks.workunit.client.0.smithi153.stdout:3/63: sync 2024-04-05T03:02:41.301 INFO:tasks.workunit.client.1.smithi203.stdout:0/434: dread dd/f15 [0,4194304] 0 2024-04-05T03:02:41.302 INFO:tasks.workunit.client.1.smithi203.stdout:0/435: read - dd/d17/d1d/d22/f77 zero size 2024-04-05T03:02:41.303 INFO:tasks.workunit.client.0.smithi153.stdout:0/83: truncate d9/db/f11 44221 0 2024-04-05T03:02:41.305 INFO:tasks.workunit.client.0.smithi153.stdout:1/74: dwrite d5/db/fe [0,4194304] 0 2024-04-05T03:02:41.307 INFO:tasks.workunit.client.0.smithi153.stdout:4/124: dwrite da/de/f14 [0,4194304] 0 2024-04-05T03:02:41.308 INFO:tasks.workunit.client.0.smithi153.stdout:3/64: mknod c14 0 2024-04-05T03:02:41.308 INFO:tasks.workunit.client.0.smithi153.stdout:0/84: write f3 [2716210,19147] 0 2024-04-05T03:02:41.309 INFO:tasks.workunit.client.1.smithi203.stdout:0/436: rmdir dd/d17/d1d/d22/d53/d61 39 2024-04-05T03:02:41.310 INFO:tasks.workunit.client.0.smithi153.stdout:4/125: readlink da/l15 0 2024-04-05T03:02:41.311 INFO:tasks.workunit.client.0.smithi153.stdout:1/75: dread - d5/dd/f13 zero size 2024-04-05T03:02:41.312 INFO:tasks.workunit.client.1.smithi203.stdout:0/437: read dd/f16 [3544002,48395] 0 2024-04-05T03:02:41.313 INFO:tasks.workunit.client.0.smithi153.stdout:6/91: getdents d4/dc/df/d16/d17 0 2024-04-05T03:02:41.313 INFO:tasks.workunit.client.1.smithi203.stdout:7/475: symlink d8/d2a/d3f/d5e/d77/l93 0 2024-04-05T03:02:41.314 INFO:tasks.workunit.client.0.smithi153.stdout:3/65: rename c2 to c15 0 2024-04-05T03:02:41.317 INFO:tasks.workunit.client.0.smithi153.stdout:3/66: chown l5 1 1 2024-04-05T03:02:41.321 INFO:tasks.workunit.client.0.smithi153.stdout:1/76: creat d5/db/d18/f1a x:0 0 0 2024-04-05T03:02:41.322 INFO:tasks.workunit.client.0.smithi153.stdout:0/85: creat d9/df/f15 x:0 0 0 2024-04-05T03:02:41.322 INFO:tasks.workunit.client.1.smithi203.stdout:0/438: creat dd/d17/d1d/d26/d41/d49/f7d x:0 0 0 2024-04-05T03:02:41.324 INFO:tasks.workunit.client.1.smithi203.stdout:6/472: link d0/da/d18/d38/d2a/l5d d0/da/d18/l87 0 2024-04-05T03:02:41.325 INFO:tasks.workunit.client.0.smithi153.stdout:2/69: truncate d9/da/fb 3649364 0 2024-04-05T03:02:41.326 INFO:tasks.workunit.client.0.smithi153.stdout:1/77: mkdir d5/dd/d1b 0 2024-04-05T03:02:41.326 INFO:tasks.workunit.client.0.smithi153.stdout:0/86: stat d9/db/f11 0 2024-04-05T03:02:41.328 INFO:tasks.workunit.client.1.smithi203.stdout:6/473: mknod d0/da/c88 0 2024-04-05T03:02:41.328 INFO:tasks.workunit.client.1.smithi203.stdout:7/476: dread f4 [0,4194304] 0 2024-04-05T03:02:41.328 INFO:tasks.workunit.client.0.smithi153.stdout:6/92: dwrite d4/fb [0,4194304] 0 2024-04-05T03:02:41.330 INFO:tasks.workunit.client.0.smithi153.stdout:6/93: chown d4/dc/f14 464 1 2024-04-05T03:02:41.330 INFO:tasks.workunit.client.1.smithi203.stdout:0/439: creat dd/d17/d1d/d22/d45/d51/d69/f7e x:0 0 0 2024-04-05T03:02:41.331 INFO:tasks.workunit.client.1.smithi203.stdout:7/477: write d8/d2a/d3f/d73/f78 [450509,100282] 0 2024-04-05T03:02:41.331 INFO:tasks.workunit.client.0.smithi153.stdout:4/126: truncate da/de/f14 128811 0 2024-04-05T03:02:41.331 INFO:tasks.workunit.client.0.smithi153.stdout:3/67: dread fd [0,4194304] 0 2024-04-05T03:02:41.333 INFO:tasks.workunit.client.1.smithi203.stdout:7/478: rmdir d8/d2a 39 2024-04-05T03:02:41.334 INFO:tasks.workunit.client.0.smithi153.stdout:3/68: write fd [998607,15647] 0 2024-04-05T03:02:41.335 INFO:tasks.workunit.client.1.smithi203.stdout:6/474: creat d0/da/d18/d38/d2a/d44/d78/f89 x:0 0 0 2024-04-05T03:02:41.335 INFO:tasks.workunit.client.0.smithi153.stdout:2/70: write d9/da/fc [1865442,67139] 0 2024-04-05T03:02:41.337 INFO:tasks.workunit.client.1.smithi203.stdout:6/475: write d0/f9 [3365172,85384] 0 2024-04-05T03:02:41.338 INFO:tasks.workunit.client.0.smithi153.stdout:6/94: creat d4/dc/df/d16/f23 x:0 0 0 2024-04-05T03:02:41.340 INFO:tasks.workunit.client.0.smithi153.stdout:2/71: dread - d9/da/d10/f1a zero size 2024-04-05T03:02:41.340 INFO:tasks.workunit.client.1.smithi203.stdout:7/479: rename d8/d32/f70 to d8/da/f94 0 2024-04-05T03:02:41.342 INFO:tasks.workunit.client.0.smithi153.stdout:4/127: link f7 da/db/f23 0 2024-04-05T03:02:41.343 INFO:tasks.workunit.client.0.smithi153.stdout:6/95: mkdir d4/dc/df/d16/d24 0 2024-04-05T03:02:41.343 INFO:tasks.workunit.client.0.smithi153.stdout:1/78: dread d5/db/fe [0,4194304] 0 2024-04-05T03:02:41.343 INFO:tasks.workunit.client.1.smithi203.stdout:0/440: dwrite dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:41.343 INFO:tasks.workunit.client.1.smithi203.stdout:7/480: write d8/da/f50 [1211166,131001] 0 2024-04-05T03:02:41.346 INFO:tasks.workunit.client.1.smithi203.stdout:6/476: mknod d0/da/d54/d4c/d7f/c8a 0 2024-04-05T03:02:41.346 INFO:tasks.workunit.client.0.smithi153.stdout:0/87: dwrite f0 [0,4194304] 0 2024-04-05T03:02:41.347 INFO:tasks.workunit.client.0.smithi153.stdout:1/79: write d5/dd/f13 [76204,40104] 0 2024-04-05T03:02:41.349 INFO:tasks.workunit.client.1.smithi203.stdout:6/477: fsync d0/da/d18/d38/d86/f5a 0 2024-04-05T03:02:41.350 INFO:tasks.workunit.client.0.smithi153.stdout:4/128: creat da/db/f24 x:0 0 0 2024-04-05T03:02:41.350 INFO:tasks.workunit.client.0.smithi153.stdout:4/129: stat f8 0 2024-04-05T03:02:41.351 INFO:tasks.workunit.client.0.smithi153.stdout:6/96: fdatasync d4/dc/f14 0 2024-04-05T03:02:41.353 INFO:tasks.workunit.client.0.smithi153.stdout:6/97: dread - d4/dc/df/d16/f21 zero size 2024-04-05T03:02:41.355 INFO:tasks.workunit.client.0.smithi153.stdout:6/98: dread - d4/dc/df/f13 zero size 2024-04-05T03:02:41.356 INFO:tasks.workunit.client.1.smithi203.stdout:6/478: rename d0/da/d18/d38/d2a/d2c/l34 to d0/da/d67/d71/l8b 0 2024-04-05T03:02:41.356 INFO:tasks.workunit.client.0.smithi153.stdout:4/130: mknod da/db/c25 0 2024-04-05T03:02:41.357 INFO:tasks.workunit.client.1.smithi203.stdout:6/479: chown d0/da/d18 4 1 2024-04-05T03:02:41.358 INFO:tasks.workunit.client.0.smithi153.stdout:1/80: dread f4 [0,4194304] 0 2024-04-05T03:02:41.362 INFO:tasks.workunit.client.0.smithi153.stdout:0/88: link d9/df/f15 d9/df/f16 0 2024-04-05T03:02:41.362 INFO:tasks.workunit.client.1.smithi203.stdout:6/480: read d0/da/d18/d38/f47 [239396,3438] 0 2024-04-05T03:02:41.363 INFO:tasks.workunit.client.0.smithi153.stdout:4/131: creat da/f26 x:0 0 0 2024-04-05T03:02:41.364 INFO:tasks.workunit.client.0.smithi153.stdout:6/99: dwrite d4/f15 [0,4194304] 0 2024-04-05T03:02:41.366 INFO:tasks.workunit.client.0.smithi153.stdout:0/89: fdatasync d9/db/f11 0 2024-04-05T03:02:41.366 INFO:tasks.workunit.client.0.smithi153.stdout:1/81: creat d5/db/d18/f1c x:0 0 0 2024-04-05T03:02:41.366 INFO:tasks.workunit.client.0.smithi153.stdout:2/72: chown d9/da/f15 781 1 2024-04-05T03:02:41.367 INFO:tasks.workunit.client.1.smithi203.stdout:6/481: unlink d0/da/d18/d38/d2a/d44/l84 0 2024-04-05T03:02:41.368 INFO:tasks.workunit.client.1.smithi203.stdout:7/481: write d8/d32/f56 [817581,29140] 0 2024-04-05T03:02:41.368 INFO:tasks.workunit.client.0.smithi153.stdout:6/100: chown d4/dc/df/l22 2433965 1 2024-04-05T03:02:41.369 INFO:tasks.workunit.client.0.smithi153.stdout:6/101: readlink d4/l1d 0 2024-04-05T03:02:41.369 INFO:tasks.workunit.client.0.smithi153.stdout:6/102: chown d4/f15 3463757 1 2024-04-05T03:02:41.370 INFO:tasks.workunit.client.1.smithi203.stdout:6/482: mkdir d0/da/d67/d71/d8c 0 2024-04-05T03:02:41.370 INFO:tasks.workunit.client.1.smithi203.stdout:6/483: stat d0/c16 0 2024-04-05T03:02:41.373 INFO:tasks.workunit.client.1.smithi203.stdout:6/484: readlink d0/da/d54/d4c/l55 0 2024-04-05T03:02:41.376 INFO:tasks.workunit.client.0.smithi153.stdout:2/73: rename f8 to d9/da/d19/f1c 0 2024-04-05T03:02:41.377 INFO:tasks.workunit.client.0.smithi153.stdout:0/90: getdents d9/df/d12 0 2024-04-05T03:02:41.377 INFO:tasks.workunit.client.1.smithi203.stdout:6/485: creat d0/da/d54/d53/f8d x:0 0 0 2024-04-05T03:02:41.383 INFO:tasks.workunit.client.1.smithi203.stdout:6/486: creat d0/da/d18/f8e x:0 0 0 2024-04-05T03:02:41.384 INFO:tasks.workunit.client.0.smithi153.stdout:2/74: dread d9/da/f18 [0,4194304] 0 2024-04-05T03:02:41.385 INFO:tasks.workunit.client.1.smithi203.stdout:7/482: dwrite d8/d2a/d3f/d52/f5b [0,4194304] 0 2024-04-05T03:02:41.386 INFO:tasks.workunit.client.1.smithi203.stdout:0/441: dread dd/d17/d1d/d26/d28/f3f [0,4194304] 0 2024-04-05T03:02:41.388 INFO:tasks.workunit.client.1.smithi203.stdout:7/483: dread - d8/d32/d5d/d6d/f8d zero size 2024-04-05T03:02:41.389 INFO:tasks.workunit.client.1.smithi203.stdout:6/487: rmdir d0/da/d18/d38/d86 39 2024-04-05T03:02:41.390 INFO:tasks.workunit.client.1.smithi203.stdout:6/488: read - d0/da/d18/f8e zero size 2024-04-05T03:02:41.391 INFO:tasks.workunit.client.0.smithi153.stdout:2/75: rename c6 to d9/da/c1d 0 2024-04-05T03:02:41.395 INFO:tasks.workunit.client.1.smithi203.stdout:7/484: mkdir d8/d32/d5d/d95 0 2024-04-05T03:02:41.395 INFO:tasks.workunit.client.1.smithi203.stdout:7/485: chown d8/f46 645 1 2024-04-05T03:02:41.396 INFO:tasks.workunit.client.0.smithi153.stdout:6/103: truncate d4/f15 2585836 0 2024-04-05T03:02:41.396 INFO:tasks.workunit.client.0.smithi153.stdout:4/132: mknod da/c27 0 2024-04-05T03:02:41.398 INFO:tasks.workunit.client.0.smithi153.stdout:2/76: mkdir d9/d1e 0 2024-04-05T03:02:41.398 INFO:tasks.workunit.client.1.smithi203.stdout:0/442: mkdir dd/d17/d1d/d22/d53/d7f 0 2024-04-05T03:02:41.400 INFO:tasks.workunit.client.1.smithi203.stdout:6/489: mknod d0/da/d18/d38/d2a/d44/c8f 0 2024-04-05T03:02:41.400 INFO:tasks.workunit.client.0.smithi153.stdout:1/82: dwrite d5/db/f11 [0,4194304] 0 2024-04-05T03:02:41.400 INFO:tasks.workunit.client.0.smithi153.stdout:6/104: creat d4/dc/df/d16/d17/f25 x:0 0 0 2024-04-05T03:02:41.402 INFO:tasks.workunit.client.1.smithi203.stdout:7/486: dwrite d8/d32/d16/d1a/d3d/f6b [0,4194304] 0 2024-04-05T03:02:41.406 INFO:tasks.workunit.client.0.smithi153.stdout:6/105: creat d4/dc/df/f26 x:0 0 0 2024-04-05T03:02:41.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/443: truncate f7 2767273 0 2024-04-05T03:02:41.407 INFO:tasks.workunit.client.0.smithi153.stdout:4/133: chown da/db/c18 1435574545 1 2024-04-05T03:02:41.408 INFO:tasks.workunit.client.0.smithi153.stdout:0/91: dwrite d9/db/f11 [0,4194304] 0 2024-04-05T03:02:41.410 INFO:tasks.workunit.client.0.smithi153.stdout:4/134: write da/db/fd [8695455,18177] 0 2024-04-05T03:02:41.410 INFO:tasks.workunit.client.0.smithi153.stdout:1/83: rename f2 to d5/dd/d1b/f1d 0 2024-04-05T03:02:41.411 INFO:tasks.workunit.client.0.smithi153.stdout:1/84: readlink d5/dd/l15 0 2024-04-05T03:02:41.412 INFO:tasks.workunit.client.0.smithi153.stdout:6/106: unlink d4/l1d 0 2024-04-05T03:02:41.413 INFO:tasks.workunit.client.0.smithi153.stdout:4/135: dread - da/db/d20/f21 zero size 2024-04-05T03:02:41.414 INFO:tasks.workunit.client.0.smithi153.stdout:4/136: write da/db/f24 [942325,3113] 0 2024-04-05T03:02:41.422 INFO:tasks.workunit.client.1.smithi203.stdout:6/490: link d0/da/d18/d38/d86/c62 d0/da/d54/d4c/c90 0 2024-04-05T03:02:41.423 INFO:tasks.workunit.client.0.smithi153.stdout:0/92: creat d9/df/f17 x:0 0 0 2024-04-05T03:02:41.423 INFO:tasks.workunit.client.1.smithi203.stdout:6/491: dread - d0/da/d18/f8e zero size 2024-04-05T03:02:41.424 INFO:tasks.workunit.client.0.smithi153.stdout:6/107: getdents d4/dc/df/d16/d24 0 2024-04-05T03:02:41.427 INFO:tasks.workunit.client.1.smithi203.stdout:6/492: creat d0/da/d54/d4c/d7f/f91 x:0 0 0 2024-04-05T03:02:41.429 INFO:tasks.workunit.client.0.smithi153.stdout:2/77: truncate d9/da/fe 1451348 0 2024-04-05T03:02:41.429 INFO:tasks.workunit.client.0.smithi153.stdout:4/137: truncate da/fc 1098040 0 2024-04-05T03:02:41.434 INFO:tasks.workunit.client.1.smithi203.stdout:6/493: dwrite d0/da/d54/d53/f8d [0,4194304] 0 2024-04-05T03:02:41.436 INFO:tasks.workunit.client.1.smithi203.stdout:6/494: chown d0/da/d54/d53/f63 3309895 1 2024-04-05T03:02:41.436 INFO:tasks.workunit.client.1.smithi203.stdout:7/487: truncate d8/d2a/d3f/d5e/d77/f87 1064091 0 2024-04-05T03:02:41.437 INFO:tasks.workunit.client.0.smithi153.stdout:5/88: sync 2024-04-05T03:02:41.437 INFO:tasks.workunit.client.0.smithi153.stdout:7/100: sync 2024-04-05T03:02:41.437 INFO:tasks.workunit.client.1.smithi203.stdout:6/495: write d0/da/d67/f28 [2842623,68355] 0 2024-04-05T03:02:41.440 INFO:tasks.workunit.client.0.smithi153.stdout:6/108: link d4/dc/c1f d4/c27 0 2024-04-05T03:02:41.441 INFO:tasks.workunit.client.1.smithi203.stdout:7/488: dread d8/d2a/d3f/d73/f7b [0,4194304] 0 2024-04-05T03:02:41.441 INFO:tasks.workunit.client.1.smithi203.stdout:0/444: write dd/d17/d1d/d26/d34/f35 [1161765,91129] 0 2024-04-05T03:02:41.443 INFO:tasks.workunit.client.1.smithi203.stdout:0/445: stat dd/d17/d1d/d22/d53/d55 0 2024-04-05T03:02:41.444 INFO:tasks.workunit.client.1.smithi203.stdout:0/446: write dd/d17/d1d/d26/d41/d49/f7d [721585,127340] 0 2024-04-05T03:02:41.447 INFO:tasks.workunit.client.0.smithi153.stdout:0/93: truncate f3 2992838 0 2024-04-05T03:02:41.448 INFO:tasks.workunit.client.0.smithi153.stdout:7/101: write d2/f6 [2958740,86054] 0 2024-04-05T03:02:41.449 INFO:tasks.workunit.client.1.smithi203.stdout:6/496: mkdir d0/da/d18/d92 0 2024-04-05T03:02:41.451 INFO:tasks.workunit.client.0.smithi153.stdout:2/78: dwrite d9/da/d10/f1a [0,4194304] 0 2024-04-05T03:02:41.453 INFO:tasks.workunit.client.0.smithi153.stdout:1/85: dwrite d5/db/fe [4194304,4194304] 0 2024-04-05T03:02:41.455 INFO:tasks.workunit.client.1.smithi203.stdout:7/489: link d8/da/f50 d8/d32/d5d/d6d/f96 0 2024-04-05T03:02:41.458 INFO:tasks.workunit.client.1.smithi203.stdout:3/413: sync 2024-04-05T03:02:41.458 INFO:tasks.workunit.client.1.smithi203.stdout:6/497: creat d0/da/d18/d38/d86/f93 x:0 0 0 2024-04-05T03:02:41.460 INFO:tasks.workunit.client.1.smithi203.stdout:3/414: write d0/dc/d1a/d1d/f5c [554473,70193] 0 2024-04-05T03:02:41.461 INFO:tasks.workunit.client.1.smithi203.stdout:6/498: dread - d0/da/d18/d38/d2a/d44/d78/f89 zero size 2024-04-05T03:02:41.462 INFO:tasks.workunit.client.0.smithi153.stdout:6/109: mknod d4/dc/df/d16/c28 0 2024-04-05T03:02:41.462 INFO:tasks.workunit.client.0.smithi153.stdout:4/138: write da/db/f13 [1670873,76204] 0 2024-04-05T03:02:41.464 INFO:tasks.workunit.client.0.smithi153.stdout:5/89: dread d0/fd [0,4194304] 0 2024-04-05T03:02:41.466 INFO:tasks.workunit.client.0.smithi153.stdout:9/32: sync 2024-04-05T03:02:41.466 INFO:tasks.workunit.client.0.smithi153.stdout:8/58: sync 2024-04-05T03:02:41.468 INFO:tasks.workunit.client.1.smithi203.stdout:6/499: creat d0/da/f94 x:0 0 0 2024-04-05T03:02:41.471 INFO:tasks.workunit.client.1.smithi203.stdout:3/415: rmdir d0/dc/d1a/d76/d35/d70 0 2024-04-05T03:02:41.472 INFO:tasks.workunit.client.0.smithi153.stdout:1/86: dwrite d5/db/fe [4194304,4194304] 0 2024-04-05T03:02:41.472 INFO:tasks.workunit.client.0.smithi153.stdout:0/94: write d9/df/f16 [548218,31362] 0 2024-04-05T03:02:41.472 INFO:tasks.workunit.client.0.smithi153.stdout:0/95: chown d9/db 3944 1 2024-04-05T03:02:41.474 INFO:tasks.workunit.client.0.smithi153.stdout:0/96: write d9/df/f17 [219704,105579] 0 2024-04-05T03:02:41.474 INFO:tasks.workunit.client.1.smithi203.stdout:6/500: mkdir d0/da/d67/d95 0 2024-04-05T03:02:41.479 INFO:tasks.workunit.client.0.smithi153.stdout:5/90: creat d0/d9/f1f x:0 0 0 2024-04-05T03:02:41.479 INFO:tasks.workunit.client.1.smithi203.stdout:3/416: creat d0/dc/d1a/d1d/d53/d5a/f77 x:0 0 0 2024-04-05T03:02:41.481 INFO:tasks.workunit.client.0.smithi153.stdout:6/110: creat d4/dc/df/d16/d24/f29 x:0 0 0 2024-04-05T03:02:41.481 INFO:tasks.workunit.client.1.smithi203.stdout:6/501: creat d0/da/f96 x:0 0 0 2024-04-05T03:02:41.483 INFO:tasks.workunit.client.1.smithi203.stdout:6/502: chown d0/da/d54/d53/f6a 0 1 2024-04-05T03:02:41.487 INFO:tasks.workunit.client.0.smithi153.stdout:6/111: chown d4/f7 513139 1 2024-04-05T03:02:41.488 INFO:tasks.workunit.client.0.smithi153.stdout:9/33: link d1/d9/fa d1/d9/fc 0 2024-04-05T03:02:41.488 INFO:tasks.workunit.client.0.smithi153.stdout:9/34: chown d1/db 12683933 1 2024-04-05T03:02:41.490 INFO:tasks.workunit.client.0.smithi153.stdout:1/87: mkdir d5/db/d1e 0 2024-04-05T03:02:41.490 INFO:tasks.workunit.client.0.smithi153.stdout:4/139: mkdir da/de/d28 0 2024-04-05T03:02:41.492 INFO:tasks.workunit.client.0.smithi153.stdout:2/79: getdents d9/da/d19 0 2024-04-05T03:02:41.492 INFO:tasks.workunit.client.0.smithi153.stdout:7/102: rename d2/da/l16 to d2/da/l21 0 2024-04-05T03:02:41.492 INFO:tasks.workunit.client.1.smithi203.stdout:7/490: write d8/f31 [1856504,78330] 0 2024-04-05T03:02:41.492 INFO:tasks.workunit.client.0.smithi153.stdout:9/35: fdatasync d1/f4 0 2024-04-05T03:02:41.493 INFO:tasks.workunit.client.1.smithi203.stdout:7/491: readlink d8/d2a/d3f/d52/l4d 0 2024-04-05T03:02:41.494 INFO:tasks.workunit.client.0.smithi153.stdout:7/103: fdatasync d2/f6 0 2024-04-05T03:02:41.496 INFO:tasks.workunit.client.0.smithi153.stdout:5/91: creat d0/d9/f20 x:0 0 0 2024-04-05T03:02:41.497 INFO:tasks.workunit.client.0.smithi153.stdout:8/59: dwrite f4 [0,4194304] 0 2024-04-05T03:02:41.500 INFO:tasks.workunit.client.0.smithi153.stdout:1/88: mknod d5/dd/c1f 0 2024-04-05T03:02:41.500 INFO:tasks.workunit.client.1.smithi203.stdout:6/503: dread d0/da/d67/f5f [0,4194304] 0 2024-04-05T03:02:41.501 INFO:tasks.workunit.client.0.smithi153.stdout:0/97: dwrite f0 [4194304,4194304] 0 2024-04-05T03:02:41.502 INFO:tasks.workunit.client.1.smithi203.stdout:7/492: truncate d8/d32/d16/d1a/d21/f27 3504382 0 2024-04-05T03:02:41.503 INFO:tasks.workunit.client.0.smithi153.stdout:4/140: dread da/de/f14 [0,4194304] 0 2024-04-05T03:02:41.503 INFO:tasks.workunit.client.1.smithi203.stdout:0/447: dread dd/d17/f48 [0,4194304] 0 2024-04-05T03:02:41.505 INFO:tasks.workunit.client.1.smithi203.stdout:6/504: mknod d0/da/d67/d71/c97 0 2024-04-05T03:02:41.505 INFO:tasks.workunit.client.0.smithi153.stdout:1/89: stat d5/dd/d1b/f1d 0 2024-04-05T03:02:41.506 INFO:tasks.workunit.client.0.smithi153.stdout:0/98: stat d9/df/f17 0 2024-04-05T03:02:41.507 INFO:tasks.workunit.client.1.smithi203.stdout:6/505: write d0/da/d67/d71/f79 [746468,111433] 0 2024-04-05T03:02:41.507 INFO:tasks.workunit.client.0.smithi153.stdout:8/60: creat d7/f10 x:0 0 0 2024-04-05T03:02:41.508 INFO:tasks.workunit.client.0.smithi153.stdout:6/112: creat d4/dc/df/d16/d17/f2a x:0 0 0 2024-04-05T03:02:41.509 INFO:tasks.workunit.client.1.smithi203.stdout:7/493: mknod d8/d32/d5d/d95/c97 0 2024-04-05T03:02:41.510 INFO:tasks.workunit.client.1.smithi203.stdout:6/506: fsync d0/da/d54/f45 0 2024-04-05T03:02:41.511 INFO:tasks.workunit.client.0.smithi153.stdout:7/104: link d2/da/d1f/f10 d2/da/d11/d12/f22 0 2024-04-05T03:02:41.511 INFO:tasks.workunit.client.0.smithi153.stdout:1/90: dread - d5/db/d18/f1a zero size 2024-04-05T03:02:41.511 INFO:tasks.workunit.client.0.smithi153.stdout:9/36: rename d1/d9/fa to d1/fd 0 2024-04-05T03:02:41.511 INFO:tasks.workunit.client.0.smithi153.stdout:6/113: read - d4/f20 zero size 2024-04-05T03:02:41.512 INFO:tasks.workunit.client.0.smithi153.stdout:4/141: write da/de/f1e [747061,128813] 0 2024-04-05T03:02:41.512 INFO:tasks.workunit.client.1.smithi203.stdout:7/494: creat d8/d32/d16/d1a/d30/f98 x:0 0 0 2024-04-05T03:02:41.513 INFO:tasks.workunit.client.0.smithi153.stdout:0/99: creat d9/df/f18 x:0 0 0 2024-04-05T03:02:41.515 INFO:tasks.workunit.client.0.smithi153.stdout:7/105: write d2/da/d11/d12/f1b [925756,14115] 0 2024-04-05T03:02:41.515 INFO:tasks.workunit.client.0.smithi153.stdout:2/80: rmdir d9/d1e 0 2024-04-05T03:02:41.517 INFO:tasks.workunit.client.1.smithi203.stdout:0/448: rename dd/d17/d1d/d26/d41 to dd/d17/d1d/d22/d80 0 2024-04-05T03:02:41.518 INFO:tasks.workunit.client.0.smithi153.stdout:8/61: creat d7/f11 x:0 0 0 2024-04-05T03:02:41.518 INFO:tasks.workunit.client.0.smithi153.stdout:4/142: dread - da/db/d20/f21 zero size 2024-04-05T03:02:41.518 INFO:tasks.workunit.client.1.smithi203.stdout:7/495: readlink d8/d2a/d3f/d5e/l7c 0 2024-04-05T03:02:41.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/507: dwrite d0/da/d18/f1d [0,4194304] 0 2024-04-05T03:02:41.521 INFO:tasks.workunit.client.1.smithi203.stdout:7/496: read - d8/d32/d16/d1a/d30/f89 zero size 2024-04-05T03:02:41.523 INFO:tasks.workunit.client.1.smithi203.stdout:0/449: mknod dd/d17/d1d/d22/d53/d7f/c81 0 2024-04-05T03:02:41.526 INFO:tasks.workunit.client.1.smithi203.stdout:6/508: rename d0/da/d18/d38/d2a/l6c to d0/da/d67/l98 0 2024-04-05T03:02:41.528 INFO:tasks.workunit.client.1.smithi203.stdout:7/497: creat d8/d32/d5d/d95/f99 x:0 0 0 2024-04-05T03:02:41.529 INFO:tasks.workunit.client.1.smithi203.stdout:0/450: unlink dd/f27 0 2024-04-05T03:02:41.530 INFO:tasks.workunit.client.1.smithi203.stdout:6/509: dread d0/da/d54/f45 [0,4194304] 0 2024-04-05T03:02:41.532 INFO:tasks.workunit.client.0.smithi153.stdout:1/91: dread d5/db/fe [4194304,4194304] 0 2024-04-05T03:02:41.532 INFO:tasks.workunit.client.0.smithi153.stdout:7/106: symlink d2/da/l23 0 2024-04-05T03:02:41.533 INFO:tasks.workunit.client.1.smithi203.stdout:7/498: creat d8/d32/d16/d1a/d3d/d59/f9a x:0 0 0 2024-04-05T03:02:41.534 INFO:tasks.workunit.client.0.smithi153.stdout:6/114: mkdir d4/dc/df/d16/d2b 0 2024-04-05T03:02:41.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/499: dread - d8/da/f94 zero size 2024-04-05T03:02:41.535 INFO:tasks.workunit.client.1.smithi203.stdout:7/500: fdatasync d8/d2a/d3f/d5e/d77/f80 0 2024-04-05T03:02:41.538 INFO:tasks.workunit.client.0.smithi153.stdout:9/37: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:41.540 INFO:tasks.workunit.client.1.smithi203.stdout:7/501: dread d8/f31 [0,4194304] 0 2024-04-05T03:02:41.540 INFO:tasks.workunit.client.0.smithi153.stdout:8/62: dwrite f4 [0,4194304] 0 2024-04-05T03:02:41.541 INFO:tasks.workunit.client.1.smithi203.stdout:7/502: dread - d8/d2a/d3f/d5e/d63/f75 zero size 2024-04-05T03:02:41.543 INFO:tasks.workunit.client.1.smithi203.stdout:0/451: getdents dd/d17/d1d/d22/d45 0 2024-04-05T03:02:41.543 INFO:tasks.workunit.client.0.smithi153.stdout:2/81: link d9/ff d9/da/d19/f1f 0 2024-04-05T03:02:41.543 INFO:tasks.workunit.client.0.smithi153.stdout:4/143: dwrite da/ff [0,4194304] 0 2024-04-05T03:02:41.544 INFO:tasks.workunit.client.0.smithi153.stdout:7/107: creat d2/da/d11/f24 x:0 0 0 2024-04-05T03:02:41.545 INFO:tasks.workunit.client.0.smithi153.stdout:9/38: mknod d1/d9/ce 0 2024-04-05T03:02:41.545 INFO:tasks.workunit.client.1.smithi203.stdout:7/503: mkdir d8/d2a/d9b 0 2024-04-05T03:02:41.546 INFO:tasks.workunit.client.0.smithi153.stdout:6/115: creat d4/dc/df/d16/d17/f2c x:0 0 0 2024-04-05T03:02:41.547 INFO:tasks.workunit.client.0.smithi153.stdout:1/92: write d5/db/f11 [40014,24571] 0 2024-04-05T03:02:41.547 INFO:tasks.workunit.client.0.smithi153.stdout:8/63: creat d7/f12 x:0 0 0 2024-04-05T03:02:41.547 INFO:tasks.workunit.client.0.smithi153.stdout:0/100: link f3 d9/d14/f19 0 2024-04-05T03:02:41.548 INFO:tasks.workunit.client.1.smithi203.stdout:7/504: mkdir d8/da/d72/d9c 0 2024-04-05T03:02:41.549 INFO:tasks.workunit.client.1.smithi203.stdout:0/452: symlink dd/d17/d1d/d22/l82 0 2024-04-05T03:02:41.549 INFO:tasks.workunit.client.0.smithi153.stdout:2/82: creat d9/da/f20 x:0 0 0 2024-04-05T03:02:41.550 INFO:tasks.workunit.client.0.smithi153.stdout:1/93: dread - d5/db/d18/f1a zero size 2024-04-05T03:02:41.550 INFO:tasks.workunit.client.1.smithi203.stdout:0/453: fsync dd/d17/d1d/d22/d80/f72 0 2024-04-05T03:02:41.550 INFO:tasks.workunit.client.0.smithi153.stdout:6/116: unlink d4/c5 0 2024-04-05T03:02:41.550 INFO:tasks.workunit.client.0.smithi153.stdout:7/108: truncate d2/da/d11/d12/f22 4644000 0 2024-04-05T03:02:41.551 INFO:tasks.workunit.client.1.smithi203.stdout:7/505: mknod d8/d2a/d3f/d5e/c9d 0 2024-04-05T03:02:41.551 INFO:tasks.workunit.client.0.smithi153.stdout:2/83: fdatasync d9/da/fc 0 2024-04-05T03:02:41.551 INFO:tasks.workunit.client.0.smithi153.stdout:8/64: mkdir d7/d13 0 2024-04-05T03:02:41.552 INFO:tasks.workunit.client.1.smithi203.stdout:7/506: stat d8/d32/d16/d35/l41 0 2024-04-05T03:02:41.553 INFO:tasks.workunit.client.1.smithi203.stdout:7/507: readlink d8/d32/d16/d1a/d3d/l58 0 2024-04-05T03:02:41.553 INFO:tasks.workunit.client.0.smithi153.stdout:7/109: write d2/f6 [7147028,95272] 0 2024-04-05T03:02:41.558 INFO:tasks.workunit.client.0.smithi153.stdout:0/101: creat d9/f1a x:0 0 0 2024-04-05T03:02:41.558 INFO:tasks.workunit.client.0.smithi153.stdout:4/144: rename da/db/f24 to da/db/d20/f29 0 2024-04-05T03:02:41.558 INFO:tasks.workunit.client.1.smithi203.stdout:6/510: dread d0/da/d54/d53/f63 [0,4194304] 0 2024-04-05T03:02:41.559 INFO:tasks.workunit.client.0.smithi153.stdout:9/39: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:41.561 INFO:tasks.workunit.client.0.smithi153.stdout:2/84: fsync d9/da/f15 0 2024-04-05T03:02:41.563 INFO:tasks.workunit.client.0.smithi153.stdout:8/65: creat d7/d13/f14 x:0 0 0 2024-04-05T03:02:41.564 INFO:tasks.workunit.client.1.smithi203.stdout:7/508: dread d8/d32/f56 [0,4194304] 0 2024-04-05T03:02:41.566 INFO:tasks.workunit.client.0.smithi153.stdout:4/145: mkdir da/d2a 0 2024-04-05T03:02:41.567 INFO:tasks.workunit.client.0.smithi153.stdout:6/117: dwrite d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:41.568 INFO:tasks.workunit.client.0.smithi153.stdout:2/85: creat d9/f21 x:0 0 0 2024-04-05T03:02:41.568 INFO:tasks.workunit.client.0.smithi153.stdout:8/66: unlink d7/f11 0 2024-04-05T03:02:41.571 INFO:tasks.workunit.client.0.smithi153.stdout:8/67: write d7/d13/f14 [882438,95932] 0 2024-04-05T03:02:41.574 INFO:tasks.workunit.client.0.smithi153.stdout:2/86: creat d9/da/d19/f22 x:0 0 0 2024-04-05T03:02:41.574 INFO:tasks.workunit.client.0.smithi153.stdout:1/94: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:41.580 INFO:tasks.workunit.client.0.smithi153.stdout:1/95: dread d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:41.580 INFO:tasks.workunit.client.0.smithi153.stdout:8/68: creat d7/d13/f15 x:0 0 0 2024-04-05T03:02:41.580 INFO:tasks.workunit.client.0.smithi153.stdout:2/87: symlink d9/da/l23 0 2024-04-05T03:02:41.584 INFO:tasks.workunit.client.0.smithi153.stdout:8/69: dread - d7/ff zero size 2024-04-05T03:02:41.585 INFO:tasks.workunit.client.0.smithi153.stdout:1/96: chown f4 90562734 1 2024-04-05T03:02:41.585 INFO:tasks.workunit.client.1.smithi203.stdout:7/509: write d8/d32/d16/d1a/d21/f27 [2601277,96132] 0 2024-04-05T03:02:41.588 INFO:tasks.workunit.client.0.smithi153.stdout:4/146: dwrite da/f26 [0,4194304] 0 2024-04-05T03:02:41.588 INFO:tasks.workunit.client.1.smithi203.stdout:7/510: creat d8/d6c/f9e x:0 0 0 2024-04-05T03:02:41.592 INFO:tasks.workunit.client.0.smithi153.stdout:8/70: dread d7/fa [0,4194304] 0 2024-04-05T03:02:41.592 INFO:tasks.workunit.client.1.smithi203.stdout:6/511: dread d0/da/d67/f28 [0,4194304] 0 2024-04-05T03:02:41.593 INFO:tasks.workunit.client.1.smithi203.stdout:7/511: creat d8/d2a/d3f/d52/d65/f9f x:0 0 0 2024-04-05T03:02:41.593 INFO:tasks.workunit.client.0.smithi153.stdout:8/71: chown f4 1968 1 2024-04-05T03:02:41.594 INFO:tasks.workunit.client.0.smithi153.stdout:8/72: dread - f3 zero size 2024-04-05T03:02:41.595 INFO:tasks.workunit.client.0.smithi153.stdout:9/40: write d1/d9/fc [46867,125160] 0 2024-04-05T03:02:41.596 INFO:tasks.workunit.client.0.smithi153.stdout:8/73: stat d7/d13 0 2024-04-05T03:02:41.596 INFO:tasks.workunit.client.1.smithi203.stdout:7/512: dread d8/d2a/d3f/d5e/d77/f80 [0,4194304] 0 2024-04-05T03:02:41.597 INFO:tasks.workunit.client.0.smithi153.stdout:4/147: write da/f17 [3825237,15277] 0 2024-04-05T03:02:41.598 INFO:tasks.workunit.client.0.smithi153.stdout:9/41: write d1/fd [451535,80414] 0 2024-04-05T03:02:41.598 INFO:tasks.workunit.client.0.smithi153.stdout:1/97: dread d5/db/f11 [0,4194304] 0 2024-04-05T03:02:41.600 INFO:tasks.workunit.client.1.smithi203.stdout:7/513: symlink d8/d32/d16/d1a/d21/la0 0 2024-04-05T03:02:41.600 INFO:tasks.workunit.client.0.smithi153.stdout:9/42: chown d1/d9/ce 50363 1 2024-04-05T03:02:41.601 INFO:tasks.workunit.client.1.smithi203.stdout:6/512: dwrite d0/da/d18/d38/f6d [0,4194304] 0 2024-04-05T03:02:41.602 INFO:tasks.workunit.client.0.smithi153.stdout:8/74: creat d7/f16 x:0 0 0 2024-04-05T03:02:41.602 INFO:tasks.workunit.client.0.smithi153.stdout:9/43: write d1/d9/fc [78553,73175] 0 2024-04-05T03:02:41.605 INFO:tasks.workunit.client.0.smithi153.stdout:1/98: mkdir d5/d8/d20 0 2024-04-05T03:02:41.606 INFO:tasks.workunit.client.0.smithi153.stdout:8/75: creat d7/f17 x:0 0 0 2024-04-05T03:02:41.608 INFO:tasks.workunit.client.1.smithi203.stdout:7/514: symlink d8/d2a/d3f/d52/d65/la1 0 2024-04-05T03:02:41.608 INFO:tasks.workunit.client.1.smithi203.stdout:6/513: creat d0/da/d18/d38/d86/f99 x:0 0 0 2024-04-05T03:02:41.609 INFO:tasks.workunit.client.1.smithi203.stdout:6/514: write d0/da/fd [2494928,78993] 0 2024-04-05T03:02:41.610 INFO:tasks.workunit.client.1.smithi203.stdout:7/515: chown d8/d32/d16/d1a/d3d/d59/f61 303 1 2024-04-05T03:02:41.611 INFO:tasks.workunit.client.0.smithi153.stdout:1/99: chown d5/db/d18/f1a 40 1 2024-04-05T03:02:41.613 INFO:tasks.workunit.client.1.smithi203.stdout:7/516: rename d8/d32/d5d/d95/f99 to d8/d32/d5d/d95/fa2 0 2024-04-05T03:02:41.614 INFO:tasks.workunit.client.0.smithi153.stdout:8/76: write d7/f12 [568099,122242] 0 2024-04-05T03:02:41.614 INFO:tasks.workunit.client.0.smithi153.stdout:4/148: dwrite da/de/f1c [0,4194304] 0 2024-04-05T03:02:41.614 INFO:tasks.workunit.client.0.smithi153.stdout:9/44: rename d1/c5 to d1/d9/cf 0 2024-04-05T03:02:41.615 INFO:tasks.workunit.client.1.smithi203.stdout:7/517: truncate d8/d32/d16/d1a/d5a/f85 4725249 0 2024-04-05T03:02:41.615 INFO:tasks.workunit.client.1.smithi203.stdout:7/518: write d8/f4c [760038,39911] 0 2024-04-05T03:02:41.618 INFO:tasks.workunit.client.0.smithi153.stdout:9/45: fsync d1/d9/fc 0 2024-04-05T03:02:41.619 INFO:tasks.workunit.client.0.smithi153.stdout:2/88: dwrite d9/da/fe [0,4194304] 0 2024-04-05T03:02:41.621 INFO:tasks.workunit.client.1.smithi203.stdout:4/341: sync 2024-04-05T03:02:41.621 INFO:tasks.workunit.client.1.smithi203.stdout:9/383: sync 2024-04-05T03:02:41.622 INFO:tasks.workunit.client.0.smithi153.stdout:1/100: rmdir d5/dd 39 2024-04-05T03:02:41.622 INFO:tasks.workunit.client.0.smithi153.stdout:1/101: chown d5 69734370 1 2024-04-05T03:02:41.622 INFO:tasks.workunit.client.0.smithi153.stdout:1/102: chown d5/f7 463 1 2024-04-05T03:02:41.623 INFO:tasks.workunit.client.0.smithi153.stdout:8/77: write f3 [459963,947] 0 2024-04-05T03:02:41.624 INFO:tasks.workunit.client.1.smithi203.stdout:4/342: write da/f4a [1430100,1360] 0 2024-04-05T03:02:41.624 INFO:tasks.workunit.client.0.smithi153.stdout:8/78: chown d7/fe 213 1 2024-04-05T03:02:41.625 INFO:tasks.workunit.client.1.smithi203.stdout:7/519: dwrite d8/d32/d16/d1a/d30/f81 [0,4194304] 0 2024-04-05T03:02:41.626 INFO:tasks.workunit.client.0.smithi153.stdout:9/46: mkdir d1/d10 0 2024-04-05T03:02:41.628 INFO:tasks.workunit.client.1.smithi203.stdout:7/520: fdatasync d8/d32/d16/d35/d64/f8e 0 2024-04-05T03:02:41.631 INFO:tasks.workunit.client.0.smithi153.stdout:2/89: truncate d9/da/fb 2308506 0 2024-04-05T03:02:41.632 INFO:tasks.workunit.client.0.smithi153.stdout:2/90: chown d9 28153 1 2024-04-05T03:02:41.634 INFO:tasks.workunit.client.0.smithi153.stdout:8/79: creat d7/d13/f18 x:0 0 0 2024-04-05T03:02:41.635 INFO:tasks.workunit.client.0.smithi153.stdout:8/80: readlink d7/lb 0 2024-04-05T03:02:41.635 INFO:tasks.workunit.client.0.smithi153.stdout:8/81: write d7/ff [805653,64651] 0 2024-04-05T03:02:41.636 INFO:tasks.workunit.client.0.smithi153.stdout:9/47: creat d1/d9/f11 x:0 0 0 2024-04-05T03:02:41.637 INFO:tasks.workunit.client.1.smithi203.stdout:7/521: rename d8/d6c/f9e to d8/d32/d16/d1a/d5a/fa3 0 2024-04-05T03:02:41.637 INFO:tasks.workunit.client.0.smithi153.stdout:2/91: creat d9/da/f24 x:0 0 0 2024-04-05T03:02:41.638 INFO:tasks.workunit.client.0.smithi153.stdout:2/92: fdatasync d9/da/fc 0 2024-04-05T03:02:41.641 INFO:tasks.workunit.client.0.smithi153.stdout:1/103: rename d5/db to d5/dd/d21 0 2024-04-05T03:02:41.641 INFO:tasks.workunit.client.0.smithi153.stdout:1/104: dread - d5/dd/d21/d18/f1c zero size 2024-04-05T03:02:41.642 INFO:tasks.workunit.client.0.smithi153.stdout:8/82: mkdir d7/d13/d19 0 2024-04-05T03:02:41.643 INFO:tasks.workunit.client.0.smithi153.stdout:1/105: chown d5/dd/d21/c17 6 1 2024-04-05T03:02:41.643 INFO:tasks.workunit.client.0.smithi153.stdout:8/83: stat d7/fe 0 2024-04-05T03:02:41.643 INFO:tasks.workunit.client.0.smithi153.stdout:9/48: symlink d1/l12 0 2024-04-05T03:02:41.644 INFO:tasks.workunit.client.0.smithi153.stdout:2/93: rename d9/da/d10/l17 to d9/da/l25 0 2024-04-05T03:02:41.644 INFO:tasks.workunit.client.1.smithi203.stdout:7/522: dwrite d8/d2a/f7e [0,4194304] 0 2024-04-05T03:02:41.645 INFO:tasks.workunit.client.0.smithi153.stdout:3/69: sync 2024-04-05T03:02:41.646 INFO:tasks.workunit.client.0.smithi153.stdout:8/84: mknod d7/d13/c1a 0 2024-04-05T03:02:41.646 INFO:tasks.workunit.client.1.smithi203.stdout:7/523: chown d8/d32/d16/d35/d64 65365 1 2024-04-05T03:02:41.651 INFO:tasks.workunit.client.1.smithi203.stdout:7/524: write d8/d32/f44 [2875510,70815] 0 2024-04-05T03:02:41.651 INFO:tasks.workunit.client.0.smithi153.stdout:2/94: symlink d9/da/d10/l26 0 2024-04-05T03:02:41.651 INFO:tasks.workunit.client.1.smithi203.stdout:9/384: dwrite d13/d1b/d33/d64/f6d [0,4194304] 0 2024-04-05T03:02:41.652 INFO:tasks.workunit.client.0.smithi153.stdout:3/70: rename le to l16 0 2024-04-05T03:02:41.656 INFO:tasks.workunit.client.0.smithi153.stdout:2/95: chown d9/da/d19/l1b 2771293 1 2024-04-05T03:02:41.661 INFO:tasks.workunit.client.1.smithi203.stdout:7/525: getdents d8/d32/d5d/d6d 0 2024-04-05T03:02:41.663 INFO:tasks.workunit.client.0.smithi153.stdout:8/85: mkdir d7/d13/d19/d1b 0 2024-04-05T03:02:41.664 INFO:tasks.workunit.client.1.smithi203.stdout:7/526: fsync d8/fe 0 2024-04-05T03:02:41.664 INFO:tasks.workunit.client.1.smithi203.stdout:9/385: dwrite d13/d1b/d33/d42/f49 [0,4194304] 0 2024-04-05T03:02:41.665 INFO:tasks.workunit.client.0.smithi153.stdout:1/106: dwrite d5/dd/d21/d18/f1a [0,4194304] 0 2024-04-05T03:02:41.666 INFO:tasks.workunit.client.0.smithi153.stdout:3/71: creat f17 x:0 0 0 2024-04-05T03:02:41.666 INFO:tasks.workunit.client.0.smithi153.stdout:9/49: rmdir d1/d9 39 2024-04-05T03:02:41.667 INFO:tasks.workunit.client.0.smithi153.stdout:9/50: chown d1/l12 2684 1 2024-04-05T03:02:41.667 INFO:tasks.workunit.client.0.smithi153.stdout:3/72: write fd [9380980,32876] 0 2024-04-05T03:02:41.669 INFO:tasks.workunit.client.0.smithi153.stdout:3/73: write f17 [934405,127635] 0 2024-04-05T03:02:41.671 INFO:tasks.workunit.client.0.smithi153.stdout:3/74: read f3 [2828902,21092] 0 2024-04-05T03:02:41.673 INFO:tasks.workunit.client.0.smithi153.stdout:8/86: dread f6 [0,4194304] 0 2024-04-05T03:02:41.674 INFO:tasks.workunit.client.1.smithi203.stdout:7/527: dwrite d8/d32/d16/d1a/f5c [0,4194304] 0 2024-04-05T03:02:41.674 INFO:tasks.workunit.client.0.smithi153.stdout:8/87: dread - d7/fe zero size 2024-04-05T03:02:41.679 INFO:tasks.workunit.client.1.smithi203.stdout:7/528: mknod d8/d32/d16/d1a/d30/ca4 0 2024-04-05T03:02:41.681 INFO:tasks.workunit.client.1.smithi203.stdout:9/386: write d13/d1b/d33/d64/f75 [2260843,4466] 0 2024-04-05T03:02:41.683 INFO:tasks.workunit.client.0.smithi153.stdout:9/51: unlink d1/c3 0 2024-04-05T03:02:41.684 INFO:tasks.workunit.client.1.smithi203.stdout:7/529: read d8/d2a/d3f/d52/f2e [1439056,16325] 0 2024-04-05T03:02:41.685 INFO:tasks.workunit.client.1.smithi203.stdout:9/387: creat d13/d1b/d33/d64/f79 x:0 0 0 2024-04-05T03:02:41.687 INFO:tasks.workunit.client.1.smithi203.stdout:7/530: rmdir d8/d32/d16/d1a/d3d 39 2024-04-05T03:02:41.688 INFO:tasks.workunit.client.1.smithi203.stdout:9/388: stat d13/d1b/c30 0 2024-04-05T03:02:41.691 INFO:tasks.workunit.client.0.smithi153.stdout:2/96: write d9/da/d19/f1f [124080,13138] 0 2024-04-05T03:02:41.691 INFO:tasks.workunit.client.1.smithi203.stdout:7/531: creat d8/d2a/d3f/d5e/fa5 x:0 0 0 2024-04-05T03:02:41.694 INFO:tasks.workunit.client.0.smithi153.stdout:2/97: write d9/da/fc [3078636,5110] 0 2024-04-05T03:02:41.695 INFO:tasks.workunit.client.0.smithi153.stdout:2/98: readlink l2 0 2024-04-05T03:02:41.696 INFO:tasks.workunit.client.1.smithi203.stdout:7/532: creat d8/d32/d16/fa6 x:0 0 0 2024-04-05T03:02:41.696 INFO:tasks.workunit.client.1.smithi203.stdout:7/533: stat d8/f33 0 2024-04-05T03:02:41.698 INFO:tasks.workunit.client.1.smithi203.stdout:7/534: write d8/d2a/d3f/d52/f5b [3671417,4789] 0 2024-04-05T03:02:41.698 INFO:tasks.workunit.client.0.smithi153.stdout:2/99: fdatasync d9/da/f24 0 2024-04-05T03:02:41.699 INFO:tasks.workunit.client.0.smithi153.stdout:9/52: dwrite d1/d9/f11 [0,4194304] 0 2024-04-05T03:02:41.702 INFO:tasks.workunit.client.0.smithi153.stdout:3/75: truncate fd 6814806 0 2024-04-05T03:02:41.704 INFO:tasks.workunit.client.0.smithi153.stdout:8/88: write d7/fa [1258849,95588] 0 2024-04-05T03:02:41.705 INFO:tasks.workunit.client.1.smithi203.stdout:9/389: truncate d13/d1b/d33/d64/f75 2298575 0 2024-04-05T03:02:41.708 INFO:tasks.workunit.client.0.smithi153.stdout:9/53: chown d1/d9/cf 1712575304 1 2024-04-05T03:02:41.709 INFO:tasks.workunit.client.1.smithi203.stdout:9/390: mkdir d13/d1b/d6b/d7a 0 2024-04-05T03:02:41.709 INFO:tasks.workunit.client.1.smithi203.stdout:7/535: dread d8/d32/d16/d1a/d3d/d59/f61 [0,4194304] 0 2024-04-05T03:02:41.709 INFO:tasks.workunit.client.0.smithi153.stdout:3/76: unlink l7 0 2024-04-05T03:02:41.709 INFO:tasks.workunit.client.0.smithi153.stdout:1/107: dwrite f4 [0,4194304] 0 2024-04-05T03:02:41.711 INFO:tasks.workunit.client.1.smithi203.stdout:7/536: readlink d8/d32/d16/d1a/d21/l48 0 2024-04-05T03:02:41.713 INFO:tasks.workunit.client.1.smithi203.stdout:7/537: chown d8/d2a/d3f/d52/d65/la1 33093772 1 2024-04-05T03:02:41.715 INFO:tasks.workunit.client.1.smithi203.stdout:7/538: chown d8/d32/d16/d1a/d21/f27 2449 1 2024-04-05T03:02:41.715 INFO:tasks.workunit.client.0.smithi153.stdout:3/77: write f3 [4041988,86155] 0 2024-04-05T03:02:41.718 INFO:tasks.workunit.client.0.smithi153.stdout:8/89: link f0 d7/d13/d19/f1c 0 2024-04-05T03:02:41.718 INFO:tasks.workunit.client.1.smithi203.stdout:9/391: dwrite d13/f24 [0,4194304] 0 2024-04-05T03:02:41.719 INFO:tasks.workunit.client.1.smithi203.stdout:7/539: write d8/d32/d16/d1a/d5a/f85 [2019706,56728] 0 2024-04-05T03:02:41.720 INFO:tasks.workunit.client.0.smithi153.stdout:2/100: getdents d9/da 0 2024-04-05T03:02:41.721 INFO:tasks.workunit.client.0.smithi153.stdout:1/108: creat d5/dd/d21/d18/d16/f22 x:0 0 0 2024-04-05T03:02:41.721 INFO:tasks.workunit.client.0.smithi153.stdout:3/78: creat f18 x:0 0 0 2024-04-05T03:02:41.723 INFO:tasks.workunit.client.1.smithi203.stdout:9/392: truncate d13/d1b/d33/d63/f6e 317932 0 2024-04-05T03:02:41.723 INFO:tasks.workunit.client.1.smithi203.stdout:7/540: symlink d8/d2a/d3f/d5e/d77/la7 0 2024-04-05T03:02:41.724 INFO:tasks.workunit.client.0.smithi153.stdout:3/79: write f17 [1059543,26420] 0 2024-04-05T03:02:41.726 INFO:tasks.workunit.client.0.smithi153.stdout:3/80: dread - f18 zero size 2024-04-05T03:02:41.728 INFO:tasks.workunit.client.1.smithi203.stdout:7/541: unlink d8/d32/d16/d1a/d21/l26 0 2024-04-05T03:02:41.729 INFO:tasks.workunit.client.0.smithi153.stdout:8/90: dread d7/d13/d19/f1c [0,4194304] 0 2024-04-05T03:02:41.733 INFO:tasks.workunit.client.1.smithi203.stdout:7/542: truncate d8/d32/f43 471703 0 2024-04-05T03:02:41.738 INFO:tasks.workunit.client.0.smithi153.stdout:8/91: truncate d7/fc 5042038 0 2024-04-05T03:02:41.741 INFO:tasks.workunit.client.1.smithi203.stdout:7/543: getdents d8/d2a/d3f/d5e/d63 0 2024-04-05T03:02:41.741 INFO:tasks.workunit.client.0.smithi153.stdout:2/101: dwrite d9/da/d19/f1f [4194304,4194304] 0 2024-04-05T03:02:41.742 INFO:tasks.workunit.client.0.smithi153.stdout:8/92: symlink d7/l1d 0 2024-04-05T03:02:41.744 INFO:tasks.workunit.client.0.smithi153.stdout:9/54: rename d1/d9/cf to d1/db/c13 0 2024-04-05T03:02:41.744 INFO:tasks.workunit.client.0.smithi153.stdout:1/109: dread d5/f7 [0,4194304] 0 2024-04-05T03:02:41.747 INFO:tasks.workunit.client.0.smithi153.stdout:3/81: dwrite f3 [0,4194304] 0 2024-04-05T03:02:41.747 INFO:tasks.workunit.client.0.smithi153.stdout:8/93: chown d7/d13/d19/f1c 261 1 2024-04-05T03:02:41.750 INFO:tasks.workunit.client.1.smithi203.stdout:9/393: dread d13/d1b/d26/d31/d38/f68 [0,4194304] 0 2024-04-05T03:02:41.751 INFO:tasks.workunit.client.0.smithi153.stdout:2/102: write d9/da/f18 [200639,82894] 0 2024-04-05T03:02:41.752 INFO:tasks.workunit.client.0.smithi153.stdout:1/110: mknod d5/dd/c23 0 2024-04-05T03:02:41.756 INFO:tasks.workunit.client.0.smithi153.stdout:8/94: mknod d7/d13/c1e 0 2024-04-05T03:02:41.756 INFO:tasks.workunit.client.1.smithi203.stdout:7/544: truncate d8/da/f1d 2108713 0 2024-04-05T03:02:41.756 INFO:tasks.workunit.client.0.smithi153.stdout:9/55: rename d1/c8 to d1/db/c14 0 2024-04-05T03:02:41.757 INFO:tasks.workunit.client.0.smithi153.stdout:3/82: dread f17 [0,4194304] 0 2024-04-05T03:02:41.758 INFO:tasks.workunit.client.0.smithi153.stdout:2/103: dread d9/da/d19/f1f [0,4194304] 0 2024-04-05T03:02:41.760 INFO:tasks.workunit.client.0.smithi153.stdout:1/111: mknod d5/dd/c24 0 2024-04-05T03:02:41.760 INFO:tasks.workunit.client.0.smithi153.stdout:3/83: write f18 [938474,34814] 0 2024-04-05T03:02:41.761 INFO:tasks.workunit.client.1.smithi203.stdout:7/545: write d8/d32/d16/d1a/d5a/f85 [4957872,82581] 0 2024-04-05T03:02:41.761 INFO:tasks.workunit.client.0.smithi153.stdout:2/104: fsync d9/da/f18 0 2024-04-05T03:02:41.762 INFO:tasks.workunit.client.1.smithi203.stdout:9/394: dwrite d13/d1b/d26/f54 [0,4194304] 0 2024-04-05T03:02:41.762 INFO:tasks.workunit.client.1.smithi203.stdout:7/546: readlink d8/d2a/d3f/d5e/d63/l88 0 2024-04-05T03:02:41.766 INFO:tasks.workunit.client.1.smithi203.stdout:7/547: mkdir d8/d2a/d3f/d73/da8 0 2024-04-05T03:02:41.768 INFO:tasks.workunit.client.0.smithi153.stdout:3/84: rename c15 to c19 0 2024-04-05T03:02:41.769 INFO:tasks.workunit.client.0.smithi153.stdout:8/95: dwrite d7/fc [0,4194304] 0 2024-04-05T03:02:41.770 INFO:tasks.workunit.client.0.smithi153.stdout:1/112: creat d5/d8/d20/f25 x:0 0 0 2024-04-05T03:02:41.770 INFO:tasks.workunit.client.1.smithi203.stdout:9/395: mkdir d13/d1b/d26/d31/d38/d59/d7b 0 2024-04-05T03:02:41.772 INFO:tasks.workunit.client.0.smithi153.stdout:9/56: dread d1/fd [0,4194304] 0 2024-04-05T03:02:41.772 INFO:tasks.workunit.client.0.smithi153.stdout:3/85: write f17 [187858,53329] 0 2024-04-05T03:02:41.773 INFO:tasks.workunit.client.0.smithi153.stdout:8/96: truncate d7/f12 1718682 0 2024-04-05T03:02:41.773 INFO:tasks.workunit.client.0.smithi153.stdout:1/113: dread - d5/dd/d21/d18/f1c zero size 2024-04-05T03:02:41.775 INFO:tasks.workunit.client.0.smithi153.stdout:3/86: write f3 [5173474,13896] 0 2024-04-05T03:02:41.775 INFO:tasks.workunit.client.0.smithi153.stdout:3/87: stat c12 0 2024-04-05T03:02:41.777 INFO:tasks.workunit.client.0.smithi153.stdout:8/97: write f6 [2595630,45606] 0 2024-04-05T03:02:41.781 INFO:tasks.workunit.client.0.smithi153.stdout:1/114: creat d5/dd/f26 x:0 0 0 2024-04-05T03:02:41.781 INFO:tasks.workunit.client.1.smithi203.stdout:9/396: dwrite d13/f1d [0,4194304] 0 2024-04-05T03:02:41.781 INFO:tasks.workunit.client.0.smithi153.stdout:1/115: fdatasync f4 0 2024-04-05T03:02:41.785 INFO:tasks.workunit.client.0.smithi153.stdout:3/88: creat f1a x:0 0 0 2024-04-05T03:02:41.787 INFO:tasks.workunit.client.0.smithi153.stdout:8/98: unlink d7/d13/d19/f1c 0 2024-04-05T03:02:41.789 INFO:tasks.workunit.client.1.smithi203.stdout:9/397: truncate d13/d1b/d26/d31/f41 214964 0 2024-04-05T03:02:41.790 INFO:tasks.workunit.client.0.smithi153.stdout:1/116: mknod d5/dd/c27 0 2024-04-05T03:02:41.790 INFO:tasks.workunit.client.1.smithi203.stdout:9/398: write d13/d1b/d26/f54 [310926,54570] 0 2024-04-05T03:02:41.793 INFO:tasks.workunit.client.0.smithi153.stdout:9/57: creat d1/f15 x:0 0 0 2024-04-05T03:02:41.794 INFO:tasks.workunit.client.0.smithi153.stdout:0/102: sync 2024-04-05T03:02:41.794 INFO:tasks.workunit.client.0.smithi153.stdout:7/110: sync 2024-04-05T03:02:41.794 INFO:tasks.workunit.client.0.smithi153.stdout:5/92: sync 2024-04-05T03:02:41.797 INFO:tasks.workunit.client.1.smithi203.stdout:9/399: read f7 [539895,27971] 0 2024-04-05T03:02:41.799 INFO:tasks.workunit.client.1.smithi203.stdout:7/548: dwrite d8/da/f1d [0,4194304] 0 2024-04-05T03:02:41.800 INFO:tasks.workunit.client.0.smithi153.stdout:1/117: dread d5/dd/d21/f11 [0,4194304] 0 2024-04-05T03:02:41.801 INFO:tasks.workunit.client.1.smithi203.stdout:9/400: symlink d13/d1b/d26/l7c 0 2024-04-05T03:02:41.802 INFO:tasks.workunit.client.0.smithi153.stdout:3/89: link l13 l1b 0 2024-04-05T03:02:41.803 INFO:tasks.workunit.client.0.smithi153.stdout:3/90: stat l11 0 2024-04-05T03:02:41.805 INFO:tasks.workunit.client.0.smithi153.stdout:1/118: dread d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:41.805 INFO:tasks.workunit.client.1.smithi203.stdout:7/549: creat d8/d32/d16/fa9 x:0 0 0 2024-04-05T03:02:41.807 INFO:tasks.workunit.client.1.smithi203.stdout:9/401: creat d13/d1b/d26/d31/d38/d77/f7d x:0 0 0 2024-04-05T03:02:41.808 INFO:tasks.workunit.client.1.smithi203.stdout:7/550: write d8/d2a/d3f/d5e/fa5 [65687,35348] 0 2024-04-05T03:02:41.808 INFO:tasks.workunit.client.0.smithi153.stdout:0/103: creat d9/df/d12/f1b x:0 0 0 2024-04-05T03:02:41.809 INFO:tasks.workunit.client.1.smithi203.stdout:9/402: chown d13/d1b/d26/d31/d38/f68 2 1 2024-04-05T03:02:41.809 INFO:tasks.workunit.client.1.smithi203.stdout:2/615: sync 2024-04-05T03:02:41.809 INFO:tasks.workunit.client.1.smithi203.stdout:5/466: sync 2024-04-05T03:02:41.809 INFO:tasks.workunit.client.1.smithi203.stdout:8/500: sync 2024-04-05T03:02:41.809 INFO:tasks.workunit.client.1.smithi203.stdout:1/426: sync 2024-04-05T03:02:41.810 INFO:tasks.workunit.client.1.smithi203.stdout:7/551: read d8/d2a/f40 [57571,97344] 0 2024-04-05T03:02:41.811 INFO:tasks.workunit.client.0.smithi153.stdout:7/111: dwrite d2/da/d11/f24 [0,4194304] 0 2024-04-05T03:02:41.813 INFO:tasks.workunit.client.0.smithi153.stdout:9/58: mkdir d1/d10/d16 0 2024-04-05T03:02:41.813 INFO:tasks.workunit.client.1.smithi203.stdout:2/616: symlink d5/d10/d2f/ld9 0 2024-04-05T03:02:41.814 INFO:tasks.workunit.client.0.smithi153.stdout:3/91: rename ca to c1c 0 2024-04-05T03:02:41.814 INFO:tasks.workunit.client.1.smithi203.stdout:5/467: creat d3/d1f/d59/d60/f99 x:0 0 0 2024-04-05T03:02:41.815 INFO:tasks.workunit.client.1.smithi203.stdout:9/403: creat d13/d1b/d26/d31/d38/d59/f7e x:0 0 0 2024-04-05T03:02:41.816 INFO:tasks.workunit.client.0.smithi153.stdout:8/99: write f0 [3052228,41674] 0 2024-04-05T03:02:41.817 INFO:tasks.workunit.client.0.smithi153.stdout:3/92: mkdir d1d 0 2024-04-05T03:02:41.817 INFO:tasks.workunit.client.0.smithi153.stdout:2/105: dwrite d9/f11 [0,4194304] 0 2024-04-05T03:02:41.818 INFO:tasks.workunit.client.1.smithi203.stdout:1/427: mknod d7/d24/d33/d48/c84 0 2024-04-05T03:02:41.818 INFO:tasks.workunit.client.0.smithi153.stdout:1/119: truncate d5/dd/d21/d18/f1c 189493 0 2024-04-05T03:02:41.819 INFO:tasks.workunit.client.0.smithi153.stdout:7/112: creat d2/da/d1f/f25 x:0 0 0 2024-04-05T03:02:41.822 INFO:tasks.workunit.client.0.smithi153.stdout:8/100: dread - d7/f16 zero size 2024-04-05T03:02:41.824 INFO:tasks.workunit.client.1.smithi203.stdout:2/617: truncate d5/d66/fcb 343474 0 2024-04-05T03:02:41.825 INFO:tasks.workunit.client.0.smithi153.stdout:0/104: mknod d9/d14/c1c 0 2024-04-05T03:02:41.827 INFO:tasks.workunit.client.0.smithi153.stdout:1/120: write d5/dd/f26 [575598,36141] 0 2024-04-05T03:02:41.828 INFO:tasks.workunit.client.0.smithi153.stdout:5/93: dwrite d0/f14 [0,4194304] 0 2024-04-05T03:02:41.828 INFO:tasks.workunit.client.1.smithi203.stdout:7/552: dwrite d8/d2a/f8c [0,4194304] 0 2024-04-05T03:02:41.828 INFO:tasks.workunit.client.0.smithi153.stdout:7/113: fdatasync d2/da/d11/f13 0 2024-04-05T03:02:41.831 INFO:tasks.workunit.client.0.smithi153.stdout:0/105: chown f0 7 1 2024-04-05T03:02:41.832 INFO:tasks.workunit.client.0.smithi153.stdout:8/101: fdatasync f0 0 2024-04-05T03:02:41.834 INFO:tasks.workunit.client.0.smithi153.stdout:9/59: dread d1/d9/f11 [0,4194304] 0 2024-04-05T03:02:41.837 INFO:tasks.workunit.client.1.smithi203.stdout:8/501: dwrite d1/d12/d17/d47/d53/f82 [0,4194304] 0 2024-04-05T03:02:41.837 INFO:tasks.workunit.client.1.smithi203.stdout:1/428: unlink d7/d24/d33/f44 0 2024-04-05T03:02:41.838 INFO:tasks.workunit.client.1.smithi203.stdout:8/502: stat d1/c33 0 2024-04-05T03:02:41.839 INFO:tasks.workunit.client.0.smithi153.stdout:5/94: creat d0/d9/f21 x:0 0 0 2024-04-05T03:02:41.840 INFO:tasks.workunit.client.1.smithi203.stdout:2/618: read d5/d10/db8/f56 [539073,110488] 0 2024-04-05T03:02:41.840 INFO:tasks.workunit.client.0.smithi153.stdout:0/106: dread d9/df/f17 [0,4194304] 0 2024-04-05T03:02:41.841 INFO:tasks.workunit.client.0.smithi153.stdout:2/106: fdatasync d9/da/d19/f1f 0 2024-04-05T03:02:41.842 INFO:tasks.workunit.client.0.smithi153.stdout:4/149: sync 2024-04-05T03:02:41.842 INFO:tasks.workunit.client.0.smithi153.stdout:6/118: sync 2024-04-05T03:02:41.843 INFO:tasks.workunit.client.1.smithi203.stdout:9/404: dwrite d13/f2a [0,4194304] 0 2024-04-05T03:02:41.843 INFO:tasks.workunit.client.0.smithi153.stdout:8/102: mknod d7/d13/d19/c1f 0 2024-04-05T03:02:41.844 INFO:tasks.workunit.client.0.smithi153.stdout:6/119: chown d4/dc/df 6157207 1 2024-04-05T03:02:41.846 INFO:tasks.workunit.client.0.smithi153.stdout:9/60: rename d1/fd to d1/d10/d16/f17 0 2024-04-05T03:02:41.847 INFO:tasks.workunit.client.0.smithi153.stdout:1/121: dread d5/dd/f26 [0,4194304] 0 2024-04-05T03:02:41.848 INFO:tasks.workunit.client.0.smithi153.stdout:8/103: write d7/fe [796531,123594] 0 2024-04-05T03:02:41.849 INFO:tasks.workunit.client.1.smithi203.stdout:7/553: rename d8/c11 to d8/d2a/caa 0 2024-04-05T03:02:41.850 INFO:tasks.workunit.client.0.smithi153.stdout:7/114: dwrite d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:41.851 INFO:tasks.workunit.client.0.smithi153.stdout:6/120: write d4/fb [2411552,129566] 0 2024-04-05T03:02:41.851 INFO:tasks.workunit.client.0.smithi153.stdout:5/95: mkdir d0/d9/d22 0 2024-04-05T03:02:41.852 INFO:tasks.workunit.client.1.smithi203.stdout:8/503: write d1/d6/d2e/f30 [1198955,97909] 0 2024-04-05T03:02:41.853 INFO:tasks.workunit.client.0.smithi153.stdout:0/107: rmdir d9/d14 39 2024-04-05T03:02:41.855 INFO:tasks.workunit.client.0.smithi153.stdout:0/108: fsync d9/db/f13 0 2024-04-05T03:02:41.855 INFO:tasks.workunit.client.0.smithi153.stdout:5/96: write d0/d9/f18 [595453,83009] 0 2024-04-05T03:02:41.856 INFO:tasks.workunit.client.1.smithi203.stdout:2/619: truncate d5/d10/d2f/f9d 591835 0 2024-04-05T03:02:41.859 INFO:tasks.workunit.client.0.smithi153.stdout:2/107: rename d9/da/d10/c16 to d9/da/d19/c27 0 2024-04-05T03:02:41.859 INFO:tasks.workunit.client.0.smithi153.stdout:9/61: readlink d1/l12 0 2024-04-05T03:02:41.859 INFO:tasks.workunit.client.0.smithi153.stdout:4/150: symlink da/de/d28/l2b 0 2024-04-05T03:02:41.861 INFO:tasks.workunit.client.1.smithi203.stdout:1/429: dwrite d7/f3d [4194304,4194304] 0 2024-04-05T03:02:41.863 INFO:tasks.workunit.client.0.smithi153.stdout:6/121: truncate d4/dc/df/d16/d17/f2a 638559 0 2024-04-05T03:02:41.863 INFO:tasks.workunit.client.0.smithi153.stdout:1/122: truncate d5/dd/d21/d18/d16/f22 84874 0 2024-04-05T03:02:41.863 INFO:tasks.workunit.client.0.smithi153.stdout:8/104: creat d7/d13/d19/f20 x:0 0 0 2024-04-05T03:02:41.867 INFO:tasks.workunit.client.1.smithi203.stdout:1/430: write d7/d14/f6c [972566,81577] 0 2024-04-05T03:02:41.868 INFO:tasks.workunit.client.0.smithi153.stdout:1/123: dread d5/dd/d21/fe [0,4194304] 0 2024-04-05T03:02:41.869 INFO:tasks.workunit.client.0.smithi153.stdout:5/97: mknod d0/d9/c23 0 2024-04-05T03:02:41.869 INFO:tasks.workunit.client.0.smithi153.stdout:3/93: truncate f3 4594123 0 2024-04-05T03:02:41.871 INFO:tasks.workunit.client.0.smithi153.stdout:7/115: chown d2/da/c1c 4758965 1 2024-04-05T03:02:41.872 INFO:tasks.workunit.client.0.smithi153.stdout:2/108: symlink d9/da/d19/l28 0 2024-04-05T03:02:41.874 INFO:tasks.workunit.client.1.smithi203.stdout:0/454: sync 2024-04-05T03:02:41.875 INFO:tasks.workunit.client.1.smithi203.stdout:3/417: sync 2024-04-05T03:02:41.878 INFO:tasks.workunit.client.0.smithi153.stdout:0/109: chown d9/d14/f19 349955310 1 2024-04-05T03:02:41.882 INFO:tasks.workunit.client.1.smithi203.stdout:8/504: dwrite d1/d12/d17/d47/d53/f82 [0,4194304] 0 2024-04-05T03:02:41.883 INFO:tasks.workunit.client.1.smithi203.stdout:5/468: dwrite d3/d4/dc/f4b [0,4194304] 0 2024-04-05T03:02:41.887 INFO:tasks.workunit.client.1.smithi203.stdout:3/418: stat d0/dc/d1a/d1d/c1e 0 2024-04-05T03:02:41.892 INFO:tasks.workunit.client.0.smithi153.stdout:0/110: mkdir d9/df/d1d 0 2024-04-05T03:02:41.893 INFO:tasks.workunit.client.0.smithi153.stdout:1/124: creat d5/dd/d21/d1e/f28 x:0 0 0 2024-04-05T03:02:41.894 INFO:tasks.workunit.client.1.smithi203.stdout:0/455: creat dd/d17/d1d/d22/d53/d7f/f83 x:0 0 0 2024-04-05T03:02:41.895 INFO:tasks.workunit.client.0.smithi153.stdout:2/109: link d9/da/f24 d9/da/d19/f29 0 2024-04-05T03:02:41.897 INFO:tasks.workunit.client.1.smithi203.stdout:9/405: dwrite d13/d1b/d26/d31/d38/d59/f7e [0,4194304] 0 2024-04-05T03:02:41.897 INFO:tasks.workunit.client.1.smithi203.stdout:8/505: unlink d1/d12/d17/d10/l36 0 2024-04-05T03:02:41.898 INFO:tasks.workunit.client.0.smithi153.stdout:7/116: mknod d2/c26 0 2024-04-05T03:02:41.898 INFO:tasks.workunit.client.1.smithi203.stdout:3/419: symlink d0/dc/d1a/d1d/d53/l78 0 2024-04-05T03:02:41.898 INFO:tasks.workunit.client.1.smithi203.stdout:7/554: dwrite d8/fe [0,4194304] 0 2024-04-05T03:02:41.899 INFO:tasks.workunit.client.0.smithi153.stdout:7/117: fdatasync d2/da/d1f/f20 0 2024-04-05T03:02:41.900 INFO:tasks.workunit.client.0.smithi153.stdout:9/62: rename d1/d9/f11 to d1/f18 0 2024-04-05T03:02:41.900 INFO:tasks.workunit.client.1.smithi203.stdout:9/406: chown f7 278257234 1 2024-04-05T03:02:41.901 INFO:tasks.workunit.client.0.smithi153.stdout:0/111: mkdir d9/df/d1e 0 2024-04-05T03:02:41.901 INFO:tasks.workunit.client.0.smithi153.stdout:9/63: write d1/f15 [396687,86736] 0 2024-04-05T03:02:41.902 INFO:tasks.workunit.client.1.smithi203.stdout:1/431: getdents d7/d14/d2b/d58 0 2024-04-05T03:02:41.907 INFO:tasks.workunit.client.0.smithi153.stdout:3/94: rename f1a to d1d/f1e 0 2024-04-05T03:02:41.908 INFO:tasks.workunit.client.0.smithi153.stdout:5/98: getdents d0/de 0 2024-04-05T03:02:41.908 INFO:tasks.workunit.client.1.smithi203.stdout:8/506: rename d1/c66 to d1/d6/c85 0 2024-04-05T03:02:41.908 INFO:tasks.workunit.client.1.smithi203.stdout:3/420: unlink d0/dc/d3f/c4d 0 2024-04-05T03:02:41.910 INFO:tasks.workunit.client.0.smithi153.stdout:0/112: dwrite d9/df/f18 [0,4194304] 0 2024-04-05T03:02:41.911 INFO:tasks.workunit.client.1.smithi203.stdout:0/456: symlink dd/d17/d1d/d22/d53/d7f/l84 0 2024-04-05T03:02:41.911 INFO:tasks.workunit.client.1.smithi203.stdout:8/507: mknod d1/d4b/d5f/c86 0 2024-04-05T03:02:41.915 INFO:tasks.workunit.client.1.smithi203.stdout:8/508: fsync d1/d12/d1a/f42 0 2024-04-05T03:02:41.916 INFO:tasks.workunit.client.1.smithi203.stdout:1/432: unlink d7/d24/d33/l41 0 2024-04-05T03:02:41.917 INFO:tasks.workunit.client.0.smithi153.stdout:1/125: rename l0 to d5/dd/d1b/l29 0 2024-04-05T03:02:41.918 INFO:tasks.workunit.client.0.smithi153.stdout:3/95: mknod d1d/c1f 0 2024-04-05T03:02:41.919 INFO:tasks.workunit.client.1.smithi203.stdout:7/555: rename d8/d32/d16/d1a/c34 to d8/d2a/d3f/d52/d65/cab 0 2024-04-05T03:02:41.919 INFO:tasks.workunit.client.1.smithi203.stdout:5/469: dwrite d3/d4/f87 [0,4194304] 0 2024-04-05T03:02:41.921 INFO:tasks.workunit.client.0.smithi153.stdout:7/118: creat d2/da/f27 x:0 0 0 2024-04-05T03:02:41.922 INFO:tasks.workunit.client.1.smithi203.stdout:7/556: fsync d8/d2a/d3f/d5e/fa5 0 2024-04-05T03:02:41.922 INFO:tasks.workunit.client.1.smithi203.stdout:1/433: chown d7/d24/d33/d48/d4b/d6b/f75 1357 1 2024-04-05T03:02:41.922 INFO:tasks.workunit.client.1.smithi203.stdout:8/509: creat d1/d4b/d68/d75/d77/f87 x:0 0 0 2024-04-05T03:02:41.922 INFO:tasks.workunit.client.0.smithi153.stdout:0/113: rmdir d9/db 39 2024-04-05T03:02:41.925 INFO:tasks.workunit.client.1.smithi203.stdout:8/510: symlink d1/d6/d2e/l88 0 2024-04-05T03:02:41.925 INFO:tasks.workunit.client.0.smithi153.stdout:3/96: rmdir d1d 39 2024-04-05T03:02:41.926 INFO:tasks.workunit.client.1.smithi203.stdout:7/557: fsync d8/d32/d5d/d6d/f96 0 2024-04-05T03:02:41.927 INFO:tasks.workunit.client.0.smithi153.stdout:3/97: read f17 [141051,128562] 0 2024-04-05T03:02:41.927 INFO:tasks.workunit.client.1.smithi203.stdout:7/558: dread - d8/d2a/d3f/d5e/d63/f75 zero size 2024-04-05T03:02:41.928 INFO:tasks.workunit.client.1.smithi203.stdout:5/470: symlink d3/l9a 0 2024-04-05T03:02:41.928 INFO:tasks.workunit.client.0.smithi153.stdout:9/64: truncate d1/f4 4990455 0 2024-04-05T03:02:41.930 INFO:tasks.workunit.client.0.smithi153.stdout:0/114: rename l4 to d9/df/l1f 0 2024-04-05T03:02:41.931 INFO:tasks.workunit.client.1.smithi203.stdout:5/471: truncate d3/d4/dc/f95 582882 0 2024-04-05T03:02:41.934 INFO:tasks.workunit.client.0.smithi153.stdout:1/126: dwrite d5/dd/d21/fe [4194304,4194304] 0 2024-04-05T03:02:41.937 INFO:tasks.workunit.client.1.smithi203.stdout:0/457: rename dd/d17/d1d/d22/c5d to dd/d17/d1d/d26/c85 0 2024-04-05T03:02:41.938 INFO:tasks.workunit.client.0.smithi153.stdout:9/65: write d1/d10/d16/f17 [1205708,111760] 0 2024-04-05T03:02:41.939 INFO:tasks.workunit.client.1.smithi203.stdout:7/559: dread d8/d2a/d3f/d73/f78 [0,4194304] 0 2024-04-05T03:02:41.939 INFO:tasks.workunit.client.1.smithi203.stdout:5/472: dwrite d3/d1f/d59/d64/f10 [4194304,4194304] 0 2024-04-05T03:02:41.940 INFO:tasks.workunit.client.0.smithi153.stdout:3/98: mknod d1d/c20 0 2024-04-05T03:02:41.941 INFO:tasks.workunit.client.0.smithi153.stdout:1/127: mknod d5/dd/c2a 0 2024-04-05T03:02:41.942 INFO:tasks.workunit.client.1.smithi203.stdout:0/458: write dd/d17/d1d/f1e [1977712,126627] 0 2024-04-05T03:02:41.943 INFO:tasks.workunit.client.0.smithi153.stdout:1/128: chown d5/dd/l15 3204637 1 2024-04-05T03:02:41.946 INFO:tasks.workunit.client.0.smithi153.stdout:1/129: write d5/dd/d21/d1e/f28 [1001600,14628] 0 2024-04-05T03:02:41.948 INFO:tasks.workunit.client.1.smithi203.stdout:5/473: rmdir d3/d1f/d59/d60 39 2024-04-05T03:02:41.949 INFO:tasks.workunit.client.0.smithi153.stdout:9/66: link d1/f18 d1/d10/f19 0 2024-04-05T03:02:41.951 INFO:tasks.workunit.client.0.smithi153.stdout:9/67: write d1/d9/fc [1223536,84743] 0 2024-04-05T03:02:41.951 INFO:tasks.workunit.client.0.smithi153.stdout:1/130: creat d5/dd/d1b/f2b x:0 0 0 2024-04-05T03:02:41.952 INFO:tasks.workunit.client.1.smithi203.stdout:1/434: rename d7/d14/d2b/f49 to d7/d24/f85 0 2024-04-05T03:02:41.952 INFO:tasks.workunit.client.0.smithi153.stdout:3/99: dwrite fd [0,4194304] 0 2024-04-05T03:02:41.955 INFO:tasks.workunit.client.1.smithi203.stdout:7/560: dwrite d8/f31 [0,4194304] 0 2024-04-05T03:02:41.955 INFO:tasks.workunit.client.0.smithi153.stdout:3/100: creat d1d/f21 x:0 0 0 2024-04-05T03:02:41.957 INFO:tasks.workunit.client.1.smithi203.stdout:0/459: dread - dd/d17/d1d/d22/d80/f72 zero size 2024-04-05T03:02:41.959 INFO:tasks.workunit.client.0.smithi153.stdout:1/131: mkdir d5/dd/d21/d18/d16/d2c 0 2024-04-05T03:02:41.966 INFO:tasks.workunit.client.1.smithi203.stdout:5/474: truncate d3/d1f/d59/d64/f3a 3065318 0 2024-04-05T03:02:41.966 INFO:tasks.workunit.client.1.smithi203.stdout:0/460: chown dd/d17/d1d/d22/d53/d55/c6e 439723 1 2024-04-05T03:02:41.968 INFO:tasks.workunit.client.0.smithi153.stdout:3/101: chown c1c 59348919 1 2024-04-05T03:02:41.969 INFO:tasks.workunit.client.0.smithi153.stdout:9/68: link d1/d9/fc d1/d10/d16/f1a 0 2024-04-05T03:02:41.970 INFO:tasks.workunit.client.1.smithi203.stdout:7/561: dwrite d8/d2a/d3f/d73/f7b [0,4194304] 0 2024-04-05T03:02:41.971 INFO:tasks.workunit.client.1.smithi203.stdout:2/620: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:41.972 INFO:tasks.workunit.client.0.smithi153.stdout:4/151: dwrite da/fc [0,4194304] 0 2024-04-05T03:02:41.973 INFO:tasks.workunit.client.0.smithi153.stdout:3/102: symlink d1d/l22 0 2024-04-05T03:02:41.973 INFO:tasks.workunit.client.0.smithi153.stdout:1/132: link d5/dd/d1b/f2b d5/dd/d21/d18/d16/f2d 0 2024-04-05T03:02:41.974 INFO:tasks.workunit.client.1.smithi203.stdout:0/461: read dd/d17/d1d/d26/d34/f47 [391019,97361] 0 2024-04-05T03:02:41.976 INFO:tasks.workunit.client.1.smithi203.stdout:7/562: mkdir d8/da/dac 0 2024-04-05T03:02:41.976 INFO:tasks.workunit.client.0.smithi153.stdout:3/103: chown d1d/f1e 74283 1 2024-04-05T03:02:41.981 INFO:tasks.workunit.client.1.smithi203.stdout:2/621: mknod d5/d10/d44/cda 0 2024-04-05T03:02:41.981 INFO:tasks.workunit.client.1.smithi203.stdout:2/622: readlink d5/d10/d35/d2d/d7c/d29/db1/lb2 0 2024-04-05T03:02:41.982 INFO:tasks.workunit.client.0.smithi153.stdout:9/69: mkdir d1/d9/d1b 0 2024-04-05T03:02:41.982 INFO:tasks.workunit.client.1.smithi203.stdout:2/623: write d5/d66/fa9 [155985,91328] 0 2024-04-05T03:02:41.983 INFO:tasks.workunit.client.1.smithi203.stdout:0/462: creat dd/d17/d1d/d22/d45/d51/f86 x:0 0 0 2024-04-05T03:02:41.984 INFO:tasks.workunit.client.0.smithi153.stdout:1/133: rmdir d5/d8/d20 39 2024-04-05T03:02:41.985 INFO:tasks.workunit.client.1.smithi203.stdout:2/624: write d5/d10/d2f/d5d/f68 [2715076,72545] 0 2024-04-05T03:02:41.985 INFO:tasks.workunit.client.0.smithi153.stdout:4/152: creat da/db/d20/f2c x:0 0 0 2024-04-05T03:02:41.986 INFO:tasks.workunit.client.0.smithi153.stdout:6/122: dwrite d4/f6 [0,4194304] 0 2024-04-05T03:02:41.989 INFO:tasks.workunit.client.0.smithi153.stdout:4/153: fsync f7 0 2024-04-05T03:02:41.989 INFO:tasks.workunit.client.0.smithi153.stdout:2/110: truncate d9/da/fe 2787683 0 2024-04-05T03:02:41.991 INFO:tasks.workunit.client.1.smithi203.stdout:0/463: creat dd/d17/d1d/d26/d28/f87 x:0 0 0 2024-04-05T03:02:41.995 INFO:tasks.workunit.client.1.smithi203.stdout:7/563: rename d8/d32/d16/d35/l74 to d8/d32/d16/d1a/d30/d90/lad 0 2024-04-05T03:02:41.996 INFO:tasks.workunit.client.1.smithi203.stdout:7/564: fsync d8/f4c 0 2024-04-05T03:02:41.996 INFO:tasks.workunit.client.1.smithi203.stdout:2/625: fdatasync d5/d66/d7d/f94 0 2024-04-05T03:02:41.997 INFO:tasks.workunit.client.0.smithi153.stdout:2/111: write d9/f21 [228706,32724] 0 2024-04-05T03:02:41.997 INFO:tasks.workunit.client.0.smithi153.stdout:9/70: creat d1/d9/d1b/f1c x:0 0 0 2024-04-05T03:02:41.998 INFO:tasks.workunit.client.1.smithi203.stdout:7/565: write d8/da/f1d [1778959,26830] 0 2024-04-05T03:02:41.999 INFO:tasks.workunit.client.0.smithi153.stdout:4/154: mkdir da/de/d28/d2d 0 2024-04-05T03:02:42.001 INFO:tasks.workunit.client.0.smithi153.stdout:1/134: symlink d5/d8/l2e 0 2024-04-05T03:02:42.003 INFO:tasks.workunit.client.0.smithi153.stdout:2/112: dread - d9/da/d19/f22 zero size 2024-04-05T03:02:42.006 INFO:tasks.workunit.client.0.smithi153.stdout:9/71: unlink d1/f15 0 2024-04-05T03:02:42.007 INFO:tasks.workunit.client.0.smithi153.stdout:6/123: dwrite d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:42.008 INFO:tasks.workunit.client.1.smithi203.stdout:7/566: dwrite d8/d32/d16/d1a/d30/f98 [0,4194304] 0 2024-04-05T03:02:42.009 INFO:tasks.workunit.client.0.smithi153.stdout:1/135: fsync d5/f9 0 2024-04-05T03:02:42.011 INFO:tasks.workunit.client.0.smithi153.stdout:0/115: dwrite f2 [0,4194304] 0 2024-04-05T03:02:42.012 INFO:tasks.workunit.client.1.smithi203.stdout:7/567: mknod d8/d32/d16/d35/cae 0 2024-04-05T03:02:42.013 INFO:tasks.workunit.client.0.smithi153.stdout:0/116: write d9/df/f15 [375587,76229] 0 2024-04-05T03:02:42.019 INFO:tasks.workunit.client.0.smithi153.stdout:3/104: dwrite f17 [0,4194304] 0 2024-04-05T03:02:42.019 INFO:tasks.workunit.client.1.smithi203.stdout:1/435: dwrite d7/f4f [0,4194304] 0 2024-04-05T03:02:42.022 INFO:tasks.workunit.client.0.smithi153.stdout:2/113: unlink d9/da/f18 0 2024-04-05T03:02:42.025 INFO:tasks.workunit.client.1.smithi203.stdout:7/568: rmdir d8/d32/d16/d1a/d30 39 2024-04-05T03:02:42.025 INFO:tasks.workunit.client.0.smithi153.stdout:6/124: mknod d4/c2d 0 2024-04-05T03:02:42.025 INFO:tasks.workunit.client.0.smithi153.stdout:2/114: fsync d9/f21 0 2024-04-05T03:02:42.025 INFO:tasks.workunit.client.1.smithi203.stdout:7/569: stat d8/d32/d16 0 2024-04-05T03:02:42.026 INFO:tasks.workunit.client.1.smithi203.stdout:0/464: truncate dd/d17/d1d/d22/d80/d49/f7d 749945 0 2024-04-05T03:02:42.026 INFO:tasks.workunit.client.0.smithi153.stdout:4/155: creat da/de/d28/d2d/f2e x:0 0 0 2024-04-05T03:02:42.026 INFO:tasks.workunit.client.0.smithi153.stdout:1/136: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:42.027 INFO:tasks.workunit.client.0.smithi153.stdout:2/115: fdatasync d9/ff 0 2024-04-05T03:02:42.028 INFO:tasks.workunit.client.1.smithi203.stdout:7/570: chown d8/d32/d16/d1a/d21 716397 1 2024-04-05T03:02:42.029 INFO:tasks.workunit.client.1.smithi203.stdout:1/436: mkdir d7/d14/d2b/d58/d86 0 2024-04-05T03:02:42.032 INFO:tasks.workunit.client.1.smithi203.stdout:7/571: dread - d8/d2a/d3f/d5e/d63/f75 zero size 2024-04-05T03:02:42.033 INFO:tasks.workunit.client.0.smithi153.stdout:0/117: readlink d9/df/l1f 0 2024-04-05T03:02:42.033 INFO:tasks.workunit.client.0.smithi153.stdout:0/118: readlink d9/df/l1f 0 2024-04-05T03:02:42.033 INFO:tasks.workunit.client.1.smithi203.stdout:2/626: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:42.034 INFO:tasks.workunit.client.1.smithi203.stdout:6/515: sync 2024-04-05T03:02:42.034 INFO:tasks.workunit.client.1.smithi203.stdout:4/343: sync 2024-04-05T03:02:42.035 INFO:tasks.workunit.client.1.smithi203.stdout:1/437: rmdir d7/d14/d2b/d58 39 2024-04-05T03:02:42.037 INFO:tasks.workunit.client.1.smithi203.stdout:7/572: read d8/d32/d16/d1a/d3d/f76 [26265,61853] 0 2024-04-05T03:02:42.037 INFO:tasks.workunit.client.0.smithi153.stdout:3/105: dwrite f18 [0,4194304] 0 2024-04-05T03:02:42.037 INFO:tasks.workunit.client.0.smithi153.stdout:9/72: mkdir d1/d1d 0 2024-04-05T03:02:42.042 INFO:tasks.workunit.client.1.smithi203.stdout:2/627: write d5/d10/d7f/fc0 [2557461,58008] 0 2024-04-05T03:02:42.042 INFO:tasks.workunit.client.1.smithi203.stdout:4/344: creat da/d29/d2e/d53/f6f x:0 0 0 2024-04-05T03:02:42.043 INFO:tasks.workunit.client.0.smithi153.stdout:6/125: creat d4/dc/f2e x:0 0 0 2024-04-05T03:02:42.046 INFO:tasks.workunit.client.1.smithi203.stdout:7/573: getdents d8/d2a/d3f/d73/da8 0 2024-04-05T03:02:42.046 INFO:tasks.workunit.client.1.smithi203.stdout:6/516: link d0/l2 d0/da/d67/d71/d8c/l9a 0 2024-04-05T03:02:42.047 INFO:tasks.workunit.client.0.smithi153.stdout:2/116: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:42.047 INFO:tasks.workunit.client.0.smithi153.stdout:9/73: symlink d1/d10/d16/l1e 0 2024-04-05T03:02:42.048 INFO:tasks.workunit.client.1.smithi203.stdout:4/345: truncate da/d29/d2e/f61 1032273 0 2024-04-05T03:02:42.051 INFO:tasks.workunit.client.1.smithi203.stdout:6/517: dread - d0/da/d18/d38/d86/f7c zero size 2024-04-05T03:02:42.051 INFO:tasks.workunit.client.1.smithi203.stdout:1/438: dwrite d7/d24/f85 [4194304,4194304] 0 2024-04-05T03:02:42.053 INFO:tasks.workunit.client.1.smithi203.stdout:6/518: chown d0 24 1 2024-04-05T03:02:42.053 INFO:tasks.workunit.client.0.smithi153.stdout:9/74: read d1/f18 [1322193,64940] 0 2024-04-05T03:02:42.054 INFO:tasks.workunit.client.1.smithi203.stdout:4/346: stat da/l20 0 2024-04-05T03:02:42.054 INFO:tasks.workunit.client.1.smithi203.stdout:1/439: write d7/f77 [232481,13070] 0 2024-04-05T03:02:42.055 INFO:tasks.workunit.client.1.smithi203.stdout:6/519: chown d0/da/d54/d53/f59 3 1 2024-04-05T03:02:42.055 INFO:tasks.workunit.client.0.smithi153.stdout:2/117: unlink d9/da/d19/f29 0 2024-04-05T03:02:42.056 INFO:tasks.workunit.client.0.smithi153.stdout:0/119: dwrite f0 [4194304,4194304] 0 2024-04-05T03:02:42.061 INFO:tasks.workunit.client.1.smithi203.stdout:2/628: fdatasync d5/d10/d35/d2d/d7c/f25 0 2024-04-05T03:02:42.067 INFO:tasks.workunit.client.1.smithi203.stdout:7/574: dwrite d8/d32/d16/d1a/d3d/f76 [0,4194304] 0 2024-04-05T03:02:42.068 INFO:tasks.workunit.client.0.smithi153.stdout:4/156: dwrite da/f1d [0,4194304] 0 2024-04-05T03:02:42.070 INFO:tasks.workunit.client.0.smithi153.stdout:2/118: symlink d9/da/d19/l2a 0 2024-04-05T03:02:42.070 INFO:tasks.workunit.client.1.smithi203.stdout:1/440: link d7/d24/d33/d48/d4b/d51/f7c d7/d24/d33/d48/d4b/d51/f87 0 2024-04-05T03:02:42.071 INFO:tasks.workunit.client.0.smithi153.stdout:3/106: dwrite fd [0,4194304] 0 2024-04-05T03:02:42.071 INFO:tasks.workunit.client.1.smithi203.stdout:7/575: rmdir d8/d32/d5d/d6d 39 2024-04-05T03:02:42.074 INFO:tasks.workunit.client.0.smithi153.stdout:4/157: stat da/db/d20/f29 0 2024-04-05T03:02:42.076 INFO:tasks.workunit.client.1.smithi203.stdout:2/629: rename d5/d63/d86/laf to d5/ldb 0 2024-04-05T03:02:42.081 INFO:tasks.workunit.client.1.smithi203.stdout:7/576: dwrite d8/d32/d16/d1a/d5a/f85 [0,4194304] 0 2024-04-05T03:02:42.081 INFO:tasks.workunit.client.0.smithi153.stdout:0/120: dwrite d9/db/f13 [0,4194304] 0 2024-04-05T03:02:42.082 INFO:tasks.workunit.client.0.smithi153.stdout:3/107: symlink d1d/l23 0 2024-04-05T03:02:42.082 INFO:tasks.workunit.client.1.smithi203.stdout:1/441: creat d7/d79/f88 x:0 0 0 2024-04-05T03:02:42.083 INFO:tasks.workunit.client.0.smithi153.stdout:6/126: dread d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:42.083 INFO:tasks.workunit.client.0.smithi153.stdout:3/108: write f17 [3155331,66260] 0 2024-04-05T03:02:42.084 INFO:tasks.workunit.client.0.smithi153.stdout:4/158: chown da/db/d20/f2c 38 1 2024-04-05T03:02:42.084 INFO:tasks.workunit.client.1.smithi203.stdout:7/577: readlink d8/d2a/d3f/d52/l3c 0 2024-04-05T03:02:42.084 INFO:tasks.workunit.client.1.smithi203.stdout:2/630: write d5/d10/d35/d36/f54 [2060757,33059] 0 2024-04-05T03:02:42.085 INFO:tasks.workunit.client.0.smithi153.stdout:3/109: chown d1d 141228396 1 2024-04-05T03:02:42.087 INFO:tasks.workunit.client.1.smithi203.stdout:7/578: write d8/d32/d16/d1a/d21/f62 [2943262,67387] 0 2024-04-05T03:02:42.088 INFO:tasks.workunit.client.0.smithi153.stdout:2/119: dread d9/f11 [0,4194304] 0 2024-04-05T03:02:42.090 INFO:tasks.workunit.client.0.smithi153.stdout:9/75: link d1/d10/d16/f17 d1/f1f 0 2024-04-05T03:02:42.092 INFO:tasks.workunit.client.0.smithi153.stdout:4/159: mkdir da/de/d2f 0 2024-04-05T03:02:42.095 INFO:tasks.workunit.client.0.smithi153.stdout:9/76: dread d1/d10/f19 [0,4194304] 0 2024-04-05T03:02:42.096 INFO:tasks.workunit.client.0.smithi153.stdout:9/77: chown d1/l12 2533113 1 2024-04-05T03:02:42.096 INFO:tasks.workunit.client.0.smithi153.stdout:3/110: rename l5 to d1d/l24 0 2024-04-05T03:02:42.097 INFO:tasks.workunit.client.0.smithi153.stdout:2/120: rename d9/l12 to d9/l2b 0 2024-04-05T03:02:42.097 INFO:tasks.workunit.client.1.smithi203.stdout:7/579: write d8/da/f94 [350526,2842] 0 2024-04-05T03:02:42.099 INFO:tasks.workunit.client.1.smithi203.stdout:0/465: dread f5 [0,4194304] 0 2024-04-05T03:02:42.099 INFO:tasks.workunit.client.1.smithi203.stdout:2/631: creat d5/d10/d35/d36/fdc x:0 0 0 2024-04-05T03:02:42.101 INFO:tasks.workunit.client.1.smithi203.stdout:2/632: truncate d5/d10/d35/d36/f58 4223043 0 2024-04-05T03:02:42.101 INFO:tasks.workunit.client.1.smithi203.stdout:7/580: readlink d8/d2a/d3f/d52/d65/la1 0 2024-04-05T03:02:42.102 INFO:tasks.workunit.client.0.smithi153.stdout:6/127: link d4/dc/df/d16/d17/f2a d4/dc/df/f2f 0 2024-04-05T03:02:42.103 INFO:tasks.workunit.client.1.smithi203.stdout:1/442: dread d7/d14/f1e [8388608,4194304] 0 2024-04-05T03:02:42.103 INFO:tasks.workunit.client.0.smithi153.stdout:9/78: creat d1/d9/f20 x:0 0 0 2024-04-05T03:02:42.105 INFO:tasks.workunit.client.0.smithi153.stdout:2/121: mknod d9/da/c2c 0 2024-04-05T03:02:42.105 INFO:tasks.workunit.client.0.smithi153.stdout:3/111: creat d1d/f25 x:0 0 0 2024-04-05T03:02:42.105 INFO:tasks.workunit.client.0.smithi153.stdout:3/112: chown f17 10704844 1 2024-04-05T03:02:42.106 INFO:tasks.workunit.client.0.smithi153.stdout:3/113: chown c1 118093486 1 2024-04-05T03:02:42.108 INFO:tasks.workunit.client.0.smithi153.stdout:0/121: getdents d9/db 0 2024-04-05T03:02:42.108 INFO:tasks.workunit.client.0.smithi153.stdout:0/122: chown d9/db/f11 1809707 1 2024-04-05T03:02:42.109 INFO:tasks.workunit.client.1.smithi203.stdout:2/633: mknod d5/d10/d35/d2d/d7c/d7a/cdd 0 2024-04-05T03:02:42.109 INFO:tasks.workunit.client.0.smithi153.stdout:9/79: read d1/d9/fc [789398,74416] 0 2024-04-05T03:02:42.109 INFO:tasks.workunit.client.0.smithi153.stdout:0/123: readlink d9/df/l1f 0 2024-04-05T03:02:42.110 INFO:tasks.workunit.client.0.smithi153.stdout:6/128: fsync d4/dc/df/d16/f21 0 2024-04-05T03:02:42.111 INFO:tasks.workunit.client.0.smithi153.stdout:6/129: chown d4/dc/df/d16/d2b 4843603 1 2024-04-05T03:02:42.111 INFO:tasks.workunit.client.0.smithi153.stdout:0/124: stat d9/df/d1d 0 2024-04-05T03:02:42.112 INFO:tasks.workunit.client.0.smithi153.stdout:0/125: chown d9/lc 121233764 1 2024-04-05T03:02:42.113 INFO:tasks.workunit.client.1.smithi203.stdout:2/634: dread - d5/d63/f75 zero size 2024-04-05T03:02:42.114 INFO:tasks.workunit.client.1.smithi203.stdout:6/520: dwrite d0/da/d67/f5f [0,4194304] 0 2024-04-05T03:02:42.116 INFO:tasks.workunit.client.0.smithi153.stdout:3/114: mkdir d1d/d26 0 2024-04-05T03:02:42.116 INFO:tasks.workunit.client.1.smithi203.stdout:7/581: mkdir d8/d32/d5d/d6d/daf 0 2024-04-05T03:02:42.116 INFO:tasks.workunit.client.1.smithi203.stdout:1/443: read d7/d14/f52 [2828587,1879] 0 2024-04-05T03:02:42.116 INFO:tasks.workunit.client.1.smithi203.stdout:0/466: dwrite f5 [0,4194304] 0 2024-04-05T03:02:42.116 INFO:tasks.workunit.client.1.smithi203.stdout:2/635: write d5/d66/fa9 [584091,1081] 0 2024-04-05T03:02:42.117 INFO:tasks.workunit.client.0.smithi153.stdout:9/80: rmdir d1/d10/d16 39 2024-04-05T03:02:42.119 INFO:tasks.workunit.client.0.smithi153.stdout:6/130: symlink d4/dc/l30 0 2024-04-05T03:02:42.123 INFO:tasks.workunit.client.0.smithi153.stdout:4/160: getdents da/de/d28/d2d 0 2024-04-05T03:02:42.123 INFO:tasks.workunit.client.1.smithi203.stdout:6/521: symlink d0/da/d54/d82/l9b 0 2024-04-05T03:02:42.123 INFO:tasks.workunit.client.1.smithi203.stdout:2/636: truncate d5/d10/d1a/d57/fac 4920738 0 2024-04-05T03:02:42.124 INFO:tasks.workunit.client.1.smithi203.stdout:0/467: chown dd/d17/d29/c3e 2 1 2024-04-05T03:02:42.124 INFO:tasks.workunit.client.1.smithi203.stdout:7/582: creat d8/d2a/d3f/d5e/d77/fb0 x:0 0 0 2024-04-05T03:02:42.125 INFO:tasks.workunit.client.1.smithi203.stdout:1/444: truncate d7/d24/d33/d48/d4b/f5c 261587 0 2024-04-05T03:02:42.126 INFO:tasks.workunit.client.1.smithi203.stdout:4/347: dread da/d29/d2e/f3e [0,4194304] 0 2024-04-05T03:02:42.127 INFO:tasks.workunit.client.0.smithi153.stdout:0/126: symlink d9/df/l20 0 2024-04-05T03:02:42.128 INFO:tasks.workunit.client.0.smithi153.stdout:3/115: mkdir d1d/d27 0 2024-04-05T03:02:42.129 INFO:tasks.workunit.client.1.smithi203.stdout:7/583: chown d8/d2a/d3f/d5e 1953 1 2024-04-05T03:02:42.130 INFO:tasks.workunit.client.1.smithi203.stdout:1/445: mknod d7/d79/c89 0 2024-04-05T03:02:42.130 INFO:tasks.workunit.client.1.smithi203.stdout:0/468: write dd/d17/d1d/d22/f65 [4474096,104668] 0 2024-04-05T03:02:42.131 INFO:tasks.workunit.client.0.smithi153.stdout:6/131: symlink d4/dc/df/d16/l31 0 2024-04-05T03:02:42.131 INFO:tasks.workunit.client.1.smithi203.stdout:4/348: chown da/d3c/d4c/l67 112696 1 2024-04-05T03:02:42.132 INFO:tasks.workunit.client.1.smithi203.stdout:4/349: chown da/d29/d2e 159594 1 2024-04-05T03:02:42.132 INFO:tasks.workunit.client.0.smithi153.stdout:4/161: mkdir da/db/d20/d30 0 2024-04-05T03:02:42.132 INFO:tasks.workunit.client.1.smithi203.stdout:2/637: unlink d5/d10/d35/d36/db6/d71/f8b 0 2024-04-05T03:02:42.133 INFO:tasks.workunit.client.0.smithi153.stdout:4/162: write da/fc [3675518,109051] 0 2024-04-05T03:02:42.134 INFO:tasks.workunit.client.0.smithi153.stdout:0/127: creat d9/df/f21 x:0 0 0 2024-04-05T03:02:42.134 INFO:tasks.workunit.client.1.smithi203.stdout:7/584: dread d8/da/f14 [0,4194304] 0 2024-04-05T03:02:42.135 INFO:tasks.workunit.client.0.smithi153.stdout:3/116: readlink l1b 0 2024-04-05T03:02:42.136 INFO:tasks.workunit.client.0.smithi153.stdout:3/117: chown d1d/l23 189049389 1 2024-04-05T03:02:42.137 INFO:tasks.workunit.client.0.smithi153.stdout:6/132: creat d4/dc/df/d16/d24/f32 x:0 0 0 2024-04-05T03:02:42.137 INFO:tasks.workunit.client.1.smithi203.stdout:7/585: readlink d8/d2a/d3f/d52/l4d 0 2024-04-05T03:02:42.137 INFO:tasks.workunit.client.1.smithi203.stdout:0/469: mknod dd/d17/c88 0 2024-04-05T03:02:42.138 INFO:tasks.workunit.client.1.smithi203.stdout:7/586: chown d8/f20 16131707 1 2024-04-05T03:02:42.140 INFO:tasks.workunit.client.1.smithi203.stdout:6/522: link d0/da/d18/d38/d2a/d2c/f4e d0/f9c 0 2024-04-05T03:02:42.140 INFO:tasks.workunit.client.0.smithi153.stdout:3/118: creat d1d/f28 x:0 0 0 2024-04-05T03:02:42.141 INFO:tasks.workunit.client.1.smithi203.stdout:1/446: getdents d7/d24/d33/d48 0 2024-04-05T03:02:42.142 INFO:tasks.workunit.client.0.smithi153.stdout:6/133: rename d4/dc/df/d16/d2b to d4/dc/df/d16/d33 0 2024-04-05T03:02:42.142 INFO:tasks.workunit.client.1.smithi203.stdout:2/638: dwrite d5/d10/d7f/fc0 [0,4194304] 0 2024-04-05T03:02:42.143 INFO:tasks.workunit.client.1.smithi203.stdout:7/587: symlink d8/lb1 0 2024-04-05T03:02:42.143 INFO:tasks.workunit.client.0.smithi153.stdout:6/134: dread - d4/dc/df/f26 zero size 2024-04-05T03:02:42.143 INFO:tasks.workunit.client.0.smithi153.stdout:9/81: link d1/d10/d16/l1e d1/d10/d16/l21 0 2024-04-05T03:02:42.144 INFO:tasks.workunit.client.0.smithi153.stdout:6/135: dread - d4/dc/df/f13 zero size 2024-04-05T03:02:42.145 INFO:tasks.workunit.client.1.smithi203.stdout:4/350: dread da/f56 [0,4194304] 0 2024-04-05T03:02:42.145 INFO:tasks.workunit.client.1.smithi203.stdout:0/470: creat dd/d17/d1d/d26/f89 x:0 0 0 2024-04-05T03:02:42.145 INFO:tasks.workunit.client.1.smithi203.stdout:1/447: rmdir d7/d24/d33/d48/d4b/d6b 39 2024-04-05T03:02:42.145 INFO:tasks.workunit.client.1.smithi203.stdout:7/588: read d8/d2a/d3f/d73/f78 [507971,126029] 0 2024-04-05T03:02:42.146 INFO:tasks.workunit.client.0.smithi153.stdout:6/136: dread d4/dc/df/f2f [0,4194304] 0 2024-04-05T03:02:42.147 INFO:tasks.workunit.client.1.smithi203.stdout:6/523: creat d0/f9d x:0 0 0 2024-04-05T03:02:42.150 INFO:tasks.workunit.client.1.smithi203.stdout:1/448: creat d7/d14/d2b/d58/d7a/f8a x:0 0 0 2024-04-05T03:02:42.150 INFO:tasks.workunit.client.0.smithi153.stdout:6/137: mknod d4/dc/df/c34 0 2024-04-05T03:02:42.151 INFO:tasks.workunit.client.1.smithi203.stdout:7/589: truncate d8/d32/f43 1090567 0 2024-04-05T03:02:42.152 INFO:tasks.workunit.client.1.smithi203.stdout:6/524: mknod d0/da/d18/d38/d2a/c9e 0 2024-04-05T03:02:42.153 INFO:tasks.workunit.client.1.smithi203.stdout:7/590: read - d8/f46 zero size 2024-04-05T03:02:42.154 INFO:tasks.workunit.client.0.smithi153.stdout:6/138: getdents d4 0 2024-04-05T03:02:42.154 INFO:tasks.workunit.client.1.smithi203.stdout:1/449: symlink d7/d24/d33/d48/d4b/d51/l8b 0 2024-04-05T03:02:42.155 INFO:tasks.workunit.client.0.smithi153.stdout:6/139: read d4/dc/df/d16/f21 [1273006,25013] 0 2024-04-05T03:02:42.155 INFO:tasks.workunit.client.0.smithi153.stdout:6/140: readlink d4/dc/l30 0 2024-04-05T03:02:42.155 INFO:tasks.workunit.client.1.smithi203.stdout:2/639: dwrite d5/d10/db8/f56 [0,4194304] 0 2024-04-05T03:02:42.157 INFO:tasks.workunit.client.0.smithi153.stdout:6/141: symlink d4/dc/df/l35 0 2024-04-05T03:02:42.157 INFO:tasks.workunit.client.1.smithi203.stdout:0/471: dwrite dd/d17/d1d/d26/f42 [0,4194304] 0 2024-04-05T03:02:42.157 INFO:tasks.workunit.client.1.smithi203.stdout:2/640: stat d5/d10/d2f/c6d 0 2024-04-05T03:02:42.158 INFO:tasks.workunit.client.1.smithi203.stdout:1/450: rename d7 to d7/d24/d33/d48/d4b/d51/d8c 22 2024-04-05T03:02:42.160 INFO:tasks.workunit.client.0.smithi153.stdout:6/142: mkdir d4/dc/df/d16/d36 0 2024-04-05T03:02:42.161 INFO:tasks.workunit.client.1.smithi203.stdout:0/472: fdatasync dd/d17/d1d/d22/d45/d51/f86 0 2024-04-05T03:02:42.162 INFO:tasks.workunit.client.1.smithi203.stdout:1/451: truncate d7/d14/f21 445440 0 2024-04-05T03:02:42.163 INFO:tasks.workunit.client.1.smithi203.stdout:1/452: readlink d7/d24/d33/l37 0 2024-04-05T03:02:42.163 INFO:tasks.workunit.client.1.smithi203.stdout:1/453: readlink d7/d24/l4c 0 2024-04-05T03:02:42.165 INFO:tasks.workunit.client.1.smithi203.stdout:6/525: getdents d0/da/d67/d71/d8c 0 2024-04-05T03:02:42.169 INFO:tasks.workunit.client.1.smithi203.stdout:1/454: rename d7/d24/d33/l37 to d7/d79/l8d 0 2024-04-05T03:02:42.170 INFO:tasks.workunit.client.1.smithi203.stdout:0/473: dwrite dd/d17/d1d/d26/d34/f47 [4194304,4194304] 0 2024-04-05T03:02:42.170 INFO:tasks.workunit.client.1.smithi203.stdout:1/455: read - d7/d79/f88 zero size 2024-04-05T03:02:42.173 INFO:tasks.workunit.client.1.smithi203.stdout:6/526: creat d0/da/d18/f9f x:0 0 0 2024-04-05T03:02:42.173 INFO:tasks.workunit.client.1.smithi203.stdout:6/527: dread - d0/da/d18/d38/d86/f2d zero size 2024-04-05T03:02:42.176 INFO:tasks.workunit.client.1.smithi203.stdout:1/456: mknod d7/d14/d2b/d58/d86/c8e 0 2024-04-05T03:02:42.183 INFO:tasks.workunit.client.0.smithi153.stdout:0/128: dread d9/df/f15 [0,4194304] 0 2024-04-05T03:02:42.183 INFO:tasks.workunit.client.1.smithi203.stdout:1/457: rename d7/d14/d2b/c31 to d7/d24/d33/c8f 0 2024-04-05T03:02:42.187 INFO:tasks.workunit.client.0.smithi153.stdout:0/129: fsync d9/d14/f19 0 2024-04-05T03:02:42.187 INFO:tasks.workunit.client.1.smithi203.stdout:1/458: mkdir d7/d90 0 2024-04-05T03:02:42.192 INFO:tasks.workunit.client.0.smithi153.stdout:0/130: rename l6 to d9/df/d1e/l22 0 2024-04-05T03:02:42.196 INFO:tasks.workunit.client.1.smithi203.stdout:2/641: fdatasync d5/d10/db8/f56 0 2024-04-05T03:02:42.196 INFO:tasks.workunit.client.0.smithi153.stdout:6/143: rmdir d4/dc/df/d16/d17 39 2024-04-05T03:02:42.197 INFO:tasks.workunit.client.1.smithi203.stdout:2/642: write d5/d63/fa3 [5847237,60028] 0 2024-04-05T03:02:42.198 INFO:tasks.workunit.client.1.smithi203.stdout:2/643: chown l2 293261106 1 2024-04-05T03:02:42.198 INFO:tasks.workunit.client.0.smithi153.stdout:6/144: write d4/dc/f2e [488538,11149] 0 2024-04-05T03:02:42.199 INFO:tasks.workunit.client.1.smithi203.stdout:2/644: fdatasync d5/d10/d35/d2d/d7c/d72/d8f/fb9 0 2024-04-05T03:02:42.200 INFO:tasks.workunit.client.0.smithi153.stdout:2/122: write d9/da/f15 [261081,57471] 0 2024-04-05T03:02:42.201 INFO:tasks.workunit.client.1.smithi203.stdout:2/645: dread d5/d66/fcb [0,4194304] 0 2024-04-05T03:02:42.202 INFO:tasks.workunit.client.0.smithi153.stdout:0/131: dwrite d9/df/f15 [0,4194304] 0 2024-04-05T03:02:42.202 INFO:tasks.workunit.client.0.smithi153.stdout:2/123: read d9/da/f15 [125638,88750] 0 2024-04-05T03:02:42.206 INFO:tasks.workunit.client.0.smithi153.stdout:0/132: creat d9/d14/f23 x:0 0 0 2024-04-05T03:02:42.207 INFO:tasks.workunit.client.1.smithi203.stdout:2/646: rmdir d5/d10/d1a/d57 39 2024-04-05T03:02:42.208 INFO:tasks.workunit.client.0.smithi153.stdout:2/124: creat d9/da/d19/f2d x:0 0 0 2024-04-05T03:02:42.209 INFO:tasks.workunit.client.0.smithi153.stdout:2/125: fdatasync d9/ff 0 2024-04-05T03:02:42.209 INFO:tasks.workunit.client.0.smithi153.stdout:6/145: dread d4/dc/f2e [0,4194304] 0 2024-04-05T03:02:42.210 INFO:tasks.workunit.client.0.smithi153.stdout:6/146: write d4/dc/df/f12 [262616,39998] 0 2024-04-05T03:02:42.211 INFO:tasks.workunit.client.0.smithi153.stdout:0/133: rename f2 to d9/df/d1d/f24 0 2024-04-05T03:02:42.212 INFO:tasks.workunit.client.1.smithi203.stdout:2/647: read d5/d66/fcb [88007,22193] 0 2024-04-05T03:02:42.213 INFO:tasks.workunit.client.1.smithi203.stdout:2/648: read d5/f9 [217743,48946] 0 2024-04-05T03:02:42.214 INFO:tasks.workunit.client.1.smithi203.stdout:7/591: dwrite d8/d32/d5d/d6d/f96 [0,4194304] 0 2024-04-05T03:02:42.216 INFO:tasks.workunit.client.1.smithi203.stdout:0/474: write dd/d17/d1d/d22/d80/d49/f7d [523276,120708] 0 2024-04-05T03:02:42.217 INFO:tasks.workunit.client.0.smithi153.stdout:2/126: dread - d9/da/f24 zero size 2024-04-05T03:02:42.219 INFO:tasks.workunit.client.0.smithi153.stdout:3/119: truncate fd 2036218 0 2024-04-05T03:02:42.220 INFO:tasks.workunit.client.0.smithi153.stdout:6/147: dwrite d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:42.220 INFO:tasks.workunit.client.1.smithi203.stdout:9/407: sync 2024-04-05T03:02:42.220 INFO:tasks.workunit.client.0.smithi153.stdout:0/134: creat d9/df/d1e/f25 x:0 0 0 2024-04-05T03:02:42.221 INFO:tasks.workunit.client.0.smithi153.stdout:9/82: write d1/f4 [3523065,51389] 0 2024-04-05T03:02:42.222 INFO:tasks.workunit.client.1.smithi203.stdout:7/592: symlink d8/d32/d16/d1a/d3d/d59/lb2 0 2024-04-05T03:02:42.227 INFO:tasks.workunit.client.1.smithi203.stdout:1/459: dwrite d7/d24/d33/d48/d4b/d6b/f75 [0,4194304] 0 2024-04-05T03:02:42.228 INFO:tasks.workunit.client.0.smithi153.stdout:2/127: dread d9/da/d10/f1a [0,4194304] 0 2024-04-05T03:02:42.228 INFO:tasks.workunit.client.0.smithi153.stdout:0/135: unlink d9/df/f21 0 2024-04-05T03:02:42.230 INFO:tasks.workunit.client.1.smithi203.stdout:1/460: fsync d7/d24/f66 0 2024-04-05T03:02:42.232 INFO:tasks.workunit.client.1.smithi203.stdout:9/408: creat d13/d1b/d26/d31/d38/d59/d7b/f7f x:0 0 0 2024-04-05T03:02:42.232 INFO:tasks.workunit.client.0.smithi153.stdout:6/148: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:42.233 INFO:tasks.workunit.client.1.smithi203.stdout:9/409: chown f7 273916135 1 2024-04-05T03:02:42.234 INFO:tasks.workunit.client.1.smithi203.stdout:7/593: mknod d8/d32/d16/d1a/d30/d90/cb3 0 2024-04-05T03:02:42.235 INFO:tasks.workunit.client.0.smithi153.stdout:9/83: truncate d1/f18 1529019 0 2024-04-05T03:02:42.237 INFO:tasks.workunit.client.1.smithi203.stdout:9/410: fdatasync d13/d1b/d26/d31/d38/f5e 0 2024-04-05T03:02:42.237 INFO:tasks.workunit.client.1.smithi203.stdout:7/594: chown d8/d32/d16/d35/d64 0 1 2024-04-05T03:02:42.239 INFO:tasks.workunit.client.0.smithi153.stdout:0/136: symlink d9/df/d1e/l26 0 2024-04-05T03:02:42.241 INFO:tasks.workunit.client.0.smithi153.stdout:2/128: dwrite d9/ff [4194304,4194304] 0 2024-04-05T03:02:42.241 INFO:tasks.workunit.client.1.smithi203.stdout:9/411: chown d13/d1b/d33/d42/d67 5004137 1 2024-04-05T03:02:42.242 INFO:tasks.workunit.client.1.smithi203.stdout:1/461: dwrite d7/f3d [8388608,4194304] 0 2024-04-05T03:02:42.243 INFO:tasks.workunit.client.0.smithi153.stdout:0/137: write d9/d14/f23 [925533,58151] 0 2024-04-05T03:02:42.243 INFO:tasks.workunit.client.0.smithi153.stdout:3/120: link f17 d1d/f29 0 2024-04-05T03:02:42.245 INFO:tasks.workunit.client.0.smithi153.stdout:0/138: write d9/df/d1e/f25 [879837,40148] 0 2024-04-05T03:02:42.245 INFO:tasks.workunit.client.1.smithi203.stdout:1/462: chown d7/d79 4382097 1 2024-04-05T03:02:42.245 INFO:tasks.workunit.client.0.smithi153.stdout:6/149: write d4/dc/df/d16/d17/f25 [985922,64050] 0 2024-04-05T03:02:42.247 INFO:tasks.workunit.client.1.smithi203.stdout:1/463: dread - d7/d24/d33/d48/d4b/d51/f7f zero size 2024-04-05T03:02:42.247 INFO:tasks.workunit.client.1.smithi203.stdout:7/595: symlink d8/d32/lb4 0 2024-04-05T03:02:42.248 INFO:tasks.workunit.client.1.smithi203.stdout:7/596: dread - d8/d32/d5d/d6d/f8d zero size 2024-04-05T03:02:42.250 INFO:tasks.workunit.client.1.smithi203.stdout:1/464: mkdir d7/d14/d2b/d58/d86/d91 0 2024-04-05T03:02:42.252 INFO:tasks.workunit.client.1.smithi203.stdout:7/597: rmdir d8/d32/d5d 39 2024-04-05T03:02:42.252 INFO:tasks.workunit.client.1.smithi203.stdout:1/465: chown d7/d14/d2b/l4e 25 1 2024-04-05T03:02:42.255 INFO:tasks.workunit.client.0.smithi153.stdout:0/139: symlink d9/df/l27 0 2024-04-05T03:02:42.256 INFO:tasks.workunit.client.0.smithi153.stdout:0/140: fdatasync d9/df/f15 0 2024-04-05T03:02:42.256 INFO:tasks.workunit.client.0.smithi153.stdout:6/150: dread d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:42.257 INFO:tasks.workunit.client.0.smithi153.stdout:0/141: read d9/df/f15 [3437541,27626] 0 2024-04-05T03:02:42.260 INFO:tasks.workunit.client.0.smithi153.stdout:6/151: creat d4/dc/df/d16/d33/f37 x:0 0 0 2024-04-05T03:02:42.260 INFO:tasks.workunit.client.1.smithi203.stdout:2/649: dread d5/d10/d35/d2d/d7c/d29/db1/fb7 [0,4194304] 0 2024-04-05T03:02:42.260 INFO:tasks.workunit.client.1.smithi203.stdout:7/598: dwrite d8/da/f50 [4194304,4194304] 0 2024-04-05T03:02:42.261 INFO:tasks.workunit.client.0.smithi153.stdout:0/142: fsync d9/d14/f19 0 2024-04-05T03:02:42.264 INFO:tasks.workunit.client.0.smithi153.stdout:9/84: rmdir d1 39 2024-04-05T03:02:42.266 INFO:tasks.workunit.client.1.smithi203.stdout:7/599: rmdir d8/da/dac 0 2024-04-05T03:02:42.269 INFO:tasks.workunit.client.0.smithi153.stdout:0/143: symlink d9/d14/l28 0 2024-04-05T03:02:42.270 INFO:tasks.workunit.client.0.smithi153.stdout:6/152: getdents d4/dc/df/d16/d36 0 2024-04-05T03:02:42.271 INFO:tasks.workunit.client.1.smithi203.stdout:7/600: mkdir d8/d32/d16/d1a/d5a/db5 0 2024-04-05T03:02:42.272 INFO:tasks.workunit.client.0.smithi153.stdout:6/153: write d4/f20 [691937,57733] 0 2024-04-05T03:02:42.272 INFO:tasks.workunit.client.0.smithi153.stdout:0/144: symlink d9/df/l29 0 2024-04-05T03:02:42.272 INFO:tasks.workunit.client.1.smithi203.stdout:2/650: dwrite d5/d10/d35/d36/db6/d87/f89 [0,4194304] 0 2024-04-05T03:02:42.273 INFO:tasks.workunit.client.0.smithi153.stdout:6/154: write d4/fb [4305320,13950] 0 2024-04-05T03:02:42.274 INFO:tasks.workunit.client.0.smithi153.stdout:9/85: mkdir d1/d10/d16/d22 0 2024-04-05T03:02:42.274 INFO:tasks.workunit.client.0.smithi153.stdout:5/99: sync 2024-04-05T03:02:42.274 INFO:tasks.workunit.client.0.smithi153.stdout:8/105: sync 2024-04-05T03:02:42.275 INFO:tasks.workunit.client.1.smithi203.stdout:7/601: symlink d8/d2a/d3f/d52/lb6 0 2024-04-05T03:02:42.275 INFO:tasks.workunit.client.0.smithi153.stdout:3/121: rmdir d1d 39 2024-04-05T03:02:42.276 INFO:tasks.workunit.client.1.smithi203.stdout:2/651: creat d5/d66/fde x:0 0 0 2024-04-05T03:02:42.278 INFO:tasks.workunit.client.0.smithi153.stdout:0/145: mkdir d9/df/d1e/d2a 0 2024-04-05T03:02:42.280 INFO:tasks.workunit.client.1.smithi203.stdout:2/652: creat d5/d66/d7d/fdf x:0 0 0 2024-04-05T03:02:42.282 INFO:tasks.workunit.client.1.smithi203.stdout:7/602: link d8/d32/f56 d8/fb7 0 2024-04-05T03:02:42.283 INFO:tasks.workunit.client.0.smithi153.stdout:2/129: truncate d9/f11 3983738 0 2024-04-05T03:02:42.284 INFO:tasks.workunit.client.1.smithi203.stdout:7/603: dread - d8/d32/d16/d1a/d5a/fa3 zero size 2024-04-05T03:02:42.284 INFO:tasks.workunit.client.0.smithi153.stdout:6/155: dwrite d4/f6 [0,4194304] 0 2024-04-05T03:02:42.285 INFO:tasks.workunit.client.1.smithi203.stdout:7/604: stat d8/d2a/d3f/d52/l4d 0 2024-04-05T03:02:42.286 INFO:tasks.workunit.client.0.smithi153.stdout:6/156: write d4/dc/df/d16/f21 [2834669,126438] 0 2024-04-05T03:02:42.287 INFO:tasks.workunit.client.1.smithi203.stdout:2/653: rename d5/d10/d1a/f5a to d5/d10/db8/dbe/fe0 0 2024-04-05T03:02:42.287 INFO:tasks.workunit.client.0.smithi153.stdout:8/106: mkdir d7/d13/d19/d1b/d21 0 2024-04-05T03:02:42.287 INFO:tasks.workunit.client.0.smithi153.stdout:6/157: readlink d4/dc/l30 0 2024-04-05T03:02:42.288 INFO:tasks.workunit.client.0.smithi153.stdout:5/100: getdents d0/de 0 2024-04-05T03:02:42.290 INFO:tasks.workunit.client.0.smithi153.stdout:5/101: write d0/fd [3256683,56659] 0 2024-04-05T03:02:42.292 INFO:tasks.workunit.client.1.smithi203.stdout:7/605: rename l7 to d8/d2a/lb8 0 2024-04-05T03:02:42.293 INFO:tasks.workunit.client.0.smithi153.stdout:2/130: mkdir d9/da/d19/d2e 0 2024-04-05T03:02:42.294 INFO:tasks.workunit.client.1.smithi203.stdout:7/606: chown d8/d2a/d3f/d52/d65/d82 394345092 1 2024-04-05T03:02:42.295 INFO:tasks.workunit.client.0.smithi153.stdout:3/122: mknod d1d/d26/c2a 0 2024-04-05T03:02:42.295 INFO:tasks.workunit.client.1.smithi203.stdout:2/654: dwrite d5/d10/d1a/f65 [0,4194304] 0 2024-04-05T03:02:42.296 INFO:tasks.workunit.client.0.smithi153.stdout:2/131: write d9/f21 [871778,81041] 0 2024-04-05T03:02:42.297 INFO:tasks.workunit.client.0.smithi153.stdout:9/86: getdents d1/d9 0 2024-04-05T03:02:42.299 INFO:tasks.workunit.client.0.smithi153.stdout:6/158: mkdir d4/d38 0 2024-04-05T03:02:42.299 INFO:tasks.workunit.client.0.smithi153.stdout:2/132: write d9/da/d19/f1f [5335210,44182] 0 2024-04-05T03:02:42.300 INFO:tasks.workunit.client.0.smithi153.stdout:8/107: dwrite f4 [4194304,4194304] 0 2024-04-05T03:02:42.300 INFO:tasks.workunit.client.0.smithi153.stdout:0/146: rmdir d9/df/d1e/d2a 0 2024-04-05T03:02:42.301 INFO:tasks.workunit.client.0.smithi153.stdout:0/147: stat d9/df 0 2024-04-05T03:02:42.303 INFO:tasks.workunit.client.1.smithi203.stdout:7/607: write d8/d32/d16/d1a/f5c [509765,63736] 0 2024-04-05T03:02:42.303 INFO:tasks.workunit.client.0.smithi153.stdout:5/102: rename d0/de/f17 to d0/d9/d22/f24 0 2024-04-05T03:02:42.305 INFO:tasks.workunit.client.1.smithi203.stdout:7/608: stat d8/d2a/d3f/d73 0 2024-04-05T03:02:42.305 INFO:tasks.workunit.client.1.smithi203.stdout:7/609: chown d8/da 37 1 2024-04-05T03:02:42.305 INFO:tasks.workunit.client.0.smithi153.stdout:9/87: chown d1/db/c13 297139 1 2024-04-05T03:02:42.307 INFO:tasks.workunit.client.0.smithi153.stdout:2/133: creat d9/da/d19/f2f x:0 0 0 2024-04-05T03:02:42.308 INFO:tasks.workunit.client.1.smithi203.stdout:7/610: truncate d8/d2a/d3f/d5e/d77/f80 324739 0 2024-04-05T03:02:42.308 INFO:tasks.workunit.client.0.smithi153.stdout:0/148: creat d9/df/d12/f2b x:0 0 0 2024-04-05T03:02:42.308 INFO:tasks.workunit.client.0.smithi153.stdout:2/134: dread - d9/da/d19/f22 zero size 2024-04-05T03:02:42.309 INFO:tasks.workunit.client.0.smithi153.stdout:8/108: creat d7/d13/d19/f22 x:0 0 0 2024-04-05T03:02:42.309 INFO:tasks.workunit.client.0.smithi153.stdout:3/123: mkdir d1d/d27/d2b 0 2024-04-05T03:02:42.313 INFO:tasks.workunit.client.0.smithi153.stdout:9/88: creat d1/d10/d16/f23 x:0 0 0 2024-04-05T03:02:42.314 INFO:tasks.workunit.client.0.smithi153.stdout:0/149: symlink d9/df/d1e/l2c 0 2024-04-05T03:02:42.315 INFO:tasks.workunit.client.0.smithi153.stdout:2/135: write d9/da/fb [1195048,121720] 0 2024-04-05T03:02:42.315 INFO:tasks.workunit.client.0.smithi153.stdout:0/150: chown d9/db/f13 1157654887 1 2024-04-05T03:02:42.316 INFO:tasks.workunit.client.0.smithi153.stdout:5/103: mkdir d0/d25 0 2024-04-05T03:02:42.317 INFO:tasks.workunit.client.0.smithi153.stdout:5/104: chown d0/d9 12543919 1 2024-04-05T03:02:42.317 INFO:tasks.workunit.client.0.smithi153.stdout:9/89: rmdir d1 39 2024-04-05T03:02:42.318 INFO:tasks.workunit.client.1.smithi203.stdout:2/655: dwrite d5/d10/d35/d2d/d7c/f25 [0,4194304] 0 2024-04-05T03:02:42.320 INFO:tasks.workunit.client.0.smithi153.stdout:8/109: link f0 d7/f23 0 2024-04-05T03:02:42.321 INFO:tasks.workunit.client.0.smithi153.stdout:8/110: fsync f5 0 2024-04-05T03:02:42.321 INFO:tasks.workunit.client.0.smithi153.stdout:8/111: fdatasync d7/ff 0 2024-04-05T03:02:42.322 INFO:tasks.workunit.client.0.smithi153.stdout:2/136: mknod d9/da/d19/c30 0 2024-04-05T03:02:42.323 INFO:tasks.workunit.client.0.smithi153.stdout:8/112: chown d7/d13/d19/c1f 192 1 2024-04-05T03:02:42.323 INFO:tasks.workunit.client.1.smithi203.stdout:2/656: rename d5/d10/d35/d2d/d7c/d7a to d5/d10/db8/dab/de1 0 2024-04-05T03:02:42.329 INFO:tasks.workunit.client.1.smithi203.stdout:7/611: dwrite d8/d32/f43 [0,4194304] 0 2024-04-05T03:02:42.330 INFO:tasks.workunit.client.1.smithi203.stdout:7/612: chown d8/d32/d16/d1a/d21/f62 111 1 2024-04-05T03:02:42.331 INFO:tasks.workunit.client.1.smithi203.stdout:7/613: chown d8/d32/d16/d1a/d3d/f6b 445 1 2024-04-05T03:02:42.335 INFO:tasks.workunit.client.1.smithi203.stdout:7/614: mknod d8/d32/d16/d92/cb9 0 2024-04-05T03:02:42.336 INFO:tasks.workunit.client.1.smithi203.stdout:7/615: chown d8/d32/d16/d1a/d5a/db5 2357191 1 2024-04-05T03:02:42.338 INFO:tasks.workunit.client.1.smithi203.stdout:7/616: truncate d8/d2a/f42 575912 0 2024-04-05T03:02:42.339 INFO:tasks.workunit.client.1.smithi203.stdout:2/657: symlink d5/d10/d35/d2d/d7c/le2 0 2024-04-05T03:02:42.340 INFO:tasks.workunit.client.1.smithi203.stdout:7/617: chown d8/d32/d16/d1a/d21/la0 106 1 2024-04-05T03:02:42.342 INFO:tasks.workunit.client.1.smithi203.stdout:2/658: write d5/d10/d1a/d57/f6b [142209,104591] 0 2024-04-05T03:02:42.342 INFO:tasks.workunit.client.1.smithi203.stdout:7/618: creat d8/d2a/d9b/fba x:0 0 0 2024-04-05T03:02:42.343 INFO:tasks.workunit.client.1.smithi203.stdout:2/659: chown d5/d10/d2f/d5d/lc9 82223168 1 2024-04-05T03:02:42.345 INFO:tasks.workunit.client.1.smithi203.stdout:7/619: creat d8/d2a/d3f/d5e/d77/fbb x:0 0 0 2024-04-05T03:02:42.345 INFO:tasks.workunit.client.1.smithi203.stdout:2/660: rmdir d5/d10/d35/d36/db6/d71 39 2024-04-05T03:02:42.346 INFO:tasks.workunit.client.1.smithi203.stdout:7/620: creat d8/d32/d16/d1a/d5a/fbc x:0 0 0 2024-04-05T03:02:42.347 INFO:tasks.workunit.client.1.smithi203.stdout:2/661: rmdir d5/d10/d7f 39 2024-04-05T03:02:42.350 INFO:tasks.workunit.client.1.smithi203.stdout:7/621: link d8/fe d8/d32/d16/d1a/d21/fbd 0 2024-04-05T03:02:42.353 INFO:tasks.workunit.client.1.smithi203.stdout:7/622: mkdir d8/d32/d5d/dbe 0 2024-04-05T03:02:42.353 INFO:tasks.workunit.client.1.smithi203.stdout:2/662: dwrite d5/d10/db8/f56 [0,4194304] 0 2024-04-05T03:02:42.354 INFO:tasks.workunit.client.1.smithi203.stdout:7/623: dread - d8/d32/d5d/d95/fa2 zero size 2024-04-05T03:02:42.355 INFO:tasks.workunit.client.1.smithi203.stdout:7/624: read d8/da/f14 [61825,8534] 0 2024-04-05T03:02:42.356 INFO:tasks.workunit.client.1.smithi203.stdout:2/663: mkdir d5/d63/d86/de3 0 2024-04-05T03:02:42.357 INFO:tasks.workunit.client.1.smithi203.stdout:7/625: read - d8/d32/d16/d35/d64/f8e zero size 2024-04-05T03:02:42.364 INFO:tasks.workunit.client.1.smithi203.stdout:2/664: dwrite d5/d10/d35/d2d/d7c/d29/d67/d9e/fd6 [0,4194304] 0 2024-04-05T03:02:42.365 INFO:tasks.workunit.client.1.smithi203.stdout:2/665: chown d5/d10/db8/dbe/fe0 1 1 2024-04-05T03:02:42.365 INFO:tasks.workunit.client.1.smithi203.stdout:2/666: read - d5/fc1 zero size 2024-04-05T03:02:42.370 INFO:tasks.workunit.client.0.smithi153.stdout:0/151: creat d9/db/f2d x:0 0 0 2024-04-05T03:02:42.370 INFO:tasks.workunit.client.0.smithi153.stdout:5/105: mknod d0/de/c26 0 2024-04-05T03:02:42.371 INFO:tasks.workunit.client.0.smithi153.stdout:8/113: write f6 [3828536,55932] 0 2024-04-05T03:02:42.372 INFO:tasks.workunit.client.0.smithi153.stdout:9/90: chown d1/d10/d16/f1a 175 1 2024-04-05T03:02:42.373 INFO:tasks.workunit.client.1.smithi203.stdout:2/667: link d5/d10/db8/f56 d5/d10/d35/d2d/d7c/d72/d8f/fe4 0 2024-04-05T03:02:42.373 INFO:tasks.workunit.client.0.smithi153.stdout:5/106: chown d0/d9/c23 864130398 1 2024-04-05T03:02:42.374 INFO:tasks.workunit.client.0.smithi153.stdout:6/159: truncate d4/f15 1618448 0 2024-04-05T03:02:42.377 INFO:tasks.workunit.client.0.smithi153.stdout:8/114: write d7/d13/d19/f22 [605199,129257] 0 2024-04-05T03:02:42.377 INFO:tasks.workunit.client.1.smithi203.stdout:2/668: getdents d5/d10/d35/d2d 0 2024-04-05T03:02:42.377 INFO:tasks.workunit.client.0.smithi153.stdout:2/137: rmdir d9/da 39 2024-04-05T03:02:42.380 INFO:tasks.workunit.client.1.smithi203.stdout:2/669: mkdir d5/d10/d35/d2d/d7c/d29/de5 0 2024-04-05T03:02:42.380 INFO:tasks.workunit.client.0.smithi153.stdout:6/160: creat d4/dc/df/f39 x:0 0 0 2024-04-05T03:02:42.380 INFO:tasks.workunit.client.0.smithi153.stdout:8/115: mknod d7/c24 0 2024-04-05T03:02:42.382 INFO:tasks.workunit.client.1.smithi203.stdout:2/670: fsync d5/d10/d1a/f34 0 2024-04-05T03:02:42.382 INFO:tasks.workunit.client.0.smithi153.stdout:6/161: stat d4/f20 0 2024-04-05T03:02:42.382 INFO:tasks.workunit.client.0.smithi153.stdout:2/138: fsync d9/da/f20 0 2024-04-05T03:02:42.383 INFO:tasks.workunit.client.1.smithi203.stdout:3/421: sync 2024-04-05T03:02:42.383 INFO:tasks.workunit.client.0.smithi153.stdout:0/152: dwrite f7 [0,4194304] 0 2024-04-05T03:02:42.383 INFO:tasks.workunit.client.0.smithi153.stdout:8/116: chown f0 1158368468 1 2024-04-05T03:02:42.384 INFO:tasks.workunit.client.1.smithi203.stdout:7/626: dread d8/d32/d16/d1a/d30/f38 [0,4194304] 0 2024-04-05T03:02:42.385 INFO:tasks.workunit.client.0.smithi153.stdout:8/117: readlink d7/l1d 0 2024-04-05T03:02:42.385 INFO:tasks.workunit.client.1.smithi203.stdout:7/627: chown d8/d2a/l5f 3754 1 2024-04-05T03:02:42.385 INFO:tasks.workunit.client.0.smithi153.stdout:9/91: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:42.386 INFO:tasks.workunit.client.1.smithi203.stdout:7/628: chown d8/f4c 0 1 2024-04-05T03:02:42.387 INFO:tasks.workunit.client.0.smithi153.stdout:9/92: chown d1/db 3 1 2024-04-05T03:02:42.387 INFO:tasks.workunit.client.0.smithi153.stdout:5/107: link d0/f14 d0/f27 0 2024-04-05T03:02:42.387 INFO:tasks.workunit.client.1.smithi203.stdout:7/629: dread - d8/d32/d16/fa9 zero size 2024-04-05T03:02:42.387 INFO:tasks.workunit.client.1.smithi203.stdout:3/422: symlink d0/dc/d38/l79 0 2024-04-05T03:02:42.389 INFO:tasks.workunit.client.0.smithi153.stdout:6/162: creat d4/dc/df/d16/d17/f3a x:0 0 0 2024-04-05T03:02:42.389 INFO:tasks.workunit.client.0.smithi153.stdout:5/108: chown d0/d9/f20 402218953 1 2024-04-05T03:02:42.389 INFO:tasks.workunit.client.1.smithi203.stdout:2/671: creat d5/d10/d35/d2d/d7c/d72/fe6 x:0 0 0 2024-04-05T03:02:42.391 INFO:tasks.workunit.client.0.smithi153.stdout:9/93: write d1/d9/f20 [894910,100866] 0 2024-04-05T03:02:42.392 INFO:tasks.workunit.client.0.smithi153.stdout:6/163: write d4/dc/df/f26 [3941374,112383] 0 2024-04-05T03:02:42.392 INFO:tasks.workunit.client.0.smithi153.stdout:5/109: write d0/d9/f18 [212937,6351] 0 2024-04-05T03:02:42.393 INFO:tasks.workunit.client.1.smithi203.stdout:3/423: symlink d0/dc/d1a/d76/d35/l7a 0 2024-04-05T03:02:42.394 INFO:tasks.workunit.client.0.smithi153.stdout:0/153: dread d9/df/f15 [0,4194304] 0 2024-04-05T03:02:42.395 INFO:tasks.workunit.client.0.smithi153.stdout:5/110: truncate d0/d9/f21 680058 0 2024-04-05T03:02:42.396 INFO:tasks.workunit.client.0.smithi153.stdout:5/111: dread - d0/f1c zero size 2024-04-05T03:02:42.397 INFO:tasks.workunit.client.1.smithi203.stdout:7/630: dwrite d8/d32/d5d/d6d/f8d [0,4194304] 0 2024-04-05T03:02:42.397 INFO:tasks.workunit.client.1.smithi203.stdout:2/672: creat d5/d10/d35/d36/db6/d71/fe7 x:0 0 0 2024-04-05T03:02:42.399 INFO:tasks.workunit.client.1.smithi203.stdout:3/424: creat d0/dc/d3f/f7b x:0 0 0 2024-04-05T03:02:42.402 INFO:tasks.workunit.client.1.smithi203.stdout:3/425: mknod d0/dc/d3f/c7c 0 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:1/466: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:4/351: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:6/528: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:9/412: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:8/511: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/475: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.1.smithi203.stdout:5/475: sync 2024-04-05T03:02:42.406 INFO:tasks.workunit.client.0.smithi153.stdout:9/94: rename d1/c2 to d1/d9/d1b/c24 0 2024-04-05T03:02:42.407 INFO:tasks.workunit.client.1.smithi203.stdout:7/631: link d8/da/f50 d8/d32/d5d/d6d/fbf 0 2024-04-05T03:02:42.409 INFO:tasks.workunit.client.1.smithi203.stdout:3/426: unlink d0/l17 0 2024-04-05T03:02:42.411 INFO:tasks.workunit.client.1.smithi203.stdout:8/512: chown d1/d6/d2e/f30 105974 1 2024-04-05T03:02:42.411 INFO:tasks.workunit.client.0.smithi153.stdout:5/112: truncate d0/f27 1528354 0 2024-04-05T03:02:42.411 INFO:tasks.workunit.client.0.smithi153.stdout:8/118: truncate d7/fa 902787 0 2024-04-05T03:02:42.412 INFO:tasks.workunit.client.1.smithi203.stdout:5/476: symlink d3/d1f/l9b 0 2024-04-05T03:02:42.412 INFO:tasks.workunit.client.1.smithi203.stdout:0/476: dread - dd/d17/d1d/d22/d53/d7f/f83 zero size 2024-04-05T03:02:42.413 INFO:tasks.workunit.client.1.smithi203.stdout:2/673: dwrite d5/d10/d1a/f65 [0,4194304] 0 2024-04-05T03:02:42.414 INFO:tasks.workunit.client.0.smithi153.stdout:6/164: rename d4/dc/df/d16/d33/f37 to d4/dc/df/d16/d17/f3b 0 2024-04-05T03:02:42.415 INFO:tasks.workunit.client.0.smithi153.stdout:6/165: fsync d4/f20 0 2024-04-05T03:02:42.418 INFO:tasks.workunit.client.0.smithi153.stdout:5/113: creat d0/d9/d22/f28 x:0 0 0 2024-04-05T03:02:42.418 INFO:tasks.workunit.client.1.smithi203.stdout:8/513: write d1/f58 [486456,8903] 0 2024-04-05T03:02:42.421 INFO:tasks.workunit.client.0.smithi153.stdout:0/154: truncate f0 1011305 0 2024-04-05T03:02:42.422 INFO:tasks.workunit.client.1.smithi203.stdout:2/674: write d5/d10/d2f/d5d/f68 [1704830,46261] 0 2024-04-05T03:02:42.423 INFO:tasks.workunit.client.1.smithi203.stdout:6/529: dread d0/da/d18/d38/f47 [0,4194304] 0 2024-04-05T03:02:42.424 INFO:tasks.workunit.client.0.smithi153.stdout:0/155: read d9/d14/f19 [272024,97566] 0 2024-04-05T03:02:42.425 INFO:tasks.workunit.client.0.smithi153.stdout:7/119: sync 2024-04-05T03:02:42.425 INFO:tasks.workunit.client.0.smithi153.stdout:4/163: sync 2024-04-05T03:02:42.425 INFO:tasks.workunit.client.0.smithi153.stdout:1/137: sync 2024-04-05T03:02:42.426 INFO:tasks.workunit.client.1.smithi203.stdout:3/427: rename d0/dc/d1a/d1d/d53/d5a/l67 to d0/dc/d1a/d76/l7d 0 2024-04-05T03:02:42.426 INFO:tasks.workunit.client.0.smithi153.stdout:6/166: rmdir d4/dc/df/d16/d24 39 2024-04-05T03:02:42.427 INFO:tasks.workunit.client.0.smithi153.stdout:9/95: creat d1/f25 x:0 0 0 2024-04-05T03:02:42.427 INFO:tasks.workunit.client.0.smithi153.stdout:2/139: dwrite d9/f11 [0,4194304] 0 2024-04-05T03:02:42.428 INFO:tasks.workunit.client.0.smithi153.stdout:5/114: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T03:02:42.428 INFO:tasks.workunit.client.0.smithi153.stdout:8/119: mknod d7/d13/d19/d1b/d21/c25 0 2024-04-05T03:02:42.429 INFO:tasks.workunit.client.0.smithi153.stdout:0/156: symlink d9/df/l2e 0 2024-04-05T03:02:42.432 INFO:tasks.workunit.client.1.smithi203.stdout:0/477: symlink dd/d17/d1d/d22/d5f/l8a 0 2024-04-05T03:02:42.433 INFO:tasks.workunit.client.0.smithi153.stdout:1/138: rmdir d5/dd 39 2024-04-05T03:02:42.433 INFO:tasks.workunit.client.1.smithi203.stdout:5/477: mknod d3/d1f/c9c 0 2024-04-05T03:02:42.434 INFO:tasks.workunit.client.0.smithi153.stdout:1/139: stat d5/f9 0 2024-04-05T03:02:42.436 INFO:tasks.workunit.client.1.smithi203.stdout:9/413: dwrite d13/d1b/d33/f3f [4194304,4194304] 0 2024-04-05T03:02:42.439 INFO:tasks.workunit.client.1.smithi203.stdout:6/530: symlink d0/da/d54/d82/la0 0 2024-04-05T03:02:42.442 INFO:tasks.workunit.client.0.smithi153.stdout:4/164: dread da/fc [0,4194304] 0 2024-04-05T03:02:42.443 INFO:tasks.workunit.client.0.smithi153.stdout:6/167: dread d4/dc/df/f12 [0,4194304] 0 2024-04-05T03:02:42.445 INFO:tasks.workunit.client.1.smithi203.stdout:1/467: dwrite d7/d14/d2b/f5a [0,4194304] 0 2024-04-05T03:02:42.445 INFO:tasks.workunit.client.0.smithi153.stdout:8/120: symlink d7/d13/d19/d1b/l26 0 2024-04-05T03:02:42.449 INFO:tasks.workunit.client.0.smithi153.stdout:9/96: creat d1/d9/d1b/f26 x:0 0 0 2024-04-05T03:02:42.449 INFO:tasks.workunit.client.0.smithi153.stdout:0/157: mknod d9/db/c2f 0 2024-04-05T03:02:42.452 INFO:tasks.workunit.client.1.smithi203.stdout:3/428: rmdir d0/dc/d1a/d1d 39 2024-04-05T03:02:42.453 INFO:tasks.workunit.client.1.smithi203.stdout:1/468: fsync d7/d14/f1b 0 2024-04-05T03:02:42.454 INFO:tasks.workunit.client.0.smithi153.stdout:2/140: dwrite d9/da/d10/f1a [0,4194304] 0 2024-04-05T03:02:42.455 INFO:tasks.workunit.client.0.smithi153.stdout:5/115: mknod d0/c29 0 2024-04-05T03:02:42.455 INFO:tasks.workunit.client.1.smithi203.stdout:0/478: creat dd/d17/d1d/d26/d28/f8b x:0 0 0 2024-04-05T03:02:42.455 INFO:tasks.workunit.client.0.smithi153.stdout:6/168: rmdir d4 39 2024-04-05T03:02:42.456 INFO:tasks.workunit.client.0.smithi153.stdout:8/121: truncate d7/d13/d19/f20 521578 0 2024-04-05T03:02:42.458 INFO:tasks.workunit.client.1.smithi203.stdout:2/675: dwrite d5/d10/d35/d36/db6/d71/f97 [0,4194304] 0 2024-04-05T03:02:42.458 INFO:tasks.workunit.client.0.smithi153.stdout:1/140: rmdir d5/dd/d21/d18 39 2024-04-05T03:02:42.458 INFO:tasks.workunit.client.0.smithi153.stdout:9/97: chown d1/f18 19 1 2024-04-05T03:02:42.459 INFO:tasks.workunit.client.0.smithi153.stdout:5/116: read - d0/d9/f1f zero size 2024-04-05T03:02:42.460 INFO:tasks.workunit.client.1.smithi203.stdout:8/514: dwrite d1/d12/f31 [0,4194304] 0 2024-04-05T03:02:42.461 INFO:tasks.workunit.client.1.smithi203.stdout:9/414: symlink d13/d1b/d33/d42/l80 0 2024-04-05T03:02:42.462 INFO:tasks.workunit.client.0.smithi153.stdout:6/169: chown d4/dc/df/d16/d17/f2a 42874474 1 2024-04-05T03:02:42.464 INFO:tasks.workunit.client.1.smithi203.stdout:8/515: fsync d1/d12/f28 0 2024-04-05T03:02:42.465 INFO:tasks.workunit.client.1.smithi203.stdout:2/676: readlink d5/d10/d35/d36/l55 0 2024-04-05T03:02:42.465 INFO:tasks.workunit.client.1.smithi203.stdout:7/632: dread d8/d32/d16/d1a/d3d/f6b [0,4194304] 0 2024-04-05T03:02:42.467 INFO:tasks.workunit.client.0.smithi153.stdout:6/170: read - d4/dc/df/d16/d17/f3a zero size 2024-04-05T03:02:42.468 INFO:tasks.workunit.client.0.smithi153.stdout:1/141: chown d5/d8/d20 1880 1 2024-04-05T03:02:42.468 INFO:tasks.workunit.client.0.smithi153.stdout:1/142: fsync f4 0 2024-04-05T03:02:42.471 INFO:tasks.workunit.client.0.smithi153.stdout:0/158: creat d9/f30 x:0 0 0 2024-04-05T03:02:42.472 INFO:tasks.workunit.client.0.smithi153.stdout:5/117: mknod d0/de/c2a 0 2024-04-05T03:02:42.472 INFO:tasks.workunit.client.0.smithi153.stdout:9/98: dwrite d1/d9/d1b/f26 [0,4194304] 0 2024-04-05T03:02:42.474 INFO:tasks.workunit.client.0.smithi153.stdout:1/143: truncate d5/dd/d1b/f1d 4954313 0 2024-04-05T03:02:42.474 INFO:tasks.workunit.client.0.smithi153.stdout:5/118: stat d0/de/l15 0 2024-04-05T03:02:42.474 INFO:tasks.workunit.client.1.smithi203.stdout:3/429: dread - d0/dc/d1a/d1d/d53/f75 zero size 2024-04-05T03:02:42.475 INFO:tasks.workunit.client.0.smithi153.stdout:5/119: chown d0 824709011 1 2024-04-05T03:02:42.475 INFO:tasks.workunit.client.0.smithi153.stdout:2/141: symlink d9/da/d19/d2e/l31 0 2024-04-05T03:02:42.477 INFO:tasks.workunit.client.0.smithi153.stdout:0/159: symlink d9/db/l31 0 2024-04-05T03:02:42.478 INFO:tasks.workunit.client.1.smithi203.stdout:2/677: symlink d5/d10/db8/dbe/le8 0 2024-04-05T03:02:42.479 INFO:tasks.workunit.client.1.smithi203.stdout:9/415: truncate d13/d1b/d26/d31/f60 607921 0 2024-04-05T03:02:42.479 INFO:tasks.workunit.client.1.smithi203.stdout:1/469: dwrite d7/d14/d2b/f5a [0,4194304] 0 2024-04-05T03:02:42.480 INFO:tasks.workunit.client.0.smithi153.stdout:5/120: rmdir d0/d9/d22 39 2024-04-05T03:02:42.480 INFO:tasks.workunit.client.0.smithi153.stdout:0/160: read d9/d14/f19 [2592875,10975] 0 2024-04-05T03:02:42.481 INFO:tasks.workunit.client.1.smithi203.stdout:5/478: dwrite d3/d4/dc/f95 [0,4194304] 0 2024-04-05T03:02:42.482 INFO:tasks.workunit.client.0.smithi153.stdout:8/122: dread d7/d13/f14 [0,4194304] 0 2024-04-05T03:02:42.482 INFO:tasks.workunit.client.0.smithi153.stdout:1/144: fdatasync d5/dd/d21/d18/f1a 0 2024-04-05T03:02:42.485 INFO:tasks.workunit.client.1.smithi203.stdout:0/479: rename dd/d17/d1d/d22/d5f to dd/d17/d1d/d26/d8c 0 2024-04-05T03:02:42.489 INFO:tasks.workunit.client.0.smithi153.stdout:9/99: getdents d1/d10 0 2024-04-05T03:02:42.491 INFO:tasks.workunit.client.1.smithi203.stdout:1/470: unlink d7/d14/f1b 0 2024-04-05T03:02:42.494 INFO:tasks.workunit.client.0.smithi153.stdout:6/171: dwrite d4/dc/df/d16/d17/f3b [0,4194304] 0 2024-04-05T03:02:42.494 INFO:tasks.workunit.client.0.smithi153.stdout:0/161: readlink d9/df/d1e/l22 0 2024-04-05T03:02:42.495 INFO:tasks.workunit.client.1.smithi203.stdout:2/678: dread d5/d63/fa3 [0,4194304] 0 2024-04-05T03:02:42.495 INFO:tasks.workunit.client.1.smithi203.stdout:7/633: rename d8/f46 to d8/d2a/d3f/d73/da8/fc0 0 2024-04-05T03:02:42.495 INFO:tasks.workunit.client.0.smithi153.stdout:8/123: rmdir d7/d13/d19/d1b 39 2024-04-05T03:02:42.497 INFO:tasks.workunit.client.1.smithi203.stdout:3/430: dwrite d0/dc/d38/f5f [0,4194304] 0 2024-04-05T03:02:42.497 INFO:tasks.workunit.client.1.smithi203.stdout:2/679: write d5/fc1 [865250,115977] 0 2024-04-05T03:02:42.497 INFO:tasks.workunit.client.1.smithi203.stdout:7/634: write d8/d2a/d3f/d5e/fa5 [463242,29887] 0 2024-04-05T03:02:42.500 INFO:tasks.workunit.client.0.smithi153.stdout:5/121: creat d0/d25/f2b x:0 0 0 2024-04-05T03:02:42.501 INFO:tasks.workunit.client.1.smithi203.stdout:2/680: truncate d5/d10/d35/d2d/d7c/d72/d8f/fe4 5016749 0 2024-04-05T03:02:42.502 INFO:tasks.workunit.client.1.smithi203.stdout:2/681: write d5/d10/d35/d36/db6/fa7 [267462,59807] 0 2024-04-05T03:02:42.503 INFO:tasks.workunit.client.1.smithi203.stdout:2/682: readlink d5/d10/d35/d36/l55 0 2024-04-05T03:02:42.504 INFO:tasks.workunit.client.0.smithi153.stdout:1/145: mkdir d5/d8/d20/d2f 0 2024-04-05T03:02:42.508 INFO:tasks.workunit.client.0.smithi153.stdout:6/172: rename d4/dc/df/d16/d17/f2c to d4/dc/f3c 0 2024-04-05T03:02:42.508 INFO:tasks.workunit.client.1.smithi203.stdout:9/416: getdents d13/d1b/d26/d31 0 2024-04-05T03:02:42.509 INFO:tasks.workunit.client.0.smithi153.stdout:0/162: dwrite d9/df/f16 [0,4194304] 0 2024-04-05T03:02:42.509 INFO:tasks.workunit.client.1.smithi203.stdout:2/683: symlink d5/d10/db8/dab/le9 0 2024-04-05T03:02:42.511 INFO:tasks.workunit.client.1.smithi203.stdout:7/635: creat d8/d2a/d3f/d52/d65/d82/fc1 x:0 0 0 2024-04-05T03:02:42.513 INFO:tasks.workunit.client.1.smithi203.stdout:0/480: getdents dd/d17/d1d/d26/d34/d56 0 2024-04-05T03:02:42.513 INFO:tasks.workunit.client.0.smithi153.stdout:1/146: mknod d5/d8/c30 0 2024-04-05T03:02:42.513 INFO:tasks.workunit.client.0.smithi153.stdout:0/163: read d9/db/fd [741111,108829] 0 2024-04-05T03:02:42.514 INFO:tasks.workunit.client.0.smithi153.stdout:8/124: symlink d7/d13/d19/l27 0 2024-04-05T03:02:42.515 INFO:tasks.workunit.client.0.smithi153.stdout:9/100: link d1/d10/f19 d1/d10/d16/f27 0 2024-04-05T03:02:42.516 INFO:tasks.workunit.client.0.smithi153.stdout:1/147: rmdir d5 39 2024-04-05T03:02:42.518 INFO:tasks.workunit.client.1.smithi203.stdout:9/417: mkdir d13/d1b/d26/d51/d81 0 2024-04-05T03:02:42.518 INFO:tasks.workunit.client.0.smithi153.stdout:5/122: dwrite d0/d9/f1e [0,4194304] 0 2024-04-05T03:02:42.518 INFO:tasks.workunit.client.0.smithi153.stdout:0/164: creat d9/df/d1e/f32 x:0 0 0 2024-04-05T03:02:42.520 INFO:tasks.workunit.client.0.smithi153.stdout:0/165: dread - d9/f30 zero size 2024-04-05T03:02:42.521 INFO:tasks.workunit.client.1.smithi203.stdout:7/636: rename d8/f31 to d8/d2a/d3f/d73/fc2 0 2024-04-05T03:02:42.522 INFO:tasks.workunit.client.1.smithi203.stdout:5/479: link d3/d5f/c6b d3/d1f/d59/c9d 0 2024-04-05T03:02:42.526 INFO:tasks.workunit.client.1.smithi203.stdout:1/471: rename d7/d24/l7e to d7/d24/l92 0 2024-04-05T03:02:42.527 INFO:tasks.workunit.client.1.smithi203.stdout:1/472: fdatasync d7/d24/f66 0 2024-04-05T03:02:42.527 INFO:tasks.workunit.client.1.smithi203.stdout:1/473: chown d7 5981 1 2024-04-05T03:02:42.528 INFO:tasks.workunit.client.0.smithi153.stdout:9/101: link d1/d9/d1b/f1c d1/d9/d1b/f28 0 2024-04-05T03:02:42.530 INFO:tasks.workunit.client.1.smithi203.stdout:0/481: mknod dd/d17/d1d/d26/c8d 0 2024-04-05T03:02:42.530 INFO:tasks.workunit.client.0.smithi153.stdout:8/125: dwrite d7/fe [0,4194304] 0 2024-04-05T03:02:42.532 INFO:tasks.workunit.client.0.smithi153.stdout:0/166: write d9/df/f17 [400494,21842] 0 2024-04-05T03:02:42.533 INFO:tasks.workunit.client.0.smithi153.stdout:1/148: mknod d5/dd/d21/d18/d16/d2c/c31 0 2024-04-05T03:02:42.533 INFO:tasks.workunit.client.1.smithi203.stdout:2/684: link d5/d10/d35/d2d/d7c/d29/d67/fd5 d5/d10/d35/d2d/d7c/d72/d8f/fea 0 2024-04-05T03:02:42.536 INFO:tasks.workunit.client.0.smithi153.stdout:5/123: dwrite d0/de/f1a [0,4194304] 0 2024-04-05T03:02:42.538 INFO:tasks.workunit.client.1.smithi203.stdout:9/418: mkdir d13/d1b/d33/d42/d67/d82 0 2024-04-05T03:02:42.539 INFO:tasks.workunit.client.1.smithi203.stdout:5/480: mkdir d3/d4/d9e 0 2024-04-05T03:02:42.540 INFO:tasks.workunit.client.1.smithi203.stdout:5/481: stat f2 0 2024-04-05T03:02:42.545 INFO:tasks.workunit.client.0.smithi153.stdout:9/102: mknod d1/d10/d16/d22/c29 0 2024-04-05T03:02:42.546 INFO:tasks.workunit.client.0.smithi153.stdout:1/149: mknod d5/dd/d21/d18/d16/d2c/c32 0 2024-04-05T03:02:42.546 INFO:tasks.workunit.client.0.smithi153.stdout:0/167: symlink d9/db/l33 0 2024-04-05T03:02:42.546 INFO:tasks.workunit.client.1.smithi203.stdout:9/419: dwrite d13/d1b/d26/d51/f52 [0,4194304] 0 2024-04-05T03:02:42.548 INFO:tasks.workunit.client.0.smithi153.stdout:8/126: link d7/fe d7/f28 0 2024-04-05T03:02:42.548 INFO:tasks.workunit.client.1.smithi203.stdout:0/482: dread dd/d17/d1d/d26/d34/f35 [0,4194304] 0 2024-04-05T03:02:42.549 INFO:tasks.workunit.client.0.smithi153.stdout:8/127: chown d7/l1d 76251475 1 2024-04-05T03:02:42.550 INFO:tasks.workunit.client.1.smithi203.stdout:0/483: dread - dd/d17/d1d/d26/d28/f8b zero size 2024-04-05T03:02:42.551 INFO:tasks.workunit.client.0.smithi153.stdout:5/124: mkdir d0/d2c 0 2024-04-05T03:02:42.552 INFO:tasks.workunit.client.1.smithi203.stdout:5/482: creat d3/d4/d9e/f9f x:0 0 0 2024-04-05T03:02:42.553 INFO:tasks.workunit.client.1.smithi203.stdout:9/420: truncate d13/d1b/d33/f65 67531 0 2024-04-05T03:02:42.557 INFO:tasks.workunit.client.1.smithi203.stdout:2/685: getdents d5/d66 0 2024-04-05T03:02:42.558 INFO:tasks.workunit.client.0.smithi153.stdout:6/173: rmdir d4/dc/df/d16/d17 39 2024-04-05T03:02:42.560 INFO:tasks.workunit.client.0.smithi153.stdout:0/168: dwrite d9/df/d1e/f25 [0,4194304] 0 2024-04-05T03:02:42.560 INFO:tasks.workunit.client.0.smithi153.stdout:8/128: symlink d7/d13/d19/l29 0 2024-04-05T03:02:42.560 INFO:tasks.workunit.client.0.smithi153.stdout:9/103: creat d1/f2a x:0 0 0 2024-04-05T03:02:42.562 INFO:tasks.workunit.client.1.smithi203.stdout:9/421: fsync d13/d1b/f21 0 2024-04-05T03:02:42.565 INFO:tasks.workunit.client.0.smithi153.stdout:9/104: chown d1 6593 1 2024-04-05T03:02:42.566 INFO:tasks.workunit.client.0.smithi153.stdout:5/125: creat d0/d2c/f2d x:0 0 0 2024-04-05T03:02:42.566 INFO:tasks.workunit.client.1.smithi203.stdout:7/637: getdents d8/d2a/d3f/d73/da8 0 2024-04-05T03:02:42.570 INFO:tasks.workunit.client.0.smithi153.stdout:1/150: dwrite d5/dd/d21/f11 [0,4194304] 0 2024-04-05T03:02:42.572 INFO:tasks.workunit.client.0.smithi153.stdout:9/105: mknod d1/d1d/c2b 0 2024-04-05T03:02:42.573 INFO:tasks.workunit.client.1.smithi203.stdout:0/484: dread dd/d17/d1d/d22/d53/f63 [4194304,4194304] 0 2024-04-05T03:02:42.574 INFO:tasks.workunit.client.0.smithi153.stdout:0/169: dread d9/df/d1e/f25 [0,4194304] 0 2024-04-05T03:02:42.574 INFO:tasks.workunit.client.1.smithi203.stdout:9/422: symlink d13/d1b/d26/d31/d38/d59/d5f/l83 0 2024-04-05T03:02:42.574 INFO:tasks.workunit.client.1.smithi203.stdout:8/516: dread d1/d12/d1a/f42 [0,4194304] 0 2024-04-05T03:02:42.574 INFO:tasks.workunit.client.0.smithi153.stdout:8/129: fsync d7/d13/f14 0 2024-04-05T03:02:42.576 INFO:tasks.workunit.client.0.smithi153.stdout:8/130: dread - d7/f10 zero size 2024-04-05T03:02:42.576 INFO:tasks.workunit.client.1.smithi203.stdout:2/686: dwrite d5/f8 [0,4194304] 0 2024-04-05T03:02:42.577 INFO:tasks.workunit.client.0.smithi153.stdout:3/124: sync 2024-04-05T03:02:42.577 INFO:tasks.workunit.client.1.smithi203.stdout:0/485: stat dd/d17/d1d/d26/d8c/l8a 0 2024-04-05T03:02:42.580 INFO:tasks.workunit.client.0.smithi153.stdout:9/106: mkdir d1/d10/d16/d22/d2c 0 2024-04-05T03:02:42.582 INFO:tasks.workunit.client.1.smithi203.stdout:2/687: truncate d5/d10/d44/fd7 677805 0 2024-04-05T03:02:42.586 INFO:tasks.workunit.client.1.smithi203.stdout:0/486: symlink dd/d17/d1d/d22/d45/d51/l8e 0 2024-04-05T03:02:42.587 INFO:tasks.workunit.client.1.smithi203.stdout:1/474: write d7/d24/f35 [1178147,125684] 0 2024-04-05T03:02:42.588 INFO:tasks.workunit.client.1.smithi203.stdout:2/688: mknod d5/d66/d7d/ceb 0 2024-04-05T03:02:42.589 INFO:tasks.workunit.client.1.smithi203.stdout:9/423: dwrite d13/d1b/f21 [4194304,4194304] 0 2024-04-05T03:02:42.590 INFO:tasks.workunit.client.1.smithi203.stdout:9/424: stat d13/d1b/d26/c2d 0 2024-04-05T03:02:42.591 INFO:tasks.workunit.client.1.smithi203.stdout:7/638: rename d8/da/d72/d9c to d8/d32/dc3 0 2024-04-05T03:02:42.593 INFO:tasks.workunit.client.1.smithi203.stdout:7/639: chown d8/d32/d16/d1a/d3d/f76 1281814 1 2024-04-05T03:02:42.594 INFO:tasks.workunit.client.1.smithi203.stdout:0/487: chown dd/d17/d1d/d22/d45/d51/l58 1102 1 2024-04-05T03:02:42.595 INFO:tasks.workunit.client.1.smithi203.stdout:5/483: dread d3/d1f/d59/d64/f3d [0,4194304] 0 2024-04-05T03:02:42.596 INFO:tasks.workunit.client.1.smithi203.stdout:5/484: fsync d3/f36 0 2024-04-05T03:02:42.596 INFO:tasks.workunit.client.1.smithi203.stdout:7/640: truncate d8/d32/d16/d1a/d5a/fa3 169163 0 2024-04-05T03:02:42.597 INFO:tasks.workunit.client.1.smithi203.stdout:2/689: creat d5/d10/d44/fec x:0 0 0 2024-04-05T03:02:42.598 INFO:tasks.workunit.client.1.smithi203.stdout:7/641: chown d8/c9 1734 1 2024-04-05T03:02:42.598 INFO:tasks.workunit.client.1.smithi203.stdout:8/517: link d1/d6/c85 d1/d12/d17/d47/d53/d69/c89 0 2024-04-05T03:02:42.601 INFO:tasks.workunit.client.1.smithi203.stdout:9/425: mknod d13/d1b/d33/d42/d67/c84 0 2024-04-05T03:02:42.602 INFO:tasks.workunit.client.1.smithi203.stdout:5/485: creat d3/d4/d94/fa0 x:0 0 0 2024-04-05T03:02:42.605 INFO:tasks.workunit.client.0.smithi153.stdout:0/170: dread d9/db/f13 [0,4194304] 0 2024-04-05T03:02:42.606 INFO:tasks.workunit.client.1.smithi203.stdout:5/486: dread - d3/d1f/d59/d64/d12/f96 zero size 2024-04-05T03:02:42.606 INFO:tasks.workunit.client.1.smithi203.stdout:7/642: dwrite d8/d32/f44 [4194304,4194304] 0 2024-04-05T03:02:42.607 INFO:tasks.workunit.client.0.smithi153.stdout:3/125: creat d1d/f2c x:0 0 0 2024-04-05T03:02:42.607 INFO:tasks.workunit.client.1.smithi203.stdout:8/518: dwrite d1/d4b/d5f/f7d [0,4194304] 0 2024-04-05T03:02:42.608 INFO:tasks.workunit.client.0.smithi153.stdout:3/126: fdatasync f18 0 2024-04-05T03:02:42.610 INFO:tasks.workunit.client.0.smithi153.stdout:6/174: write d4/dc/df/f12 [1030684,49212] 0 2024-04-05T03:02:42.613 INFO:tasks.workunit.client.0.smithi153.stdout:8/131: dwrite d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:42.615 INFO:tasks.workunit.client.1.smithi203.stdout:7/643: chown d8/f3e 5935 1 2024-04-05T03:02:42.616 INFO:tasks.workunit.client.1.smithi203.stdout:7/644: write d8/d32/d16/d1a/d21/f22 [1520571,125968] 0 2024-04-05T03:02:42.617 INFO:tasks.workunit.client.0.smithi153.stdout:5/126: rename d0/la to d0/d9/d22/l2e 0 2024-04-05T03:02:42.618 INFO:tasks.workunit.client.1.smithi203.stdout:0/488: link dd/d17/d1d/d26/d34/d56/l73 dd/d17/d1d/d22/d80/d49/l8f 0 2024-04-05T03:02:42.619 INFO:tasks.workunit.client.1.smithi203.stdout:7/645: write d8/d2a/d3f/d5e/fa5 [1173277,44722] 0 2024-04-05T03:02:42.623 INFO:tasks.workunit.client.0.smithi153.stdout:0/171: rmdir d9/df/d1e 39 2024-04-05T03:02:42.623 INFO:tasks.workunit.client.0.smithi153.stdout:9/107: dread d1/f1f [0,4194304] 0 2024-04-05T03:02:42.623 INFO:tasks.workunit.client.1.smithi203.stdout:5/487: mknod d3/d4/ca1 0 2024-04-05T03:02:42.626 INFO:tasks.workunit.client.0.smithi153.stdout:5/127: write d0/d2c/f2d [583182,14670] 0 2024-04-05T03:02:42.627 INFO:tasks.workunit.client.0.smithi153.stdout:8/132: dwrite f3 [0,4194304] 0 2024-04-05T03:02:42.628 INFO:tasks.workunit.client.0.smithi153.stdout:3/127: creat d1d/d27/d2b/f2d x:0 0 0 2024-04-05T03:02:42.629 INFO:tasks.workunit.client.0.smithi153.stdout:0/172: unlink d9/lc 0 2024-04-05T03:02:42.629 INFO:tasks.workunit.client.0.smithi153.stdout:5/128: fsync d0/d9/f21 0 2024-04-05T03:02:42.629 INFO:tasks.workunit.client.1.smithi203.stdout:8/519: dwrite d1/d4b/d68/d75/d77/f87 [0,4194304] 0 2024-04-05T03:02:42.633 INFO:tasks.workunit.client.1.smithi203.stdout:4/352: sync 2024-04-05T03:02:42.633 INFO:tasks.workunit.client.1.smithi203.stdout:6/531: sync 2024-04-05T03:02:42.634 INFO:tasks.workunit.client.0.smithi153.stdout:9/108: mknod d1/d10/d16/d22/d2c/c2d 0 2024-04-05T03:02:42.636 INFO:tasks.workunit.client.1.smithi203.stdout:7/646: symlink d8/d32/d16/d1a/d5a/db5/lc4 0 2024-04-05T03:02:42.637 INFO:tasks.workunit.client.0.smithi153.stdout:6/175: dwrite d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:42.637 INFO:tasks.workunit.client.0.smithi153.stdout:3/128: mknod d1d/c2e 0 2024-04-05T03:02:42.638 INFO:tasks.workunit.client.1.smithi203.stdout:5/488: mkdir d3/d1f/d59/d60/da2 0 2024-04-05T03:02:42.639 INFO:tasks.workunit.client.1.smithi203.stdout:1/475: dwrite d7/f80 [0,4194304] 0 2024-04-05T03:02:42.640 INFO:tasks.workunit.client.0.smithi153.stdout:0/173: mkdir d9/df/d12/d34 0 2024-04-05T03:02:42.641 INFO:tasks.workunit.client.0.smithi153.stdout:5/129: creat d0/d25/f2f x:0 0 0 2024-04-05T03:02:42.642 INFO:tasks.workunit.client.1.smithi203.stdout:8/520: symlink d1/d12/d2f/d5b/l8a 0 2024-04-05T03:02:42.644 INFO:tasks.workunit.client.0.smithi153.stdout:6/176: dread d4/dc/df/f2f [0,4194304] 0 2024-04-05T03:02:42.646 INFO:tasks.workunit.client.1.smithi203.stdout:4/353: dread da/d29/d2e/f3e [0,4194304] 0 2024-04-05T03:02:42.647 INFO:tasks.workunit.client.1.smithi203.stdout:5/489: mknod d3/d5f/d92/ca3 0 2024-04-05T03:02:42.652 INFO:tasks.workunit.client.1.smithi203.stdout:6/532: getdents d0/da/d18/d38 0 2024-04-05T03:02:42.653 INFO:tasks.workunit.client.0.smithi153.stdout:0/174: creat d9/df/d1e/f35 x:0 0 0 2024-04-05T03:02:42.653 INFO:tasks.workunit.client.1.smithi203.stdout:4/354: symlink da/d29/d2e/d57/l70 0 2024-04-05T03:02:42.654 INFO:tasks.workunit.client.0.smithi153.stdout:5/130: dwrite d0/d2c/f2d [0,4194304] 0 2024-04-05T03:02:42.655 INFO:tasks.workunit.client.0.smithi153.stdout:8/133: getdents d7/d13/d19/d1b 0 2024-04-05T03:02:42.655 INFO:tasks.workunit.client.1.smithi203.stdout:4/355: write da/f13 [3588777,46959] 0 2024-04-05T03:02:42.656 INFO:tasks.workunit.client.1.smithi203.stdout:8/521: dwrite d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:42.656 INFO:tasks.workunit.client.0.smithi153.stdout:9/109: creat d1/f2e x:0 0 0 2024-04-05T03:02:42.656 INFO:tasks.workunit.client.0.smithi153.stdout:3/129: dread f18 [0,4194304] 0 2024-04-05T03:02:42.657 INFO:tasks.workunit.client.1.smithi203.stdout:5/490: link d3/d4/l7c d3/d1f/la4 0 2024-04-05T03:02:42.659 INFO:tasks.workunit.client.1.smithi203.stdout:4/356: readlink da/d3c/d4c/l4f 0 2024-04-05T03:02:42.659 INFO:tasks.workunit.client.0.smithi153.stdout:0/175: rmdir d9/d14 39 2024-04-05T03:02:42.661 INFO:tasks.workunit.client.0.smithi153.stdout:5/131: dread d0/d2c/f2d [0,4194304] 0 2024-04-05T03:02:42.661 INFO:tasks.workunit.client.0.smithi153.stdout:9/110: symlink d1/d10/l2f 0 2024-04-05T03:02:42.662 INFO:tasks.workunit.client.0.smithi153.stdout:6/177: link d4/dc/df/c1c d4/dc/c3d 0 2024-04-05T03:02:42.665 INFO:tasks.workunit.client.1.smithi203.stdout:6/533: dwrite d0/da/d18/d38/d2a/d2c/f46 [0,4194304] 0 2024-04-05T03:02:42.665 INFO:tasks.workunit.client.0.smithi153.stdout:8/134: unlink d7/lb 0 2024-04-05T03:02:42.665 INFO:tasks.workunit.client.0.smithi153.stdout:3/130: creat d1d/d27/d2b/f2f x:0 0 0 2024-04-05T03:02:42.667 INFO:tasks.workunit.client.1.smithi203.stdout:6/534: write d0/da/d54/d4c/f5b [1429899,75444] 0 2024-04-05T03:02:42.668 INFO:tasks.workunit.client.0.smithi153.stdout:5/132: symlink d0/de/l30 0 2024-04-05T03:02:42.668 INFO:tasks.workunit.client.0.smithi153.stdout:8/135: write d7/f17 [760973,54477] 0 2024-04-05T03:02:42.669 INFO:tasks.workunit.client.0.smithi153.stdout:6/178: write d4/dc/f2e [140471,115108] 0 2024-04-05T03:02:42.669 INFO:tasks.workunit.client.1.smithi203.stdout:6/535: write d0/da/d54/d4c/f77 [651609,21250] 0 2024-04-05T03:02:42.670 INFO:tasks.workunit.client.1.smithi203.stdout:8/522: creat d1/d12/d2f/d55/f8b x:0 0 0 2024-04-05T03:02:42.671 INFO:tasks.workunit.client.0.smithi153.stdout:5/133: write d0/de/f1a [4468541,49574] 0 2024-04-05T03:02:42.674 INFO:tasks.workunit.client.1.smithi203.stdout:0/489: truncate dd/d17/d1d/f37 3310608 0 2024-04-05T03:02:42.678 INFO:tasks.workunit.client.1.smithi203.stdout:4/357: read da/f3a [762275,125539] 0 2024-04-05T03:02:42.678 INFO:tasks.workunit.client.1.smithi203.stdout:8/523: symlink d1/d12/d17/d47/d53/d69/l8c 0 2024-04-05T03:02:42.681 INFO:tasks.workunit.client.1.smithi203.stdout:8/524: rmdir d1/d6/d2e/d5c 39 2024-04-05T03:02:42.682 INFO:tasks.workunit.client.0.smithi153.stdout:8/136: mkdir d7/d2a 0 2024-04-05T03:02:42.683 INFO:tasks.workunit.client.1.smithi203.stdout:6/536: dwrite d0/f3 [0,4194304] 0 2024-04-05T03:02:42.683 INFO:tasks.workunit.client.1.smithi203.stdout:0/490: write dd/d17/d1d/d26/f89 [283484,6587] 0 2024-04-05T03:02:42.686 INFO:tasks.workunit.client.1.smithi203.stdout:0/491: truncate dd/d17/d1d/d22/d80/d49/f7d 1172471 0 2024-04-05T03:02:42.687 INFO:tasks.workunit.client.1.smithi203.stdout:6/537: unlink d0/da/d18/d38/d86/f99 0 2024-04-05T03:02:42.688 INFO:tasks.workunit.client.0.smithi153.stdout:6/179: mkdir d4/dc/d3e 0 2024-04-05T03:02:42.689 INFO:tasks.workunit.client.1.smithi203.stdout:8/525: symlink d1/d4b/d68/d75/d77/l8d 0 2024-04-05T03:02:42.693 INFO:tasks.workunit.client.1.smithi203.stdout:0/492: rmdir dd/d17/d1d/d26/d34/d56 39 2024-04-05T03:02:42.693 INFO:tasks.workunit.client.1.smithi203.stdout:6/538: mkdir d0/da/d54/d4c/da1 0 2024-04-05T03:02:42.694 INFO:tasks.workunit.client.1.smithi203.stdout:8/526: dread d1/d4b/d5f/f7d [0,4194304] 0 2024-04-05T03:02:42.695 INFO:tasks.workunit.client.0.smithi153.stdout:5/134: link d0/d9/f21 d0/d25/f31 0 2024-04-05T03:02:42.697 INFO:tasks.workunit.client.0.smithi153.stdout:6/180: mknod d4/dc/df/d16/d24/c3f 0 2024-04-05T03:02:42.698 INFO:tasks.workunit.client.0.smithi153.stdout:0/176: write f0 [1954325,58099] 0 2024-04-05T03:02:42.699 INFO:tasks.workunit.client.1.smithi203.stdout:6/539: rmdir d0/da/d18/d38/d2a 39 2024-04-05T03:02:42.701 INFO:tasks.workunit.client.0.smithi153.stdout:6/181: mknod d4/c40 0 2024-04-05T03:02:42.702 INFO:tasks.workunit.client.1.smithi203.stdout:8/527: mknod d1/d12/d17/c8e 0 2024-04-05T03:02:42.702 INFO:tasks.workunit.client.1.smithi203.stdout:0/493: mknod dd/d17/d1d/d22/c90 0 2024-04-05T03:02:42.702 INFO:tasks.workunit.client.1.smithi203.stdout:1/476: dread d7/d14/f76 [0,4194304] 0 2024-04-05T03:02:42.703 INFO:tasks.workunit.client.1.smithi203.stdout:8/528: chown d1/d6/f41 64255 1 2024-04-05T03:02:42.706 INFO:tasks.workunit.client.0.smithi153.stdout:0/177: mkdir d9/db/d36 0 2024-04-05T03:02:42.707 INFO:tasks.workunit.client.1.smithi203.stdout:8/529: mkdir d1/d6/d8f 0 2024-04-05T03:02:42.709 INFO:tasks.workunit.client.1.smithi203.stdout:0/494: creat dd/d17/d1d/d22/d45/d51/d69/f91 x:0 0 0 2024-04-05T03:02:42.709 INFO:tasks.workunit.client.0.smithi153.stdout:8/137: getdents d7 0 2024-04-05T03:02:42.710 INFO:tasks.workunit.client.0.smithi153.stdout:6/182: dwrite d4/f6 [0,4194304] 0 2024-04-05T03:02:42.711 INFO:tasks.workunit.client.0.smithi153.stdout:5/135: rename d0/d9/d22/f24 to d0/d9/f32 0 2024-04-05T03:02:42.712 INFO:tasks.workunit.client.0.smithi153.stdout:3/131: dwrite f3 [4194304,4194304] 0 2024-04-05T03:02:42.713 INFO:tasks.workunit.client.1.smithi203.stdout:5/491: dread d3/f36 [16777216,4194304] 0 2024-04-05T03:02:42.714 INFO:tasks.workunit.client.0.smithi153.stdout:3/132: fdatasync f18 0 2024-04-05T03:02:42.714 INFO:tasks.workunit.client.0.smithi153.stdout:0/178: creat d9/db/f37 x:0 0 0 2024-04-05T03:02:42.716 INFO:tasks.workunit.client.1.smithi203.stdout:1/477: rename d7/f27 to d7/d14/f93 0 2024-04-05T03:02:42.717 INFO:tasks.workunit.client.0.smithi153.stdout:8/138: creat d7/f2b x:0 0 0 2024-04-05T03:02:42.718 INFO:tasks.workunit.client.1.smithi203.stdout:1/478: dread - d7/d24/d33/d48/d4b/d51/f7f zero size 2024-04-05T03:02:42.718 INFO:tasks.workunit.client.0.smithi153.stdout:8/139: fsync d7/f16 0 2024-04-05T03:02:42.720 INFO:tasks.workunit.client.0.smithi153.stdout:6/183: symlink d4/dc/df/d16/d24/l41 0 2024-04-05T03:02:42.720 INFO:tasks.workunit.client.0.smithi153.stdout:9/111: write d1/d10/d16/f1a [2312335,30171] 0 2024-04-05T03:02:42.722 INFO:tasks.workunit.client.1.smithi203.stdout:1/479: write d7/d79/f7d [176745,130051] 0 2024-04-05T03:02:42.724 INFO:tasks.workunit.client.1.smithi203.stdout:1/480: chown d7/d90 7014 1 2024-04-05T03:02:42.724 INFO:tasks.workunit.client.1.smithi203.stdout:0/495: dread f7 [0,4194304] 0 2024-04-05T03:02:42.725 INFO:tasks.workunit.client.0.smithi153.stdout:3/133: write d1d/f21 [263442,2191] 0 2024-04-05T03:02:42.726 INFO:tasks.workunit.client.1.smithi203.stdout:0/496: fdatasync dd/d17/d1d/d22/d80/f72 0 2024-04-05T03:02:42.726 INFO:tasks.workunit.client.0.smithi153.stdout:0/179: dread d9/db/fd [0,4194304] 0 2024-04-05T03:02:42.727 INFO:tasks.workunit.client.0.smithi153.stdout:5/136: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T03:02:42.729 INFO:tasks.workunit.client.0.smithi153.stdout:8/140: creat d7/d2a/f2c x:0 0 0 2024-04-05T03:02:42.731 INFO:tasks.workunit.client.0.smithi153.stdout:9/112: chown d1/f18 193 1 2024-04-05T03:02:42.731 INFO:tasks.workunit.client.0.smithi153.stdout:3/134: symlink d1d/l30 0 2024-04-05T03:02:42.732 INFO:tasks.workunit.client.0.smithi153.stdout:9/113: read d1/d9/fc [1188651,125507] 0 2024-04-05T03:02:42.733 INFO:tasks.workunit.client.1.smithi203.stdout:1/481: dwrite d7/d24/f66 [0,4194304] 0 2024-04-05T03:02:42.733 INFO:tasks.workunit.client.0.smithi153.stdout:6/184: mknod d4/dc/df/d16/d17/c42 0 2024-04-05T03:02:42.734 INFO:tasks.workunit.client.0.smithi153.stdout:5/137: creat d0/d9/d22/f33 x:0 0 0 2024-04-05T03:02:42.739 INFO:tasks.workunit.client.0.smithi153.stdout:5/138: symlink d0/d2c/l34 0 2024-04-05T03:02:42.739 INFO:tasks.workunit.client.0.smithi153.stdout:9/114: creat d1/d9/d1b/f30 x:0 0 0 2024-04-05T03:02:42.741 INFO:tasks.workunit.client.1.smithi203.stdout:5/492: link d3/d4/dc/l5c d3/d1f/d49/la5 0 2024-04-05T03:02:42.742 INFO:tasks.workunit.client.0.smithi153.stdout:5/139: read - d0/d9/d22/f33 zero size 2024-04-05T03:02:42.742 INFO:tasks.workunit.client.1.smithi203.stdout:5/493: chown d3/d1f/d59/d64/l34 2 1 2024-04-05T03:02:42.745 INFO:tasks.workunit.client.0.smithi153.stdout:0/180: rename d9/df/d1e/l22 to d9/df/l38 0 2024-04-05T03:02:42.750 INFO:tasks.workunit.client.0.smithi153.stdout:6/185: dread d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:42.756 INFO:tasks.workunit.client.0.smithi153.stdout:7/120: sync 2024-04-05T03:02:42.756 INFO:tasks.workunit.client.0.smithi153.stdout:9/115: dwrite d1/d9/d1b/f30 [0,4194304] 0 2024-04-05T03:02:42.758 INFO:tasks.workunit.client.0.smithi153.stdout:0/181: rmdir d9/df/d12/d34 0 2024-04-05T03:02:42.760 INFO:tasks.workunit.client.0.smithi153.stdout:0/182: chown f7 81 1 2024-04-05T03:02:42.762 INFO:tasks.workunit.client.0.smithi153.stdout:8/141: dwrite d7/f23 [4194304,4194304] 0 2024-04-05T03:02:42.764 INFO:tasks.workunit.client.1.smithi203.stdout:6/540: write d0/da/d18/d38/d2a/d2c/f4e [82151,22109] 0 2024-04-05T03:02:42.764 INFO:tasks.workunit.client.0.smithi153.stdout:8/142: write f0 [3687610,111084] 0 2024-04-05T03:02:42.764 INFO:tasks.workunit.client.0.smithi153.stdout:6/186: rename d4/dc/df/d16/c28 to d4/dc/c43 0 2024-04-05T03:02:42.765 INFO:tasks.workunit.client.0.smithi153.stdout:7/121: rename d2/da to d2/da/d1f/d28 22 2024-04-05T03:02:42.766 INFO:tasks.workunit.client.0.smithi153.stdout:9/116: fdatasync d1/d9/fc 0 2024-04-05T03:02:42.771 INFO:tasks.workunit.client.0.smithi153.stdout:9/117: write d1/d10/d16/f23 [394353,127289] 0 2024-04-05T03:02:42.772 INFO:tasks.workunit.client.0.smithi153.stdout:8/143: dread f5 [0,4194304] 0 2024-04-05T03:02:42.774 INFO:tasks.workunit.client.0.smithi153.stdout:7/122: rmdir d2/da 39 2024-04-05T03:02:42.774 INFO:tasks.workunit.client.1.smithi203.stdout:6/541: dwrite d0/da/d18/d38/d2a/d2c/f46 [0,4194304] 0 2024-04-05T03:02:42.775 INFO:tasks.workunit.client.0.smithi153.stdout:8/144: dread d7/d13/d19/f20 [0,4194304] 0 2024-04-05T03:02:42.775 INFO:tasks.workunit.client.1.smithi203.stdout:6/542: write d0/f3 [7592685,102746] 0 2024-04-05T03:02:42.776 INFO:tasks.workunit.client.0.smithi153.stdout:8/145: read - d7/d2a/f2c zero size 2024-04-05T03:02:42.777 INFO:tasks.workunit.client.1.smithi203.stdout:6/543: write d0/da/d54/d4c/f77 [37920,33258] 0 2024-04-05T03:02:42.778 INFO:tasks.workunit.client.0.smithi153.stdout:7/123: readlink d2/da/l23 0 2024-04-05T03:02:42.779 INFO:tasks.workunit.client.1.smithi203.stdout:5/494: mknod d3/d4/d94/ca6 0 2024-04-05T03:02:42.779 INFO:tasks.workunit.client.1.smithi203.stdout:5/495: chown d3/d1f/d59/d64/f1c 105847 1 2024-04-05T03:02:42.782 INFO:tasks.workunit.client.0.smithi153.stdout:7/124: fdatasync d2/da/d11/d12/f19 0 2024-04-05T03:02:42.782 INFO:tasks.workunit.client.0.smithi153.stdout:7/125: fsync d2/f5 0 2024-04-05T03:02:42.783 INFO:tasks.workunit.client.0.smithi153.stdout:6/187: mkdir d4/dc/df/d16/d44 0 2024-04-05T03:02:42.784 INFO:tasks.workunit.client.0.smithi153.stdout:8/146: unlink d7/f12 0 2024-04-05T03:02:42.784 INFO:tasks.workunit.client.0.smithi153.stdout:9/118: dwrite d1/d9/fc [0,4194304] 0 2024-04-05T03:02:42.787 INFO:tasks.workunit.client.1.smithi203.stdout:6/544: dwrite d0/da/d18/f1d [0,4194304] 0 2024-04-05T03:02:42.787 INFO:tasks.workunit.client.1.smithi203.stdout:5/496: dread d3/d1f/d59/d64/f4f [0,4194304] 0 2024-04-05T03:02:42.789 INFO:tasks.workunit.client.1.smithi203.stdout:5/497: write d3/f18 [3604342,103881] 0 2024-04-05T03:02:42.790 INFO:tasks.workunit.client.1.smithi203.stdout:5/498: readlink d3/d4/dc/l14 0 2024-04-05T03:02:42.792 INFO:tasks.workunit.client.0.smithi153.stdout:5/140: truncate d0/de/f10 313484 0 2024-04-05T03:02:42.792 INFO:tasks.workunit.client.0.smithi153.stdout:8/147: creat d7/d2a/f2d x:0 0 0 2024-04-05T03:02:42.793 INFO:tasks.workunit.client.0.smithi153.stdout:5/141: readlink d0/de/l30 0 2024-04-05T03:02:42.793 INFO:tasks.workunit.client.1.smithi203.stdout:1/482: creat d7/d24/d33/d48/d4b/f94 x:0 0 0 2024-04-05T03:02:42.797 INFO:tasks.workunit.client.0.smithi153.stdout:8/148: write d7/d13/f18 [3563396,21045] 0 2024-04-05T03:02:42.802 INFO:tasks.workunit.client.0.smithi153.stdout:7/126: symlink d2/da/l29 0 2024-04-05T03:02:42.802 INFO:tasks.workunit.client.0.smithi153.stdout:0/183: write d9/db/fd [457774,36125] 0 2024-04-05T03:02:42.802 INFO:tasks.workunit.client.0.smithi153.stdout:6/188: mkdir d4/dc/d3e/d45 0 2024-04-05T03:02:42.802 INFO:tasks.workunit.client.1.smithi203.stdout:5/499: write d3/d1f/f93 [287763,63840] 0 2024-04-05T03:02:42.803 INFO:tasks.workunit.client.0.smithi153.stdout:7/127: chown d2/da/cd 15801542 1 2024-04-05T03:02:42.803 INFO:tasks.workunit.client.0.smithi153.stdout:0/184: read d9/df/f15 [994356,70613] 0 2024-04-05T03:02:42.804 INFO:tasks.workunit.client.0.smithi153.stdout:5/142: readlink d0/d9/d22/l2e 0 2024-04-05T03:02:42.806 INFO:tasks.workunit.client.0.smithi153.stdout:7/128: dread - d2/da/d1f/f20 zero size 2024-04-05T03:02:42.806 INFO:tasks.workunit.client.0.smithi153.stdout:8/149: dread d7/fe [0,4194304] 0 2024-04-05T03:02:42.806 INFO:tasks.workunit.client.0.smithi153.stdout:6/189: unlink d4/dc/l10 0 2024-04-05T03:02:42.807 INFO:tasks.workunit.client.1.smithi203.stdout:6/545: link d0/da/d18/d38/f6d d0/da/d67/fa2 0 2024-04-05T03:02:42.807 INFO:tasks.workunit.client.0.smithi153.stdout:0/185: dread - d9/df/d1e/f35 zero size 2024-04-05T03:02:42.807 INFO:tasks.workunit.client.0.smithi153.stdout:5/143: rmdir d0/d2c 39 2024-04-05T03:02:42.809 INFO:tasks.workunit.client.1.smithi203.stdout:5/500: dwrite d3/f36 [12582912,4194304] 0 2024-04-05T03:02:42.809 INFO:tasks.workunit.client.1.smithi203.stdout:6/546: symlink d0/da/d67/d71/la3 0 2024-04-05T03:02:42.809 INFO:tasks.workunit.client.1.smithi203.stdout:3/431: sync 2024-04-05T03:02:42.813 INFO:tasks.workunit.client.0.smithi153.stdout:6/190: rmdir d4/dc/df/d16 39 2024-04-05T03:02:42.814 INFO:tasks.workunit.client.0.smithi153.stdout:6/191: write d4/dc/df/f12 [1291050,106606] 0 2024-04-05T03:02:42.821 INFO:tasks.workunit.client.0.smithi153.stdout:7/129: mknod d2/da/c2a 0 2024-04-05T03:02:42.821 INFO:tasks.workunit.client.0.smithi153.stdout:0/186: dread d9/df/f15 [0,4194304] 0 2024-04-05T03:02:42.822 INFO:tasks.workunit.client.0.smithi153.stdout:6/192: dread d4/f20 [0,4194304] 0 2024-04-05T03:02:42.823 INFO:tasks.workunit.client.1.smithi203.stdout:5/501: rename d3/d1f/d59/d64/d12/c77 to d3/d5f/ca7 0 2024-04-05T03:02:42.823 INFO:tasks.workunit.client.0.smithi153.stdout:6/193: chown d4/fb 35 1 2024-04-05T03:02:42.824 INFO:tasks.workunit.client.1.smithi203.stdout:3/432: dwrite d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:42.826 INFO:tasks.workunit.client.0.smithi153.stdout:9/119: write d1/d9/d1b/f28 [398683,81367] 0 2024-04-05T03:02:42.826 INFO:tasks.workunit.client.0.smithi153.stdout:7/130: rmdir d2/da/d11/d12 39 2024-04-05T03:02:42.827 INFO:tasks.workunit.client.1.smithi203.stdout:5/502: write d3/d1f/d59/d64/f44 [1480284,40509] 0 2024-04-05T03:02:42.830 INFO:tasks.workunit.client.0.smithi153.stdout:0/187: dwrite d9/df/d1e/f32 [0,4194304] 0 2024-04-05T03:02:42.832 INFO:tasks.workunit.client.0.smithi153.stdout:6/194: creat d4/dc/f46 x:0 0 0 2024-04-05T03:02:42.835 INFO:tasks.workunit.client.1.smithi203.stdout:3/433: dwrite d0/d21/f6c [0,4194304] 0 2024-04-05T03:02:42.835 INFO:tasks.workunit.client.0.smithi153.stdout:8/150: truncate f6 1276565 0 2024-04-05T03:02:42.840 INFO:tasks.workunit.client.1.smithi203.stdout:5/503: getdents d3/d1f/d59/d60/da2 0 2024-04-05T03:02:42.842 INFO:tasks.workunit.client.1.smithi203.stdout:3/434: creat d0/dc/d38/d5d/d66/f7e x:0 0 0 2024-04-05T03:02:42.844 INFO:tasks.workunit.client.1.smithi203.stdout:3/435: stat d0/dc/d38/d5d 0 2024-04-05T03:02:42.845 INFO:tasks.workunit.client.0.smithi153.stdout:7/131: rename d2/da/f27 to d2/da/d11/f2b 0 2024-04-05T03:02:42.847 INFO:tasks.workunit.client.0.smithi153.stdout:8/151: dwrite d7/d2a/f2c [0,4194304] 0 2024-04-05T03:02:42.848 INFO:tasks.workunit.client.0.smithi153.stdout:6/195: dread d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:42.850 INFO:tasks.workunit.client.1.smithi203.stdout:5/504: truncate d3/d1f/d59/d64/f3a 427750 0 2024-04-05T03:02:42.851 INFO:tasks.workunit.client.1.smithi203.stdout:5/505: write d3/d1f/f93 [466611,92197] 0 2024-04-05T03:02:42.858 INFO:tasks.workunit.client.1.smithi203.stdout:5/506: dwrite d3/f7a [0,4194304] 0 2024-04-05T03:02:42.858 INFO:tasks.workunit.client.0.smithi153.stdout:5/144: truncate d0/de/f10 77093 0 2024-04-05T03:02:42.860 INFO:tasks.workunit.client.1.smithi203.stdout:5/507: chown d3/d1f/d59/d60/f74 14255 1 2024-04-05T03:02:42.861 INFO:tasks.workunit.client.1.smithi203.stdout:5/508: readlink d3/l9a 0 2024-04-05T03:02:42.861 INFO:tasks.workunit.client.0.smithi153.stdout:6/196: link d4/dc/f2e d4/dc/df/d16/d33/f47 0 2024-04-05T03:02:42.861 INFO:tasks.workunit.client.0.smithi153.stdout:0/188: write d9/df/f15 [4067170,121700] 0 2024-04-05T03:02:42.862 INFO:tasks.workunit.client.1.smithi203.stdout:0/497: rmdir dd/d17/d1d/d22/d45 39 2024-04-05T03:02:42.863 INFO:tasks.workunit.client.0.smithi153.stdout:7/132: link d2/da/d1f/f25 d2/da/d11/f2c 0 2024-04-05T03:02:42.867 INFO:tasks.workunit.client.0.smithi153.stdout:5/145: rename d0/d9/f32 to d0/d25/f35 0 2024-04-05T03:02:42.868 INFO:tasks.workunit.client.0.smithi153.stdout:5/146: dread - d0/d25/f2b zero size 2024-04-05T03:02:42.868 INFO:tasks.workunit.client.0.smithi153.stdout:6/197: chown d4/f15 1217 1 2024-04-05T03:02:42.869 INFO:tasks.workunit.client.0.smithi153.stdout:5/147: stat d0/d9/f1f 0 2024-04-05T03:02:42.871 INFO:tasks.workunit.client.0.smithi153.stdout:5/148: fdatasync d0/d9/d22/f33 0 2024-04-05T03:02:42.871 INFO:tasks.workunit.client.0.smithi153.stdout:7/133: chown d2/da/d11/d12/f1e 23 1 2024-04-05T03:02:42.871 INFO:tasks.workunit.client.0.smithi153.stdout:0/189: write d9/df/d1d/f24 [839227,73945] 0 2024-04-05T03:02:42.871 INFO:tasks.workunit.client.1.smithi203.stdout:1/483: dwrite d7/d24/d33/d48/d4b/d51/f87 [0,4194304] 0 2024-04-05T03:02:42.871 INFO:tasks.workunit.client.1.smithi203.stdout:6/547: symlink d0/da/d18/d38/d2a/la4 0 2024-04-05T03:02:42.873 INFO:tasks.workunit.client.0.smithi153.stdout:5/149: write d0/d25/f2b [960999,60844] 0 2024-04-05T03:02:42.876 INFO:tasks.workunit.client.0.smithi153.stdout:8/152: write f6 [1104668,42306] 0 2024-04-05T03:02:42.876 INFO:tasks.workunit.client.1.smithi203.stdout:1/484: write d7/d79/f7d [752841,124102] 0 2024-04-05T03:02:42.877 INFO:tasks.workunit.client.1.smithi203.stdout:5/509: dwrite d3/d4/f71 [0,4194304] 0 2024-04-05T03:02:42.877 INFO:tasks.workunit.client.1.smithi203.stdout:3/436: write d0/d41/f4a [2937770,41107] 0 2024-04-05T03:02:42.878 INFO:tasks.workunit.client.1.smithi203.stdout:6/548: write d0/da/d67/f7a [285556,90143] 0 2024-04-05T03:02:42.882 INFO:tasks.workunit.client.1.smithi203.stdout:6/549: readlink d0/da/l74 0 2024-04-05T03:02:42.884 INFO:tasks.workunit.client.1.smithi203.stdout:0/498: dwrite dd/d17/d1d/d22/d80/d49/f7d [0,4194304] 0 2024-04-05T03:02:42.887 INFO:tasks.workunit.client.0.smithi153.stdout:6/198: mknod d4/dc/df/d16/d17/c48 0 2024-04-05T03:02:42.888 INFO:tasks.workunit.client.1.smithi203.stdout:5/510: read d3/f4d [326795,72023] 0 2024-04-05T03:02:42.888 INFO:tasks.workunit.client.0.smithi153.stdout:6/199: chown d4/dc/l30 70213117 1 2024-04-05T03:02:42.890 INFO:tasks.workunit.client.1.smithi203.stdout:3/437: rmdir d0/dc/d38/d5d/d66 39 2024-04-05T03:02:42.891 INFO:tasks.workunit.client.0.smithi153.stdout:2/142: sync 2024-04-05T03:02:42.891 INFO:tasks.workunit.client.0.smithi153.stdout:4/165: sync 2024-04-05T03:02:42.891 INFO:tasks.workunit.client.0.smithi153.stdout:1/151: sync 2024-04-05T03:02:42.893 INFO:tasks.workunit.client.0.smithi153.stdout:4/166: write da/db/f1a [227710,12712] 0 2024-04-05T03:02:42.893 INFO:tasks.workunit.client.0.smithi153.stdout:5/150: creat d0/d25/f36 x:0 0 0 2024-04-05T03:02:42.894 INFO:tasks.workunit.client.1.smithi203.stdout:6/550: dwrite d0/da/d18/d38/d86/f7c [0,4194304] 0 2024-04-05T03:02:42.895 INFO:tasks.workunit.client.0.smithi153.stdout:4/167: write da/db/fd [8702423,89464] 0 2024-04-05T03:02:42.895 INFO:tasks.workunit.client.0.smithi153.stdout:1/152: fsync d5/dd/d1b/f1d 0 2024-04-05T03:02:42.898 INFO:tasks.workunit.client.1.smithi203.stdout:0/499: unlink dd/d17/d1d/d22/d53/d55/l5a 0 2024-04-05T03:02:42.903 INFO:tasks.workunit.client.1.smithi203.stdout:5/511: mknod d3/d1f/d59/d64/d12/ca8 0 2024-04-05T03:02:42.903 INFO:tasks.workunit.client.1.smithi203.stdout:6/551: rmdir d0/da/d18/d38/d2a 39 2024-04-05T03:02:42.904 INFO:tasks.workunit.client.0.smithi153.stdout:8/153: mknod d7/c2e 0 2024-04-05T03:02:42.904 INFO:tasks.workunit.client.0.smithi153.stdout:8/154: stat d7/l1d 0 2024-04-05T03:02:42.904 INFO:tasks.workunit.client.0.smithi153.stdout:6/200: symlink d4/dc/df/d16/d33/l49 0 2024-04-05T03:02:42.910 INFO:tasks.workunit.client.1.smithi203.stdout:3/438: creat d0/f7f x:0 0 0 2024-04-05T03:02:42.912 INFO:tasks.workunit.client.0.smithi153.stdout:5/151: dread d0/d2c/f2d [0,4194304] 0 2024-04-05T03:02:42.912 INFO:tasks.workunit.client.0.smithi153.stdout:7/134: rename d2/da/d1f/f25 to d2/da/d11/f2d 0 2024-04-05T03:02:42.914 INFO:tasks.workunit.client.1.smithi203.stdout:5/512: mknod d3/d1f/d59/d64/ca9 0 2024-04-05T03:02:42.915 INFO:tasks.workunit.client.0.smithi153.stdout:0/190: getdents d9/df/d1e 0 2024-04-05T03:02:42.915 INFO:tasks.workunit.client.0.smithi153.stdout:0/191: chown d9/f30 23 1 2024-04-05T03:02:42.916 INFO:tasks.workunit.client.0.smithi153.stdout:5/152: mkdir d0/de/d37 0 2024-04-05T03:02:42.916 INFO:tasks.workunit.client.1.smithi203.stdout:5/513: chown d3/d1f/la4 126 1 2024-04-05T03:02:42.918 INFO:tasks.workunit.client.0.smithi153.stdout:2/143: rename d9/da/c2c to d9/da/d10/c32 0 2024-04-05T03:02:42.919 INFO:tasks.workunit.client.1.smithi203.stdout:5/514: fsync d3/d1f/f62 0 2024-04-05T03:02:42.920 INFO:tasks.workunit.client.0.smithi153.stdout:2/144: truncate d9/da/d10/f1a 4590369 0 2024-04-05T03:02:42.921 INFO:tasks.workunit.client.1.smithi203.stdout:6/552: dread d0/da/f61 [0,4194304] 0 2024-04-05T03:02:42.922 INFO:tasks.workunit.client.0.smithi153.stdout:6/201: link d4/dc/df/c1c d4/dc/c4a 0 2024-04-05T03:02:42.923 INFO:tasks.workunit.client.0.smithi153.stdout:5/153: mkdir d0/de/d37/d38 0 2024-04-05T03:02:42.925 INFO:tasks.workunit.client.1.smithi203.stdout:6/553: creat d0/da/d18/d38/d2a/d2c/fa5 x:0 0 0 2024-04-05T03:02:42.927 INFO:tasks.workunit.client.0.smithi153.stdout:4/168: rename da/l15 to da/db/d20/d30/l31 0 2024-04-05T03:02:42.927 INFO:tasks.workunit.client.0.smithi153.stdout:2/145: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:42.927 INFO:tasks.workunit.client.1.smithi203.stdout:5/515: creat d3/faa x:0 0 0 2024-04-05T03:02:42.928 INFO:tasks.workunit.client.0.smithi153.stdout:2/146: fdatasync d9/f21 0 2024-04-05T03:02:42.929 INFO:tasks.workunit.client.1.smithi203.stdout:6/554: creat d0/da/d67/d71/fa6 x:0 0 0 2024-04-05T03:02:42.931 INFO:tasks.workunit.client.1.smithi203.stdout:0/500: dread dd/d17/d1d/d22/d53/d61/f6a [0,4194304] 0 2024-04-05T03:02:42.931 INFO:tasks.workunit.client.0.smithi153.stdout:0/192: link f0 d9/df/f39 0 2024-04-05T03:02:42.931 INFO:tasks.workunit.client.1.smithi203.stdout:5/516: fsync d3/d1f/f62 0 2024-04-05T03:02:42.932 INFO:tasks.workunit.client.1.smithi203.stdout:5/517: fsync d3/f36 0 2024-04-05T03:02:42.932 INFO:tasks.workunit.client.1.smithi203.stdout:6/555: write d0/da/d54/d53/f63 [767168,118843] 0 2024-04-05T03:02:42.933 INFO:tasks.workunit.client.0.smithi153.stdout:6/202: mkdir d4/dc/df/d16/d17/d4b 0 2024-04-05T03:02:42.936 INFO:tasks.workunit.client.0.smithi153.stdout:2/147: dwrite d9/f11 [0,4194304] 0 2024-04-05T03:02:42.937 INFO:tasks.workunit.client.0.smithi153.stdout:1/153: truncate d5/dd/d1b/f1d 3023990 0 2024-04-05T03:02:42.937 INFO:tasks.workunit.client.0.smithi153.stdout:5/154: mkdir d0/de/d37/d39 0 2024-04-05T03:02:42.938 INFO:tasks.workunit.client.1.smithi203.stdout:5/518: unlink d3/d1f/d49/c6e 0 2024-04-05T03:02:42.940 INFO:tasks.workunit.client.0.smithi153.stdout:5/155: write d0/f1c [493387,64100] 0 2024-04-05T03:02:42.941 INFO:tasks.workunit.client.1.smithi203.stdout:6/556: mkdir d0/da/d18/d38/d2a/d44/d78/da7 0 2024-04-05T03:02:42.941 INFO:tasks.workunit.client.0.smithi153.stdout:5/156: write d0/d9/d22/f28 [111930,16482] 0 2024-04-05T03:02:42.942 INFO:tasks.workunit.client.0.smithi153.stdout:8/155: rename d7/d13/c1a to d7/d2a/c2f 0 2024-04-05T03:02:42.944 INFO:tasks.workunit.client.1.smithi203.stdout:5/519: chown d3/d4/dc/c6d 99304 1 2024-04-05T03:02:42.946 INFO:tasks.workunit.client.1.smithi203.stdout:5/520: write d3/d1f/d59/d60/f99 [972112,16102] 0 2024-04-05T03:02:42.947 INFO:tasks.workunit.client.1.smithi203.stdout:6/557: rename d0/da/d54/d53/l80 to d0/da/d67/la8 0 2024-04-05T03:02:42.947 INFO:tasks.workunit.client.0.smithi153.stdout:1/154: mkdir d5/dd/d21/d18/d16/d33 0 2024-04-05T03:02:42.948 INFO:tasks.workunit.client.0.smithi153.stdout:5/157: creat d0/d2c/f3a x:0 0 0 2024-04-05T03:02:42.949 INFO:tasks.workunit.client.0.smithi153.stdout:8/156: dwrite d7/fc [0,4194304] 0 2024-04-05T03:02:42.949 INFO:tasks.workunit.client.0.smithi153.stdout:5/158: chown d0/d25 914048 1 2024-04-05T03:02:42.950 INFO:tasks.workunit.client.0.smithi153.stdout:1/155: chown d5/dd/d1b/f2b 1082830 1 2024-04-05T03:02:42.951 INFO:tasks.workunit.client.1.smithi203.stdout:5/521: creat d3/d4/d9e/fab x:0 0 0 2024-04-05T03:02:42.952 INFO:tasks.workunit.client.1.smithi203.stdout:6/558: getdents d0/da/d18/d92 0 2024-04-05T03:02:42.953 INFO:tasks.workunit.client.1.smithi203.stdout:6/559: chown d0/da/d54/d53/f63 2020890 1 2024-04-05T03:02:42.954 INFO:tasks.workunit.client.1.smithi203.stdout:5/522: mkdir d3/d4/dc/d67/dac 0 2024-04-05T03:02:42.954 INFO:tasks.workunit.client.0.smithi153.stdout:8/157: mknod d7/d13/d19/d1b/d21/c30 0 2024-04-05T03:02:42.955 INFO:tasks.workunit.client.1.smithi203.stdout:5/523: dread - d3/d4/d9e/fab zero size 2024-04-05T03:02:42.955 INFO:tasks.workunit.client.1.smithi203.stdout:6/560: rmdir d0/da/d54/d4c 39 2024-04-05T03:02:42.959 INFO:tasks.workunit.client.0.smithi153.stdout:7/135: rename d2/da/d11/d12/f1b to d2/da/d11/f2e 0 2024-04-05T03:02:42.960 INFO:tasks.workunit.client.1.smithi203.stdout:5/524: creat d3/d1f/d59/d60/fad x:0 0 0 2024-04-05T03:02:42.961 INFO:tasks.workunit.client.0.smithi153.stdout:1/156: fdatasync d5/d8/ff 0 2024-04-05T03:02:42.962 INFO:tasks.workunit.client.1.smithi203.stdout:5/525: symlink d3/d1f/d49/lae 0 2024-04-05T03:02:42.963 INFO:tasks.workunit.client.1.smithi203.stdout:6/561: creat d0/da/d54/d4c/da1/fa9 x:0 0 0 2024-04-05T03:02:42.964 INFO:tasks.workunit.client.0.smithi153.stdout:5/159: creat d0/de/d37/d39/f3b x:0 0 0 2024-04-05T03:02:42.965 INFO:tasks.workunit.client.1.smithi203.stdout:5/526: mknod d3/d4/caf 0 2024-04-05T03:02:42.966 INFO:tasks.workunit.client.1.smithi203.stdout:5/527: fsync d3/d5f/f75 0 2024-04-05T03:02:42.966 INFO:tasks.workunit.client.1.smithi203.stdout:6/562: write d0/da/d54/d53/f6a [807306,35287] 0 2024-04-05T03:02:42.969 INFO:tasks.workunit.client.0.smithi153.stdout:1/157: dwrite d5/dd/d21/d18/f1a [4194304,4194304] 0 2024-04-05T03:02:42.969 INFO:tasks.workunit.client.1.smithi203.stdout:5/528: unlink d3/d4/d94/fa0 0 2024-04-05T03:02:42.970 INFO:tasks.workunit.client.0.smithi153.stdout:6/203: rename d4/dc/df/d16/d17/d4b to d4/d38/d4c 0 2024-04-05T03:02:42.971 INFO:tasks.workunit.client.0.smithi153.stdout:0/193: rmdir d9 39 2024-04-05T03:02:42.972 INFO:tasks.workunit.client.1.smithi203.stdout:5/529: symlink d3/d4/d9e/lb0 0 2024-04-05T03:02:42.973 INFO:tasks.workunit.client.0.smithi153.stdout:7/136: symlink d2/da/d1f/l2f 0 2024-04-05T03:02:42.977 INFO:tasks.workunit.client.0.smithi153.stdout:8/158: mkdir d7/d13/d19/d31 0 2024-04-05T03:02:42.984 INFO:tasks.workunit.client.0.smithi153.stdout:1/158: mknod d5/dd/c34 0 2024-04-05T03:02:42.986 INFO:tasks.workunit.client.0.smithi153.stdout:2/148: rename d9/f21 to d9/da/d10/f33 0 2024-04-05T03:02:42.987 INFO:tasks.workunit.client.1.smithi203.stdout:5/530: dwrite d3/d1f/d59/d64/f4f [0,4194304] 0 2024-04-05T03:02:42.988 INFO:tasks.workunit.client.0.smithi153.stdout:4/169: truncate da/de/f1c 2177407 0 2024-04-05T03:02:42.991 INFO:tasks.workunit.client.1.smithi203.stdout:5/531: symlink d3/d4/dc/lb1 0 2024-04-05T03:02:42.994 INFO:tasks.workunit.client.1.smithi203.stdout:5/532: mknod d3/d4/dc/cb2 0 2024-04-05T03:02:42.995 INFO:tasks.workunit.client.0.smithi153.stdout:8/159: unlink d7/f10 0 2024-04-05T03:02:42.995 INFO:tasks.workunit.client.1.smithi203.stdout:5/533: chown d3/d1f/d59/d64/d12/c83 1 1 2024-04-05T03:02:42.996 INFO:tasks.workunit.client.1.smithi203.stdout:5/534: write d3/d1f/f3b [1130958,44292] 0 2024-04-05T03:02:42.997 INFO:tasks.workunit.client.0.smithi153.stdout:0/194: dwrite d9/df/d12/f1b [0,4194304] 0 2024-04-05T03:02:42.998 INFO:tasks.workunit.client.0.smithi153.stdout:0/195: chown d9/db/c2f 14 1 2024-04-05T03:02:42.999 INFO:tasks.workunit.client.0.smithi153.stdout:6/204: mkdir d4/dc/df/d16/d36/d4d 0 2024-04-05T03:02:43.000 INFO:tasks.workunit.client.1.smithi203.stdout:5/535: symlink d3/d1f/lb3 0 2024-04-05T03:02:43.001 INFO:tasks.workunit.client.1.smithi203.stdout:5/536: chown d3/d4/d9e 1 1 2024-04-05T03:02:43.001 INFO:tasks.workunit.client.0.smithi153.stdout:2/149: mknod d9/da/d10/c34 0 2024-04-05T03:02:43.005 INFO:tasks.workunit.client.0.smithi153.stdout:7/137: mknod d2/c30 0 2024-04-05T03:02:43.006 INFO:tasks.workunit.client.1.smithi203.stdout:5/537: link d3/d1f/d59/d64/f63 d3/d1f/d59/d60/fb4 0 2024-04-05T03:02:43.009 INFO:tasks.workunit.client.0.smithi153.stdout:7/138: dread d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:43.009 INFO:tasks.workunit.client.1.smithi203.stdout:5/538: mkdir d3/d4/dc/d67/db5 0 2024-04-05T03:02:43.010 INFO:tasks.workunit.client.0.smithi153.stdout:6/205: mkdir d4/dc/df/d16/d24/d4e 0 2024-04-05T03:02:43.010 INFO:tasks.workunit.client.0.smithi153.stdout:1/159: dread d5/dd/d21/f11 [0,4194304] 0 2024-04-05T03:02:43.010 INFO:tasks.workunit.client.1.smithi203.stdout:5/539: write d3/d5f/f75 [498718,32186] 0 2024-04-05T03:02:43.011 INFO:tasks.workunit.client.0.smithi153.stdout:2/150: truncate d9/da/f24 767003 0 2024-04-05T03:02:43.012 INFO:tasks.workunit.client.0.smithi153.stdout:2/151: chown d9/da/c1d 11706550 1 2024-04-05T03:02:43.012 INFO:tasks.workunit.client.0.smithi153.stdout:2/152: read d9/ff [5605221,33750] 0 2024-04-05T03:02:43.014 INFO:tasks.workunit.client.0.smithi153.stdout:5/160: rename d0/f8 to d0/f3c 0 2024-04-05T03:02:43.014 INFO:tasks.workunit.client.0.smithi153.stdout:2/153: write d9/da/fc [5145916,126116] 0 2024-04-05T03:02:43.015 INFO:tasks.workunit.client.1.smithi203.stdout:5/540: getdents d3/d1f/d59/d64 0 2024-04-05T03:02:43.017 INFO:tasks.workunit.client.0.smithi153.stdout:8/160: link d7/d13/f18 d7/d13/d19/d31/f32 0 2024-04-05T03:02:43.019 INFO:tasks.workunit.client.1.smithi203.stdout:5/541: creat d3/d4/dc/d67/dac/fb6 x:0 0 0 2024-04-05T03:02:43.020 INFO:tasks.workunit.client.0.smithi153.stdout:4/170: rmdir da/de/d2f 0 2024-04-05T03:02:43.023 INFO:tasks.workunit.client.0.smithi153.stdout:6/206: unlink d4/dc/f14 0 2024-04-05T03:02:43.025 INFO:tasks.workunit.client.1.smithi203.stdout:5/542: dwrite d3/d1f/d59/d60/f99 [0,4194304] 0 2024-04-05T03:02:43.030 INFO:tasks.workunit.client.1.smithi203.stdout:5/543: symlink d3/lb7 0 2024-04-05T03:02:43.030 INFO:tasks.workunit.client.0.smithi153.stdout:5/161: fdatasync d0/d2c/f2d 0 2024-04-05T03:02:43.031 INFO:tasks.workunit.client.0.smithi153.stdout:8/161: mkdir d7/d13/d19/d1b/d33 0 2024-04-05T03:02:43.031 INFO:tasks.workunit.client.0.smithi153.stdout:1/160: dwrite d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:43.032 INFO:tasks.workunit.client.1.smithi203.stdout:5/544: mknod d3/d5f/d92/cb8 0 2024-04-05T03:02:43.033 INFO:tasks.workunit.client.0.smithi153.stdout:0/196: dread d9/db/f11 [0,4194304] 0 2024-04-05T03:02:43.035 INFO:tasks.workunit.client.0.smithi153.stdout:4/171: read f4 [2478872,94807] 0 2024-04-05T03:02:43.035 INFO:tasks.workunit.client.0.smithi153.stdout:6/207: mknod d4/dc/df/d16/d44/c4f 0 2024-04-05T03:02:43.036 INFO:tasks.workunit.client.1.smithi203.stdout:5/545: mknod d3/d5f/cb9 0 2024-04-05T03:02:43.037 INFO:tasks.workunit.client.0.smithi153.stdout:8/162: fdatasync d7/fa 0 2024-04-05T03:02:43.041 INFO:tasks.workunit.client.0.smithi153.stdout:5/162: rename d0/f1c to d0/d2c/f3d 0 2024-04-05T03:02:43.042 INFO:tasks.workunit.client.0.smithi153.stdout:0/197: dread d9/df/f18 [0,4194304] 0 2024-04-05T03:02:43.044 INFO:tasks.workunit.client.1.smithi203.stdout:5/546: rename d3/d1f/l5a to d3/d4/dc/lba 0 2024-04-05T03:02:43.044 INFO:tasks.workunit.client.1.smithi203.stdout:5/547: chown d3/d4/l7d 35 1 2024-04-05T03:02:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:42 smithi203 ceph-mon[39911]: pgmap v6: 65 pgs: 65 active+clean; 892 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 23 MiB/s rd, 54 MiB/s wr, 96 op/s 2024-04-05T03:02:43.046 INFO:tasks.workunit.client.0.smithi153.stdout:6/208: symlink d4/dc/l50 0 2024-04-05T03:02:43.047 INFO:tasks.workunit.client.0.smithi153.stdout:8/163: symlink d7/d13/l34 0 2024-04-05T03:02:43.048 INFO:tasks.workunit.client.0.smithi153.stdout:1/161: symlink d5/d8/d20/d2f/l35 0 2024-04-05T03:02:43.048 INFO:tasks.workunit.client.1.smithi203.stdout:5/548: rename d3/d1f/d59/d64/d12/c4c to d3/d1f/d49/cbb 0 2024-04-05T03:02:43.049 INFO:tasks.workunit.client.0.smithi153.stdout:0/198: rmdir d9/df/d1e 39 2024-04-05T03:02:43.049 INFO:tasks.workunit.client.0.smithi153.stdout:1/162: truncate d5/dd/d1b/f2b 620293 0 2024-04-05T03:02:43.050 INFO:tasks.workunit.client.0.smithi153.stdout:1/163: chown d5/dd/c27 105206 1 2024-04-05T03:02:43.050 INFO:tasks.workunit.client.1.smithi203.stdout:5/549: readlink d3/d1f/d59/l86 0 2024-04-05T03:02:43.051 INFO:tasks.workunit.client.0.smithi153.stdout:6/209: creat d4/f51 x:0 0 0 2024-04-05T03:02:43.052 INFO:tasks.workunit.client.0.smithi153.stdout:6/210: stat d4/dc/df/d16/d17 0 2024-04-05T03:02:43.053 INFO:tasks.workunit.client.0.smithi153.stdout:8/164: rename d7/d13/d19/f20 to d7/d13/d19/d1b/f35 0 2024-04-05T03:02:43.054 INFO:tasks.workunit.client.0.smithi153.stdout:5/163: symlink d0/l3e 0 2024-04-05T03:02:43.054 INFO:tasks.workunit.client.0.smithi153.stdout:8/165: write d7/d13/f15 [274112,107676] 0 2024-04-05T03:02:43.055 INFO:tasks.workunit.client.0.smithi153.stdout:5/164: write d0/d9/f18 [1431349,124009] 0 2024-04-05T03:02:43.056 INFO:tasks.workunit.client.0.smithi153.stdout:6/211: mkdir d4/dc/df/d16/d36/d52 0 2024-04-05T03:02:43.057 INFO:tasks.workunit.client.1.smithi203.stdout:5/550: dwrite d3/d1f/f93 [4194304,4194304] 0 2024-04-05T03:02:43.058 INFO:tasks.workunit.client.1.smithi203.stdout:5/551: chown d3/d1f/d59/l86 5142842 1 2024-04-05T03:02:43.059 INFO:tasks.workunit.client.0.smithi153.stdout:4/172: getdents da 0 2024-04-05T03:02:43.059 INFO:tasks.workunit.client.1.smithi203.stdout:5/552: chown d3/d1f/d59/d64/d12/f46 60 1 2024-04-05T03:02:43.064 INFO:tasks.workunit.client.0.smithi153.stdout:5/165: mkdir d0/d2c/d3f 0 2024-04-05T03:02:43.067 INFO:tasks.workunit.client.0.smithi153.stdout:4/173: creat da/db/d20/f32 x:0 0 0 2024-04-05T03:02:43.067 INFO:tasks.workunit.client.0.smithi153.stdout:4/174: dread - da/db/d20/f21 zero size 2024-04-05T03:02:43.069 INFO:tasks.workunit.client.1.smithi203.stdout:2/690: sync 2024-04-05T03:02:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:42 smithi153 ceph-mon[29381]: pgmap v6: 65 pgs: 65 active+clean; 892 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 23 MiB/s rd, 54 MiB/s wr, 96 op/s 2024-04-05T03:02:43.073 INFO:tasks.workunit.client.0.smithi153.stdout:5/166: rename d0/de/c2a to d0/d2c/c40 0 2024-04-05T03:02:43.074 INFO:tasks.workunit.client.0.smithi153.stdout:4/175: readlink l6 0 2024-04-05T03:02:43.074 INFO:tasks.workunit.client.0.smithi153.stdout:8/166: link d7/d13/d19/l29 d7/d2a/l36 0 2024-04-05T03:02:43.075 INFO:tasks.workunit.client.0.smithi153.stdout:0/199: link d9/df/d1e/l26 d9/df/d1e/l3a 0 2024-04-05T03:02:43.076 INFO:tasks.workunit.client.0.smithi153.stdout:5/167: chown d0/de/f10 362286 1 2024-04-05T03:02:43.078 INFO:tasks.workunit.client.0.smithi153.stdout:8/167: creat d7/d2a/f37 x:0 0 0 2024-04-05T03:02:43.079 INFO:tasks.workunit.client.0.smithi153.stdout:8/168: chown d7/d13/d19/d1b/d21/c25 1198 1 2024-04-05T03:02:43.080 INFO:tasks.workunit.client.0.smithi153.stdout:8/169: write f4 [4702258,75875] 0 2024-04-05T03:02:43.080 INFO:tasks.workunit.client.0.smithi153.stdout:0/200: unlink d9/df/l1f 0 2024-04-05T03:02:43.080 INFO:tasks.workunit.client.0.smithi153.stdout:6/212: rmdir d4 39 2024-04-05T03:02:43.080 INFO:tasks.workunit.client.0.smithi153.stdout:0/201: truncate d9/f1a 1022591 0 2024-04-05T03:02:43.081 INFO:tasks.workunit.client.0.smithi153.stdout:4/176: link da/db/d20/f21 da/d2a/f33 0 2024-04-05T03:02:43.085 INFO:tasks.workunit.client.0.smithi153.stdout:0/202: creat d9/df/d1d/f3b x:0 0 0 2024-04-05T03:02:43.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/177: dread da/db/d20/f29 [0,4194304] 0 2024-04-05T03:02:43.087 INFO:tasks.workunit.client.0.smithi153.stdout:8/170: symlink d7/d13/d19/l38 0 2024-04-05T03:02:43.087 INFO:tasks.workunit.client.0.smithi153.stdout:4/178: write da/f26 [2057109,68711] 0 2024-04-05T03:02:43.087 INFO:tasks.workunit.client.0.smithi153.stdout:8/171: readlink d7/d13/d19/l38 0 2024-04-05T03:02:43.089 INFO:tasks.workunit.client.0.smithi153.stdout:6/213: getdents d4/d38/d4c 0 2024-04-05T03:02:43.093 INFO:tasks.workunit.client.0.smithi153.stdout:4/179: mkdir da/db/d20/d34 0 2024-04-05T03:02:43.094 INFO:tasks.workunit.client.0.smithi153.stdout:4/180: chown da/de/l10 27906 1 2024-04-05T03:02:43.094 INFO:tasks.workunit.client.0.smithi153.stdout:4/181: fsync da/db/f1a 0 2024-04-05T03:02:43.095 INFO:tasks.workunit.client.0.smithi153.stdout:4/182: chown da/de/f19 72694 1 2024-04-05T03:02:43.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/172: link f4 d7/d13/d19/d1b/d21/f39 0 2024-04-05T03:02:43.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/173: readlink d7/d13/l34 0 2024-04-05T03:02:43.098 INFO:tasks.workunit.client.0.smithi153.stdout:0/203: rmdir d9/db/d36 0 2024-04-05T03:02:43.101 INFO:tasks.workunit.client.0.smithi153.stdout:8/174: dread f3 [0,4194304] 0 2024-04-05T03:02:43.102 INFO:tasks.workunit.client.0.smithi153.stdout:4/183: rmdir da/db 39 2024-04-05T03:02:43.103 INFO:tasks.workunit.client.0.smithi153.stdout:0/204: symlink d9/l3c 0 2024-04-05T03:02:43.103 INFO:tasks.workunit.client.0.smithi153.stdout:0/205: stat d9/df/f17 0 2024-04-05T03:02:43.107 INFO:tasks.workunit.client.0.smithi153.stdout:8/175: creat d7/d13/d19/d1b/d33/f3a x:0 0 0 2024-04-05T03:02:43.109 INFO:tasks.workunit.client.0.smithi153.stdout:0/206: link d9/f30 d9/df/d12/f3d 0 2024-04-05T03:02:43.110 INFO:tasks.workunit.client.0.smithi153.stdout:8/176: truncate d7/d13/d19/d1b/d21/f39 3064158 0 2024-04-05T03:02:43.111 INFO:tasks.workunit.client.0.smithi153.stdout:0/207: mknod d9/df/d1e/c3e 0 2024-04-05T03:02:43.112 INFO:tasks.workunit.client.0.smithi153.stdout:8/177: mknod d7/d13/c3b 0 2024-04-05T03:02:43.112 INFO:tasks.workunit.client.0.smithi153.stdout:0/208: readlink d9/df/l20 0 2024-04-05T03:02:43.113 INFO:tasks.workunit.client.0.smithi153.stdout:0/209: readlink d9/df/d1e/l2c 0 2024-04-05T03:02:43.114 INFO:tasks.workunit.client.0.smithi153.stdout:0/210: chown d9/db/c2f 95 1 2024-04-05T03:02:43.114 INFO:tasks.workunit.client.0.smithi153.stdout:4/184: dwrite f4 [0,4194304] 0 2024-04-05T03:02:43.121 INFO:tasks.workunit.client.0.smithi153.stdout:0/211: creat d9/d14/f3f x:0 0 0 2024-04-05T03:02:43.123 INFO:tasks.workunit.client.0.smithi153.stdout:0/212: creat d9/d14/f40 x:0 0 0 2024-04-05T03:02:43.124 INFO:tasks.workunit.client.0.smithi153.stdout:0/213: dread - d9/df/d1d/f3b zero size 2024-04-05T03:02:43.127 INFO:tasks.workunit.client.0.smithi153.stdout:0/214: mknod d9/df/d1e/c41 0 2024-04-05T03:02:43.127 INFO:tasks.workunit.client.0.smithi153.stdout:8/178: dwrite d7/d13/d19/d31/f32 [0,4194304] 0 2024-04-05T03:02:43.128 INFO:tasks.workunit.client.0.smithi153.stdout:0/215: dread - d9/df/d1e/f35 zero size 2024-04-05T03:02:43.129 INFO:tasks.workunit.client.0.smithi153.stdout:8/179: write d7/d13/f18 [1455256,41664] 0 2024-04-05T03:02:43.131 INFO:tasks.workunit.client.0.smithi153.stdout:0/216: mkdir d9/db/d42 0 2024-04-05T03:02:43.139 INFO:tasks.workunit.client.0.smithi153.stdout:3/135: sync 2024-04-05T03:02:43.140 INFO:tasks.workunit.client.0.smithi153.stdout:0/217: dwrite d9/df/d12/f2b [0,4194304] 0 2024-04-05T03:02:43.142 INFO:tasks.workunit.client.1.smithi203.stdout:2/691: mknod d5/d10/ced 0 2024-04-05T03:02:43.145 INFO:tasks.workunit.client.0.smithi153.stdout:0/218: mkdir d9/db/d42/d43 0 2024-04-05T03:02:43.146 INFO:tasks.workunit.client.0.smithi153.stdout:3/136: link d1d/d26/c2a d1d/c31 0 2024-04-05T03:02:43.147 INFO:tasks.workunit.client.1.smithi203.stdout:5/553: dwrite d3/d1f/d59/d64/f1c [4194304,4194304] 0 2024-04-05T03:02:43.147 INFO:tasks.workunit.client.1.smithi203.stdout:2/692: dread d5/d10/d44/fd7 [0,4194304] 0 2024-04-05T03:02:43.150 INFO:tasks.workunit.client.1.smithi203.stdout:2/693: fsync d5/d10/d35/d36/db6/f37 0 2024-04-05T03:02:43.151 INFO:tasks.workunit.client.0.smithi153.stdout:4/185: dread da/db/f23 [0,4194304] 0 2024-04-05T03:02:43.151 INFO:tasks.workunit.client.1.smithi203.stdout:5/554: write d3/d4/dc/d67/f7f [8914102,122020] 0 2024-04-05T03:02:43.156 INFO:tasks.workunit.client.0.smithi153.stdout:3/137: dwrite f3 [0,4194304] 0 2024-04-05T03:02:43.156 INFO:tasks.workunit.client.0.smithi153.stdout:0/219: dwrite d9/df/d12/f2b [0,4194304] 0 2024-04-05T03:02:43.159 INFO:tasks.workunit.client.0.smithi153.stdout:4/186: rmdir da/db/d20/d34 0 2024-04-05T03:02:43.162 INFO:tasks.workunit.client.0.smithi153.stdout:3/138: mkdir d1d/d32 0 2024-04-05T03:02:43.163 INFO:tasks.workunit.client.0.smithi153.stdout:0/220: write d9/df/d1e/f25 [4406398,44008] 0 2024-04-05T03:02:43.164 INFO:tasks.workunit.client.0.smithi153.stdout:4/187: mkdir da/db/d20/d35 0 2024-04-05T03:02:43.165 INFO:tasks.workunit.client.0.smithi153.stdout:3/139: mknod d1d/d26/c33 0 2024-04-05T03:02:43.166 INFO:tasks.workunit.client.1.smithi203.stdout:5/555: write d3/d1f/d59/d60/f76 [573471,82025] 0 2024-04-05T03:02:43.167 INFO:tasks.workunit.client.0.smithi153.stdout:3/140: write d1d/f28 [425670,40365] 0 2024-04-05T03:02:43.171 INFO:tasks.workunit.client.0.smithi153.stdout:0/221: dread d9/db/f11 [0,4194304] 0 2024-04-05T03:02:43.172 INFO:tasks.workunit.client.0.smithi153.stdout:3/141: creat d1d/d27/f34 x:0 0 0 2024-04-05T03:02:43.172 INFO:tasks.workunit.client.0.smithi153.stdout:4/188: symlink da/db/d20/d35/l36 0 2024-04-05T03:02:43.173 INFO:tasks.workunit.client.0.smithi153.stdout:3/142: dread - d1d/d27/d2b/f2f zero size 2024-04-05T03:02:43.174 INFO:tasks.workunit.client.0.smithi153.stdout:3/143: dread - d1d/f1e zero size 2024-04-05T03:02:43.174 INFO:tasks.workunit.client.0.smithi153.stdout:0/222: creat d9/db/f44 x:0 0 0 2024-04-05T03:02:43.175 INFO:tasks.workunit.client.0.smithi153.stdout:0/223: fdatasync d9/df/d1d/f3b 0 2024-04-05T03:02:43.175 INFO:tasks.workunit.client.0.smithi153.stdout:4/189: chown da/f1d 7 1 2024-04-05T03:02:43.179 INFO:tasks.workunit.client.0.smithi153.stdout:3/144: dread f3 [4194304,4194304] 0 2024-04-05T03:02:43.179 INFO:tasks.workunit.client.0.smithi153.stdout:0/224: mkdir d9/df/d1e/d45 0 2024-04-05T03:02:43.180 INFO:tasks.workunit.client.0.smithi153.stdout:3/145: dread - d1d/f25 zero size 2024-04-05T03:02:43.180 INFO:tasks.workunit.client.0.smithi153.stdout:4/190: dread - da/de/f11 zero size 2024-04-05T03:02:43.180 INFO:tasks.workunit.client.0.smithi153.stdout:0/225: dread - d9/db/f2d zero size 2024-04-05T03:02:43.181 INFO:tasks.workunit.client.0.smithi153.stdout:3/146: chown l16 175 1 2024-04-05T03:02:43.184 INFO:tasks.workunit.client.0.smithi153.stdout:4/191: creat da/db/d20/d30/f37 x:0 0 0 2024-04-05T03:02:43.186 INFO:tasks.workunit.client.0.smithi153.stdout:3/147: rename d1d/f2c to d1d/d27/d2b/f35 0 2024-04-05T03:02:43.187 INFO:tasks.workunit.client.0.smithi153.stdout:3/148: chown d1d/d27/d2b/f2f 9147509 1 2024-04-05T03:02:43.189 INFO:tasks.workunit.client.0.smithi153.stdout:4/192: dwrite f7 [0,4194304] 0 2024-04-05T03:02:43.189 INFO:tasks.workunit.client.0.smithi153.stdout:3/149: creat d1d/f36 x:0 0 0 2024-04-05T03:02:43.200 INFO:tasks.workunit.client.0.smithi153.stdout:4/193: dread da/de/f1c [0,4194304] 0 2024-04-05T03:02:43.202 INFO:tasks.workunit.client.0.smithi153.stdout:4/194: fsync da/fc 0 2024-04-05T03:02:43.208 INFO:tasks.workunit.client.0.smithi153.stdout:0/226: dwrite d9/df/d12/f3d [0,4194304] 0 2024-04-05T03:02:43.208 INFO:tasks.workunit.client.0.smithi153.stdout:0/227: fdatasync d9/d14/f40 0 2024-04-05T03:02:43.210 INFO:tasks.workunit.client.0.smithi153.stdout:0/228: truncate d9/db/f2d 728207 0 2024-04-05T03:02:43.210 INFO:tasks.workunit.client.0.smithi153.stdout:4/195: fsync da/db/d20/d30/f37 0 2024-04-05T03:02:43.213 INFO:tasks.workunit.client.0.smithi153.stdout:0/229: rmdir d9/db/d42 39 2024-04-05T03:02:43.214 INFO:tasks.workunit.client.0.smithi153.stdout:0/230: write d9/df/d12/f1b [3982777,8681] 0 2024-04-05T03:02:43.214 INFO:tasks.workunit.client.0.smithi153.stdout:4/196: mkdir da/db/d20/d35/d38 0 2024-04-05T03:02:43.218 INFO:tasks.workunit.client.0.smithi153.stdout:4/197: mknod da/c39 0 2024-04-05T03:02:43.218 INFO:tasks.workunit.client.0.smithi153.stdout:4/198: fdatasync f4 0 2024-04-05T03:02:43.218 INFO:tasks.workunit.client.0.smithi153.stdout:0/231: dread f7 [0,4194304] 0 2024-04-05T03:02:43.219 INFO:tasks.workunit.client.0.smithi153.stdout:0/232: readlink d9/db/l31 0 2024-04-05T03:02:43.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/199: symlink da/db/d20/d35/l3a 0 2024-04-05T03:02:43.221 INFO:tasks.workunit.client.0.smithi153.stdout:0/233: stat d9/df/d1e/l26 0 2024-04-05T03:02:43.222 INFO:tasks.workunit.client.0.smithi153.stdout:4/200: mkdir da/db/d20/d30/d3b 0 2024-04-05T03:02:43.224 INFO:tasks.workunit.client.0.smithi153.stdout:0/234: creat d9/df/f46 x:0 0 0 2024-04-05T03:02:43.225 INFO:tasks.workunit.client.0.smithi153.stdout:4/201: mkdir da/db/d20/d3c 0 2024-04-05T03:02:43.227 INFO:tasks.workunit.client.0.smithi153.stdout:4/202: creat da/db/d20/d30/d3b/f3d x:0 0 0 2024-04-05T03:02:43.230 INFO:tasks.workunit.client.0.smithi153.stdout:0/235: dwrite d9/df/d12/f2b [0,4194304] 0 2024-04-05T03:02:43.233 INFO:tasks.workunit.client.0.smithi153.stdout:0/236: unlink d9/df/l27 0 2024-04-05T03:02:43.233 INFO:tasks.workunit.client.0.smithi153.stdout:0/237: stat d9/df/l20 0 2024-04-05T03:02:43.234 INFO:tasks.workunit.client.0.smithi153.stdout:4/203: dwrite da/f17 [0,4194304] 0 2024-04-05T03:02:43.234 INFO:tasks.workunit.client.0.smithi153.stdout:0/238: chown d9/df/d1e 970523 1 2024-04-05T03:02:43.239 INFO:tasks.workunit.client.0.smithi153.stdout:4/204: mkdir da/db/d20/d35/d38/d3e 0 2024-04-05T03:02:43.269 INFO:tasks.workunit.client.0.smithi153.stdout:0/239: truncate d9/df/f15 671915 0 2024-04-05T03:02:43.269 INFO:tasks.workunit.client.0.smithi153.stdout:4/205: truncate da/db/d20/f29 479432 0 2024-04-05T03:02:43.271 INFO:tasks.workunit.client.0.smithi153.stdout:4/206: truncate da/db/d20/d30/d3b/f3d 206154 0 2024-04-05T03:02:43.272 INFO:tasks.workunit.client.0.smithi153.stdout:9/120: sync 2024-04-05T03:02:43.273 INFO:tasks.workunit.client.0.smithi153.stdout:0/240: rmdir d9/df 39 2024-04-05T03:02:43.277 INFO:tasks.workunit.client.0.smithi153.stdout:9/121: creat d1/d10/d16/d22/d2c/f31 x:0 0 0 2024-04-05T03:02:43.277 INFO:tasks.workunit.client.0.smithi153.stdout:0/241: readlink d9/df/l2e 0 2024-04-05T03:02:43.278 INFO:tasks.workunit.client.0.smithi153.stdout:4/207: mknod da/db/d20/d3c/c3f 0 2024-04-05T03:02:43.281 INFO:tasks.workunit.client.0.smithi153.stdout:4/208: creat da/de/d28/d2d/f40 x:0 0 0 2024-04-05T03:02:43.285 INFO:tasks.workunit.client.0.smithi153.stdout:4/209: write da/db/d20/f21 [671545,717] 0 2024-04-05T03:02:43.286 INFO:tasks.workunit.client.0.smithi153.stdout:4/210: stat da/de/f19 0 2024-04-05T03:02:43.287 INFO:tasks.workunit.client.0.smithi153.stdout:4/211: fsync da/d2a/f33 0 2024-04-05T03:02:43.289 INFO:tasks.workunit.client.0.smithi153.stdout:0/242: dwrite d9/d14/f19 [0,4194304] 0 2024-04-05T03:02:43.289 INFO:tasks.workunit.client.0.smithi153.stdout:4/212: dread - da/de/f19 zero size 2024-04-05T03:02:43.293 INFO:tasks.workunit.client.0.smithi153.stdout:0/243: unlink d9/df/l29 0 2024-04-05T03:02:43.295 INFO:tasks.workunit.client.0.smithi153.stdout:0/244: mkdir d9/df/d1d/d47 0 2024-04-05T03:02:43.299 INFO:tasks.workunit.client.1.smithi203.stdout:9/426: sync 2024-04-05T03:02:43.301 INFO:tasks.workunit.client.1.smithi203.stdout:9/427: truncate d13/d1b/d26/d31/f41 140427 0 2024-04-05T03:02:43.306 INFO:tasks.workunit.client.0.smithi153.stdout:0/245: truncate d9/df/d1e/f32 1060552 0 2024-04-05T03:02:43.307 INFO:tasks.workunit.client.0.smithi153.stdout:0/246: chown d9/ce 15944836 1 2024-04-05T03:02:43.309 INFO:tasks.workunit.client.0.smithi153.stdout:0/247: fsync d9/df/f18 0 2024-04-05T03:02:43.311 INFO:tasks.workunit.client.0.smithi153.stdout:0/248: mknod d9/df/d12/c48 0 2024-04-05T03:02:43.312 INFO:tasks.workunit.client.0.smithi153.stdout:0/249: fsync d9/db/fd 0 2024-04-05T03:02:43.316 INFO:tasks.workunit.client.0.smithi153.stdout:0/250: creat d9/db/f49 x:0 0 0 2024-04-05T03:02:43.317 INFO:tasks.workunit.client.0.smithi153.stdout:0/251: write d9/db/f49 [496977,1040] 0 2024-04-05T03:02:43.318 INFO:tasks.workunit.client.0.smithi153.stdout:0/252: write d9/d14/f3f [917902,79035] 0 2024-04-05T03:02:43.322 INFO:tasks.workunit.client.1.smithi203.stdout:9/428: dread d13/f24 [0,4194304] 0 2024-04-05T03:02:43.325 INFO:tasks.workunit.client.0.smithi153.stdout:0/253: dwrite d9/db/f37 [0,4194304] 0 2024-04-05T03:02:43.327 INFO:tasks.workunit.client.1.smithi203.stdout:9/429: rename f7 to d13/d1b/d26/d31/d38/d59/f85 0 2024-04-05T03:02:43.329 INFO:tasks.workunit.client.0.smithi153.stdout:0/254: creat d9/db/d42/d43/f4a x:0 0 0 2024-04-05T03:02:43.332 INFO:tasks.workunit.client.1.smithi203.stdout:9/430: getdents d13 0 2024-04-05T03:02:43.334 INFO:tasks.workunit.client.0.smithi153.stdout:0/255: creat d9/db/f4b x:0 0 0 2024-04-05T03:02:43.337 INFO:tasks.workunit.client.1.smithi203.stdout:9/431: link d13/d1b/d33/l43 d13/d1b/d26/d31/l86 0 2024-04-05T03:02:43.338 INFO:tasks.workunit.client.1.smithi203.stdout:9/432: write d13/d1b/d26/d51/f53 [398645,65619] 0 2024-04-05T03:02:43.339 INFO:tasks.workunit.client.1.smithi203.stdout:9/433: read d13/d1b/d33/f65 [21584,79331] 0 2024-04-05T03:02:43.341 INFO:tasks.workunit.client.0.smithi153.stdout:0/256: dwrite d9/df/d1d/f24 [0,4194304] 0 2024-04-05T03:02:43.341 INFO:tasks.workunit.client.1.smithi203.stdout:9/434: creat d13/d1b/d6b/f87 x:0 0 0 2024-04-05T03:02:43.342 INFO:tasks.workunit.client.1.smithi203.stdout:9/435: write d13/d1b/d26/d51/f52 [1971228,39465] 0 2024-04-05T03:02:43.343 INFO:tasks.workunit.client.0.smithi153.stdout:0/257: write d9/d14/f3f [257573,22031] 0 2024-04-05T03:02:43.346 INFO:tasks.workunit.client.0.smithi153.stdout:0/258: dread d9/db/f49 [0,4194304] 0 2024-04-05T03:02:43.347 INFO:tasks.workunit.client.0.smithi153.stdout:0/259: chown d9/db/f4b 0 1 2024-04-05T03:02:43.348 INFO:tasks.workunit.client.0.smithi153.stdout:0/260: write f3 [4102256,38389] 0 2024-04-05T03:02:43.349 INFO:tasks.workunit.client.0.smithi153.stdout:0/261: chown d9/df/d12/c48 350972 1 2024-04-05T03:02:43.352 INFO:tasks.workunit.client.0.smithi153.stdout:0/262: symlink d9/df/d1e/l4c 0 2024-04-05T03:02:43.353 INFO:tasks.workunit.client.0.smithi153.stdout:0/263: chown d9/df/d1e/c41 8002 1 2024-04-05T03:02:43.353 INFO:tasks.workunit.client.1.smithi203.stdout:9/436: symlink d13/d1b/d26/d31/l88 0 2024-04-05T03:02:43.354 INFO:tasks.workunit.client.0.smithi153.stdout:0/264: write d9/db/f37 [3746402,39199] 0 2024-04-05T03:02:43.355 INFO:tasks.workunit.client.0.smithi153.stdout:0/265: write d9/df/d12/f2b [165551,109627] 0 2024-04-05T03:02:43.358 INFO:tasks.workunit.client.1.smithi203.stdout:9/437: dwrite fd [0,4194304] 0 2024-04-05T03:02:43.359 INFO:tasks.workunit.client.1.smithi203.stdout:9/438: write d13/d1b/f21 [6068779,88441] 0 2024-04-05T03:02:43.362 INFO:tasks.workunit.client.1.smithi203.stdout:9/439: chown d13/d1b/d26/d31/d38/c40 2283991 1 2024-04-05T03:02:43.364 INFO:tasks.workunit.client.1.smithi203.stdout:9/440: creat d13/d5c/f89 x:0 0 0 2024-04-05T03:02:43.366 INFO:tasks.workunit.client.1.smithi203.stdout:9/441: creat d13/d1b/d26/d31/d38/d59/d7b/f8a x:0 0 0 2024-04-05T03:02:43.368 INFO:tasks.workunit.client.1.smithi203.stdout:9/442: link d13/d1b/d33/d42/l80 d13/d1b/d33/d63/l8b 0 2024-04-05T03:02:43.370 INFO:tasks.workunit.client.1.smithi203.stdout:9/443: mknod d13/d1b/d6b/d7a/c8c 0 2024-04-05T03:02:43.392 INFO:tasks.workunit.client.1.smithi203.stdout:9/444: truncate d13/d1b/f47 3674295 0 2024-04-05T03:02:43.398 INFO:tasks.workunit.client.0.smithi153.stdout:0/266: dread d9/df/f39 [0,4194304] 0 2024-04-05T03:02:43.406 INFO:tasks.workunit.client.1.smithi203.stdout:9/445: write d13/d1b/d33/d42/f58 [235799,15071] 0 2024-04-05T03:02:43.412 INFO:tasks.workunit.client.1.smithi203.stdout:9/446: dwrite d13/d1b/d26/d31/d38/f68 [0,4194304] 0 2024-04-05T03:02:43.414 INFO:tasks.workunit.client.1.smithi203.stdout:9/447: mknod d13/d1b/d33/d63/c8d 0 2024-04-05T03:02:43.418 INFO:tasks.workunit.client.1.smithi203.stdout:9/448: creat d13/d1b/d33/d42/d67/d82/f8e x:0 0 0 2024-04-05T03:02:43.422 INFO:tasks.workunit.client.1.smithi203.stdout:9/449: read f10 [1258562,18991] 0 2024-04-05T03:02:43.426 INFO:tasks.workunit.client.1.smithi203.stdout:9/450: getdents d13/d1b/d26/d51 0 2024-04-05T03:02:43.434 INFO:tasks.workunit.client.1.smithi203.stdout:9/451: truncate d13/d1b/d33/d42/f49 1057754 0 2024-04-05T03:02:43.437 INFO:tasks.workunit.client.1.smithi203.stdout:9/452: dread fd [0,4194304] 0 2024-04-05T03:02:43.437 INFO:tasks.workunit.client.1.smithi203.stdout:9/453: dread - d13/d1b/d26/d31/d38/d59/d7b/f8a zero size 2024-04-05T03:02:43.439 INFO:tasks.workunit.client.1.smithi203.stdout:9/454: write d13/d1b/d26/d31/f3e [1088400,78554] 0 2024-04-05T03:02:43.445 INFO:tasks.workunit.client.1.smithi203.stdout:9/455: dwrite d13/d1b/d26/d31/d38/f68 [0,4194304] 0 2024-04-05T03:02:43.450 INFO:tasks.workunit.client.1.smithi203.stdout:9/456: rename d13/d1b/d26/d31/l86 to d13/d1b/d26/d51/d81/l8f 0 2024-04-05T03:02:43.455 INFO:tasks.workunit.client.1.smithi203.stdout:7/647: sync 2024-04-05T03:02:43.456 INFO:tasks.workunit.client.1.smithi203.stdout:7/648: chown d8/d32/d16/d1a/d21/c83 2 1 2024-04-05T03:02:43.460 INFO:tasks.workunit.client.1.smithi203.stdout:9/457: write d13/d1b/f47 [3736578,119349] 0 2024-04-05T03:02:43.465 INFO:tasks.workunit.client.1.smithi203.stdout:9/458: dwrite d13/d1b/d26/d31/d38/d59/f85 [0,4194304] 0 2024-04-05T03:02:43.466 INFO:tasks.workunit.client.1.smithi203.stdout:9/459: stat d13/l2c 0 2024-04-05T03:02:43.468 INFO:tasks.workunit.client.1.smithi203.stdout:9/460: rename fd to d13/d1b/d26/f90 0 2024-04-05T03:02:43.476 INFO:tasks.workunit.client.1.smithi203.stdout:9/461: truncate f2 52510 0 2024-04-05T03:02:43.477 INFO:tasks.workunit.client.1.smithi203.stdout:9/462: write d13/f1d [5070733,99058] 0 2024-04-05T03:02:43.507 INFO:tasks.workunit.client.1.smithi203.stdout:9/463: chown d13/c73 60479 1 2024-04-05T03:02:43.513 INFO:tasks.workunit.client.1.smithi203.stdout:9/464: dwrite d13/d1b/d26/d31/d38/f5e [0,4194304] 0 2024-04-05T03:02:43.515 INFO:tasks.workunit.client.1.smithi203.stdout:9/465: rmdir d13/d1b/d26/d31 39 2024-04-05T03:02:43.516 INFO:tasks.workunit.client.1.smithi203.stdout:9/466: dread - d13/d1b/d33/f69 zero size 2024-04-05T03:02:43.519 INFO:tasks.workunit.client.1.smithi203.stdout:9/467: read d13/d1b/d33/d42/f49 [612508,89520] 0 2024-04-05T03:02:43.520 INFO:tasks.workunit.client.1.smithi203.stdout:9/468: fsync d13/d1b/d26/f6f 0 2024-04-05T03:02:43.525 INFO:tasks.workunit.client.1.smithi203.stdout:9/469: getdents d13/d1b/d26/d31 0 2024-04-05T03:02:43.526 INFO:tasks.workunit.client.0.smithi153.stdout:2/154: sync 2024-04-05T03:02:43.526 INFO:tasks.workunit.client.0.smithi153.stdout:7/139: sync 2024-04-05T03:02:43.527 INFO:tasks.workunit.client.0.smithi153.stdout:2/155: readlink d9/da/l23 0 2024-04-05T03:02:43.527 INFO:tasks.workunit.client.1.smithi203.stdout:9/470: stat d13/d1b/d26/d31/d38/d59/d7b/f7f 0 2024-04-05T03:02:43.527 INFO:tasks.workunit.client.1.smithi203.stdout:9/471: chown d13/d1b/d26/l28 190 1 2024-04-05T03:02:43.528 INFO:tasks.workunit.client.0.smithi153.stdout:2/156: stat d9/l2b 0 2024-04-05T03:02:43.529 INFO:tasks.workunit.client.1.smithi203.stdout:9/472: creat d13/d1b/d26/d51/f91 x:0 0 0 2024-04-05T03:02:43.530 INFO:tasks.workunit.client.1.smithi203.stdout:9/473: stat d13/d1b/d33/d63 0 2024-04-05T03:02:43.530 INFO:tasks.workunit.client.0.smithi153.stdout:7/140: mkdir d2/da/d11/d12/d31 0 2024-04-05T03:02:43.531 INFO:tasks.workunit.client.0.smithi153.stdout:7/141: write d2/f1d [4345100,26438] 0 2024-04-05T03:02:43.531 INFO:tasks.workunit.client.1.smithi203.stdout:9/474: dread d13/d1b/d26/d31/f60 [0,4194304] 0 2024-04-05T03:02:43.532 INFO:tasks.workunit.client.0.smithi153.stdout:7/142: truncate d2/da/d11/f24 4571341 0 2024-04-05T03:02:43.537 INFO:tasks.workunit.client.0.smithi153.stdout:7/143: creat d2/f32 x:0 0 0 2024-04-05T03:02:43.539 INFO:tasks.workunit.client.0.smithi153.stdout:7/144: creat d2/da/d11/d12/f33 x:0 0 0 2024-04-05T03:02:43.540 INFO:tasks.workunit.client.0.smithi153.stdout:7/145: chown d2/da/d11/d12 1049149040 1 2024-04-05T03:02:43.541 INFO:tasks.workunit.client.0.smithi153.stdout:2/157: dwrite d9/da/fe [0,4194304] 0 2024-04-05T03:02:43.542 INFO:tasks.workunit.client.0.smithi153.stdout:2/158: write d9/da/d19/f22 [448282,41202] 0 2024-04-05T03:02:43.543 INFO:tasks.workunit.client.1.smithi203.stdout:9/475: dread d13/d1b/d26/d31/d38/d59/f7e [0,4194304] 0 2024-04-05T03:02:43.543 INFO:tasks.workunit.client.0.smithi153.stdout:2/159: rename d9 to d9/da/d19/d35 22 2024-04-05T03:02:43.545 INFO:tasks.workunit.client.0.smithi153.stdout:2/160: write d9/da/f20 [45761,35171] 0 2024-04-05T03:02:43.546 INFO:tasks.workunit.client.0.smithi153.stdout:2/161: write d9/f11 [4063022,45451] 0 2024-04-05T03:02:43.547 INFO:tasks.workunit.client.1.smithi203.stdout:9/476: rmdir d13/d1b/d26/d31/d38/d59/d5f 39 2024-04-05T03:02:43.548 INFO:tasks.workunit.client.0.smithi153.stdout:2/162: truncate d9/da/f20 978457 0 2024-04-05T03:02:43.551 INFO:tasks.workunit.client.0.smithi153.stdout:2/163: mkdir d9/da/d36 0 2024-04-05T03:02:43.551 INFO:tasks.workunit.client.1.smithi203.stdout:9/477: truncate d13/d1b/d26/d31/f41 713049 0 2024-04-05T03:02:43.552 INFO:tasks.workunit.client.0.smithi153.stdout:2/164: chown d9/da/d19/f2d 0 1 2024-04-05T03:02:43.553 INFO:tasks.workunit.client.1.smithi203.stdout:9/478: write d13/f20 [2838399,53194] 0 2024-04-05T03:02:43.554 INFO:tasks.workunit.client.0.smithi153.stdout:7/146: dwrite d2/da/d1f/f10 [0,4194304] 0 2024-04-05T03:02:43.555 INFO:tasks.workunit.client.0.smithi153.stdout:2/165: write d9/da/d19/f1c [1097445,6000] 0 2024-04-05T03:02:43.558 INFO:tasks.workunit.client.0.smithi153.stdout:2/166: rmdir d9/da/d10 39 2024-04-05T03:02:43.559 INFO:tasks.workunit.client.0.smithi153.stdout:2/167: readlink d9/da/d19/d2e/l31 0 2024-04-05T03:02:43.559 INFO:tasks.workunit.client.0.smithi153.stdout:7/147: mknod d2/c34 0 2024-04-05T03:02:43.560 INFO:tasks.workunit.client.1.smithi203.stdout:9/479: fsync d13/f20 0 2024-04-05T03:02:43.560 INFO:tasks.workunit.client.1.smithi203.stdout:4/358: sync 2024-04-05T03:02:43.563 INFO:tasks.workunit.client.0.smithi153.stdout:7/148: mknod d2/da/d11/c35 0 2024-04-05T03:02:43.564 INFO:tasks.workunit.client.0.smithi153.stdout:7/149: read d2/da/d11/f2e [556362,63380] 0 2024-04-05T03:02:43.564 INFO:tasks.workunit.client.0.smithi153.stdout:2/168: getdents d9 0 2024-04-05T03:02:43.565 INFO:tasks.workunit.client.1.smithi203.stdout:9/480: rename d13/d1b/d33/c3a to d13/d1b/d33/d42/d67/d82/c92 0 2024-04-05T03:02:43.565 INFO:tasks.workunit.client.0.smithi153.stdout:7/150: stat d2/da/d11/d12/d31 0 2024-04-05T03:02:43.566 INFO:tasks.workunit.client.1.smithi203.stdout:4/359: mkdir da/d71 0 2024-04-05T03:02:43.566 INFO:tasks.workunit.client.0.smithi153.stdout:2/169: write d9/da/f20 [15825,125860] 0 2024-04-05T03:02:43.568 INFO:tasks.workunit.client.1.smithi203.stdout:4/360: write da/d29/d6a/f6b [572117,7163] 0 2024-04-05T03:02:43.568 INFO:tasks.workunit.client.1.smithi203.stdout:4/361: chown da/f1b 52630 1 2024-04-05T03:02:43.569 INFO:tasks.workunit.client.1.smithi203.stdout:4/362: write f9 [4433197,7890] 0 2024-04-05T03:02:43.570 INFO:tasks.workunit.client.0.smithi153.stdout:1/164: sync 2024-04-05T03:02:43.572 INFO:tasks.workunit.client.0.smithi153.stdout:7/151: mknod d2/da/c36 0 2024-04-05T03:02:43.572 INFO:tasks.workunit.client.0.smithi153.stdout:5/168: sync 2024-04-05T03:02:43.573 INFO:tasks.workunit.client.0.smithi153.stdout:7/152: chown d2/da/d11/d12/f22 6799907 1 2024-04-05T03:02:43.576 INFO:tasks.workunit.client.1.smithi203.stdout:4/363: rename da/l36 to da/d29/l72 0 2024-04-05T03:02:43.577 INFO:tasks.workunit.client.0.smithi153.stdout:7/153: read d2/f6 [5604970,111891] 0 2024-04-05T03:02:43.578 INFO:tasks.workunit.client.0.smithi153.stdout:5/169: link d0/d9/f18 d0/d9/d22/f41 0 2024-04-05T03:02:43.579 INFO:tasks.workunit.client.1.smithi203.stdout:4/364: unlink da/f1e 0 2024-04-05T03:02:43.579 INFO:tasks.workunit.client.0.smithi153.stdout:7/154: truncate d2/da/d1f/f20 845732 0 2024-04-05T03:02:43.579 INFO:tasks.workunit.client.1.smithi203.stdout:4/365: chown da/f56 4268379 1 2024-04-05T03:02:43.583 INFO:tasks.workunit.client.1.smithi203.stdout:9/481: truncate f8 2119680 0 2024-04-05T03:02:43.583 INFO:tasks.workunit.client.1.smithi203.stdout:9/482: stat c9 0 2024-04-05T03:02:43.584 INFO:tasks.workunit.client.0.smithi153.stdout:5/170: rmdir d0/d9/d22 39 2024-04-05T03:02:43.590 INFO:tasks.workunit.client.0.smithi153.stdout:5/171: truncate d0/de/f10 290963 0 2024-04-05T03:02:43.594 INFO:tasks.workunit.client.0.smithi153.stdout:1/165: dwrite d5/f7 [4194304,4194304] 0 2024-04-05T03:02:43.595 INFO:tasks.workunit.client.0.smithi153.stdout:2/170: dwrite d9/ff [8388608,4194304] 0 2024-04-05T03:02:43.597 INFO:tasks.workunit.client.0.smithi153.stdout:1/166: write d5/dd/f13 [801043,111339] 0 2024-04-05T03:02:43.599 INFO:tasks.workunit.client.0.smithi153.stdout:1/167: read d5/dd/d21/d18/d16/f22 [19113,65344] 0 2024-04-05T03:02:43.603 INFO:tasks.workunit.client.1.smithi203.stdout:9/483: truncate d13/d1b/d26/d51/f53 2994239 0 2024-04-05T03:02:43.606 INFO:tasks.workunit.client.0.smithi153.stdout:7/155: dread d2/da/d11/f24 [0,4194304] 0 2024-04-05T03:02:43.610 INFO:tasks.workunit.client.0.smithi153.stdout:2/171: mkdir d9/da/d37 0 2024-04-05T03:02:43.611 INFO:tasks.workunit.client.1.smithi203.stdout:9/484: creat d13/d1b/d26/f93 x:0 0 0 2024-04-05T03:02:43.613 INFO:tasks.workunit.client.1.smithi203.stdout:9/485: symlink d13/d1b/d33/d42/d67/l94 0 2024-04-05T03:02:43.615 INFO:tasks.workunit.client.0.smithi153.stdout:2/172: mkdir d9/da/d36/d38 0 2024-04-05T03:02:43.617 INFO:tasks.workunit.client.0.smithi153.stdout:2/173: chown d9/ff 119368149 1 2024-04-05T03:02:43.622 INFO:tasks.workunit.client.0.smithi153.stdout:7/156: dread d2/f9 [0,4194304] 0 2024-04-05T03:02:43.623 INFO:tasks.workunit.client.0.smithi153.stdout:1/168: dwrite d5/dd/d1b/f1d [0,4194304] 0 2024-04-05T03:02:43.630 INFO:tasks.workunit.client.0.smithi153.stdout:7/157: creat d2/da/d11/d12/d31/f37 x:0 0 0 2024-04-05T03:02:43.633 INFO:tasks.workunit.client.0.smithi153.stdout:1/169: dread d5/d8/ff [0,4194304] 0 2024-04-05T03:02:43.636 INFO:tasks.workunit.client.0.smithi153.stdout:2/174: dwrite d9/da/d19/f2d [0,4194304] 0 2024-04-05T03:02:43.638 INFO:tasks.workunit.client.0.smithi153.stdout:7/158: truncate d2/f32 965250 0 2024-04-05T03:02:43.638 INFO:tasks.workunit.client.1.smithi203.stdout:9/486: creat d13/d1b/d26/d31/d38/d77/f95 x:0 0 0 2024-04-05T03:02:43.641 INFO:tasks.workunit.client.0.smithi153.stdout:1/170: creat d5/dd/d21/d18/d16/f36 x:0 0 0 2024-04-05T03:02:43.642 INFO:tasks.workunit.client.1.smithi203.stdout:9/487: mknod d13/c96 0 2024-04-05T03:02:43.644 INFO:tasks.workunit.client.0.smithi153.stdout:2/175: dwrite d9/da/d19/f1c [0,4194304] 0 2024-04-05T03:02:43.645 INFO:tasks.workunit.client.0.smithi153.stdout:7/159: rename d2/da/d11/d12/f33 to d2/da/d1f/f38 0 2024-04-05T03:02:43.645 INFO:tasks.workunit.client.0.smithi153.stdout:1/171: creat d5/dd/d21/d18/f37 x:0 0 0 2024-04-05T03:02:43.645 INFO:tasks.workunit.client.1.smithi203.stdout:9/488: creat d13/d1b/d26/d51/f97 x:0 0 0 2024-04-05T03:02:43.646 INFO:tasks.workunit.client.1.smithi203.stdout:9/489: chown d13/l1a 438400 1 2024-04-05T03:02:43.647 INFO:tasks.workunit.client.1.smithi203.stdout:9/490: fsync d13/d1b/f21 0 2024-04-05T03:02:43.649 INFO:tasks.workunit.client.1.smithi203.stdout:9/491: mknod d13/d1b/d33/d64/c98 0 2024-04-05T03:02:43.650 INFO:tasks.workunit.client.0.smithi153.stdout:1/172: creat d5/d8/d20/f38 x:0 0 0 2024-04-05T03:02:43.651 INFO:tasks.workunit.client.1.smithi203.stdout:9/492: truncate fb 5088657 0 2024-04-05T03:02:43.651 INFO:tasks.workunit.client.0.smithi153.stdout:2/176: mkdir d9/da/d36/d38/d39 0 2024-04-05T03:02:43.651 INFO:tasks.workunit.client.0.smithi153.stdout:7/160: mkdir d2/da/d11/d39 0 2024-04-05T03:02:43.652 INFO:tasks.workunit.client.0.smithi153.stdout:1/173: dread - d5/d8/d20/f25 zero size 2024-04-05T03:02:43.652 INFO:tasks.workunit.client.0.smithi153.stdout:2/177: chown d9/da/fb 2928 1 2024-04-05T03:02:43.653 INFO:tasks.workunit.client.0.smithi153.stdout:7/161: fdatasync d2/da/d11/d12/d31/f37 0 2024-04-05T03:02:43.655 INFO:tasks.workunit.client.1.smithi203.stdout:9/493: creat d13/d1b/d26/d31/f99 x:0 0 0 2024-04-05T03:02:43.655 INFO:tasks.workunit.client.0.smithi153.stdout:7/162: chown d2/da/d11/c35 26808 1 2024-04-05T03:02:43.656 INFO:tasks.workunit.client.0.smithi153.stdout:1/174: symlink d5/dd/l39 0 2024-04-05T03:02:43.658 INFO:tasks.workunit.client.0.smithi153.stdout:2/178: creat d9/da/d19/d2e/f3a x:0 0 0 2024-04-05T03:02:43.659 INFO:tasks.workunit.client.0.smithi153.stdout:1/175: rmdir d5 39 2024-04-05T03:02:43.659 INFO:tasks.workunit.client.0.smithi153.stdout:7/163: mkdir d2/da/d3a 0 2024-04-05T03:02:43.660 INFO:tasks.workunit.client.1.smithi203.stdout:9/494: symlink d13/d1b/d26/d51/l9a 0 2024-04-05T03:02:43.661 INFO:tasks.workunit.client.1.smithi203.stdout:9/495: write d13/d1b/d26/d51/f52 [95932,108381] 0 2024-04-05T03:02:43.662 INFO:tasks.workunit.client.0.smithi153.stdout:1/176: truncate d5/dd/d21/d18/f37 103321 0 2024-04-05T03:02:43.663 INFO:tasks.workunit.client.0.smithi153.stdout:1/177: fdatasync d5/f7 0 2024-04-05T03:02:43.664 INFO:tasks.workunit.client.0.smithi153.stdout:2/179: link d9/da/d19/l2a d9/da/d36/d38/l3b 0 2024-04-05T03:02:43.665 INFO:tasks.workunit.client.0.smithi153.stdout:2/180: fdatasync d9/da/fe 0 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:3/439: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:0/501: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:5/556: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:6/563: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:2/694: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:8/530: sync 2024-04-05T03:02:43.670 INFO:tasks.workunit.client.1.smithi203.stdout:1/485: sync 2024-04-05T03:02:43.671 INFO:tasks.workunit.client.0.smithi153.stdout:2/181: dread d9/da/fc [0,4194304] 0 2024-04-05T03:02:43.672 INFO:tasks.workunit.client.1.smithi203.stdout:6/564: fdatasync d0/da/d54/d53/f6a 0 2024-04-05T03:02:43.674 INFO:tasks.workunit.client.1.smithi203.stdout:6/565: fsync d0/da/d18/d38/d2a/d44/d78/f89 0 2024-04-05T03:02:43.675 INFO:tasks.workunit.client.1.smithi203.stdout:1/486: stat d7/d24/d33/d48/d4b/c73 0 2024-04-05T03:02:43.676 INFO:tasks.workunit.client.1.smithi203.stdout:1/487: stat d7/c22 0 2024-04-05T03:02:43.677 INFO:tasks.workunit.client.0.smithi153.stdout:1/178: truncate d5/dd/d21/f11 454069 0 2024-04-05T03:02:43.680 INFO:tasks.workunit.client.0.smithi153.stdout:1/179: truncate d5/f9 4929832 0 2024-04-05T03:02:43.680 INFO:tasks.workunit.client.0.smithi153.stdout:2/182: dwrite d9/da/fe [0,4194304] 0 2024-04-05T03:02:43.682 INFO:tasks.workunit.client.0.smithi153.stdout:2/183: write d9/da/fe [3271965,110042] 0 2024-04-05T03:02:43.684 INFO:tasks.workunit.client.0.smithi153.stdout:1/180: read d5/f7 [3521117,98236] 0 2024-04-05T03:02:43.685 INFO:tasks.workunit.client.0.smithi153.stdout:2/184: symlink d9/da/d36/l3c 0 2024-04-05T03:02:43.685 INFO:tasks.workunit.client.0.smithi153.stdout:7/164: dread d2/f6 [4194304,4194304] 0 2024-04-05T03:02:43.686 INFO:tasks.workunit.client.0.smithi153.stdout:7/165: chown d2/da/d11/f2e 38 1 2024-04-05T03:02:43.687 INFO:tasks.workunit.client.0.smithi153.stdout:7/166: chown d2/da/d11/d12/d31 32 1 2024-04-05T03:02:43.688 INFO:tasks.workunit.client.1.smithi203.stdout:4/366: sync 2024-04-05T03:02:43.688 INFO:tasks.workunit.client.1.smithi203.stdout:7/649: sync 2024-04-05T03:02:43.689 INFO:tasks.workunit.client.0.smithi153.stdout:2/185: chown d9/da/d10/c32 55084 1 2024-04-05T03:02:43.692 INFO:tasks.workunit.client.1.smithi203.stdout:8/531: mkdir d1/d90 0 2024-04-05T03:02:43.694 INFO:tasks.workunit.client.1.smithi203.stdout:0/502: creat dd/d17/d1d/d22/d53/d61/f92 x:0 0 0 2024-04-05T03:02:43.694 INFO:tasks.workunit.client.0.smithi153.stdout:2/186: rename d9/da/d19/f2d to d9/da/d37/f3d 0 2024-04-05T03:02:43.694 INFO:tasks.workunit.client.1.smithi203.stdout:3/440: mkdir d0/dc/d38/d5d/d66/d80 0 2024-04-05T03:02:43.695 INFO:tasks.workunit.client.0.smithi153.stdout:2/187: stat l2 0 2024-04-05T03:02:43.695 INFO:tasks.workunit.client.1.smithi203.stdout:4/367: write da/d29/f42 [2650420,6053] 0 2024-04-05T03:02:43.698 INFO:tasks.workunit.client.1.smithi203.stdout:9/496: dread d13/d1b/f21 [0,4194304] 0 2024-04-05T03:02:43.698 INFO:tasks.workunit.client.0.smithi153.stdout:1/181: dwrite d5/dd/d1b/f1d [0,4194304] 0 2024-04-05T03:02:43.699 INFO:tasks.workunit.client.1.smithi203.stdout:3/441: chown d0/dc/d38/f5f 1761038588 1 2024-04-05T03:02:43.699 INFO:tasks.workunit.client.0.smithi153.stdout:2/188: chown d9/da/d10/c13 189056731 1 2024-04-05T03:02:43.700 INFO:tasks.workunit.client.1.smithi203.stdout:1/488: creat d7/d14/d2b/f95 x:0 0 0 2024-04-05T03:02:43.700 INFO:tasks.workunit.client.1.smithi203.stdout:2/695: read d5/d10/d35/d2d/d7c/d29/d67/f77 [2108691,82032] 0 2024-04-05T03:02:43.701 INFO:tasks.workunit.client.1.smithi203.stdout:7/650: mkdir d8/d2a/d3f/d5e/d63/dc5 0 2024-04-05T03:02:43.702 INFO:tasks.workunit.client.0.smithi153.stdout:2/189: stat d9/da 0 2024-04-05T03:02:43.703 INFO:tasks.workunit.client.1.smithi203.stdout:8/532: dread d1/d12/d17/d10/f40 [4194304,4194304] 0 2024-04-05T03:02:43.704 INFO:tasks.workunit.client.1.smithi203.stdout:0/503: write dd/d17/d1d/d26/f4f [35103,85857] 0 2024-04-05T03:02:43.704 INFO:tasks.workunit.client.1.smithi203.stdout:8/533: readlink d1/l48 0 2024-04-05T03:02:43.705 INFO:tasks.workunit.client.0.smithi153.stdout:2/190: write d9/da/d19/f2f [972447,45296] 0 2024-04-05T03:02:43.708 INFO:tasks.workunit.client.0.smithi153.stdout:1/182: unlink d5/d8/c14 0 2024-04-05T03:02:43.708 INFO:tasks.workunit.client.1.smithi203.stdout:0/504: chown dd/d17/d1d/d22/d80/f72 1 1 2024-04-05T03:02:43.709 INFO:tasks.workunit.client.1.smithi203.stdout:4/368: dwrite da/d29/f5b [0,4194304] 0 2024-04-05T03:02:43.711 INFO:tasks.workunit.client.0.smithi153.stdout:7/167: dwrite d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:43.712 INFO:tasks.workunit.client.0.smithi153.stdout:2/191: mknod d9/da/d10/c3e 0 2024-04-05T03:02:43.713 INFO:tasks.workunit.client.1.smithi203.stdout:0/505: fdatasync dd/d17/d1d/d22/f77 0 2024-04-05T03:02:43.714 INFO:tasks.workunit.client.1.smithi203.stdout:8/534: dread d1/d12/f28 [0,4194304] 0 2024-04-05T03:02:43.715 INFO:tasks.workunit.client.1.smithi203.stdout:4/369: dread - da/d29/d6a/f6c zero size 2024-04-05T03:02:43.718 INFO:tasks.workunit.client.0.smithi153.stdout:7/168: rename d2/f3 to d2/da/d11/f3b 0 2024-04-05T03:02:43.720 INFO:tasks.workunit.client.1.smithi203.stdout:7/651: rmdir d8/d2a/d3f/d52/d65 39 2024-04-05T03:02:43.723 INFO:tasks.workunit.client.0.smithi153.stdout:7/169: unlink d2/da/d11/f2e 0 2024-04-05T03:02:43.724 INFO:tasks.workunit.client.0.smithi153.stdout:2/192: link d9/da/d10/l14 d9/da/d10/l3f 0 2024-04-05T03:02:43.726 INFO:tasks.workunit.client.1.smithi203.stdout:1/489: mknod d7/c96 0 2024-04-05T03:02:43.726 INFO:tasks.workunit.client.0.smithi153.stdout:2/193: symlink d9/da/d19/l40 0 2024-04-05T03:02:43.728 INFO:tasks.workunit.client.0.smithi153.stdout:7/170: rename d2/da/d11/f2c to d2/da/d1f/f3c 0 2024-04-05T03:02:43.732 INFO:tasks.workunit.client.1.smithi203.stdout:2/696: mkdir d5/d10/d35/dee 0 2024-04-05T03:02:43.733 INFO:tasks.workunit.client.1.smithi203.stdout:4/370: creat da/d3c/d4c/f73 x:0 0 0 2024-04-05T03:02:43.734 INFO:tasks.workunit.client.1.smithi203.stdout:4/371: chown da/d3c/c51 147947 1 2024-04-05T03:02:43.736 INFO:tasks.workunit.client.0.smithi153.stdout:7/171: read d2/da/d11/d12/f1e [3792381,107138] 0 2024-04-05T03:02:43.736 INFO:tasks.workunit.client.1.smithi203.stdout:8/535: unlink d1/d6/d2e/l88 0 2024-04-05T03:02:43.737 INFO:tasks.workunit.client.0.smithi153.stdout:2/194: dread d9/da/d10/f33 [0,4194304] 0 2024-04-05T03:02:43.738 INFO:tasks.workunit.client.0.smithi153.stdout:7/172: fsync d2/da/d11/d12/f22 0 2024-04-05T03:02:43.739 INFO:tasks.workunit.client.1.smithi203.stdout:7/652: unlink d8/d2a/d3f/d52/f45 0 2024-04-05T03:02:43.741 INFO:tasks.workunit.client.0.smithi153.stdout:7/173: creat d2/da/d1f/f3d x:0 0 0 2024-04-05T03:02:43.741 INFO:tasks.workunit.client.0.smithi153.stdout:1/183: dread f4 [0,4194304] 0 2024-04-05T03:02:43.742 INFO:tasks.workunit.client.0.smithi153.stdout:2/195: rename d9/da/d10/c34 to d9/da/d36/d38/d39/c41 0 2024-04-05T03:02:43.742 INFO:tasks.workunit.client.1.smithi203.stdout:0/506: mkdir dd/d93 0 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:9/122: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:3/150: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:4/213: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:0/267: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:6/214: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:8/180: sync 2024-04-05T03:02:43.743 INFO:tasks.workunit.client.0.smithi153.stdout:7/174: dread - d2/da/d1f/f38 zero size 2024-04-05T03:02:43.745 INFO:tasks.workunit.client.0.smithi153.stdout:1/184: unlink d5/dd/l15 0 2024-04-05T03:02:43.749 INFO:tasks.workunit.client.0.smithi153.stdout:2/196: creat d9/da/d19/d2e/f42 x:0 0 0 2024-04-05T03:02:43.751 INFO:tasks.workunit.client.0.smithi153.stdout:7/175: creat d2/da/d1f/f3e x:0 0 0 2024-04-05T03:02:43.751 INFO:tasks.workunit.client.0.smithi153.stdout:6/215: rename d4/c2d to d4/dc/df/d16/d24/c53 0 2024-04-05T03:02:43.753 INFO:tasks.workunit.client.1.smithi203.stdout:1/490: creat d7/d14/d2b/d58/d86/d91/f97 x:0 0 0 2024-04-05T03:02:43.754 INFO:tasks.workunit.client.0.smithi153.stdout:4/214: unlink da/db/d20/d30/f37 0 2024-04-05T03:02:43.755 INFO:tasks.workunit.client.1.smithi203.stdout:7/653: dwrite d8/f33 [4194304,4194304] 0 2024-04-05T03:02:43.755 INFO:tasks.workunit.client.0.smithi153.stdout:3/151: dwrite d1d/f1e [0,4194304] 0 2024-04-05T03:02:43.756 INFO:tasks.workunit.client.1.smithi203.stdout:3/442: rmdir d0 39 2024-04-05T03:02:43.757 INFO:tasks.workunit.client.1.smithi203.stdout:7/654: write d8/d32/f44 [2787309,90825] 0 2024-04-05T03:02:43.759 INFO:tasks.workunit.client.0.smithi153.stdout:9/123: mknod d1/d10/d16/d22/d2c/c32 0 2024-04-05T03:02:43.762 INFO:tasks.workunit.client.1.smithi203.stdout:9/497: dread d13/d1b/d26/d31/f62 [0,4194304] 0 2024-04-05T03:02:43.763 INFO:tasks.workunit.client.1.smithi203.stdout:0/507: symlink dd/d17/d1d/d22/d53/l94 0 2024-04-05T03:02:43.764 INFO:tasks.workunit.client.1.smithi203.stdout:9/498: write d13/d1b/d26/f93 [924818,95785] 0 2024-04-05T03:02:43.766 INFO:tasks.workunit.client.0.smithi153.stdout:3/152: truncate d1d/d27/d2b/f35 288257 0 2024-04-05T03:02:43.766 INFO:tasks.workunit.client.1.smithi203.stdout:9/499: write d13/d1b/d26/d31/f99 [605170,74490] 0 2024-04-05T03:02:43.767 INFO:tasks.workunit.client.0.smithi153.stdout:6/216: dread d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:43.767 INFO:tasks.workunit.client.0.smithi153.stdout:9/124: rename d1/d1d/c2b to d1/d1d/c33 0 2024-04-05T03:02:43.768 INFO:tasks.workunit.client.0.smithi153.stdout:3/153: fdatasync d1d/d27/f34 0 2024-04-05T03:02:43.770 INFO:tasks.workunit.client.0.smithi153.stdout:7/176: dread d2/f6 [4194304,4194304] 0 2024-04-05T03:02:43.770 INFO:tasks.workunit.client.1.smithi203.stdout:0/508: dwrite dd/d17/d1d/d22/d80/d49/f7d [0,4194304] 0 2024-04-05T03:02:43.772 INFO:tasks.workunit.client.0.smithi153.stdout:3/154: write d1d/f36 [840247,39051] 0 2024-04-05T03:02:43.772 INFO:tasks.workunit.client.0.smithi153.stdout:4/215: mkdir da/db/d20/d35/d41 0 2024-04-05T03:02:43.775 INFO:tasks.workunit.client.0.smithi153.stdout:4/216: fsync da/f17 0 2024-04-05T03:02:43.776 INFO:tasks.workunit.client.0.smithi153.stdout:4/217: fsync da/de/d28/d2d/f40 0 2024-04-05T03:02:43.777 INFO:tasks.workunit.client.1.smithi203.stdout:4/372: creat da/f74 x:0 0 0 2024-04-05T03:02:43.777 INFO:tasks.workunit.client.0.smithi153.stdout:4/218: truncate da/f17 5041109 0 2024-04-05T03:02:43.781 INFO:tasks.workunit.client.1.smithi203.stdout:8/536: rename d1/c81 to d1/c91 0 2024-04-05T03:02:43.781 INFO:tasks.workunit.client.1.smithi203.stdout:8/537: readlink d1/d12/d1a/l60 0 2024-04-05T03:02:43.782 INFO:tasks.workunit.client.1.smithi203.stdout:7/655: readlink d8/d2a/d3f/d52/d65/la1 0 2024-04-05T03:02:43.783 INFO:tasks.workunit.client.0.smithi153.stdout:7/177: rmdir d2/da/d11/d12/d31 39 2024-04-05T03:02:43.783 INFO:tasks.workunit.client.1.smithi203.stdout:3/443: dread d0/dc/d1a/d1d/f2f [0,4194304] 0 2024-04-05T03:02:43.785 INFO:tasks.workunit.client.1.smithi203.stdout:3/444: dread - d0/dc/d1a/d1d/d53/f75 zero size 2024-04-05T03:02:43.785 INFO:tasks.workunit.client.0.smithi153.stdout:4/219: dwrite f4 [0,4194304] 0 2024-04-05T03:02:43.786 INFO:tasks.workunit.client.0.smithi153.stdout:7/178: stat d2/c34 0 2024-04-05T03:02:43.787 INFO:tasks.workunit.client.1.smithi203.stdout:3/445: truncate d0/dc/d1a/d1d/d43/f5e 952964 0 2024-04-05T03:02:43.788 INFO:tasks.workunit.client.1.smithi203.stdout:3/446: chown d0/dc/d1a/d76/d35/l7a 94232 1 2024-04-05T03:02:43.793 INFO:tasks.workunit.client.1.smithi203.stdout:2/697: link d5/d10/d35/d2d/d7c/d29/db1/lb2 d5/d10/d35/d2d/lef 0 2024-04-05T03:02:43.794 INFO:tasks.workunit.client.1.smithi203.stdout:2/698: chown d5/d10/d35/d2d/d7c/d29/db1/fb7 44 1 2024-04-05T03:02:43.794 INFO:tasks.workunit.client.1.smithi203.stdout:2/699: chown d5/d66/d7d/fdf 3520351 1 2024-04-05T03:02:43.798 INFO:tasks.workunit.client.0.smithi153.stdout:3/155: fdatasync d1d/f1e 0 2024-04-05T03:02:43.800 INFO:tasks.workunit.client.0.smithi153.stdout:3/156: stat d1d/f28 0 2024-04-05T03:02:43.801 INFO:tasks.workunit.client.1.smithi203.stdout:4/373: mknod da/d29/d6a/c75 0 2024-04-05T03:02:43.802 INFO:tasks.workunit.client.1.smithi203.stdout:4/374: write da/f1b [5989598,27687] 0 2024-04-05T03:02:43.805 INFO:tasks.workunit.client.0.smithi153.stdout:1/185: dwrite d5/dd/d21/d18/d16/f2d [0,4194304] 0 2024-04-05T03:02:43.807 INFO:tasks.workunit.client.0.smithi153.stdout:8/181: dwrite d7/d13/d19/d1b/f35 [0,4194304] 0 2024-04-05T03:02:43.807 INFO:tasks.workunit.client.1.smithi203.stdout:8/538: rmdir d1/d12/d17/d47 39 2024-04-05T03:02:43.808 INFO:tasks.workunit.client.0.smithi153.stdout:6/217: write d4/dc/df/d16/d33/f47 [97463,64050] 0 2024-04-05T03:02:43.810 INFO:tasks.workunit.client.0.smithi153.stdout:3/157: truncate d1d/f21 1060330 0 2024-04-05T03:02:43.811 INFO:tasks.workunit.client.0.smithi153.stdout:0/268: dwrite d9/df/f15 [0,4194304] 0 2024-04-05T03:02:43.813 INFO:tasks.workunit.client.0.smithi153.stdout:0/269: fdatasync d9/f30 0 2024-04-05T03:02:43.814 INFO:tasks.workunit.client.1.smithi203.stdout:7/656: mknod d8/da/d72/cc6 0 2024-04-05T03:02:43.815 INFO:tasks.workunit.client.0.smithi153.stdout:4/220: mknod da/d2a/c42 0 2024-04-05T03:02:43.816 INFO:tasks.workunit.client.1.smithi203.stdout:7/657: chown d8/d32/d16/d1a/d30/ca4 153912101 1 2024-04-05T03:02:43.820 INFO:tasks.workunit.client.0.smithi153.stdout:8/182: rmdir d7/d2a 39 2024-04-05T03:02:43.821 INFO:tasks.workunit.client.0.smithi153.stdout:1/186: dwrite d5/dd/d21/d18/f1a [0,4194304] 0 2024-04-05T03:02:43.823 INFO:tasks.workunit.client.0.smithi153.stdout:1/187: fdatasync d5/dd/f13 0 2024-04-05T03:02:43.826 INFO:tasks.workunit.client.1.smithi203.stdout:3/447: rename d0/dc/d3f to d0/d21/d81 0 2024-04-05T03:02:43.826 INFO:tasks.workunit.client.1.smithi203.stdout:7/658: dwrite d8/d32/d16/d1a/d30/f81 [4194304,4194304] 0 2024-04-05T03:02:43.828 INFO:tasks.workunit.client.1.smithi203.stdout:7/659: chown d8/d32/d16 57 1 2024-04-05T03:02:43.829 INFO:tasks.workunit.client.1.smithi203.stdout:7/660: stat d8/d2a/d3f/d52/d65/la1 0 2024-04-05T03:02:43.830 INFO:tasks.workunit.client.0.smithi153.stdout:7/179: creat d2/da/d11/f3f x:0 0 0 2024-04-05T03:02:43.830 INFO:tasks.workunit.client.0.smithi153.stdout:3/158: mknod d1d/d32/c37 0 2024-04-05T03:02:43.831 INFO:tasks.workunit.client.0.smithi153.stdout:6/218: rename d4/dc/d3e to d4/dc/df/d16/d17/d54 0 2024-04-05T03:02:43.832 INFO:tasks.workunit.client.0.smithi153.stdout:0/270: dread d9/df/d1e/f25 [0,4194304] 0 2024-04-05T03:02:43.833 INFO:tasks.workunit.client.1.smithi203.stdout:0/509: mkdir dd/d17/d1d/d95 0 2024-04-05T03:02:43.834 INFO:tasks.workunit.client.0.smithi153.stdout:6/219: fsync d4/dc/df/d16/d24/f32 0 2024-04-05T03:02:43.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/188: rmdir d5 39 2024-04-05T03:02:43.835 INFO:tasks.workunit.client.1.smithi203.stdout:0/510: write dd/d17/d1d/d22/d80/f72 [4710,53330] 0 2024-04-05T03:02:43.836 INFO:tasks.workunit.client.0.smithi153.stdout:8/183: dwrite d7/d13/d19/d31/f32 [0,4194304] 0 2024-04-05T03:02:43.837 INFO:tasks.workunit.client.0.smithi153.stdout:8/184: truncate d7/f2b 796502 0 2024-04-05T03:02:43.840 INFO:tasks.workunit.client.0.smithi153.stdout:4/221: link f7 da/db/d20/d30/f43 0 2024-04-05T03:02:43.840 INFO:tasks.workunit.client.0.smithi153.stdout:4/222: chown da/de 1 1 2024-04-05T03:02:43.841 INFO:tasks.workunit.client.0.smithi153.stdout:3/159: mkdir d1d/d38 0 2024-04-05T03:02:43.842 INFO:tasks.workunit.client.1.smithi203.stdout:4/375: write da/f37 [1959914,111709] 0 2024-04-05T03:02:43.845 INFO:tasks.workunit.client.1.smithi203.stdout:0/511: dwrite dd/d17/d1d/d22/d80/f4b [0,4194304] 0 2024-04-05T03:02:43.848 INFO:tasks.workunit.client.0.smithi153.stdout:6/220: chown d4/dc/df/d16/d24/c53 32 1 2024-04-05T03:02:43.848 INFO:tasks.workunit.client.0.smithi153.stdout:8/185: fsync d7/fa 0 2024-04-05T03:02:43.848 INFO:tasks.workunit.client.0.smithi153.stdout:4/223: unlink da/db/c18 0 2024-04-05T03:02:43.850 INFO:tasks.workunit.client.0.smithi153.stdout:6/221: stat d4/dc/df/f13 0 2024-04-05T03:02:43.850 INFO:tasks.workunit.client.0.smithi153.stdout:4/224: fsync da/db/d20/f2c 0 2024-04-05T03:02:43.850 INFO:tasks.workunit.client.1.smithi203.stdout:3/448: rmdir d0/d21 39 2024-04-05T03:02:43.851 INFO:tasks.workunit.client.0.smithi153.stdout:1/189: truncate d5/dd/d21/d18/d16/f36 22746 0 2024-04-05T03:02:43.851 INFO:tasks.workunit.client.1.smithi203.stdout:3/449: chown d0/dc/d1a/d1d/d43/c68 2 1 2024-04-05T03:02:43.852 INFO:tasks.workunit.client.1.smithi203.stdout:9/500: mkdir d13/d1b/d26/d31/d38/d59/d9b 0 2024-04-05T03:02:43.853 INFO:tasks.workunit.client.1.smithi203.stdout:3/450: chown d0/dc/d1a/d1d/d43/f4c 15278 1 2024-04-05T03:02:43.853 INFO:tasks.workunit.client.0.smithi153.stdout:7/180: creat d2/f40 x:0 0 0 2024-04-05T03:02:43.854 INFO:tasks.workunit.client.1.smithi203.stdout:2/700: mknod d5/d10/d7f/cf0 0 2024-04-05T03:02:43.854 INFO:tasks.workunit.client.1.smithi203.stdout:3/451: chown d0/dc/d1a/d76/c4f 3 1 2024-04-05T03:02:43.857 INFO:tasks.workunit.client.0.smithi153.stdout:1/190: dread d5/dd/d21/f11 [0,4194304] 0 2024-04-05T03:02:43.858 INFO:tasks.workunit.client.0.smithi153.stdout:4/225: creat da/de/d28/f44 x:0 0 0 2024-04-05T03:02:43.858 INFO:tasks.workunit.client.1.smithi203.stdout:9/501: stat d13/d1b/d26/d51/f53 0 2024-04-05T03:02:43.859 INFO:tasks.workunit.client.0.smithi153.stdout:8/186: write d7/f28 [4530165,61247] 0 2024-04-05T03:02:43.859 INFO:tasks.workunit.client.0.smithi153.stdout:7/181: rmdir d2/da/d1f 39 2024-04-05T03:02:43.860 INFO:tasks.workunit.client.0.smithi153.stdout:7/182: chown d2/da/d11/d12/f19 472 1 2024-04-05T03:02:43.860 INFO:tasks.workunit.client.1.smithi203.stdout:4/376: dwrite da/f22 [0,4194304] 0 2024-04-05T03:02:43.862 INFO:tasks.workunit.client.0.smithi153.stdout:4/226: read da/db/d20/d30/f43 [3461111,3146] 0 2024-04-05T03:02:43.862 INFO:tasks.workunit.client.0.smithi153.stdout:7/183: chown d2/da/d3a 94439 1 2024-04-05T03:02:43.862 INFO:tasks.workunit.client.1.smithi203.stdout:5/557: sync 2024-04-05T03:02:43.863 INFO:tasks.workunit.client.1.smithi203.stdout:3/452: creat d0/dc/d1a/d1d/d53/f82 x:0 0 0 2024-04-05T03:02:43.864 INFO:tasks.workunit.client.1.smithi203.stdout:7/661: link d8/da/d72/cc6 d8/da/d72/cc7 0 2024-04-05T03:02:43.866 INFO:tasks.workunit.client.0.smithi153.stdout:6/222: dread d4/dc/df/d16/d33/f47 [0,4194304] 0 2024-04-05T03:02:43.867 INFO:tasks.workunit.client.0.smithi153.stdout:4/227: dread da/f17 [0,4194304] 0 2024-04-05T03:02:43.868 INFO:tasks.workunit.client.1.smithi203.stdout:2/701: rmdir d5/d10/d35/d36 39 2024-04-05T03:02:43.868 INFO:tasks.workunit.client.0.smithi153.stdout:7/184: rename d2/da/d11/d12 to d2/da/d3a/d41 0 2024-04-05T03:02:43.869 INFO:tasks.workunit.client.0.smithi153.stdout:3/160: link lf d1d/d27/l39 0 2024-04-05T03:02:43.870 INFO:tasks.workunit.client.0.smithi153.stdout:8/187: symlink d7/d2a/l3c 0 2024-04-05T03:02:43.871 INFO:tasks.workunit.client.1.smithi203.stdout:1/491: sync 2024-04-05T03:02:43.871 INFO:tasks.workunit.client.1.smithi203.stdout:6/566: sync 2024-04-05T03:02:43.871 INFO:tasks.workunit.client.1.smithi203.stdout:8/539: getdents d1/d12/d1a 0 2024-04-05T03:02:43.874 INFO:tasks.workunit.client.0.smithi153.stdout:6/223: rmdir d4/dc/df 39 2024-04-05T03:02:43.874 INFO:tasks.workunit.client.1.smithi203.stdout:5/558: readlink d3/d1f/d49/la5 0 2024-04-05T03:02:43.874 INFO:tasks.workunit.client.1.smithi203.stdout:6/567: fsync d0/da/d54/f4b 0 2024-04-05T03:02:43.875 INFO:tasks.workunit.client.0.smithi153.stdout:8/188: dread d7/fe [0,4194304] 0 2024-04-05T03:02:43.876 INFO:tasks.workunit.client.1.smithi203.stdout:8/540: write d1/d12/d1a/f2c [512969,30643] 0 2024-04-05T03:02:43.876 INFO:tasks.workunit.client.1.smithi203.stdout:1/492: write d7/d24/f66 [1514650,4347] 0 2024-04-05T03:02:43.878 INFO:tasks.workunit.client.1.smithi203.stdout:3/453: mkdir d0/d21/d83 0 2024-04-05T03:02:43.882 INFO:tasks.workunit.client.1.smithi203.stdout:5/559: read d3/d1f/f93 [1121047,47724] 0 2024-04-05T03:02:43.882 INFO:tasks.workunit.client.1.smithi203.stdout:1/493: readlink d7/l10 0 2024-04-05T03:02:43.883 INFO:tasks.workunit.client.1.smithi203.stdout:2/702: dwrite d5/d10/d7f/fc0 [0,4194304] 0 2024-04-05T03:02:43.886 INFO:tasks.workunit.client.0.smithi153.stdout:6/224: chown d4/dc/df/d16/d17/f3a 23 1 2024-04-05T03:02:43.887 INFO:tasks.workunit.client.0.smithi153.stdout:6/225: dread - d4/dc/df/d16/d17/f3a zero size 2024-04-05T03:02:43.888 INFO:tasks.workunit.client.0.smithi153.stdout:7/185: mkdir d2/da/d42 0 2024-04-05T03:02:43.888 INFO:tasks.workunit.client.1.smithi203.stdout:9/502: dread d13/f2a [0,4194304] 0 2024-04-05T03:02:43.890 INFO:tasks.workunit.client.0.smithi153.stdout:4/228: rename da/db/d20/d30 to da/db/d20/d45 0 2024-04-05T03:02:43.893 INFO:tasks.workunit.client.0.smithi153.stdout:8/189: chown d7/f8 29006195 1 2024-04-05T03:02:43.894 INFO:tasks.workunit.client.1.smithi203.stdout:6/568: creat d0/da/d18/d38/d2a/d44/faa x:0 0 0 2024-04-05T03:02:43.894 INFO:tasks.workunit.client.1.smithi203.stdout:8/541: mkdir d1/d6/d2e/d92 0 2024-04-05T03:02:43.895 INFO:tasks.workunit.client.1.smithi203.stdout:5/560: dwrite d3/d4/d9e/f9f [0,4194304] 0 2024-04-05T03:02:43.896 INFO:tasks.workunit.client.0.smithi153.stdout:3/161: mknod d1d/d38/c3a 0 2024-04-05T03:02:43.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/703: mknod d5/d66/d7d/cf1 0 2024-04-05T03:02:43.897 INFO:tasks.workunit.client.0.smithi153.stdout:3/162: fsync d1d/d27/f34 0 2024-04-05T03:02:43.901 INFO:tasks.workunit.client.1.smithi203.stdout:5/561: read - d3/d4/f8e zero size 2024-04-05T03:02:43.902 INFO:tasks.workunit.client.1.smithi203.stdout:0/512: write dd/f44 [384867,52257] 0 2024-04-05T03:02:43.903 INFO:tasks.workunit.client.0.smithi153.stdout:8/190: dwrite d7/fd [0,4194304] 0 2024-04-05T03:02:43.906 INFO:tasks.workunit.client.1.smithi203.stdout:5/562: chown d3/d1f/d59/d64/f91 28 1 2024-04-05T03:02:43.906 INFO:tasks.workunit.client.1.smithi203.stdout:9/503: rmdir d13/d1b/d26 39 2024-04-05T03:02:43.907 INFO:tasks.workunit.client.0.smithi153.stdout:0/271: truncate f7 2511795 0 2024-04-05T03:02:43.908 INFO:tasks.workunit.client.1.smithi203.stdout:7/662: write d8/fb7 [1867919,94041] 0 2024-04-05T03:02:43.909 INFO:tasks.workunit.client.1.smithi203.stdout:6/569: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:43.910 INFO:tasks.workunit.client.1.smithi203.stdout:4/377: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:43.910 INFO:tasks.workunit.client.1.smithi203.stdout:8/542: creat d1/d6/f93 x:0 0 0 2024-04-05T03:02:43.910 INFO:tasks.workunit.client.0.smithi153.stdout:4/229: creat da/db/f46 x:0 0 0 2024-04-05T03:02:43.910 INFO:tasks.workunit.client.0.smithi153.stdout:5/172: sync 2024-04-05T03:02:43.910 INFO:tasks.workunit.client.0.smithi153.stdout:3/163: dwrite d1d/f25 [0,4194304] 0 2024-04-05T03:02:43.911 INFO:tasks.workunit.client.1.smithi203.stdout:8/543: readlink d1/l2 0 2024-04-05T03:02:43.913 INFO:tasks.workunit.client.1.smithi203.stdout:9/504: write d13/d1b/d33/d64/f6d [128186,19549] 0 2024-04-05T03:02:43.917 INFO:tasks.workunit.client.1.smithi203.stdout:1/494: dwrite d7/d24/d33/d48/d4b/d51/f7c [0,4194304] 0 2024-04-05T03:02:43.917 INFO:tasks.workunit.client.1.smithi203.stdout:1/495: stat d7/c22 0 2024-04-05T03:02:43.917 INFO:tasks.workunit.client.1.smithi203.stdout:7/663: readlink d8/d32/d16/d1a/d3d/d59/lb2 0 2024-04-05T03:02:43.918 INFO:tasks.workunit.client.0.smithi153.stdout:8/191: symlink d7/d13/l3d 0 2024-04-05T03:02:43.919 INFO:tasks.workunit.client.1.smithi203.stdout:8/544: unlink d1/d12/f28 0 2024-04-05T03:02:43.919 INFO:tasks.workunit.client.1.smithi203.stdout:3/454: getdents d0/dc/d1a/d76 0 2024-04-05T03:02:43.920 INFO:tasks.workunit.client.0.smithi153.stdout:4/230: fsync da/db/f23 0 2024-04-05T03:02:43.920 INFO:tasks.workunit.client.1.smithi203.stdout:8/545: write d1/d6/f56 [5226318,10646] 0 2024-04-05T03:02:43.921 INFO:tasks.workunit.client.1.smithi203.stdout:0/513: mknod dd/d17/d1d/d26/d8c/d6d/c96 0 2024-04-05T03:02:43.923 INFO:tasks.workunit.client.0.smithi153.stdout:8/192: read d7/d13/d19/d31/f32 [1372319,130603] 0 2024-04-05T03:02:43.924 INFO:tasks.workunit.client.1.smithi203.stdout:2/704: rename d5/d10/d35/d2d/d7c/c5c to d5/d10/d35/d2d/d7c/d29/d82/cf2 0 2024-04-05T03:02:43.926 INFO:tasks.workunit.client.0.smithi153.stdout:7/186: truncate d2/da/d1f/f10 1133546 0 2024-04-05T03:02:43.926 INFO:tasks.workunit.client.0.smithi153.stdout:1/191: truncate d5/dd/d21/d18/f1a 4256902 0 2024-04-05T03:02:43.928 INFO:tasks.workunit.client.1.smithi203.stdout:4/378: read - da/d3c/f44 zero size 2024-04-05T03:02:43.929 INFO:tasks.workunit.client.0.smithi153.stdout:0/272: dread d9/d14/f23 [0,4194304] 0 2024-04-05T03:02:43.930 INFO:tasks.workunit.client.0.smithi153.stdout:0/273: fsync d9/df/d12/f3d 0 2024-04-05T03:02:43.931 INFO:tasks.workunit.client.1.smithi203.stdout:1/496: readlink d7/l55 0 2024-04-05T03:02:43.932 INFO:tasks.workunit.client.0.smithi153.stdout:0/274: chown d9/df/d12/f1b 792568783 1 2024-04-05T03:02:43.933 INFO:tasks.workunit.client.1.smithi203.stdout:1/497: fsync d7/f80 0 2024-04-05T03:02:43.935 INFO:tasks.workunit.client.0.smithi153.stdout:4/231: creat da/d2a/f47 x:0 0 0 2024-04-05T03:02:43.936 INFO:tasks.workunit.client.0.smithi153.stdout:8/193: rename d7/f17 to d7/d13/d19/d31/f3e 0 2024-04-05T03:02:43.937 INFO:tasks.workunit.client.0.smithi153.stdout:4/232: dread - da/de/d28/d2d/f2e zero size 2024-04-05T03:02:43.939 INFO:tasks.workunit.client.1.smithi203.stdout:2/705: truncate d5/d63/f6e 930300 0 2024-04-05T03:02:43.942 INFO:tasks.workunit.client.1.smithi203.stdout:9/505: dwrite d13/d1b/d26/d31/d38/d59/d7b/f7f [0,4194304] 0 2024-04-05T03:02:43.943 INFO:tasks.workunit.client.0.smithi153.stdout:3/164: fdatasync d1d/f25 0 2024-04-05T03:02:43.948 INFO:tasks.workunit.client.0.smithi153.stdout:6/226: dwrite d4/dc/df/f2f [0,4194304] 0 2024-04-05T03:02:43.953 INFO:tasks.workunit.client.1.smithi203.stdout:6/570: getdents d0/da/d54/d82 0 2024-04-05T03:02:43.960 INFO:tasks.workunit.client.0.smithi153.stdout:1/192: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:43.962 INFO:tasks.workunit.client.1.smithi203.stdout:4/379: dread da/f13 [0,4194304] 0 2024-04-05T03:02:43.966 INFO:tasks.workunit.client.0.smithi153.stdout:1/193: dread f4 [0,4194304] 0 2024-04-05T03:02:43.967 INFO:tasks.workunit.client.1.smithi203.stdout:2/706: dwrite d5/d10/d35/d36/f54 [4194304,4194304] 0 2024-04-05T03:02:43.968 INFO:tasks.workunit.client.1.smithi203.stdout:9/506: dread d13/d1b/d26/d31/d38/f68 [0,4194304] 0 2024-04-05T03:02:43.970 INFO:tasks.workunit.client.0.smithi153.stdout:1/194: dread f4 [0,4194304] 0 2024-04-05T03:02:43.972 INFO:tasks.workunit.client.1.smithi203.stdout:8/546: truncate d1/d6/d2e/f6b 811579 0 2024-04-05T03:02:43.974 INFO:tasks.workunit.client.1.smithi203.stdout:3/455: dread d0/d21/f45 [0,4194304] 0 2024-04-05T03:02:43.979 INFO:tasks.workunit.client.1.smithi203.stdout:4/380: dwrite da/f37 [0,4194304] 0 2024-04-05T03:02:43.979 INFO:tasks.workunit.client.1.smithi203.stdout:8/547: read d1/f63 [3911180,37653] 0 2024-04-05T03:02:43.979 INFO:tasks.workunit.client.1.smithi203.stdout:7/664: dread d8/d32/f44 [0,4194304] 0 2024-04-05T03:02:43.981 INFO:tasks.workunit.client.1.smithi203.stdout:8/548: chown d1/d12/d2f/d1f/f50 1222201 1 2024-04-05T03:02:43.981 INFO:tasks.workunit.client.0.smithi153.stdout:8/194: mknod d7/d13/d19/d1b/d21/c3f 0 2024-04-05T03:02:43.981 INFO:tasks.workunit.client.0.smithi153.stdout:4/233: dwrite da/ff [0,4194304] 0 2024-04-05T03:02:43.983 INFO:tasks.workunit.client.1.smithi203.stdout:1/498: creat d7/d24/d33/d48/d4b/d6b/f98 x:0 0 0 2024-04-05T03:02:43.985 INFO:tasks.workunit.client.0.smithi153.stdout:4/234: fdatasync da/db/d20/f21 0 2024-04-05T03:02:43.985 INFO:tasks.workunit.client.1.smithi203.stdout:6/571: symlink d0/da/d18/d38/d2a/lab 0 2024-04-05T03:02:43.985 INFO:tasks.workunit.client.1.smithi203.stdout:3/456: fdatasync d0/d21/d81/f58 0 2024-04-05T03:02:43.987 INFO:tasks.workunit.client.1.smithi203.stdout:9/507: creat d13/d1b/d33/d64/f9c x:0 0 0 2024-04-05T03:02:43.987 INFO:tasks.workunit.client.1.smithi203.stdout:0/514: dread dd/f14 [0,4194304] 0 2024-04-05T03:02:43.990 INFO:tasks.workunit.client.1.smithi203.stdout:3/457: truncate d0/dc/d1a/d1d/d53/f75 554392 0 2024-04-05T03:02:43.994 INFO:tasks.workunit.client.1.smithi203.stdout:2/707: link d5/f6f d5/d10/db8/dab/de1/ff3 0 2024-04-05T03:02:43.995 INFO:tasks.workunit.client.0.smithi153.stdout:8/195: dread d7/d13/d19/d31/f3e [0,4194304] 0 2024-04-05T03:02:43.996 INFO:tasks.workunit.client.1.smithi203.stdout:6/572: getdents d0/da/d67/d71/d8c 0 2024-04-05T03:02:43.997 INFO:tasks.workunit.client.0.smithi153.stdout:5/173: getdents d0/de 0 2024-04-05T03:02:43.997 INFO:tasks.workunit.client.0.smithi153.stdout:4/235: dwrite da/de/d28/d2d/f40 [0,4194304] 0 2024-04-05T03:02:43.997 INFO:tasks.workunit.client.0.smithi153.stdout:7/187: creat d2/da/d3a/d41/d31/f43 x:0 0 0 2024-04-05T03:02:43.998 INFO:tasks.workunit.client.1.smithi203.stdout:2/708: write d5/d10/db8/dab/de1/f9a [2959231,49550] 0 2024-04-05T03:02:43.999 INFO:tasks.workunit.client.0.smithi153.stdout:4/236: stat da/de/d28/l2b 0 2024-04-05T03:02:44.002 INFO:tasks.workunit.client.1.smithi203.stdout:8/549: rename d1/d12/d2f/d55/f8b to d1/d6/d2e/f94 0 2024-04-05T03:02:44.002 INFO:tasks.workunit.client.0.smithi153.stdout:7/188: dread - d2/da/d11/f3f zero size 2024-04-05T03:02:44.003 INFO:tasks.workunit.client.1.smithi203.stdout:1/499: creat d7/f99 x:0 0 0 2024-04-05T03:02:44.003 INFO:tasks.workunit.client.0.smithi153.stdout:3/165: mknod d1d/d38/c3b 0 2024-04-05T03:02:44.004 INFO:tasks.workunit.client.0.smithi153.stdout:3/166: fdatasync d1d/f1e 0 2024-04-05T03:02:44.006 INFO:tasks.workunit.client.0.smithi153.stdout:6/227: unlink d4/dc/df/f2f 0 2024-04-05T03:02:44.006 INFO:tasks.workunit.client.1.smithi203.stdout:6/573: symlink d0/da/d54/d4c/da1/lac 0 2024-04-05T03:02:44.007 INFO:tasks.workunit.client.0.smithi153.stdout:0/275: mkdir d9/df/d4d 0 2024-04-05T03:02:44.008 INFO:tasks.workunit.client.0.smithi153.stdout:5/174: dwrite d0/de/f1a [4194304,4194304] 0 2024-04-05T03:02:44.008 INFO:tasks.workunit.client.1.smithi203.stdout:3/458: link d0/dc/d1a/d76/d35/l4e d0/d21/d83/l84 0 2024-04-05T03:02:44.009 INFO:tasks.workunit.client.0.smithi153.stdout:1/195: mkdir d5/dd/d21/d18/d16/d3a 0 2024-04-05T03:02:44.010 INFO:tasks.workunit.client.1.smithi203.stdout:9/508: dwrite d13/d1b/d26/d51/f91 [0,4194304] 0 2024-04-05T03:02:44.011 INFO:tasks.workunit.client.1.smithi203.stdout:8/550: creat d1/d4b/d68/f95 x:0 0 0 2024-04-05T03:02:44.012 INFO:tasks.workunit.client.1.smithi203.stdout:2/709: dread d5/d10/d44/fd7 [0,4194304] 0 2024-04-05T03:02:44.013 INFO:tasks.workunit.client.1.smithi203.stdout:6/574: chown d0/da/l74 60 1 2024-04-05T03:02:44.015 INFO:tasks.workunit.client.1.smithi203.stdout:3/459: fdatasync d0/d21/d81/f58 0 2024-04-05T03:02:44.017 INFO:tasks.workunit.client.1.smithi203.stdout:8/551: write d1/d6/f7c [823045,90845] 0 2024-04-05T03:02:44.019 INFO:tasks.workunit.client.1.smithi203.stdout:9/509: unlink d13/d1b/d26/d51/c5b 0 2024-04-05T03:02:44.023 INFO:tasks.workunit.client.0.smithi153.stdout:4/237: rmdir da/db/d20/d45/d3b 39 2024-04-05T03:02:44.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/552: fsync d1/d12/d2f/f61 0 2024-04-05T03:02:44.026 INFO:tasks.workunit.client.1.smithi203.stdout:6/575: chown d0/da/d54/d4c 976 1 2024-04-05T03:02:44.029 INFO:tasks.workunit.client.0.smithi153.stdout:3/167: mkdir d1d/d26/d3c 0 2024-04-05T03:02:44.030 INFO:tasks.workunit.client.1.smithi203.stdout:9/510: write d13/f24 [3255050,61569] 0 2024-04-05T03:02:44.032 INFO:tasks.workunit.client.1.smithi203.stdout:0/515: dread dd/d17/d1d/d26/d28/f4e [0,4194304] 0 2024-04-05T03:02:44.035 INFO:tasks.workunit.client.1.smithi203.stdout:3/460: unlink d0/dc/d1a/d1d/l71 0 2024-04-05T03:02:44.037 INFO:tasks.workunit.client.1.smithi203.stdout:9/511: mkdir d13/d5c/d9d 0 2024-04-05T03:02:44.037 INFO:tasks.workunit.client.1.smithi203.stdout:6/576: creat d0/da/d67/d71/fad x:0 0 0 2024-04-05T03:02:44.038 INFO:tasks.workunit.client.1.smithi203.stdout:3/461: chown d0/dc/d34 907996 1 2024-04-05T03:02:44.039 INFO:tasks.workunit.client.1.smithi203.stdout:5/563: sync 2024-04-05T03:02:44.040 INFO:tasks.workunit.client.1.smithi203.stdout:9/512: fdatasync d13/d1b/d26/f93 0 2024-04-05T03:02:44.040 INFO:tasks.workunit.client.0.smithi153.stdout:6/228: creat d4/dc/df/d16/d36/d52/f55 x:0 0 0 2024-04-05T03:02:44.042 INFO:tasks.workunit.client.1.smithi203.stdout:0/516: rename dd/d17/d1d/d22/d53/d55/l59 to dd/d17/d1d/d26/d8c/d6d/l97 0 2024-04-05T03:02:44.044 INFO:tasks.workunit.client.0.smithi153.stdout:5/175: dread d0/d9/f1e [0,4194304] 0 2024-04-05T03:02:44.046 INFO:tasks.workunit.client.1.smithi203.stdout:8/553: creat d1/d12/d17/d47/d53/f96 x:0 0 0 2024-04-05T03:02:44.048 INFO:tasks.workunit.client.1.smithi203.stdout:7/665: dread d8/d32/f18 [0,4194304] 0 2024-04-05T03:02:44.048 INFO:tasks.workunit.client.1.smithi203.stdout:9/513: truncate d13/d1b/d33/d42/d67/d82/f8e 140568 0 2024-04-05T03:02:44.048 INFO:tasks.workunit.client.1.smithi203.stdout:4/381: dread da/f4a [0,4194304] 0 2024-04-05T03:02:44.048 INFO:tasks.workunit.client.0.smithi153.stdout:2/197: sync 2024-04-05T03:02:44.049 INFO:tasks.workunit.client.0.smithi153.stdout:9/125: sync 2024-04-05T03:02:44.049 INFO:tasks.workunit.client.0.smithi153.stdout:9/126: write d1/d9/d1b/f1c [232632,120206] 0 2024-04-05T03:02:44.050 INFO:tasks.workunit.client.1.smithi203.stdout:5/564: rmdir d3/d4 39 2024-04-05T03:02:44.050 INFO:tasks.workunit.client.0.smithi153.stdout:0/276: creat d9/f4e x:0 0 0 2024-04-05T03:02:44.052 INFO:tasks.workunit.client.1.smithi203.stdout:7/666: read d8/d2a/d3f/d73/f78 [233711,77232] 0 2024-04-05T03:02:44.053 INFO:tasks.workunit.client.1.smithi203.stdout:3/462: mknod d0/d41/c85 0 2024-04-05T03:02:44.054 INFO:tasks.workunit.client.1.smithi203.stdout:5/565: rmdir d3/d1f/d59 39 2024-04-05T03:02:44.055 INFO:tasks.workunit.client.0.smithi153.stdout:6/229: write d4/dc/df/d16/d24/f29 [1259376,62111] 0 2024-04-05T03:02:44.056 INFO:tasks.workunit.client.0.smithi153.stdout:1/196: link d5/d8/c30 d5/dd/c3b 0 2024-04-05T03:02:44.058 INFO:tasks.workunit.client.1.smithi203.stdout:7/667: dread - d8/d2a/d3f/d52/d65/d82/fc1 zero size 2024-04-05T03:02:44.058 INFO:tasks.workunit.client.0.smithi153.stdout:8/196: getdents d7 0 2024-04-05T03:02:44.058 INFO:tasks.workunit.client.0.smithi153.stdout:9/127: dwrite d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:02:44.059 INFO:tasks.workunit.client.1.smithi203.stdout:9/514: rename d13/d1b/d26/d31/d38 to d13/d1b/d6b/d9e 0 2024-04-05T03:02:44.059 INFO:tasks.workunit.client.0.smithi153.stdout:5/176: symlink d0/d2c/l42 0 2024-04-05T03:02:44.059 INFO:tasks.workunit.client.0.smithi153.stdout:6/230: read d4/f15 [546542,25807] 0 2024-04-05T03:02:44.060 INFO:tasks.workunit.client.1.smithi203.stdout:0/517: creat dd/d17/d1d/d26/f98 x:0 0 0 2024-04-05T03:02:44.061 INFO:tasks.workunit.client.0.smithi153.stdout:9/128: fsync d1/f25 0 2024-04-05T03:02:44.062 INFO:tasks.workunit.client.1.smithi203.stdout:2/710: dread d5/d10/d35/d2d/d7c/d29/d67/f77 [0,4194304] 0 2024-04-05T03:02:44.064 INFO:tasks.workunit.client.1.smithi203.stdout:8/554: mkdir d1/d12/d1a/d6d/d97 0 2024-04-05T03:02:44.064 INFO:tasks.workunit.client.0.smithi153.stdout:7/189: getdents d2/da/d1f 0 2024-04-05T03:02:44.065 INFO:tasks.workunit.client.1.smithi203.stdout:5/566: creat d3/d5f/d92/fbc x:0 0 0 2024-04-05T03:02:44.066 INFO:tasks.workunit.client.1.smithi203.stdout:7/668: mkdir d8/d32/dc3/dc8 0 2024-04-05T03:02:44.068 INFO:tasks.workunit.client.1.smithi203.stdout:5/567: write d3/f18 [5103603,85999] 0 2024-04-05T03:02:44.069 INFO:tasks.workunit.client.0.smithi153.stdout:2/198: creat d9/da/d19/d2e/f43 x:0 0 0 2024-04-05T03:02:44.069 INFO:tasks.workunit.client.0.smithi153.stdout:2/199: fdatasync d9/da/fe 0 2024-04-05T03:02:44.069 INFO:tasks.workunit.client.1.smithi203.stdout:0/518: dread dd/d17/d1d/d26/f89 [0,4194304] 0 2024-04-05T03:02:44.069 INFO:tasks.workunit.client.1.smithi203.stdout:6/577: rename d0/da/d18/l31 to d0/da/d18/lae 0 2024-04-05T03:02:44.069 INFO:tasks.workunit.client.0.smithi153.stdout:2/200: write d9/da/d10/f1a [4205499,32810] 0 2024-04-05T03:02:44.070 INFO:tasks.workunit.client.1.smithi203.stdout:8/555: truncate d1/d12/d17/d10/f6e 169988 0 2024-04-05T03:02:44.072 INFO:tasks.workunit.client.0.smithi153.stdout:0/277: rmdir d9/db/d42/d43 39 2024-04-05T03:02:44.073 INFO:tasks.workunit.client.0.smithi153.stdout:0/278: dread d9/db/f49 [0,4194304] 0 2024-04-05T03:02:44.073 INFO:tasks.workunit.client.1.smithi203.stdout:3/463: symlink d0/dc/d38/d5d/d66/d80/l86 0 2024-04-05T03:02:44.074 INFO:tasks.workunit.client.0.smithi153.stdout:0/279: fsync d9/df/d1d/f3b 0 2024-04-05T03:02:44.074 INFO:tasks.workunit.client.1.smithi203.stdout:4/382: dwrite da/f4a [0,4194304] 0 2024-04-05T03:02:44.075 INFO:tasks.workunit.client.0.smithi153.stdout:0/280: chown d9/df/f46 34 1 2024-04-05T03:02:44.075 INFO:tasks.workunit.client.1.smithi203.stdout:8/556: read d1/d12/d17/d10/f6e [99084,42438] 0 2024-04-05T03:02:44.076 INFO:tasks.workunit.client.1.smithi203.stdout:5/568: fsync d3/d1f/d49/f57 0 2024-04-05T03:02:44.076 INFO:tasks.workunit.client.1.smithi203.stdout:2/711: fdatasync d5/d10/d35/d2d/d7c/d72/d8f/fea 0 2024-04-05T03:02:44.078 INFO:tasks.workunit.client.1.smithi203.stdout:3/464: chown d0/f7f 223581360 1 2024-04-05T03:02:44.080 INFO:tasks.workunit.client.1.smithi203.stdout:4/383: write da/f26 [2017259,39618] 0 2024-04-05T03:02:44.081 INFO:tasks.workunit.client.1.smithi203.stdout:0/519: fsync dd/d17/f48 0 2024-04-05T03:02:44.082 INFO:tasks.workunit.client.1.smithi203.stdout:3/465: stat d0/dc/d1a/d1d/d53/f73 0 2024-04-05T03:02:44.085 INFO:tasks.workunit.client.1.smithi203.stdout:1/500: dread d7/fe [4194304,4194304] 0 2024-04-05T03:02:44.085 INFO:tasks.workunit.client.1.smithi203.stdout:7/669: mkdir d8/d32/d5d/dbe/dc9 0 2024-04-05T03:02:44.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/238: creat da/f48 x:0 0 0 2024-04-05T03:02:44.087 INFO:tasks.workunit.client.0.smithi153.stdout:9/129: chown d1/d10/f19 10181 1 2024-04-05T03:02:44.087 INFO:tasks.workunit.client.0.smithi153.stdout:4/239: dread - da/de/f19 zero size 2024-04-05T03:02:44.088 INFO:tasks.workunit.client.1.smithi203.stdout:7/670: write d8/d2a/d3f/d5e/d77/fb0 [871638,100148] 0 2024-04-05T03:02:44.089 INFO:tasks.workunit.client.1.smithi203.stdout:2/712: rmdir d5/d10/d35/d2d/d7c/d29/d67/d9e 39 2024-04-05T03:02:44.089 INFO:tasks.workunit.client.0.smithi153.stdout:5/177: mknod d0/d2c/c43 0 2024-04-05T03:02:44.091 INFO:tasks.workunit.client.0.smithi153.stdout:3/168: truncate d1d/f21 855595 0 2024-04-05T03:02:44.092 INFO:tasks.workunit.client.0.smithi153.stdout:7/190: dread d2/da/d3a/d41/f22 [0,4194304] 0 2024-04-05T03:02:44.094 INFO:tasks.workunit.client.1.smithi203.stdout:3/466: rmdir d0/dc/d38 39 2024-04-05T03:02:44.094 INFO:tasks.workunit.client.1.smithi203.stdout:3/467: dread - d0/d21/d81/f5b zero size 2024-04-05T03:02:44.095 INFO:tasks.workunit.client.0.smithi153.stdout:2/201: mkdir d9/d44 0 2024-04-05T03:02:44.095 INFO:tasks.workunit.client.1.smithi203.stdout:0/520: mknod dd/d17/d1d/d26/d8c/d6d/c99 0 2024-04-05T03:02:44.096 INFO:tasks.workunit.client.0.smithi153.stdout:0/281: mknod d9/d14/c4f 0 2024-04-05T03:02:44.098 INFO:tasks.workunit.client.0.smithi153.stdout:1/197: creat d5/dd/d21/d18/d16/d33/f3c x:0 0 0 2024-04-05T03:02:44.098 INFO:tasks.workunit.client.0.smithi153.stdout:8/197: write d7/d13/f18 [2225417,33191] 0 2024-04-05T03:02:44.098 INFO:tasks.workunit.client.1.smithi203.stdout:1/501: creat d7/d14/d2b/d58/f9a x:0 0 0 2024-04-05T03:02:44.098 INFO:tasks.workunit.client.1.smithi203.stdout:9/515: write f8 [236383,14822] 0 2024-04-05T03:02:44.100 INFO:tasks.workunit.client.0.smithi153.stdout:9/130: symlink d1/d10/d16/l34 0 2024-04-05T03:02:44.100 INFO:tasks.workunit.client.0.smithi153.stdout:1/198: chown d5/d8/ff 0 1 2024-04-05T03:02:44.100 INFO:tasks.workunit.client.1.smithi203.stdout:9/516: truncate d13/d1b/d33/d64/f9c 95658 0 2024-04-05T03:02:44.101 INFO:tasks.workunit.client.1.smithi203.stdout:9/517: stat d13/d1b/d26/l45 0 2024-04-05T03:02:44.106 INFO:tasks.workunit.client.1.smithi203.stdout:8/557: rename d1/d6/c85 to d1/d6/d2e/d5c/c98 0 2024-04-05T03:02:44.107 INFO:tasks.workunit.client.0.smithi153.stdout:0/282: creat d9/df/d1e/f50 x:0 0 0 2024-04-05T03:02:44.107 INFO:tasks.workunit.client.1.smithi203.stdout:5/569: symlink d3/d4/lbd 0 2024-04-05T03:02:44.108 INFO:tasks.workunit.client.0.smithi153.stdout:6/231: creat d4/dc/df/f56 x:0 0 0 2024-04-05T03:02:44.108 INFO:tasks.workunit.client.0.smithi153.stdout:8/198: dwrite d7/d13/d19/d1b/f35 [0,4194304] 0 2024-04-05T03:02:44.110 INFO:tasks.workunit.client.0.smithi153.stdout:0/283: chown d9/ce 101187 1 2024-04-05T03:02:44.111 INFO:tasks.workunit.client.1.smithi203.stdout:0/521: symlink dd/d17/d1d/d22/d53/d61/l9a 0 2024-04-05T03:02:44.113 INFO:tasks.workunit.client.1.smithi203.stdout:1/502: symlink d7/d24/d33/d48/d4b/d51/l9b 0 2024-04-05T03:02:44.116 INFO:tasks.workunit.client.1.smithi203.stdout:9/518: creat d13/d5c/f9f x:0 0 0 2024-04-05T03:02:44.116 INFO:tasks.workunit.client.0.smithi153.stdout:9/131: truncate d1/d10/f19 1790210 0 2024-04-05T03:02:44.117 INFO:tasks.workunit.client.1.smithi203.stdout:4/384: rename da/d3c/d4c/l55 to da/d29/d2e/d53/l76 0 2024-04-05T03:02:44.118 INFO:tasks.workunit.client.0.smithi153.stdout:3/169: write d1d/f29 [4092231,79319] 0 2024-04-05T03:02:44.118 INFO:tasks.workunit.client.0.smithi153.stdout:7/191: write d2/da/d1f/f3c [631994,120707] 0 2024-04-05T03:02:44.121 INFO:tasks.workunit.client.1.smithi203.stdout:5/570: dwrite d3/d4/d9e/fab [0,4194304] 0 2024-04-05T03:02:44.121 INFO:tasks.workunit.client.0.smithi153.stdout:3/170: read d1d/f36 [436513,101402] 0 2024-04-05T03:02:44.122 INFO:tasks.workunit.client.1.smithi203.stdout:7/671: dread d8/d2a/d3f/d52/f5b [0,4194304] 0 2024-04-05T03:02:44.124 INFO:tasks.workunit.client.0.smithi153.stdout:1/199: mknod d5/dd/d21/d18/d16/d2c/c3d 0 2024-04-05T03:02:44.126 INFO:tasks.workunit.client.0.smithi153.stdout:0/284: readlink d9/df/l38 0 2024-04-05T03:02:44.127 INFO:tasks.workunit.client.0.smithi153.stdout:8/199: symlink d7/d2a/l40 0 2024-04-05T03:02:44.129 INFO:tasks.workunit.client.1.smithi203.stdout:9/519: symlink d13/d1b/d33/d64/la0 0 2024-04-05T03:02:44.130 INFO:tasks.workunit.client.1.smithi203.stdout:8/558: fdatasync d1/d6/d2e/f6b 0 2024-04-05T03:02:44.131 INFO:tasks.workunit.client.0.smithi153.stdout:9/132: symlink d1/d10/d16/l35 0 2024-04-05T03:02:44.133 INFO:tasks.workunit.client.0.smithi153.stdout:4/240: truncate da/f26 1839342 0 2024-04-05T03:02:44.133 INFO:tasks.workunit.client.0.smithi153.stdout:9/133: fsync d1/d9/d1b/f30 0 2024-04-05T03:02:44.134 INFO:tasks.workunit.client.1.smithi203.stdout:7/672: rename d8/d2a to d8/d32/d5d/d6d/dca 0 2024-04-05T03:02:44.135 INFO:tasks.workunit.client.0.smithi153.stdout:5/178: write d0/f14 [822269,26186] 0 2024-04-05T03:02:44.135 INFO:tasks.workunit.client.1.smithi203.stdout:0/522: symlink dd/d17/d1d/d22/d53/l9b 0 2024-04-05T03:02:44.136 INFO:tasks.workunit.client.1.smithi203.stdout:2/713: dread d5/d10/d35/d73/f95 [0,4194304] 0 2024-04-05T03:02:44.136 INFO:tasks.workunit.client.1.smithi203.stdout:4/385: mknod da/c77 0 2024-04-05T03:02:44.137 INFO:tasks.workunit.client.0.smithi153.stdout:3/171: rename f17 to d1d/d27/d2b/f3d 0 2024-04-05T03:02:44.137 INFO:tasks.workunit.client.1.smithi203.stdout:3/468: dwrite d0/dc/f54 [4194304,4194304] 0 2024-04-05T03:02:44.140 INFO:tasks.workunit.client.1.smithi203.stdout:2/714: write d5/d10/d35/fad [96836,66242] 0 2024-04-05T03:02:44.140 INFO:tasks.workunit.client.1.smithi203.stdout:1/503: creat d7/f9c x:0 0 0 2024-04-05T03:02:44.141 INFO:tasks.workunit.client.1.smithi203.stdout:8/559: mkdir d1/d6/d2e/d5c/d99 0 2024-04-05T03:02:44.142 INFO:tasks.workunit.client.0.smithi153.stdout:8/200: creat d7/d13/d19/d31/f41 x:0 0 0 2024-04-05T03:02:44.142 INFO:tasks.workunit.client.0.smithi153.stdout:7/192: dwrite d2/f9 [0,4194304] 0 2024-04-05T03:02:44.143 INFO:tasks.workunit.client.1.smithi203.stdout:5/571: fdatasync d3/d4/d9e/fab 0 2024-04-05T03:02:44.145 INFO:tasks.workunit.client.0.smithi153.stdout:4/241: symlink da/db/l49 0 2024-04-05T03:02:44.145 INFO:tasks.workunit.client.1.smithi203.stdout:7/673: creat d8/da/d72/fcb x:0 0 0 2024-04-05T03:02:44.147 INFO:tasks.workunit.client.0.smithi153.stdout:6/232: dread d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:44.147 INFO:tasks.workunit.client.1.smithi203.stdout:4/386: fdatasync da/f13 0 2024-04-05T03:02:44.148 INFO:tasks.workunit.client.1.smithi203.stdout:0/523: write dd/d17/d1d/d22/d45/d51/f86 [167750,64317] 0 2024-04-05T03:02:44.150 INFO:tasks.workunit.client.1.smithi203.stdout:4/387: write da/d29/d2e/d53/f6f [147295,4452] 0 2024-04-05T03:02:44.152 INFO:tasks.workunit.client.0.smithi153.stdout:9/134: mkdir d1/d10/d36 0 2024-04-05T03:02:44.152 INFO:tasks.workunit.client.1.smithi203.stdout:4/388: write da/d3c/d4c/f5c [877858,6944] 0 2024-04-05T03:02:44.155 INFO:tasks.workunit.client.0.smithi153.stdout:4/242: truncate da/de/f11 581755 0 2024-04-05T03:02:44.157 INFO:tasks.workunit.client.1.smithi203.stdout:1/504: fdatasync d7/d24/d33/d48/d4b/d6b/f98 0 2024-04-05T03:02:44.157 INFO:tasks.workunit.client.0.smithi153.stdout:5/179: dwrite d0/d25/f35 [0,4194304] 0 2024-04-05T03:02:44.158 INFO:tasks.workunit.client.1.smithi203.stdout:5/572: symlink d3/d4/d94/lbe 0 2024-04-05T03:02:44.161 INFO:tasks.workunit.client.0.smithi153.stdout:1/200: rmdir d5/dd 39 2024-04-05T03:02:44.161 INFO:tasks.workunit.client.0.smithi153.stdout:6/233: symlink d4/dc/df/d16/d36/d4d/l57 0 2024-04-05T03:02:44.168 INFO:tasks.workunit.client.1.smithi203.stdout:3/469: symlink d0/dc/d1a/l87 0 2024-04-05T03:02:44.169 INFO:tasks.workunit.client.0.smithi153.stdout:9/135: read d1/d10/d16/f27 [554500,97689] 0 2024-04-05T03:02:44.170 INFO:tasks.workunit.client.0.smithi153.stdout:9/136: stat d1/d10/d16/l35 0 2024-04-05T03:02:44.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/201: link d7/d13/d19/d1b/d33/f3a d7/d13/d19/d31/f42 0 2024-04-05T03:02:44.171 INFO:tasks.workunit.client.0.smithi153.stdout:9/137: chown d1/f18 79794698 1 2024-04-05T03:02:44.171 INFO:tasks.workunit.client.1.smithi203.stdout:2/715: mkdir d5/d10/d35/df4 0 2024-04-05T03:02:44.172 INFO:tasks.workunit.client.1.smithi203.stdout:2/716: chown d5/d10/d2f/ld9 0 1 2024-04-05T03:02:44.173 INFO:tasks.workunit.client.1.smithi203.stdout:0/524: write dd/d17/d1d/d26/d34/f35 [2572315,86889] 0 2024-04-05T03:02:44.173 INFO:tasks.workunit.client.1.smithi203.stdout:4/389: mkdir da/d29/d2e/d53/d78 0 2024-04-05T03:02:44.175 INFO:tasks.workunit.client.1.smithi203.stdout:8/560: mknod d1/d4b/d5f/d74/c9a 0 2024-04-05T03:02:44.177 INFO:tasks.workunit.client.1.smithi203.stdout:5/573: unlink d3/c9 0 2024-04-05T03:02:44.177 INFO:tasks.workunit.client.1.smithi203.stdout:4/390: read da/f3a [822087,85856] 0 2024-04-05T03:02:44.178 INFO:tasks.workunit.client.1.smithi203.stdout:8/561: write d1/f34 [3515659,116384] 0 2024-04-05T03:02:44.178 INFO:tasks.workunit.client.0.smithi153.stdout:2/202: truncate d9/da/fc 3647019 0 2024-04-05T03:02:44.180 INFO:tasks.workunit.client.0.smithi153.stdout:8/202: symlink d7/d13/d19/d1b/d21/l43 0 2024-04-05T03:02:44.180 INFO:tasks.workunit.client.1.smithi203.stdout:9/520: truncate d13/d1b/d6b/d9e/d59/f7e 1577088 0 2024-04-05T03:02:44.181 INFO:tasks.workunit.client.0.smithi153.stdout:6/234: symlink d4/dc/df/l58 0 2024-04-05T03:02:44.183 INFO:tasks.workunit.client.0.smithi153.stdout:7/193: truncate d2/f1d 4260447 0 2024-04-05T03:02:44.183 INFO:tasks.workunit.client.1.smithi203.stdout:2/717: write d5/d10/d35/d2d/d7c/d29/db1/fb7 [1935719,17313] 0 2024-04-05T03:02:44.183 INFO:tasks.workunit.client.1.smithi203.stdout:7/674: write d8/d32/d5d/d6d/f96 [1741756,114364] 0 2024-04-05T03:02:44.183 INFO:tasks.workunit.client.0.smithi153.stdout:6/235: readlink d4/dc/df/d16/d36/d4d/l57 0 2024-04-05T03:02:44.184 INFO:tasks.workunit.client.0.smithi153.stdout:1/201: creat d5/dd/d21/d18/d16/d33/f3e x:0 0 0 2024-04-05T03:02:44.184 INFO:tasks.workunit.client.1.smithi203.stdout:3/470: creat d0/d69/f88 x:0 0 0 2024-04-05T03:02:44.185 INFO:tasks.workunit.client.0.smithi153.stdout:7/194: stat d2/da/d3a/d41/d31/f37 0 2024-04-05T03:02:44.187 INFO:tasks.workunit.client.1.smithi203.stdout:3/471: stat d0/d41/f4a 0 2024-04-05T03:02:44.188 INFO:tasks.workunit.client.1.smithi203.stdout:2/718: chown d5/d10/db8/dab/le9 358870 1 2024-04-05T03:02:44.188 INFO:tasks.workunit.client.1.smithi203.stdout:1/505: dwrite d7/d14/f52 [0,4194304] 0 2024-04-05T03:02:44.189 INFO:tasks.workunit.client.1.smithi203.stdout:5/574: chown d3/d1f/d49/cbb 987624975 1 2024-04-05T03:02:44.189 INFO:tasks.workunit.client.1.smithi203.stdout:0/525: creat dd/d17/d1d/d22/f9c x:0 0 0 2024-04-05T03:02:44.189 INFO:tasks.workunit.client.0.smithi153.stdout:4/243: dread da/f1d [0,4194304] 0 2024-04-05T03:02:44.190 INFO:tasks.workunit.client.0.smithi153.stdout:2/203: creat d9/da/d36/d38/d39/f45 x:0 0 0 2024-04-05T03:02:44.192 INFO:tasks.workunit.client.1.smithi203.stdout:7/675: symlink d8/d32/d5d/d6d/dca/lcc 0 2024-04-05T03:02:44.195 INFO:tasks.workunit.client.0.smithi153.stdout:1/202: unlink d5/dd/d21/d1e/f28 0 2024-04-05T03:02:44.195 INFO:tasks.workunit.client.1.smithi203.stdout:8/562: dwrite d1/d12/d1a/f2c [4194304,4194304] 0 2024-04-05T03:02:44.196 INFO:tasks.workunit.client.0.smithi153.stdout:9/138: write d1/d10/d16/f27 [811285,46809] 0 2024-04-05T03:02:44.197 INFO:tasks.workunit.client.1.smithi203.stdout:7/676: dread - d8/d32/d16/d1a/d3d/d59/f9a zero size 2024-04-05T03:02:44.198 INFO:tasks.workunit.client.0.smithi153.stdout:4/244: dread f4 [0,4194304] 0 2024-04-05T03:02:44.199 INFO:tasks.workunit.client.0.smithi153.stdout:4/245: readlink da/db/d20/d35/l3a 0 2024-04-05T03:02:44.201 INFO:tasks.workunit.client.0.smithi153.stdout:5/180: dwrite d0/de/f10 [0,4194304] 0 2024-04-05T03:02:44.201 INFO:tasks.workunit.client.0.smithi153.stdout:3/172: link d1d/f21 d1d/f3e 0 2024-04-05T03:02:44.201 INFO:tasks.workunit.client.0.smithi153.stdout:4/246: write da/db/d20/f2c [629610,106846] 0 2024-04-05T03:02:44.202 INFO:tasks.workunit.client.1.smithi203.stdout:7/677: readlink d8/d32/d5d/d6d/dca/d3f/d5e/d77/la7 0 2024-04-05T03:02:44.202 INFO:tasks.workunit.client.1.smithi203.stdout:0/526: creat dd/d17/d1d/d22/d53/d61/f9d x:0 0 0 2024-04-05T03:02:44.203 INFO:tasks.workunit.client.0.smithi153.stdout:7/195: fsync d2/da/d3a/d41/f22 0 2024-04-05T03:02:44.211 INFO:tasks.workunit.client.0.smithi153.stdout:4/247: creat da/de/d28/d2d/f4a x:0 0 0 2024-04-05T03:02:44.213 INFO:tasks.workunit.client.0.smithi153.stdout:4/248: write da/de/d28/d2d/f40 [1565030,49351] 0 2024-04-05T03:02:44.215 INFO:tasks.workunit.client.0.smithi153.stdout:2/204: unlink d9/l2b 0 2024-04-05T03:02:44.217 INFO:tasks.workunit.client.1.smithi203.stdout:0/527: write dd/d17/d1d/d22/d53/d61/f6a [1668992,88961] 0 2024-04-05T03:02:44.218 INFO:tasks.workunit.client.0.smithi153.stdout:6/236: link d4/dc/df/l58 d4/dc/df/d16/d33/l59 0 2024-04-05T03:02:44.218 INFO:tasks.workunit.client.0.smithi153.stdout:3/173: getdents d1d/d26/d3c 0 2024-04-05T03:02:44.220 INFO:tasks.workunit.client.0.smithi153.stdout:6/237: fsync d4/dc/df/f26 0 2024-04-05T03:02:44.221 INFO:tasks.workunit.client.1.smithi203.stdout:8/563: dwrite d1/d12/d1a/f2c [8388608,4194304] 0 2024-04-05T03:02:44.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/249: dread da/db/f23 [0,4194304] 0 2024-04-05T03:02:44.222 INFO:tasks.workunit.client.1.smithi203.stdout:5/575: dwrite d3/d1f/d59/d64/f1c [4194304,4194304] 0 2024-04-05T03:02:44.224 INFO:tasks.workunit.client.1.smithi203.stdout:3/472: write d0/f23 [1183939,11065] 0 2024-04-05T03:02:44.225 INFO:tasks.workunit.client.0.smithi153.stdout:7/196: creat d2/da/d42/f44 x:0 0 0 2024-04-05T03:02:44.226 INFO:tasks.workunit.client.1.smithi203.stdout:2/719: dwrite d5/d10/d35/d2d/f50 [4194304,4194304] 0 2024-04-05T03:02:44.229 INFO:tasks.workunit.client.0.smithi153.stdout:2/205: chown d9/ff 2 1 2024-04-05T03:02:44.231 INFO:tasks.workunit.client.1.smithi203.stdout:0/528: chown dd/d17/d1d/d26/f42 416555 1 2024-04-05T03:02:44.233 INFO:tasks.workunit.client.0.smithi153.stdout:4/250: unlink da/de/d28/d2d/f40 0 2024-04-05T03:02:44.234 INFO:tasks.workunit.client.1.smithi203.stdout:2/720: rename d5/f8d to d5/d10/d2f/d5d/ff5 0 2024-04-05T03:02:44.236 INFO:tasks.workunit.client.0.smithi153.stdout:7/197: creat d2/da/d3a/d41/d31/f45 x:0 0 0 2024-04-05T03:02:44.240 INFO:tasks.workunit.client.1.smithi203.stdout:2/721: fdatasync d5/f98 0 2024-04-05T03:02:44.241 INFO:tasks.workunit.client.0.smithi153.stdout:3/174: unlink d1d/d27/l39 0 2024-04-05T03:02:44.241 INFO:tasks.workunit.client.1.smithi203.stdout:0/529: rename dd/d17/d1d/d22/d45/d51/l8e to dd/d17/d1d/d22/d45/d51/d69/l9e 0 2024-04-05T03:02:44.242 INFO:tasks.workunit.client.1.smithi203.stdout:9/521: dwrite f2 [0,4194304] 0 2024-04-05T03:02:44.243 INFO:tasks.workunit.client.1.smithi203.stdout:3/473: dread d0/dc/d1a/d1d/d53/f75 [0,4194304] 0 2024-04-05T03:02:44.244 INFO:tasks.workunit.client.0.smithi153.stdout:6/238: link d4/f20 d4/dc/df/d16/d24/d4e/f5a 0 2024-04-05T03:02:44.244 INFO:tasks.workunit.client.1.smithi203.stdout:2/722: mkdir d5/d10/d44/df6 0 2024-04-05T03:02:44.248 INFO:tasks.workunit.client.0.smithi153.stdout:2/206: creat d9/da/d19/f46 x:0 0 0 2024-04-05T03:02:44.249 INFO:tasks.workunit.client.0.smithi153.stdout:7/198: fdatasync d2/da/d11/f24 0 2024-04-05T03:02:44.250 INFO:tasks.workunit.client.1.smithi203.stdout:2/723: stat d5/d10/d35/d2d/d7c/d72/d8f/fa5 0 2024-04-05T03:02:44.251 INFO:tasks.workunit.client.1.smithi203.stdout:1/506: dread d7/d14/f93 [0,4194304] 0 2024-04-05T03:02:44.251 INFO:tasks.workunit.client.0.smithi153.stdout:4/251: dread da/db/d20/f2c [0,4194304] 0 2024-04-05T03:02:44.251 INFO:tasks.workunit.client.1.smithi203.stdout:0/530: dwrite dd/d17/d1d/d26/f4f [0,4194304] 0 2024-04-05T03:02:44.252 INFO:tasks.workunit.client.0.smithi153.stdout:7/199: write d2/da/d1f/f3c [1758485,125424] 0 2024-04-05T03:02:44.253 INFO:tasks.workunit.client.1.smithi203.stdout:0/531: write dd/d17/f40 [187000,46072] 0 2024-04-05T03:02:44.258 INFO:tasks.workunit.client.1.smithi203.stdout:2/724: mknod d5/d10/d35/d2d/d7c/d29/db1/cf7 0 2024-04-05T03:02:44.260 INFO:tasks.workunit.client.1.smithi203.stdout:1/507: mkdir d7/d14/d9d 0 2024-04-05T03:02:44.267 INFO:tasks.workunit.client.0.smithi153.stdout:7/200: link d2/f32 d2/da/d3a/f46 0 2024-04-05T03:02:44.268 INFO:tasks.workunit.client.1.smithi203.stdout:7/678: truncate d8/d32/d16/d1a/d30/f38 3371395 0 2024-04-05T03:02:44.268 INFO:tasks.workunit.client.1.smithi203.stdout:1/508: fdatasync d7/d14/f93 0 2024-04-05T03:02:44.270 INFO:tasks.workunit.client.0.smithi153.stdout:6/239: dread d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:44.272 INFO:tasks.workunit.client.1.smithi203.stdout:1/509: mknod d7/d24/c9e 0 2024-04-05T03:02:44.273 INFO:tasks.workunit.client.1.smithi203.stdout:7/679: truncate d8/d32/d16/d1a/d21/fbd 2657236 0 2024-04-05T03:02:44.274 INFO:tasks.workunit.client.1.smithi203.stdout:2/725: dread d5/fc1 [0,4194304] 0 2024-04-05T03:02:44.277 INFO:tasks.workunit.client.1.smithi203.stdout:2/726: mkdir d5/d10/d35/d36/d41/df8 0 2024-04-05T03:02:44.280 INFO:tasks.workunit.client.1.smithi203.stdout:2/727: chown d5/d10/d7f/fc0 219261 1 2024-04-05T03:02:44.280 INFO:tasks.workunit.client.1.smithi203.stdout:7/680: dwrite d8/d32/d5d/d6d/dca/d3f/d5e/fa5 [0,4194304] 0 2024-04-05T03:02:44.284 INFO:tasks.workunit.client.1.smithi203.stdout:2/728: creat d5/d10/d2f/d5d/ff9 x:0 0 0 2024-04-05T03:02:44.285 INFO:tasks.workunit.client.1.smithi203.stdout:7/681: mkdir d8/d32/d5d/d6d/dca/d3f/d5e/dcd 0 2024-04-05T03:02:44.288 INFO:tasks.workunit.client.1.smithi203.stdout:2/729: rmdir d5/d10/d35/d2d/d7c/d29/d67 39 2024-04-05T03:02:44.292 INFO:tasks.workunit.client.1.smithi203.stdout:0/532: dread f6 [0,4194304] 0 2024-04-05T03:02:44.292 INFO:tasks.workunit.client.1.smithi203.stdout:7/682: creat d8/d32/d16/d1a/d3d/fce x:0 0 0 2024-04-05T03:02:44.293 INFO:tasks.workunit.client.1.smithi203.stdout:0/533: write dd/d17/d1d/d26/f67 [167412,73706] 0 2024-04-05T03:02:44.293 INFO:tasks.workunit.client.1.smithi203.stdout:2/730: rmdir d5/d63/d86/de3 0 2024-04-05T03:02:44.295 INFO:tasks.workunit.client.1.smithi203.stdout:0/534: stat dd/d17/d1d/d22/d80/d49 0 2024-04-05T03:02:44.296 INFO:tasks.workunit.client.1.smithi203.stdout:1/510: dread d7/f1c [0,4194304] 0 2024-04-05T03:02:44.300 INFO:tasks.workunit.client.1.smithi203.stdout:7/683: rename d8/d32/d16/d1a/d30/f38 to d8/d32/d16/d35/fcf 0 2024-04-05T03:02:44.300 INFO:tasks.workunit.client.1.smithi203.stdout:2/731: read d5/d10/d1a/d57/fac [2992305,17557] 0 2024-04-05T03:02:44.302 INFO:tasks.workunit.client.1.smithi203.stdout:7/684: write d8/d32/d16/d1a/d30/f98 [5100859,86400] 0 2024-04-05T03:02:44.311 INFO:tasks.workunit.client.0.smithi153.stdout:2/207: dwrite d9/da/d19/d2e/f42 [0,4194304] 0 2024-04-05T03:02:44.312 INFO:tasks.workunit.client.0.smithi153.stdout:3/175: dread d1d/f21 [0,4194304] 0 2024-04-05T03:02:44.313 INFO:tasks.workunit.client.0.smithi153.stdout:7/201: dread - d2/da/d11/f2b zero size 2024-04-05T03:02:44.317 INFO:tasks.workunit.client.0.smithi153.stdout:3/176: chown d1d/f1e 11311638 1 2024-04-05T03:02:44.317 INFO:tasks.workunit.client.0.smithi153.stdout:9/139: dread d1/d10/d16/f17 [0,4194304] 0 2024-04-05T03:02:44.317 INFO:tasks.workunit.client.0.smithi153.stdout:4/252: dwrite da/db/d20/f2c [0,4194304] 0 2024-04-05T03:02:44.321 INFO:tasks.workunit.client.1.smithi203.stdout:2/732: creat d5/d10/d35/d36/ffa x:0 0 0 2024-04-05T03:02:44.322 INFO:tasks.workunit.client.0.smithi153.stdout:7/202: fdatasync d2/da/d1f/f38 0 2024-04-05T03:02:44.322 INFO:tasks.workunit.client.1.smithi203.stdout:7/685: mknod d8/d32/d5d/d6d/dca/d3f/d5e/d77/cd0 0 2024-04-05T03:02:44.323 INFO:tasks.workunit.client.0.smithi153.stdout:4/253: mknod da/db/d20/d3c/c4b 0 2024-04-05T03:02:44.324 INFO:tasks.workunit.client.0.smithi153.stdout:3/177: mkdir d1d/d26/d3c/d3f 0 2024-04-05T03:02:44.330 INFO:tasks.workunit.client.1.smithi203.stdout:7/686: write d8/d32/d16/d1a/d5a/fa3 [400554,97320] 0 2024-04-05T03:02:44.330 INFO:tasks.workunit.client.0.smithi153.stdout:7/203: mkdir d2/da/d47 0 2024-04-05T03:02:44.332 INFO:tasks.workunit.client.1.smithi203.stdout:7/687: truncate d8/d32/d5d/d6d/dca/f84 273198 0 2024-04-05T03:02:44.332 INFO:tasks.workunit.client.0.smithi153.stdout:2/208: dwrite d9/da/d10/f33 [0,4194304] 0 2024-04-05T03:02:44.333 INFO:tasks.workunit.client.1.smithi203.stdout:1/511: link d7/c22 d7/d90/c9f 0 2024-04-05T03:02:44.336 INFO:tasks.workunit.client.0.smithi153.stdout:3/178: unlink c9 0 2024-04-05T03:02:44.336 INFO:tasks.workunit.client.0.smithi153.stdout:2/209: write d9/da/d19/d2e/f43 [955505,52024] 0 2024-04-05T03:02:44.337 INFO:tasks.workunit.client.0.smithi153.stdout:9/140: truncate d1/d9/d1b/f26 3186076 0 2024-04-05T03:02:44.338 INFO:tasks.workunit.client.0.smithi153.stdout:6/240: dread d4/fb [0,4194304] 0 2024-04-05T03:02:44.339 INFO:tasks.workunit.client.0.smithi153.stdout:7/204: symlink d2/da/d47/l48 0 2024-04-05T03:02:44.341 INFO:tasks.workunit.client.0.smithi153.stdout:4/254: write da/de/f11 [500895,111317] 0 2024-04-05T03:02:44.342 INFO:tasks.workunit.client.0.smithi153.stdout:3/179: mknod d1d/d26/d3c/d3f/c40 0 2024-04-05T03:02:44.342 INFO:tasks.workunit.client.0.smithi153.stdout:6/241: chown d4/dc/df/d16/d17/c42 658613 1 2024-04-05T03:02:44.345 INFO:tasks.workunit.client.1.smithi203.stdout:0/535: truncate dd/d17/d1d/d26/d34/f35 323690 0 2024-04-05T03:02:44.348 INFO:tasks.workunit.client.0.smithi153.stdout:3/180: mkdir d1d/d41 0 2024-04-05T03:02:44.350 INFO:tasks.workunit.client.1.smithi203.stdout:0/536: mknod dd/d93/c9f 0 2024-04-05T03:02:44.354 INFO:tasks.workunit.client.0.smithi153.stdout:6/242: rename d4/dc/c3d to d4/dc/df/c5b 0 2024-04-05T03:02:44.354 INFO:tasks.workunit.client.0.smithi153.stdout:9/141: dwrite d1/d10/d16/f17 [0,4194304] 0 2024-04-05T03:02:44.356 INFO:tasks.workunit.client.1.smithi203.stdout:7/688: write d8/d32/d5d/d6d/dca/d3f/d5e/fa5 [5147371,98856] 0 2024-04-05T03:02:44.357 INFO:tasks.workunit.client.0.smithi153.stdout:2/210: dwrite d9/da/d19/f46 [0,4194304] 0 2024-04-05T03:02:44.358 INFO:tasks.workunit.client.0.smithi153.stdout:6/243: stat d4/dc/df/d16/d24/f29 0 2024-04-05T03:02:44.358 INFO:tasks.workunit.client.0.smithi153.stdout:2/211: write d9/da/f20 [1677047,87065] 0 2024-04-05T03:02:44.359 INFO:tasks.workunit.client.1.smithi203.stdout:7/689: write d8/d32/d5d/d6d/dca/d3f/d5e/fa5 [2390380,29247] 0 2024-04-05T03:02:44.360 INFO:tasks.workunit.client.1.smithi203.stdout:7/690: fsync d8/f4c 0 2024-04-05T03:02:44.361 INFO:tasks.workunit.client.0.smithi153.stdout:3/181: rename d1d/d26/c33 to d1d/d38/c42 0 2024-04-05T03:02:44.362 INFO:tasks.workunit.client.1.smithi203.stdout:2/733: dread d5/d10/d35/d2d/d7c/d29/d67/d9e/fd6 [0,4194304] 0 2024-04-05T03:02:44.363 INFO:tasks.workunit.client.0.smithi153.stdout:2/212: write d9/da/d36/d38/d39/f45 [618934,28904] 0 2024-04-05T03:02:44.363 INFO:tasks.workunit.client.1.smithi203.stdout:7/691: chown d8/d32/d16/d1a/d3d/d59/f61 19 1 2024-04-05T03:02:44.364 INFO:tasks.workunit.client.1.smithi203.stdout:2/734: write d5/d66/d7d/fdf [428771,104486] 0 2024-04-05T03:02:44.366 INFO:tasks.workunit.client.0.smithi153.stdout:4/255: write da/f17 [4163569,48258] 0 2024-04-05T03:02:44.368 INFO:tasks.workunit.client.1.smithi203.stdout:6/578: sync 2024-04-05T03:02:44.371 INFO:tasks.workunit.client.0.smithi153.stdout:7/205: dwrite d2/f32 [0,4194304] 0 2024-04-05T03:02:44.372 INFO:tasks.workunit.client.1.smithi203.stdout:7/692: chown d8/d32/d5d/d6d/dca/d3f/d52/d65/cab 186841441 1 2024-04-05T03:02:44.374 INFO:tasks.workunit.client.1.smithi203.stdout:0/537: dwrite dd/d17/f79 [0,4194304] 0 2024-04-05T03:02:44.378 INFO:tasks.workunit.client.1.smithi203.stdout:0/538: write dd/d17/d1d/f1e [292001,75865] 0 2024-04-05T03:02:44.379 INFO:tasks.workunit.client.1.smithi203.stdout:1/512: dread d7/f2a [0,4194304] 0 2024-04-05T03:02:44.380 INFO:tasks.workunit.client.0.smithi153.stdout:7/206: dwrite d2/da/d3a/d41/d31/f43 [0,4194304] 0 2024-04-05T03:02:44.381 INFO:tasks.workunit.client.1.smithi203.stdout:2/735: symlink d5/d10/d35/d2d/d7c/d29/de5/lfb 0 2024-04-05T03:02:44.381 INFO:tasks.workunit.client.0.smithi153.stdout:9/142: rename d1/d10/d16/d22/d2c/c32 to d1/d1d/c37 0 2024-04-05T03:02:44.384 INFO:tasks.workunit.client.1.smithi203.stdout:7/693: creat d8/d32/dc3/dc8/fd1 x:0 0 0 2024-04-05T03:02:44.387 INFO:tasks.workunit.client.0.smithi153.stdout:3/182: symlink d1d/d27/l43 0 2024-04-05T03:02:44.390 INFO:tasks.workunit.client.0.smithi153.stdout:4/256: symlink da/l4c 0 2024-04-05T03:02:44.391 INFO:tasks.workunit.client.0.smithi153.stdout:4/257: chown c2 1045528 1 2024-04-05T03:02:44.391 INFO:tasks.workunit.client.1.smithi203.stdout:1/513: symlink d7/d24/d33/d48/la0 0 2024-04-05T03:02:44.392 INFO:tasks.workunit.client.1.smithi203.stdout:0/539: dwrite dd/d17/d1d/d22/d45/d51/d69/f91 [0,4194304] 0 2024-04-05T03:02:44.394 INFO:tasks.workunit.client.1.smithi203.stdout:6/579: dread d0/da/d67/f7a [0,4194304] 0 2024-04-05T03:02:44.395 INFO:tasks.workunit.client.1.smithi203.stdout:2/736: creat d5/d10/d35/d2d/d7c/d72/ffc x:0 0 0 2024-04-05T03:02:44.395 INFO:tasks.workunit.client.1.smithi203.stdout:1/514: write d7/d14/f76 [2797736,36049] 0 2024-04-05T03:02:44.398 INFO:tasks.workunit.client.0.smithi153.stdout:7/207: rename d2/da/d1f to d2/da/d11/d49 0 2024-04-05T03:02:44.401 INFO:tasks.workunit.client.1.smithi203.stdout:7/694: dwrite d8/d32/d16/d1a/d21/f27 [0,4194304] 0 2024-04-05T03:02:44.405 INFO:tasks.workunit.client.1.smithi203.stdout:6/580: mkdir d0/da/d54/d4c/daf 0 2024-04-05T03:02:44.405 INFO:tasks.workunit.client.0.smithi153.stdout:4/258: creat da/f4d x:0 0 0 2024-04-05T03:02:44.406 INFO:tasks.workunit.client.1.smithi203.stdout:2/737: creat d5/d10/db8/ffd x:0 0 0 2024-04-05T03:02:44.408 INFO:tasks.workunit.client.1.smithi203.stdout:1/515: rename d7/d14/f6c to d7/d24/d33/d48/fa1 0 2024-04-05T03:02:44.409 INFO:tasks.workunit.client.1.smithi203.stdout:0/540: stat dd/d17/d1d/d26/d28/l5b 0 2024-04-05T03:02:44.409 INFO:tasks.workunit.client.1.smithi203.stdout:2/738: truncate d5/d10/d1a/fa6 905705 0 2024-04-05T03:02:44.412 INFO:tasks.workunit.client.1.smithi203.stdout:7/695: link d8/d32/d16/d1a/d3d/l58 d8/d32/d5d/d6d/dca/d3f/d73/ld2 0 2024-04-05T03:02:44.413 INFO:tasks.workunit.client.1.smithi203.stdout:0/541: write dd/d17/d29/f3c [1744531,84339] 0 2024-04-05T03:02:44.415 INFO:tasks.workunit.client.1.smithi203.stdout:6/581: link d0/f3 d0/da/d54/fb0 0 2024-04-05T03:02:44.416 INFO:tasks.workunit.client.1.smithi203.stdout:7/696: mknod d8/d32/d5d/d6d/dca/d3f/d73/cd3 0 2024-04-05T03:02:44.417 INFO:tasks.workunit.client.1.smithi203.stdout:0/542: write dd/d17/d1d/d22/d45/d51/d69/f91 [5137178,48004] 0 2024-04-05T03:02:44.422 INFO:tasks.workunit.client.1.smithi203.stdout:1/516: rename d7/d24/l54 to d7/d24/d33/d48/d4b/d6b/la2 0 2024-04-05T03:02:44.422 INFO:tasks.workunit.client.1.smithi203.stdout:1/517: chown d7/d24/d33/l5d 3 1 2024-04-05T03:02:44.423 INFO:tasks.workunit.client.1.smithi203.stdout:2/739: link d5/d10/d1a/f34 d5/d10/d2f/d5d/ffe 0 2024-04-05T03:02:44.424 INFO:tasks.workunit.client.1.smithi203.stdout:7/697: symlink d8/d32/d5d/d6d/dca/d3f/d5e/ld4 0 2024-04-05T03:02:44.425 INFO:tasks.workunit.client.1.smithi203.stdout:0/543: creat dd/d17/d1d/d22/d53/d7f/fa0 x:0 0 0 2024-04-05T03:02:44.425 INFO:tasks.workunit.client.1.smithi203.stdout:1/518: dread d7/d14/f29 [0,4194304] 0 2024-04-05T03:02:44.426 INFO:tasks.workunit.client.1.smithi203.stdout:6/582: creat d0/da/d18/d38/d2a/fb1 x:0 0 0 2024-04-05T03:02:44.426 INFO:tasks.workunit.client.1.smithi203.stdout:2/740: symlink d5/d10/d35/df4/lff 0 2024-04-05T03:02:44.427 INFO:tasks.workunit.client.1.smithi203.stdout:1/519: dread - d7/d14/d2b/d58/f9a zero size 2024-04-05T03:02:44.429 INFO:tasks.workunit.client.1.smithi203.stdout:1/520: readlink d7/d14/d2b/l4e 0 2024-04-05T03:02:44.429 INFO:tasks.workunit.client.1.smithi203.stdout:7/698: getdents d8/d32/d16/d1a/d30 0 2024-04-05T03:02:44.430 INFO:tasks.workunit.client.1.smithi203.stdout:1/521: chown d7/d24/f85 1263829 1 2024-04-05T03:02:44.430 INFO:tasks.workunit.client.1.smithi203.stdout:0/544: truncate dd/d17/d1d/d26/d34/f64 973584 0 2024-04-05T03:02:44.431 INFO:tasks.workunit.client.1.smithi203.stdout:2/741: symlink d5/d10/d35/df4/l100 0 2024-04-05T03:02:44.432 INFO:tasks.workunit.client.1.smithi203.stdout:7/699: symlink d8/d32/d5d/d6d/dca/d9b/ld5 0 2024-04-05T03:02:44.434 INFO:tasks.workunit.client.1.smithi203.stdout:0/545: creat dd/d17/d1d/d22/d53/d55/fa1 x:0 0 0 2024-04-05T03:02:44.437 INFO:tasks.workunit.client.1.smithi203.stdout:0/546: write dd/d17/d1d/d22/d45/d51/f86 [1057857,123038] 0 2024-04-05T03:02:44.438 INFO:tasks.workunit.client.1.smithi203.stdout:1/522: creat d7/d14/d2b/fa3 x:0 0 0 2024-04-05T03:02:44.439 INFO:tasks.workunit.client.1.smithi203.stdout:7/700: creat d8/d32/d5d/dbe/dc9/fd6 x:0 0 0 2024-04-05T03:02:44.441 INFO:tasks.workunit.client.1.smithi203.stdout:6/583: dread d0/da/d54/d53/f59 [0,4194304] 0 2024-04-05T03:02:44.443 INFO:tasks.workunit.client.1.smithi203.stdout:2/742: creat d5/d10/d35/f101 x:0 0 0 2024-04-05T03:02:44.443 INFO:tasks.workunit.client.1.smithi203.stdout:6/584: fsync d0/da/fd 0 2024-04-05T03:02:44.446 INFO:tasks.workunit.client.1.smithi203.stdout:7/701: dwrite d8/f4c [0,4194304] 0 2024-04-05T03:02:44.450 INFO:tasks.workunit.client.1.smithi203.stdout:1/523: creat d7/d14/d83/fa4 x:0 0 0 2024-04-05T03:02:44.453 INFO:tasks.workunit.client.1.smithi203.stdout:1/524: truncate d7/d24/d33/d48/d4b/d6b/f98 14325 0 2024-04-05T03:02:44.454 INFO:tasks.workunit.client.1.smithi203.stdout:6/585: write d0/da/f61 [1104148,9240] 0 2024-04-05T03:02:44.456 INFO:tasks.workunit.client.1.smithi203.stdout:2/743: mkdir d5/d10/d35/d2d/d7c/d29/d82/d102 0 2024-04-05T03:02:44.457 INFO:tasks.workunit.client.1.smithi203.stdout:7/702: creat d8/d32/d16/d1a/d5a/db5/fd7 x:0 0 0 2024-04-05T03:02:44.458 INFO:tasks.workunit.client.1.smithi203.stdout:7/703: dread - d8/d32/d5d/dbe/dc9/fd6 zero size 2024-04-05T03:02:44.461 INFO:tasks.workunit.client.1.smithi203.stdout:6/586: symlink d0/lb2 0 2024-04-05T03:02:44.462 INFO:tasks.workunit.client.1.smithi203.stdout:6/587: dread - d0/da/d18/d38/d2a/fb1 zero size 2024-04-05T03:02:44.462 INFO:tasks.workunit.client.1.smithi203.stdout:1/525: mknod d7/d24/d33/d48/d4b/ca5 0 2024-04-05T03:02:44.464 INFO:tasks.workunit.client.1.smithi203.stdout:7/704: write d8/d32/d16/d1a/d3d/d59/f61 [1016497,15223] 0 2024-04-05T03:02:44.465 INFO:tasks.workunit.client.1.smithi203.stdout:2/744: rename d5/d10/d35/c84 to d5/d10/d35/d73/c103 0 2024-04-05T03:02:44.468 INFO:tasks.workunit.client.1.smithi203.stdout:7/705: mkdir d8/d32/dc3/dd8 0 2024-04-05T03:02:44.469 INFO:tasks.workunit.client.1.smithi203.stdout:1/526: getdents d7/d14/d9d 0 2024-04-05T03:02:44.470 INFO:tasks.workunit.client.1.smithi203.stdout:0/547: truncate dd/d17/d1d/d26/f4f 3910925 0 2024-04-05T03:02:44.471 INFO:tasks.workunit.client.1.smithi203.stdout:6/588: dread d0/da/f43 [0,4194304] 0 2024-04-05T03:02:44.472 INFO:tasks.workunit.client.1.smithi203.stdout:7/706: write d8/d32/d5d/d6d/dca/d3f/d73/f7b [9432868,27334] 0 2024-04-05T03:02:44.473 INFO:tasks.workunit.client.1.smithi203.stdout:0/548: readlink dd/d17/d1d/d26/d34/l60 0 2024-04-05T03:02:44.475 INFO:tasks.workunit.client.1.smithi203.stdout:2/745: dwrite d5/d10/d35/d36/f3b [0,4194304] 0 2024-04-05T03:02:44.476 INFO:tasks.workunit.client.1.smithi203.stdout:7/707: write d8/da/f4e [2358321,25469] 0 2024-04-05T03:02:44.478 INFO:tasks.workunit.client.0.smithi153.stdout:4/259: fsync f7 0 2024-04-05T03:02:44.479 INFO:tasks.workunit.client.0.smithi153.stdout:4/260: fsync da/db/d20/f21 0 2024-04-05T03:02:44.479 INFO:tasks.workunit.client.0.smithi153.stdout:3/183: chown c10 187866466 1 2024-04-05T03:02:44.482 INFO:tasks.workunit.client.1.smithi203.stdout:0/549: creat dd/d17/d1d/d22/d53/d61/fa2 x:0 0 0 2024-04-05T03:02:44.483 INFO:tasks.workunit.client.1.smithi203.stdout:1/527: dwrite d7/d24/d33/d48/d4b/f5c [0,4194304] 0 2024-04-05T03:02:44.486 INFO:tasks.workunit.client.0.smithi153.stdout:6/244: dwrite d4/dc/df/d16/d24/d4e/f5a [0,4194304] 0 2024-04-05T03:02:44.487 INFO:tasks.workunit.client.1.smithi203.stdout:2/746: dwrite d5/d63/fa3 [0,4194304] 0 2024-04-05T03:02:44.490 INFO:tasks.workunit.client.1.smithi203.stdout:7/708: mknod d8/d32/d5d/d6d/dca/d3f/d5e/d63/cd9 0 2024-04-05T03:02:44.493 INFO:tasks.workunit.client.1.smithi203.stdout:1/528: dread d7/f77 [0,4194304] 0 2024-04-05T03:02:44.493 INFO:tasks.workunit.client.1.smithi203.stdout:0/550: mknod dd/d17/d1d/d22/d80/ca3 0 2024-04-05T03:02:44.495 INFO:tasks.workunit.client.1.smithi203.stdout:1/529: write d7/d14/f52 [2552333,58241] 0 2024-04-05T03:02:44.497 INFO:tasks.workunit.client.1.smithi203.stdout:1/530: write d7/d24/d33/d48/d4b/d51/f87 [2478475,119018] 0 2024-04-05T03:02:44.498 INFO:tasks.workunit.client.1.smithi203.stdout:6/589: link d0/da/d18/d38/d2a/d44/d78/f89 d0/da/d54/d53/fb3 0 2024-04-05T03:02:44.500 INFO:tasks.workunit.client.1.smithi203.stdout:7/709: read d8/d32/d5d/d6d/dca/f8c [2134269,17709] 0 2024-04-05T03:02:44.501 INFO:tasks.workunit.client.1.smithi203.stdout:2/747: readlink d5/l16 0 2024-04-05T03:02:44.506 INFO:tasks.workunit.client.1.smithi203.stdout:7/710: dwrite d8/f33 [4194304,4194304] 0 2024-04-05T03:02:44.507 INFO:tasks.workunit.client.1.smithi203.stdout:1/531: fdatasync d7/d24/f34 0 2024-04-05T03:02:44.511 INFO:tasks.workunit.client.1.smithi203.stdout:2/748: unlink d5/d10/c12 0 2024-04-05T03:02:44.515 INFO:tasks.workunit.client.1.smithi203.stdout:2/749: dread d5/d10/d35/d36/db6/f27 [4194304,4194304] 0 2024-04-05T03:02:44.517 INFO:tasks.workunit.client.1.smithi203.stdout:2/750: truncate d5/d66/fde 636527 0 2024-04-05T03:02:44.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/590: truncate d0/da/fd 1225581 0 2024-04-05T03:02:44.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/591: write d0/da/f13 [3566805,43690] 0 2024-04-05T03:02:44.519 INFO:tasks.workunit.client.1.smithi203.stdout:6/592: chown d0/da/d67/d71/f79 331 1 2024-04-05T03:02:44.523 INFO:tasks.workunit.client.0.smithi153.stdout:0/285: sync 2024-04-05T03:02:44.524 INFO:tasks.workunit.client.1.smithi203.stdout:7/711: truncate d8/d32/d5d/f69 3467749 0 2024-04-05T03:02:44.526 INFO:tasks.workunit.client.1.smithi203.stdout:2/751: dread - d5/d10/db8/dab/de1/ff3 zero size 2024-04-05T03:02:44.530 INFO:tasks.workunit.client.1.smithi203.stdout:0/551: getdents dd/d17/d1d/d22/d53/d61 0 2024-04-05T03:02:44.533 INFO:tasks.workunit.client.1.smithi203.stdout:6/593: mkdir d0/db4 0 2024-04-05T03:02:44.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/712: getdents d8/d32/d5d/d6d/dca/d3f/d73/da8 0 2024-04-05T03:02:44.536 INFO:tasks.workunit.client.1.smithi203.stdout:2/752: mkdir d5/d10/d35/d2d/d7c/d29/d67/d104 0 2024-04-05T03:02:44.537 INFO:tasks.workunit.client.1.smithi203.stdout:0/552: dwrite dd/f44 [0,4194304] 0 2024-04-05T03:02:44.539 INFO:tasks.workunit.client.1.smithi203.stdout:2/753: chown d5/d10/d35/d36/db6/d71 2029 1 2024-04-05T03:02:44.540 INFO:tasks.workunit.client.1.smithi203.stdout:0/553: fsync dd/d17/d1d/d22/d53/d55/fa1 0 2024-04-05T03:02:44.541 INFO:tasks.workunit.client.1.smithi203.stdout:4/391: sync 2024-04-05T03:02:44.543 INFO:tasks.workunit.client.1.smithi203.stdout:1/532: dwrite d7/d24/d33/f56 [0,4194304] 0 2024-04-05T03:02:44.543 INFO:tasks.workunit.client.0.smithi153.stdout:4/261: unlink da/c39 0 2024-04-05T03:02:44.545 INFO:tasks.workunit.client.0.smithi153.stdout:4/262: write da/de/f11 [219157,34926] 0 2024-04-05T03:02:44.546 INFO:tasks.workunit.client.0.smithi153.stdout:3/184: creat d1d/d26/d3c/f44 x:0 0 0 2024-04-05T03:02:44.547 INFO:tasks.workunit.client.1.smithi203.stdout:7/713: creat d8/d32/d5d/d6d/daf/fda x:0 0 0 2024-04-05T03:02:44.550 INFO:tasks.workunit.client.1.smithi203.stdout:2/754: dwrite d5/d10/d1a/d57/f6b [0,4194304] 0 2024-04-05T03:02:44.553 INFO:tasks.workunit.client.1.smithi203.stdout:6/594: truncate d0/da/f13 4029418 0 2024-04-05T03:02:44.553 INFO:tasks.workunit.client.0.smithi153.stdout:2/213: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:44.553 INFO:tasks.workunit.client.1.smithi203.stdout:0/554: mknod dd/d17/d1d/d22/d80/ca4 0 2024-04-05T03:02:44.558 INFO:tasks.workunit.client.0.smithi153.stdout:0/286: unlink d9/df/d1e/f35 0 2024-04-05T03:02:44.558 INFO:tasks.workunit.client.1.smithi203.stdout:4/392: creat da/d3c/d4c/f79 x:0 0 0 2024-04-05T03:02:44.559 INFO:tasks.workunit.client.1.smithi203.stdout:0/555: stat dd/d17/d1d/d22/d53/d55/c6e 0 2024-04-05T03:02:44.559 INFO:tasks.workunit.client.1.smithi203.stdout:7/714: dwrite d8/f3e [0,4194304] 0 2024-04-05T03:02:44.560 INFO:tasks.workunit.client.1.smithi203.stdout:1/533: mkdir d7/d14/d2b/d58/d86/d91/da6 0 2024-04-05T03:02:44.561 INFO:tasks.workunit.client.0.smithi153.stdout:7/208: dwrite d2/da/d11/f18 [0,4194304] 0 2024-04-05T03:02:44.562 INFO:tasks.workunit.client.0.smithi153.stdout:3/185: creat d1d/d32/f45 x:0 0 0 2024-04-05T03:02:44.563 INFO:tasks.workunit.client.0.smithi153.stdout:0/287: dread d9/f1a [0,4194304] 0 2024-04-05T03:02:44.564 INFO:tasks.workunit.client.0.smithi153.stdout:7/209: write d2/da/d11/d49/f3e [192459,101679] 0 2024-04-05T03:02:44.564 INFO:tasks.workunit.client.0.smithi153.stdout:6/245: getdents d4/d38/d4c 0 2024-04-05T03:02:44.565 INFO:tasks.workunit.client.0.smithi153.stdout:2/214: fdatasync d9/da/d37/f3d 0 2024-04-05T03:02:44.566 INFO:tasks.workunit.client.1.smithi203.stdout:4/393: chown da/d29/c4b 11883389 1 2024-04-05T03:02:44.567 INFO:tasks.workunit.client.0.smithi153.stdout:3/186: chown d1d/d38/c3b 171079 1 2024-04-05T03:02:44.569 INFO:tasks.workunit.client.0.smithi153.stdout:3/187: truncate d1d/d27/d2b/f2d 880509 0 2024-04-05T03:02:44.572 INFO:tasks.workunit.client.1.smithi203.stdout:4/394: creat da/d29/d2e/f7a x:0 0 0 2024-04-05T03:02:44.575 INFO:tasks.workunit.client.1.smithi203.stdout:6/595: getdents d0/da/d18 0 2024-04-05T03:02:44.577 INFO:tasks.workunit.client.0.smithi153.stdout:3/188: creat d1d/d26/d3c/f46 x:0 0 0 2024-04-05T03:02:44.578 INFO:tasks.workunit.client.0.smithi153.stdout:4/263: dread da/de/f11 [0,4194304] 0 2024-04-05T03:02:44.579 INFO:tasks.workunit.client.0.smithi153.stdout:6/246: chown d4/dc/df/d16/d33/l59 67735772 1 2024-04-05T03:02:44.581 INFO:tasks.workunit.client.0.smithi153.stdout:0/288: dread d9/d14/f3f [0,4194304] 0 2024-04-05T03:02:44.581 INFO:tasks.workunit.client.1.smithi203.stdout:6/596: link d0/da/c88 d0/da/d67/d71/d8c/cb5 0 2024-04-05T03:02:44.582 INFO:tasks.workunit.client.0.smithi153.stdout:2/215: getdents d9/da 0 2024-04-05T03:02:44.583 INFO:tasks.workunit.client.0.smithi153.stdout:7/210: getdents d2 0 2024-04-05T03:02:44.586 INFO:tasks.workunit.client.1.smithi203.stdout:6/597: rename d0/da/d67/d71/c97 to d0/da/d54/d82/cb6 0 2024-04-05T03:02:44.587 INFO:tasks.workunit.client.0.smithi153.stdout:2/216: fsync d9/da/f15 0 2024-04-05T03:02:44.587 INFO:tasks.workunit.client.0.smithi153.stdout:7/211: chown d2/f9 502371321 1 2024-04-05T03:02:44.590 INFO:tasks.workunit.client.0.smithi153.stdout:4/264: dwrite da/db/d20/f32 [0,4194304] 0 2024-04-05T03:02:44.592 INFO:tasks.workunit.client.0.smithi153.stdout:3/189: getdents d1d/d27 0 2024-04-05T03:02:44.593 INFO:tasks.workunit.client.0.smithi153.stdout:0/289: rmdir d9/df/d4d 0 2024-04-05T03:02:44.595 INFO:tasks.workunit.client.0.smithi153.stdout:3/190: write d1d/f28 [830539,119435] 0 2024-04-05T03:02:44.597 INFO:tasks.workunit.client.1.smithi203.stdout:1/534: dread d7/f80 [0,4194304] 0 2024-04-05T03:02:44.598 INFO:tasks.workunit.client.0.smithi153.stdout:2/217: dwrite d9/da/d19/d2e/f42 [0,4194304] 0 2024-04-05T03:02:44.602 INFO:tasks.workunit.client.1.smithi203.stdout:1/535: symlink d7/d24/d33/d48/d4b/d51/la7 0 2024-04-05T03:02:44.604 INFO:tasks.workunit.client.1.smithi203.stdout:7/715: write d8/d32/d5d/d6d/dca/f8c [4112540,46788] 0 2024-04-05T03:02:44.604 INFO:tasks.workunit.client.1.smithi203.stdout:0/556: truncate dd/d17/f79 244731 0 2024-04-05T03:02:44.604 INFO:tasks.workunit.client.0.smithi153.stdout:4/265: rename da/db/f13 to da/db/d20/d45/f4e 0 2024-04-05T03:02:44.605 INFO:tasks.workunit.client.1.smithi203.stdout:2/755: dread d5/d10/d35/d2d/d7c/d72/d8f/fe4 [0,4194304] 0 2024-04-05T03:02:44.605 INFO:tasks.workunit.client.1.smithi203.stdout:4/395: write da/d29/d2e/f45 [917396,105532] 0 2024-04-05T03:02:44.605 INFO:tasks.workunit.client.1.smithi203.stdout:1/536: write d7/d14/d2b/d58/f9a [260020,50024] 0 2024-04-05T03:02:44.606 INFO:tasks.workunit.client.1.smithi203.stdout:0/557: readlink dd/d17/d1d/d22/d53/d7f/l84 0 2024-04-05T03:02:44.607 INFO:tasks.workunit.client.1.smithi203.stdout:7/716: symlink d8/da/d72/ldb 0 2024-04-05T03:02:44.607 INFO:tasks.workunit.client.0.smithi153.stdout:3/191: creat d1d/d26/f47 x:0 0 0 2024-04-05T03:02:44.609 INFO:tasks.workunit.client.1.smithi203.stdout:1/537: read - d7/f99 zero size 2024-04-05T03:02:44.609 INFO:tasks.workunit.client.0.smithi153.stdout:3/192: truncate d1d/d26/f47 205823 0 2024-04-05T03:02:44.609 INFO:tasks.workunit.client.0.smithi153.stdout:2/218: symlink d9/da/d36/d38/l47 0 2024-04-05T03:02:44.609 INFO:tasks.workunit.client.1.smithi203.stdout:4/396: chown da/d29/d6a/f6c 1182260122 1 2024-04-05T03:02:44.610 INFO:tasks.workunit.client.1.smithi203.stdout:6/598: rename d0/da/d67 to d0/da/d54/d4c/da1/db7 0 2024-04-05T03:02:44.611 INFO:tasks.workunit.client.0.smithi153.stdout:0/290: mknod d9/df/d1e/d45/c51 0 2024-04-05T03:02:44.612 INFO:tasks.workunit.client.1.smithi203.stdout:7/717: creat d8/d32/d16/d92/fdc x:0 0 0 2024-04-05T03:02:44.613 INFO:tasks.workunit.client.0.smithi153.stdout:0/291: fdatasync d9/d14/f40 0 2024-04-05T03:02:44.613 INFO:tasks.workunit.client.1.smithi203.stdout:0/558: read dd/d17/d1d/d26/f67 [8610,72829] 0 2024-04-05T03:02:44.614 INFO:tasks.workunit.client.0.smithi153.stdout:6/247: dwrite d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:44.614 INFO:tasks.workunit.client.1.smithi203.stdout:4/397: mknod da/d29/d2e/c7b 0 2024-04-05T03:02:44.614 INFO:tasks.workunit.client.1.smithi203.stdout:7/718: chown d8/da/cb 552 1 2024-04-05T03:02:44.615 INFO:tasks.workunit.client.1.smithi203.stdout:2/756: truncate d5/d10/d2f/d5d/ff9 986348 0 2024-04-05T03:02:44.615 INFO:tasks.workunit.client.0.smithi153.stdout:0/292: chown f3 4 1 2024-04-05T03:02:44.616 INFO:tasks.workunit.client.0.smithi153.stdout:3/193: mknod d1d/d26/c48 0 2024-04-05T03:02:44.617 INFO:tasks.workunit.client.0.smithi153.stdout:2/219: rmdir d9/da 39 2024-04-05T03:02:44.618 INFO:tasks.workunit.client.0.smithi153.stdout:3/194: dread - d1d/d27/d2b/f2f zero size 2024-04-05T03:02:44.619 INFO:tasks.workunit.client.0.smithi153.stdout:0/293: creat d9/df/d1e/f52 x:0 0 0 2024-04-05T03:02:44.620 INFO:tasks.workunit.client.1.smithi203.stdout:1/538: truncate d7/d24/f34 4306771 0 2024-04-05T03:02:44.621 INFO:tasks.workunit.client.1.smithi203.stdout:6/599: unlink d0/da/d18/f1d 0 2024-04-05T03:02:44.622 INFO:tasks.workunit.client.0.smithi153.stdout:3/195: chown d1d/c20 17916 1 2024-04-05T03:02:44.622 INFO:tasks.workunit.client.0.smithi153.stdout:3/196: fdatasync d1d/f1e 0 2024-04-05T03:02:44.623 INFO:tasks.workunit.client.0.smithi153.stdout:6/248: dread d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:44.623 INFO:tasks.workunit.client.0.smithi153.stdout:4/266: fsync f8 0 2024-04-05T03:02:44.628 INFO:tasks.workunit.client.1.smithi203.stdout:7/719: dwrite d8/d32/dc3/dc8/fd1 [0,4194304] 0 2024-04-05T03:02:44.630 INFO:tasks.workunit.client.0.smithi153.stdout:6/249: mknod d4/dc/df/d16/d17/c5c 0 2024-04-05T03:02:44.633 INFO:tasks.workunit.client.1.smithi203.stdout:6/600: dwrite d0/da/d54/d4c/da1/db7/d71/f79 [0,4194304] 0 2024-04-05T03:02:44.634 INFO:tasks.workunit.client.0.smithi153.stdout:2/220: rename d9/da/d10 to d9/da/d19/d2e/d48 0 2024-04-05T03:02:44.635 INFO:tasks.workunit.client.1.smithi203.stdout:6/601: write d0/da/d54/d53/f59 [470467,1144] 0 2024-04-05T03:02:44.635 INFO:tasks.workunit.client.1.smithi203.stdout:4/398: creat da/d3c/d4c/f7c x:0 0 0 2024-04-05T03:02:44.638 INFO:tasks.workunit.client.0.smithi153.stdout:0/294: dwrite d9/db/f49 [0,4194304] 0 2024-04-05T03:02:44.639 INFO:tasks.workunit.client.1.smithi203.stdout:2/757: creat d5/d10/d35/d2d/d7c/d29/d82/f105 x:0 0 0 2024-04-05T03:02:44.639 INFO:tasks.workunit.client.1.smithi203.stdout:1/539: rename d7/f78 to d7/d14/d2b/d58/fa8 0 2024-04-05T03:02:44.641 INFO:tasks.workunit.client.0.smithi153.stdout:6/250: mknod d4/dc/df/d16/d36/d52/c5d 0 2024-04-05T03:02:44.643 INFO:tasks.workunit.client.1.smithi203.stdout:7/720: mkdir d8/d32/d5d/dbe/dc9/ddd 0 2024-04-05T03:02:44.646 INFO:tasks.workunit.client.1.smithi203.stdout:0/559: creat dd/d17/fa5 x:0 0 0 2024-04-05T03:02:44.646 INFO:tasks.workunit.client.0.smithi153.stdout:0/295: dwrite d9/db/f37 [0,4194304] 0 2024-04-05T03:02:44.652 INFO:tasks.workunit.client.1.smithi203.stdout:0/560: dwrite dd/d17/d1d/d22/f77 [0,4194304] 0 2024-04-05T03:02:44.653 INFO:tasks.workunit.client.1.smithi203.stdout:7/721: mknod d8/d32/d16/d35/d64/cde 0 2024-04-05T03:02:44.654 INFO:tasks.workunit.client.1.smithi203.stdout:4/399: creat da/d29/d2e/d53/d78/f7d x:0 0 0 2024-04-05T03:02:44.655 INFO:tasks.workunit.client.1.smithi203.stdout:7/722: fdatasync d8/d32/d5d/d6d/dca/d3f/d5e/d63/f75 0 2024-04-05T03:02:44.657 INFO:tasks.workunit.client.1.smithi203.stdout:5/576: sync 2024-04-05T03:02:44.657 INFO:tasks.workunit.client.1.smithi203.stdout:8/564: sync 2024-04-05T03:02:44.658 INFO:tasks.workunit.client.0.smithi153.stdout:0/296: dwrite d9/db/f4b [0,4194304] 0 2024-04-05T03:02:44.659 INFO:tasks.workunit.client.1.smithi203.stdout:1/540: symlink d7/d24/la9 0 2024-04-05T03:02:44.660 INFO:tasks.workunit.client.1.smithi203.stdout:5/577: read d3/f18 [1484789,10869] 0 2024-04-05T03:02:44.660 INFO:tasks.workunit.client.1.smithi203.stdout:1/541: fsync d7/d14/d83/fa4 0 2024-04-05T03:02:44.664 INFO:tasks.workunit.client.0.smithi153.stdout:6/251: mkdir d4/dc/df/d16/d24/d5e 0 2024-04-05T03:02:44.664 INFO:tasks.workunit.client.0.smithi153.stdout:0/297: dread d9/df/f39 [0,4194304] 0 2024-04-05T03:02:44.666 INFO:tasks.workunit.client.0.smithi153.stdout:6/252: chown d4/dc/df/f39 3353522 1 2024-04-05T03:02:44.666 INFO:tasks.workunit.client.0.smithi153.stdout:4/267: write da/db/d20/f29 [7976,57904] 0 2024-04-05T03:02:44.669 INFO:tasks.workunit.client.0.smithi153.stdout:0/298: creat d9/d14/f53 x:0 0 0 2024-04-05T03:02:44.670 INFO:tasks.workunit.client.0.smithi153.stdout:6/253: rmdir d4/dc/df/d16/d24/d4e 39 2024-04-05T03:02:44.673 INFO:tasks.workunit.client.0.smithi153.stdout:4/268: creat da/db/d20/f4f x:0 0 0 2024-04-05T03:02:44.673 INFO:tasks.workunit.client.1.smithi203.stdout:1/542: mknod d7/d79/caa 0 2024-04-05T03:02:44.677 INFO:tasks.workunit.client.0.smithi153.stdout:6/254: mkdir d4/dc/df/d16/d5f 0 2024-04-05T03:02:44.678 INFO:tasks.workunit.client.0.smithi153.stdout:0/299: link d9/df/f15 d9/df/d1e/f54 0 2024-04-05T03:02:44.678 INFO:tasks.workunit.client.0.smithi153.stdout:6/255: fdatasync d4/dc/df/d16/d24/f29 0 2024-04-05T03:02:44.680 INFO:tasks.workunit.client.1.smithi203.stdout:5/578: truncate d3/f36 2085343 0 2024-04-05T03:02:44.682 INFO:tasks.workunit.client.0.smithi153.stdout:4/269: creat da/db/d20/d35/d38/d3e/f50 x:0 0 0 2024-04-05T03:02:44.683 INFO:tasks.workunit.client.0.smithi153.stdout:6/256: dread d4/fb [0,4194304] 0 2024-04-05T03:02:44.684 INFO:tasks.workunit.client.0.smithi153.stdout:0/300: link d9/c10 d9/df/d12/c55 0 2024-04-05T03:02:44.687 INFO:tasks.workunit.client.1.smithi203.stdout:1/543: symlink d7/d14/lab 0 2024-04-05T03:02:44.687 INFO:tasks.workunit.client.0.smithi153.stdout:4/270: mknod da/db/d20/d45/d3b/c51 0 2024-04-05T03:02:44.690 INFO:tasks.workunit.client.1.smithi203.stdout:8/565: getdents d1/d12/d1a/d6d 0 2024-04-05T03:02:44.693 INFO:tasks.workunit.client.1.smithi203.stdout:8/566: fsync d1/d6/f45 0 2024-04-05T03:02:44.698 INFO:tasks.workunit.client.0.smithi153.stdout:0/301: dwrite d9/df/d1d/f24 [0,4194304] 0 2024-04-05T03:02:44.698 INFO:tasks.workunit.client.0.smithi153.stdout:6/257: dwrite d4/dc/df/d16/d17/f3a [0,4194304] 0 2024-04-05T03:02:44.700 INFO:tasks.workunit.client.1.smithi203.stdout:2/758: dread d5/d10/d35/d36/db6/f3f [0,4194304] 0 2024-04-05T03:02:44.700 INFO:tasks.workunit.client.1.smithi203.stdout:2/759: fsync d5/d63/f75 0 2024-04-05T03:02:44.701 INFO:tasks.workunit.client.0.smithi153.stdout:4/271: link da/d2a/c42 da/db/d20/d35/d38/d3e/c52 0 2024-04-05T03:02:44.703 INFO:tasks.workunit.client.0.smithi153.stdout:0/302: mkdir d9/df/d1d/d47/d56 0 2024-04-05T03:02:44.705 INFO:tasks.workunit.client.0.smithi153.stdout:0/303: write d9/df/f46 [375097,77537] 0 2024-04-05T03:02:44.710 INFO:tasks.workunit.client.0.smithi153.stdout:2/221: dread d9/f11 [0,4194304] 0 2024-04-05T03:02:44.713 INFO:tasks.workunit.client.0.smithi153.stdout:0/304: creat d9/d14/f57 x:0 0 0 2024-04-05T03:02:44.713 INFO:tasks.workunit.client.0.smithi153.stdout:4/272: dread f4 [0,4194304] 0 2024-04-05T03:02:44.713 INFO:tasks.workunit.client.0.smithi153.stdout:0/305: readlink d9/db/l31 0 2024-04-05T03:02:44.714 INFO:tasks.workunit.client.0.smithi153.stdout:0/306: read d9/df/d1e/f25 [2042033,4090] 0 2024-04-05T03:02:44.715 INFO:tasks.workunit.client.0.smithi153.stdout:4/273: mknod da/db/d20/d35/d38/c53 0 2024-04-05T03:02:44.716 INFO:tasks.workunit.client.1.smithi203.stdout:8/567: dread d1/d12/f31 [0,4194304] 0 2024-04-05T03:02:44.716 INFO:tasks.workunit.client.0.smithi153.stdout:2/222: rmdir d9/da/d36/d38 39 2024-04-05T03:02:44.719 INFO:tasks.workunit.client.0.smithi153.stdout:2/223: readlink d9/da/d19/l1b 0 2024-04-05T03:02:44.723 INFO:tasks.workunit.client.0.smithi153.stdout:0/307: rmdir d9/df/d1d/d47/d56 0 2024-04-05T03:02:44.724 INFO:tasks.workunit.client.1.smithi203.stdout:1/544: link d7/lb d7/d14/d2b/d58/lac 0 2024-04-05T03:02:44.725 INFO:tasks.workunit.client.1.smithi203.stdout:8/568: mknod d1/d6/d2e/d5c/c9b 0 2024-04-05T03:02:44.727 INFO:tasks.workunit.client.0.smithi153.stdout:4/274: dwrite da/de/f1e [0,4194304] 0 2024-04-05T03:02:44.727 INFO:tasks.workunit.client.0.smithi153.stdout:2/224: chown d9/da/d19/d2e/d48/l3f 4892060 1 2024-04-05T03:02:44.727 INFO:tasks.workunit.client.1.smithi203.stdout:2/760: mknod d5/d10/d35/d36/d41/c106 0 2024-04-05T03:02:44.728 INFO:tasks.workunit.client.1.smithi203.stdout:1/545: write d7/d14/d2b/f5a [2444121,11137] 0 2024-04-05T03:02:44.728 INFO:tasks.workunit.client.1.smithi203.stdout:6/602: dwrite d0/da/d54/d53/fb3 [0,4194304] 0 2024-04-05T03:02:44.729 INFO:tasks.workunit.client.0.smithi153.stdout:0/308: mkdir d9/df/d1e/d45/d58 0 2024-04-05T03:02:44.729 INFO:tasks.workunit.client.1.smithi203.stdout:2/761: stat d5/d10/d35/d2d/d7c/d29/d82 0 2024-04-05T03:02:44.730 INFO:tasks.workunit.client.1.smithi203.stdout:0/561: write dd/d17/d1d/d26/d34/f64 [494128,46547] 0 2024-04-05T03:02:44.731 INFO:tasks.workunit.client.0.smithi153.stdout:2/225: write d9/da/fb [3189173,121469] 0 2024-04-05T03:02:44.732 INFO:tasks.workunit.client.1.smithi203.stdout:0/562: write dd/d17/d1d/d26/d28/f8b [499100,275] 0 2024-04-05T03:02:44.733 INFO:tasks.workunit.client.0.smithi153.stdout:2/226: chown d9/da/d19/f22 463054 1 2024-04-05T03:02:44.735 INFO:tasks.workunit.client.1.smithi203.stdout:8/569: rename d1/d12/d2f/d1f/d83 to d1/d12/d17/d47/d53/d69/d9c 0 2024-04-05T03:02:44.735 INFO:tasks.workunit.client.1.smithi203.stdout:2/762: rmdir d5/d10/db8/dbe 39 2024-04-05T03:02:44.736 INFO:tasks.workunit.client.1.smithi203.stdout:8/570: readlink d1/d12/d2f/l4c 0 2024-04-05T03:02:44.736 INFO:tasks.workunit.client.0.smithi153.stdout:4/275: dread da/db/d20/f21 [0,4194304] 0 2024-04-05T03:02:44.744 INFO:tasks.workunit.client.0.smithi153.stdout:0/309: link d9/df/f18 d9/df/f59 0 2024-04-05T03:02:44.745 INFO:tasks.workunit.client.1.smithi203.stdout:7/723: dwrite d8/d32/d5d/d6d/dca/d3f/d5e/d77/f87 [0,4194304] 0 2024-04-05T03:02:44.745 INFO:tasks.workunit.client.0.smithi153.stdout:1/203: sync 2024-04-05T03:02:44.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/203: sync 2024-04-05T03:02:44.746 INFO:tasks.workunit.client.0.smithi153.stdout:9/143: sync 2024-04-05T03:02:44.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/181: sync 2024-04-05T03:02:44.746 INFO:tasks.workunit.client.1.smithi203.stdout:1/546: creat d7/d24/d33/fad x:0 0 0 2024-04-05T03:02:44.747 INFO:tasks.workunit.client.1.smithi203.stdout:7/724: write d8/d32/d5d/d6d/f96 [1693888,47647] 0 2024-04-05T03:02:44.748 INFO:tasks.workunit.client.0.smithi153.stdout:6/258: dread d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:44.748 INFO:tasks.workunit.client.1.smithi203.stdout:8/571: mknod d1/d12/d17/d47/d53/d69/d9c/c9d 0 2024-04-05T03:02:44.756 INFO:tasks.workunit.client.0.smithi153.stdout:9/144: dread d1/f1f [0,4194304] 0 2024-04-05T03:02:44.758 INFO:tasks.workunit.client.0.smithi153.stdout:8/204: dread d7/d13/d19/f22 [0,4194304] 0 2024-04-05T03:02:44.759 INFO:tasks.workunit.client.0.smithi153.stdout:5/182: creat d0/f44 x:0 0 0 2024-04-05T03:02:44.759 INFO:tasks.workunit.client.1.smithi203.stdout:7/725: mknod d8/d32/d5d/d95/cdf 0 2024-04-05T03:02:44.759 INFO:tasks.workunit.client.1.smithi203.stdout:8/572: symlink d1/d4b/d68/l9e 0 2024-04-05T03:02:44.760 INFO:tasks.workunit.client.1.smithi203.stdout:8/573: chown d1/d12/d17/c8e 212 1 2024-04-05T03:02:44.761 INFO:tasks.workunit.client.0.smithi153.stdout:0/310: rename d9/db/c2f to d9/db/d42/c5a 0 2024-04-05T03:02:44.761 INFO:tasks.workunit.client.0.smithi153.stdout:6/259: dwrite d4/dc/df/f56 [0,4194304] 0 2024-04-05T03:02:44.763 INFO:tasks.workunit.client.0.smithi153.stdout:6/260: read d4/f20 [2256458,14855] 0 2024-04-05T03:02:44.766 INFO:tasks.workunit.client.0.smithi153.stdout:8/205: rename d7/d13/d19/d31/f3e to d7/d2a/f44 0 2024-04-05T03:02:44.766 INFO:tasks.workunit.client.0.smithi153.stdout:0/311: write d9/d14/f19 [649145,91894] 0 2024-04-05T03:02:44.768 INFO:tasks.workunit.client.0.smithi153.stdout:9/145: dread d1/d10/d16/f1a [0,4194304] 0 2024-04-05T03:02:44.768 INFO:tasks.workunit.client.0.smithi153.stdout:2/227: truncate d9/da/d19/f22 215084 0 2024-04-05T03:02:44.768 INFO:tasks.workunit.client.0.smithi153.stdout:4/276: write da/db/f23 [2343623,15062] 0 2024-04-05T03:02:44.769 INFO:tasks.workunit.client.0.smithi153.stdout:5/183: mknod d0/c45 0 2024-04-05T03:02:44.770 INFO:tasks.workunit.client.0.smithi153.stdout:2/228: fsync d9/da/d19/d2e/d48/f1a 0 2024-04-05T03:02:44.771 INFO:tasks.workunit.client.0.smithi153.stdout:0/312: mknod d9/df/d1e/c5b 0 2024-04-05T03:02:44.772 INFO:tasks.workunit.client.0.smithi153.stdout:2/229: write d9/da/d19/f1f [11836965,21283] 0 2024-04-05T03:02:44.772 INFO:tasks.workunit.client.0.smithi153.stdout:0/313: fsync d9/db/f37 0 2024-04-05T03:02:44.773 INFO:tasks.workunit.client.0.smithi153.stdout:4/277: symlink da/de/l54 0 2024-04-05T03:02:44.774 INFO:tasks.workunit.client.0.smithi153.stdout:9/146: creat d1/d10/d16/f38 x:0 0 0 2024-04-05T03:02:44.776 INFO:tasks.workunit.client.0.smithi153.stdout:6/261: creat d4/dc/df/d16/d24/d5e/f60 x:0 0 0 2024-04-05T03:02:44.778 INFO:tasks.workunit.client.0.smithi153.stdout:2/230: write d9/da/fc [280720,98942] 0 2024-04-05T03:02:44.783 INFO:tasks.workunit.client.0.smithi153.stdout:5/184: rename d0/fd to d0/f46 0 2024-04-05T03:02:44.784 INFO:tasks.workunit.client.0.smithi153.stdout:6/262: chown d4/dc/df/d16/d17/f3b 1 1 2024-04-05T03:02:44.785 INFO:tasks.workunit.client.0.smithi153.stdout:4/278: dwrite da/db/d20/d45/f43 [0,4194304] 0 2024-04-05T03:02:44.786 INFO:tasks.workunit.client.0.smithi153.stdout:0/314: link d9/df/d1e/c5b d9/df/d12/c5c 0 2024-04-05T03:02:44.788 INFO:tasks.workunit.client.0.smithi153.stdout:9/147: dwrite d1/d9/d1b/f30 [4194304,4194304] 0 2024-04-05T03:02:44.789 INFO:tasks.workunit.client.0.smithi153.stdout:5/185: fdatasync d0/f44 0 2024-04-05T03:02:44.790 INFO:tasks.workunit.client.1.smithi203.stdout:8/574: mknod d1/d12/d17/d10/c9f 0 2024-04-05T03:02:44.790 INFO:tasks.workunit.client.1.smithi203.stdout:7/726: creat d8/d32/d5d/dbe/dc9/fe0 x:0 0 0 2024-04-05T03:02:44.791 INFO:tasks.workunit.client.1.smithi203.stdout:8/575: write d1/f34 [3851889,62186] 0 2024-04-05T03:02:44.792 INFO:tasks.workunit.client.1.smithi203.stdout:7/727: write d8/da/d72/fcb [705604,124145] 0 2024-04-05T03:02:44.792 INFO:tasks.workunit.client.0.smithi153.stdout:2/231: read d9/da/fc [3582179,27956] 0 2024-04-05T03:02:44.793 INFO:tasks.workunit.client.1.smithi203.stdout:8/576: stat d1/d4b/d68/f95 0 2024-04-05T03:02:44.794 INFO:tasks.workunit.client.0.smithi153.stdout:0/315: read d9/db/f2d [510282,97086] 0 2024-04-05T03:02:44.796 INFO:tasks.workunit.client.0.smithi153.stdout:0/316: fsync d9/df/d1d/f24 0 2024-04-05T03:02:44.797 INFO:tasks.workunit.client.0.smithi153.stdout:5/186: write d0/d9/f20 [226831,129028] 0 2024-04-05T03:02:44.798 INFO:tasks.workunit.client.0.smithi153.stdout:6/263: symlink d4/dc/df/d16/d33/l61 0 2024-04-05T03:02:44.802 INFO:tasks.workunit.client.1.smithi203.stdout:8/577: dwrite d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:44.806 INFO:tasks.workunit.client.1.smithi203.stdout:1/547: getdents d7 0 2024-04-05T03:02:44.807 INFO:tasks.workunit.client.1.smithi203.stdout:8/578: mkdir d1/d12/d1a/da0 0 2024-04-05T03:02:44.809 INFO:tasks.workunit.client.1.smithi203.stdout:1/548: dread - d7/d24/d33/d48/d4b/d51/f7f zero size 2024-04-05T03:02:44.809 INFO:tasks.workunit.client.0.smithi153.stdout:9/148: dwrite d1/d9/d1b/f28 [0,4194304] 0 2024-04-05T03:02:44.811 INFO:tasks.workunit.client.1.smithi203.stdout:8/579: write d1/d12/d17/d47/d53/f96 [955318,119720] 0 2024-04-05T03:02:44.811 INFO:tasks.workunit.client.0.smithi153.stdout:4/279: symlink da/db/d20/d3c/l55 0 2024-04-05T03:02:44.812 INFO:tasks.workunit.client.0.smithi153.stdout:9/149: write d1/d10/d16/d22/d2c/f31 [2562827,51664] 0 2024-04-05T03:02:44.814 INFO:tasks.workunit.client.1.smithi203.stdout:0/563: dwrite dd/d17/d1d/d26/d34/f35 [0,4194304] 0 2024-04-05T03:02:44.818 INFO:tasks.workunit.client.0.smithi153.stdout:6/264: creat d4/dc/df/d16/d24/d5e/f62 x:0 0 0 2024-04-05T03:02:44.819 INFO:tasks.workunit.client.0.smithi153.stdout:5/187: write d0/d9/f1e [2954339,124687] 0 2024-04-05T03:02:44.821 INFO:tasks.workunit.client.0.smithi153.stdout:4/280: mkdir da/de/d56 0 2024-04-05T03:02:44.821 INFO:tasks.workunit.client.1.smithi203.stdout:1/549: dwrite d7/d24/d33/f36 [8388608,4194304] 0 2024-04-05T03:02:44.826 INFO:tasks.workunit.client.1.smithi203.stdout:1/550: fdatasync d7/d14/d2b/d58/fa8 0 2024-04-05T03:02:44.827 INFO:tasks.workunit.client.0.smithi153.stdout:2/232: creat d9/da/f49 x:0 0 0 2024-04-05T03:02:44.827 INFO:tasks.workunit.client.0.smithi153.stdout:5/188: write d0/d9/f1e [1058346,26033] 0 2024-04-05T03:02:44.831 INFO:tasks.workunit.client.0.smithi153.stdout:8/206: dwrite d7/d13/d19/d31/f42 [0,4194304] 0 2024-04-05T03:02:44.833 INFO:tasks.workunit.client.0.smithi153.stdout:0/317: dwrite d9/d14/f23 [0,4194304] 0 2024-04-05T03:02:44.834 INFO:tasks.workunit.client.1.smithi203.stdout:0/564: dread dd/d17/d1d/d26/f4f [0,4194304] 0 2024-04-05T03:02:44.834 INFO:tasks.workunit.client.1.smithi203.stdout:1/551: creat d7/d14/d2b/fae x:0 0 0 2024-04-05T03:02:44.835 INFO:tasks.workunit.client.1.smithi203.stdout:0/565: chown dd/d17/d1d/d22/d53/d55/fa1 248 1 2024-04-05T03:02:44.835 INFO:tasks.workunit.client.0.smithi153.stdout:6/265: write d4/dc/df/d16/f21 [1779890,18812] 0 2024-04-05T03:02:44.837 INFO:tasks.workunit.client.1.smithi203.stdout:1/552: write d7/d24/d33/d48/fa1 [1672355,122464] 0 2024-04-05T03:02:44.837 INFO:tasks.workunit.client.0.smithi153.stdout:4/281: creat da/de/d28/d2d/f57 x:0 0 0 2024-04-05T03:02:44.839 INFO:tasks.workunit.client.1.smithi203.stdout:0/566: rmdir dd/d17/d1d/d22/d45/d51 39 2024-04-05T03:02:44.841 INFO:tasks.workunit.client.0.smithi153.stdout:8/207: mknod d7/d13/d19/d1b/c45 0 2024-04-05T03:02:44.842 INFO:tasks.workunit.client.0.smithi153.stdout:5/189: symlink d0/d2c/d3f/l47 0 2024-04-05T03:02:44.843 INFO:tasks.workunit.client.1.smithi203.stdout:1/553: read d7/d24/d33/f64 [722487,43495] 0 2024-04-05T03:02:44.844 INFO:tasks.workunit.client.0.smithi153.stdout:4/282: creat da/db/d20/d35/f58 x:0 0 0 2024-04-05T03:02:44.845 INFO:tasks.workunit.client.1.smithi203.stdout:0/567: symlink dd/d17/d1d/d26/d8c/d6d/la6 0 2024-04-05T03:02:44.846 INFO:tasks.workunit.client.0.smithi153.stdout:6/266: truncate d4/dc/df/d16/d24/d4e/f5a 1610987 0 2024-04-05T03:02:44.851 INFO:tasks.workunit.client.0.smithi153.stdout:6/267: fdatasync d4/dc/df/f13 0 2024-04-05T03:02:44.851 INFO:tasks.workunit.client.0.smithi153.stdout:8/208: write f5 [5169463,40088] 0 2024-04-05T03:02:44.851 INFO:tasks.workunit.client.0.smithi153.stdout:5/190: rmdir d0/de/d37/d39 39 2024-04-05T03:02:44.854 INFO:tasks.workunit.client.1.smithi203.stdout:0/568: dwrite dd/d17/d29/f57 [0,4194304] 0 2024-04-05T03:02:44.855 INFO:tasks.workunit.client.0.smithi153.stdout:4/283: dwrite f7 [4194304,4194304] 0 2024-04-05T03:02:44.857 INFO:tasks.workunit.client.0.smithi153.stdout:4/284: write da/de/f1e [3304528,98339] 0 2024-04-05T03:02:44.858 INFO:tasks.workunit.client.0.smithi153.stdout:9/150: link d1/d10/f19 d1/d10/f39 0 2024-04-05T03:02:44.863 INFO:tasks.workunit.client.0.smithi153.stdout:8/209: mkdir d7/d46 0 2024-04-05T03:02:44.865 INFO:tasks.workunit.client.0.smithi153.stdout:5/191: stat d0/f5 0 2024-04-05T03:02:44.866 INFO:tasks.workunit.client.1.smithi203.stdout:0/569: dread dd/d17/d29/f3c [0,4194304] 0 2024-04-05T03:02:44.867 INFO:tasks.workunit.client.0.smithi153.stdout:9/151: write d1/d10/f39 [1623005,100793] 0 2024-04-05T03:02:44.867 INFO:tasks.workunit.client.0.smithi153.stdout:6/268: unlink d4/dc/df/d16/d17/c1a 0 2024-04-05T03:02:44.868 INFO:tasks.workunit.client.0.smithi153.stdout:4/285: link da/f17 da/db/d20/d35/d38/f59 0 2024-04-05T03:02:44.870 INFO:tasks.workunit.client.1.smithi203.stdout:3/474: sync 2024-04-05T03:02:44.870 INFO:tasks.workunit.client.1.smithi203.stdout:9/522: sync 2024-04-05T03:02:44.870 INFO:tasks.workunit.client.1.smithi203.stdout:7/728: truncate d8/da/f94 146510 0 2024-04-05T03:02:44.871 INFO:tasks.workunit.client.0.smithi153.stdout:5/192: unlink d0/f46 0 2024-04-05T03:02:44.872 INFO:tasks.workunit.client.0.smithi153.stdout:5/193: dread - d0/f44 zero size 2024-04-05T03:02:44.873 INFO:tasks.workunit.client.1.smithi203.stdout:0/570: dread dd/d17/d1d/d22/f77 [0,4194304] 0 2024-04-05T03:02:44.873 INFO:tasks.workunit.client.0.smithi153.stdout:4/286: truncate da/db/d20/d35/d38/d3e/f50 629128 0 2024-04-05T03:02:44.874 INFO:tasks.workunit.client.0.smithi153.stdout:9/152: write d1/d9/fc [211974,89215] 0 2024-04-05T03:02:44.874 INFO:tasks.workunit.client.1.smithi203.stdout:9/523: mkdir d13/d1b/d26/d51/da1 0 2024-04-05T03:02:44.876 INFO:tasks.workunit.client.1.smithi203.stdout:0/571: truncate f7 3298663 0 2024-04-05T03:02:44.879 INFO:tasks.workunit.client.1.smithi203.stdout:7/729: creat d8/d32/d16/fe1 x:0 0 0 2024-04-05T03:02:44.879 INFO:tasks.workunit.client.0.smithi153.stdout:5/194: creat d0/d2c/f48 x:0 0 0 2024-04-05T03:02:44.880 INFO:tasks.workunit.client.1.smithi203.stdout:9/524: dread d13/f4a [4194304,4194304] 0 2024-04-05T03:02:44.881 INFO:tasks.workunit.client.0.smithi153.stdout:9/153: mkdir d1/d3a 0 2024-04-05T03:02:44.882 INFO:tasks.workunit.client.1.smithi203.stdout:0/572: creat dd/d17/d1d/d26/d34/d56/fa7 x:0 0 0 2024-04-05T03:02:44.883 INFO:tasks.workunit.client.0.smithi153.stdout:4/287: dwrite da/de/d28/d2d/f4a [0,4194304] 0 2024-04-05T03:02:44.883 INFO:tasks.workunit.client.0.smithi153.stdout:5/195: write d0/d2c/f3a [911409,126768] 0 2024-04-05T03:02:44.885 INFO:tasks.workunit.client.1.smithi203.stdout:8/580: dwrite d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:44.885 INFO:tasks.workunit.client.1.smithi203.stdout:0/573: write f5 [4706437,80028] 0 2024-04-05T03:02:44.885 INFO:tasks.workunit.client.0.smithi153.stdout:4/288: chown f9 0 1 2024-04-05T03:02:44.885 INFO:tasks.workunit.client.0.smithi153.stdout:0/318: write d9/df/d1e/f54 [5098794,120236] 0 2024-04-05T03:02:44.885 INFO:tasks.workunit.client.1.smithi203.stdout:1/554: dread d7/f2d [0,4194304] 0 2024-04-05T03:02:44.888 INFO:tasks.workunit.client.1.smithi203.stdout:9/525: write d13/d1b/d6b/d9e/d77/f95 [980680,129908] 0 2024-04-05T03:02:44.889 INFO:tasks.workunit.client.1.smithi203.stdout:1/555: fdatasync d7/d14/d2b/d58/f9a 0 2024-04-05T03:02:44.890 INFO:tasks.workunit.client.1.smithi203.stdout:7/730: write d8/d32/d5d/d6d/dca/d3f/d5e/d77/f80 [245524,83469] 0 2024-04-05T03:02:44.894 INFO:tasks.workunit.client.1.smithi203.stdout:0/574: write dd/d17/d1d/d26/d34/f35 [4525814,40188] 0 2024-04-05T03:02:44.895 INFO:tasks.workunit.client.1.smithi203.stdout:1/556: readlink d7/d24/d33/d48/d4b/d51/la7 0 2024-04-05T03:02:44.896 INFO:tasks.workunit.client.1.smithi203.stdout:7/731: mknod d8/da/ce2 0 2024-04-05T03:02:44.897 INFO:tasks.workunit.client.1.smithi203.stdout:7/732: readlink d8/lb1 0 2024-04-05T03:02:44.901 INFO:tasks.workunit.client.1.smithi203.stdout:7/733: dread d8/da/f14 [0,4194304] 0 2024-04-05T03:02:44.902 INFO:tasks.workunit.client.1.smithi203.stdout:0/575: rmdir dd/d17/d1d/d22/d80/d49 39 2024-04-05T03:02:44.902 INFO:tasks.workunit.client.1.smithi203.stdout:9/526: symlink d13/d1b/d26/d51/da1/la2 0 2024-04-05T03:02:44.902 INFO:tasks.workunit.client.1.smithi203.stdout:8/581: dread d1/d12/d17/d10/f49 [0,4194304] 0 2024-04-05T03:02:44.903 INFO:tasks.workunit.client.1.smithi203.stdout:1/557: mknod d7/d24/caf 0 2024-04-05T03:02:44.905 INFO:tasks.workunit.client.1.smithi203.stdout:7/734: mkdir d8/d32/d16/d1a/d3d/d59/de3 0 2024-04-05T03:02:44.908 INFO:tasks.workunit.client.1.smithi203.stdout:8/582: symlink d1/d12/d2f/d1f/la1 0 2024-04-05T03:02:44.908 INFO:tasks.workunit.client.1.smithi203.stdout:9/527: mknod d13/d1b/d6b/ca3 0 2024-04-05T03:02:44.909 INFO:tasks.workunit.client.1.smithi203.stdout:0/576: rename dd/d17/d29/f57 to dd/d17/d1d/d26/d8c/fa8 0 2024-04-05T03:02:44.909 INFO:tasks.workunit.client.0.smithi153.stdout:9/154: mknod d1/d10/c3b 0 2024-04-05T03:02:44.910 INFO:tasks.workunit.client.1.smithi203.stdout:8/583: write d1/d12/d1a/f46 [630076,87347] 0 2024-04-05T03:02:44.911 INFO:tasks.workunit.client.1.smithi203.stdout:7/735: mknod d8/d32/d16/d1a/d30/ce4 0 2024-04-05T03:02:44.914 INFO:tasks.workunit.client.0.smithi153.stdout:8/210: chown d7/d13/d19/l29 574424 1 2024-04-05T03:02:44.914 INFO:tasks.workunit.client.0.smithi153.stdout:5/196: dread d0/d9/d22/f41 [0,4194304] 0 2024-04-05T03:02:44.915 INFO:tasks.workunit.client.0.smithi153.stdout:4/289: symlink da/db/d20/d35/d41/l5a 0 2024-04-05T03:02:44.915 INFO:tasks.workunit.client.0.smithi153.stdout:8/211: stat d7/d13/d19/d1b/d21/l43 0 2024-04-05T03:02:44.916 INFO:tasks.workunit.client.0.smithi153.stdout:6/269: dread d4/dc/df/d16/d17/f3b [0,4194304] 0 2024-04-05T03:02:44.919 INFO:tasks.workunit.client.1.smithi203.stdout:0/577: unlink dd/d17/c7a 0 2024-04-05T03:02:44.921 INFO:tasks.workunit.client.1.smithi203.stdout:1/558: dwrite d7/d24/d33/d48/fa1 [0,4194304] 0 2024-04-05T03:02:44.923 INFO:tasks.workunit.client.0.smithi153.stdout:0/319: rmdir d9/df/d1d/d47 0 2024-04-05T03:02:44.924 INFO:tasks.workunit.client.0.smithi153.stdout:0/320: truncate d9/df/d1e/f52 103413 0 2024-04-05T03:02:44.924 INFO:tasks.workunit.client.0.smithi153.stdout:5/197: unlink d0/lf 0 2024-04-05T03:02:44.926 INFO:tasks.workunit.client.1.smithi203.stdout:7/736: dwrite d8/d32/d16/d1a/d5a/db5/fd7 [0,4194304] 0 2024-04-05T03:02:44.927 INFO:tasks.workunit.client.0.smithi153.stdout:5/198: truncate d0/f44 998970 0 2024-04-05T03:02:44.927 INFO:tasks.workunit.client.1.smithi203.stdout:3/475: dread d0/f5 [0,4194304] 0 2024-04-05T03:02:44.928 INFO:tasks.workunit.client.0.smithi153.stdout:4/290: mknod da/db/d20/d35/d41/c5b 0 2024-04-05T03:02:44.928 INFO:tasks.workunit.client.0.smithi153.stdout:8/212: read d7/f2b [142390,45752] 0 2024-04-05T03:02:44.928 INFO:tasks.workunit.client.1.smithi203.stdout:9/528: getdents d13/d1b/d6b/d7a 0 2024-04-05T03:02:44.930 INFO:tasks.workunit.client.1.smithi203.stdout:1/559: dread d7/d14/f29 [0,4194304] 0 2024-04-05T03:02:44.930 INFO:tasks.workunit.client.1.smithi203.stdout:8/584: getdents d1/d12/d17/d10 0 2024-04-05T03:02:44.931 INFO:tasks.workunit.client.1.smithi203.stdout:3/476: chown d0/dc/d1a/d1d/d43 2 1 2024-04-05T03:02:44.932 INFO:tasks.workunit.client.1.smithi203.stdout:8/585: stat d1/d6/d2e/f6b 0 2024-04-05T03:02:44.932 INFO:tasks.workunit.client.0.smithi153.stdout:9/155: rename d1/d9/d1b/c24 to d1/d3a/c3c 0 2024-04-05T03:02:44.933 INFO:tasks.workunit.client.1.smithi203.stdout:3/477: fsync d0/d21/d81/f5b 0 2024-04-05T03:02:44.934 INFO:tasks.workunit.client.1.smithi203.stdout:0/578: creat dd/d17/d1d/d26/d8c/d6d/fa9 x:0 0 0 2024-04-05T03:02:44.935 INFO:tasks.workunit.client.0.smithi153.stdout:5/199: mkdir d0/d9/d22/d49 0 2024-04-05T03:02:44.936 INFO:tasks.workunit.client.1.smithi203.stdout:3/478: chown d0/dc/d34/f61 3210 1 2024-04-05T03:02:44.936 INFO:tasks.workunit.client.0.smithi153.stdout:8/213: fsync d7/fd 0 2024-04-05T03:02:44.940 INFO:tasks.workunit.client.0.smithi153.stdout:4/291: creat da/db/d20/d35/d38/f5c x:0 0 0 2024-04-05T03:02:44.941 INFO:tasks.workunit.client.1.smithi203.stdout:1/560: dread d7/d14/f93 [0,4194304] 0 2024-04-05T03:02:44.941 INFO:tasks.workunit.client.0.smithi153.stdout:4/292: chown da/db/l49 60 1 2024-04-05T03:02:44.942 INFO:tasks.workunit.client.1.smithi203.stdout:3/479: chown d0/dc/d38 381134 1 2024-04-05T03:02:44.943 INFO:tasks.workunit.client.0.smithi153.stdout:9/156: dwrite d1/d9/d1b/f28 [0,4194304] 0 2024-04-05T03:02:44.944 INFO:tasks.workunit.client.0.smithi153.stdout:4/293: write da/de/d28/d2d/f57 [316935,73694] 0 2024-04-05T03:02:44.945 INFO:tasks.workunit.client.1.smithi203.stdout:7/737: dwrite d8/d32/d5d/d6d/dca/d3f/d52/d65/d82/fc1 [0,4194304] 0 2024-04-05T03:02:44.953 INFO:tasks.workunit.client.0.smithi153.stdout:9/157: dread d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:02:44.954 INFO:tasks.workunit.client.1.smithi203.stdout:0/579: mkdir dd/d17/d1d/d22/d53/daa 0 2024-04-05T03:02:44.955 INFO:tasks.workunit.client.0.smithi153.stdout:9/158: readlink d1/d10/d16/l35 0 2024-04-05T03:02:44.956 INFO:tasks.workunit.client.1.smithi203.stdout:0/580: write dd/d17/d1d/d26/d28/f87 [1041290,5288] 0 2024-04-05T03:02:44.957 INFO:tasks.workunit.client.0.smithi153.stdout:8/214: symlink d7/d46/l47 0 2024-04-05T03:02:44.958 INFO:tasks.workunit.client.1.smithi203.stdout:3/480: mknod d0/dc/d38/d5d/d66/d80/c89 0 2024-04-05T03:02:44.958 INFO:tasks.workunit.client.1.smithi203.stdout:8/586: write d1/d12/f6c [2649569,30888] 0 2024-04-05T03:02:44.959 INFO:tasks.workunit.client.1.smithi203.stdout:0/581: fdatasync dd/d17/d1d/d22/d53/d55/fa1 0 2024-04-05T03:02:44.962 INFO:tasks.workunit.client.0.smithi153.stdout:6/270: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:44.963 INFO:tasks.workunit.client.1.smithi203.stdout:7/738: mknod d8/d32/d5d/d6d/dca/d3f/d73/da8/ce5 0 2024-04-05T03:02:44.963 INFO:tasks.workunit.client.0.smithi153.stdout:0/321: truncate d9/df/f16 4466068 0 2024-04-05T03:02:44.965 INFO:tasks.workunit.client.0.smithi153.stdout:9/159: link d1/d10/c3b d1/d9/d1b/c3d 0 2024-04-05T03:02:44.965 INFO:tasks.workunit.client.1.smithi203.stdout:3/481: stat d0/dc/d1a/d76/d35/l4e 0 2024-04-05T03:02:44.968 INFO:tasks.workunit.client.1.smithi203.stdout:3/482: dread - d0/d21/d81/f58 zero size 2024-04-05T03:02:44.968 INFO:tasks.workunit.client.1.smithi203.stdout:1/561: rename d7/d24/d33/c63 to d7/d24/cb0 0 2024-04-05T03:02:44.968 INFO:tasks.workunit.client.1.smithi203.stdout:8/587: mkdir d1/d90/da2 0 2024-04-05T03:02:44.969 INFO:tasks.workunit.client.0.smithi153.stdout:5/200: dwrite d0/f16 [0,4194304] 0 2024-04-05T03:02:44.971 INFO:tasks.workunit.client.0.smithi153.stdout:8/215: dread d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:44.971 INFO:tasks.workunit.client.1.smithi203.stdout:0/582: dwrite dd/d17/d1d/d26/d34/f64 [0,4194304] 0 2024-04-05T03:02:44.972 INFO:tasks.workunit.client.0.smithi153.stdout:9/160: write d1/d10/d16/f23 [837168,46357] 0 2024-04-05T03:02:44.972 INFO:tasks.workunit.client.0.smithi153.stdout:6/271: fsync d4/dc/df/d16/f21 0 2024-04-05T03:02:44.973 INFO:tasks.workunit.client.1.smithi203.stdout:8/588: fsync d1/d4b/d68/f95 0 2024-04-05T03:02:44.973 INFO:tasks.workunit.client.0.smithi153.stdout:9/161: chown d1 15986970 1 2024-04-05T03:02:44.973 INFO:tasks.workunit.client.0.smithi153.stdout:8/216: fsync f0 0 2024-04-05T03:02:44.973 INFO:tasks.workunit.client.0.smithi153.stdout:9/162: chown d1/f4 65436 1 2024-04-05T03:02:44.974 INFO:tasks.workunit.client.0.smithi153.stdout:8/217: chown d7 10 1 2024-04-05T03:02:44.974 INFO:tasks.workunit.client.1.smithi203.stdout:0/583: fsync dd/d17/d1d/d26/f98 0 2024-04-05T03:02:44.977 INFO:tasks.workunit.client.0.smithi153.stdout:0/322: dread d9/db/f37 [0,4194304] 0 2024-04-05T03:02:44.979 INFO:tasks.workunit.client.0.smithi153.stdout:7/212: sync 2024-04-05T03:02:44.982 INFO:tasks.workunit.client.0.smithi153.stdout:8/218: truncate d7/f2b 426907 0 2024-04-05T03:02:44.982 INFO:tasks.workunit.client.1.smithi203.stdout:8/589: mknod d1/d12/ca3 0 2024-04-05T03:02:44.984 INFO:tasks.workunit.client.1.smithi203.stdout:8/590: chown d1/d6/d2e/f6b 1116 1 2024-04-05T03:02:44.985 INFO:tasks.workunit.client.1.smithi203.stdout:0/584: rename dd/d17/d1d/d26/d34/f64 to dd/d17/d1d/d26/d34/d56/fab 0 2024-04-05T03:02:44.985 INFO:tasks.workunit.client.0.smithi153.stdout:5/201: rename d0/d2c/c40 to d0/de/c4a 0 2024-04-05T03:02:44.986 INFO:tasks.workunit.client.0.smithi153.stdout:6/272: dread d4/dc/df/f12 [0,4194304] 0 2024-04-05T03:02:44.987 INFO:tasks.workunit.client.1.smithi203.stdout:0/585: chown dd/d17/d1d/c39 933681 1 2024-04-05T03:02:44.988 INFO:tasks.workunit.client.1.smithi203.stdout:1/562: link d7/d24/f34 d7/d14/d2b/d58/fb1 0 2024-04-05T03:02:44.990 INFO:tasks.workunit.client.1.smithi203.stdout:1/563: truncate d7/d24/f6e 834199 0 2024-04-05T03:02:44.991 INFO:tasks.workunit.client.0.smithi153.stdout:8/219: write d7/d13/f18 [917996,126821] 0 2024-04-05T03:02:44.992 INFO:tasks.workunit.client.0.smithi153.stdout:7/213: truncate d2/da/d11/d49/f10 721485 0 2024-04-05T03:02:44.992 INFO:tasks.workunit.client.1.smithi203.stdout:7/739: dread d8/d32/d5d/f69 [0,4194304] 0 2024-04-05T03:02:44.994 INFO:tasks.workunit.client.0.smithi153.stdout:8/220: symlink d7/d13/l48 0 2024-04-05T03:02:44.995 INFO:tasks.workunit.client.0.smithi153.stdout:7/214: rename d2/da/d47/l48 to d2/da/d47/l4a 0 2024-04-05T03:02:44.997 INFO:tasks.workunit.client.0.smithi153.stdout:5/202: creat d0/de/d37/f4b x:0 0 0 2024-04-05T03:02:44.997 INFO:tasks.workunit.client.0.smithi153.stdout:7/215: dread - d2/da/d11/f2b zero size 2024-04-05T03:02:44.998 INFO:tasks.workunit.client.0.smithi153.stdout:6/273: mknod d4/d38/d4c/c63 0 2024-04-05T03:02:44.999 INFO:tasks.workunit.client.0.smithi153.stdout:8/221: symlink d7/d13/d19/d31/l49 0 2024-04-05T03:02:45.000 INFO:tasks.workunit.client.0.smithi153.stdout:6/274: write d4/dc/df/d16/f21 [3200485,46270] 0 2024-04-05T03:02:45.001 INFO:tasks.workunit.client.0.smithi153.stdout:5/203: mkdir d0/d2c/d4c 0 2024-04-05T03:02:45.002 INFO:tasks.workunit.client.0.smithi153.stdout:5/204: stat d0/f3c 0 2024-04-05T03:02:45.002 INFO:tasks.workunit.client.1.smithi203.stdout:7/740: dwrite d8/d32/d5d/d6d/dca/d3f/d73/da8/fc0 [0,4194304] 0 2024-04-05T03:02:45.003 INFO:tasks.workunit.client.0.smithi153.stdout:6/275: write d4/dc/df/d16/f23 [70530,89741] 0 2024-04-05T03:02:45.004 INFO:tasks.workunit.client.0.smithi153.stdout:8/222: creat d7/d13/d19/d31/f4a x:0 0 0 2024-04-05T03:02:45.004 INFO:tasks.workunit.client.1.smithi203.stdout:7/741: write d8/f17 [8360786,96420] 0 2024-04-05T03:02:45.005 INFO:tasks.workunit.client.0.smithi153.stdout:6/276: readlink d4/dc/df/d16/d33/l49 0 2024-04-05T03:02:45.006 INFO:tasks.workunit.client.0.smithi153.stdout:7/216: link d2/da/d11/f13 d2/da/d42/f4b 0 2024-04-05T03:02:45.006 INFO:tasks.workunit.client.0.smithi153.stdout:8/223: write d7/d13/f15 [1274991,64207] 0 2024-04-05T03:02:45.009 INFO:tasks.workunit.client.1.smithi203.stdout:7/742: truncate d8/d32/d5d/d6d/dca/d3f/d5e/d77/f80 1353285 0 2024-04-05T03:02:45.009 INFO:tasks.workunit.client.0.smithi153.stdout:5/205: mkdir d0/d25/d4d 0 2024-04-05T03:02:45.010 INFO:tasks.workunit.client.1.smithi203.stdout:7/743: chown d8/d32/d5d/d6d/dca/d3f/d5e/c9d 19983304 1 2024-04-05T03:02:45.012 INFO:tasks.workunit.client.0.smithi153.stdout:9/163: write d1/d9/d1b/f26 [3587503,31089] 0 2024-04-05T03:02:45.013 INFO:tasks.workunit.client.1.smithi203.stdout:3/483: truncate d0/dc/f54 1469642 0 2024-04-05T03:02:45.014 INFO:tasks.workunit.client.0.smithi153.stdout:8/224: symlink d7/d13/d19/d1b/d21/l4b 0 2024-04-05T03:02:45.016 INFO:tasks.workunit.client.0.smithi153.stdout:9/164: creat d1/d10/d16/d22/f3e x:0 0 0 2024-04-05T03:02:45.017 INFO:tasks.workunit.client.0.smithi153.stdout:9/165: read d1/d9/d1b/f28 [1050863,18940] 0 2024-04-05T03:02:45.019 INFO:tasks.workunit.client.1.smithi203.stdout:8/591: dread d1/d12/f6c [0,4194304] 0 2024-04-05T03:02:45.019 INFO:tasks.workunit.client.1.smithi203.stdout:7/744: dwrite d8/d32/d5d/d6d/f8f [0,4194304] 0 2024-04-05T03:02:45.021 INFO:tasks.workunit.client.1.smithi203.stdout:1/564: truncate d7/d14/d2b/f72 542587 0 2024-04-05T03:02:45.023 INFO:tasks.workunit.client.1.smithi203.stdout:7/745: write d8/d32/d5d/d6d/dca/d3f/d73/f7b [7248027,15725] 0 2024-04-05T03:02:45.024 INFO:tasks.workunit.client.1.smithi203.stdout:3/484: symlink d0/d21/d83/l8a 0 2024-04-05T03:02:45.025 INFO:tasks.workunit.client.1.smithi203.stdout:1/565: write d7/d24/d33/f56 [4911682,106200] 0 2024-04-05T03:02:45.026 INFO:tasks.workunit.client.1.smithi203.stdout:3/485: chown d0/d69 10 1 2024-04-05T03:02:45.027 INFO:tasks.workunit.client.0.smithi153.stdout:9/166: creat d1/d9/d1b/f3f x:0 0 0 2024-04-05T03:02:45.028 INFO:tasks.workunit.client.0.smithi153.stdout:9/167: fdatasync d1/d9/fc 0 2024-04-05T03:02:45.029 INFO:tasks.workunit.client.1.smithi203.stdout:3/486: fdatasync d0/dc/d1a/d1d/d43/f51 0 2024-04-05T03:02:45.029 INFO:tasks.workunit.client.1.smithi203.stdout:8/592: read d1/d12/d2f/d1f/f23 [624201,10390] 0 2024-04-05T03:02:45.030 INFO:tasks.workunit.client.0.smithi153.stdout:5/206: rename d0/d2c/d4c to d0/d4e 0 2024-04-05T03:02:45.031 INFO:tasks.workunit.client.1.smithi203.stdout:8/593: chown d1 1468 1 2024-04-05T03:02:45.032 INFO:tasks.workunit.client.1.smithi203.stdout:7/746: unlink d8/da/f94 0 2024-04-05T03:02:45.033 INFO:tasks.workunit.client.1.smithi203.stdout:8/594: readlink d1/d6/l29 0 2024-04-05T03:02:45.034 INFO:tasks.workunit.client.0.smithi153.stdout:9/168: creat d1/d10/d16/f40 x:0 0 0 2024-04-05T03:02:45.035 INFO:tasks.workunit.client.0.smithi153.stdout:7/217: truncate d2/da/d11/f13 1004316 0 2024-04-05T03:02:45.036 INFO:tasks.workunit.client.1.smithi203.stdout:0/586: dread dd/d17/d1d/d22/d80/d49/f7d [0,4194304] 0 2024-04-05T03:02:45.041 INFO:tasks.workunit.client.1.smithi203.stdout:1/566: dwrite d7/d14/d2b/d58/f9a [0,4194304] 0 2024-04-05T03:02:45.042 INFO:tasks.workunit.client.0.smithi153.stdout:9/169: dwrite d1/f1f [0,4194304] 0 2024-04-05T03:02:45.042 INFO:tasks.workunit.client.1.smithi203.stdout:8/595: read d1/d12/d17/d10/f49 [8976,130423] 0 2024-04-05T03:02:45.045 INFO:tasks.workunit.client.0.smithi153.stdout:5/207: creat d0/de/d37/f4f x:0 0 0 2024-04-05T03:02:45.045 INFO:tasks.workunit.client.1.smithi203.stdout:3/487: rename d0/dc/d1a/d1d/d53/d5a/f77 to d0/dc/d38/d5d/f8b 0 2024-04-05T03:02:45.046 INFO:tasks.workunit.client.1.smithi203.stdout:8/596: write d1/f58 [1365823,90845] 0 2024-04-05T03:02:45.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:44 smithi203 ceph-mon[39911]: pgmap v7: 65 pgs: 65 active+clean; 1.4 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 57 MiB/s rd, 179 MiB/s wr, 351 op/s 2024-04-05T03:02:45.051 INFO:tasks.workunit.client.1.smithi203.stdout:3/488: chown d0/d21/d81/c63 21 1 2024-04-05T03:02:45.051 INFO:tasks.workunit.client.0.smithi153.stdout:9/170: dread d1/d10/d16/f23 [0,4194304] 0 2024-04-05T03:02:45.052 INFO:tasks.workunit.client.0.smithi153.stdout:7/218: rename d2/da/d11/c35 to d2/da/c4c 0 2024-04-05T03:02:45.054 INFO:tasks.workunit.client.1.smithi203.stdout:7/747: dread d8/d32/d5d/d6d/dca/d3f/d5e/d77/f80 [0,4194304] 0 2024-04-05T03:02:45.055 INFO:tasks.workunit.client.0.smithi153.stdout:5/208: mknod d0/d2c/c50 0 2024-04-05T03:02:45.055 INFO:tasks.workunit.client.0.smithi153.stdout:8/225: dread d7/fc [0,4194304] 0 2024-04-05T03:02:45.055 INFO:tasks.workunit.client.1.smithi203.stdout:8/597: mkdir d1/d6/d2e/d92/da4 0 2024-04-05T03:02:45.056 INFO:tasks.workunit.client.1.smithi203.stdout:7/748: chown d8/d32/d5d/dbe/dc9/fd6 449 1 2024-04-05T03:02:45.057 INFO:tasks.workunit.client.0.smithi153.stdout:7/219: unlink d2/c8 0 2024-04-05T03:02:45.058 INFO:tasks.workunit.client.1.smithi203.stdout:5/579: sync 2024-04-05T03:02:45.058 INFO:tasks.workunit.client.1.smithi203.stdout:4/400: sync 2024-04-05T03:02:45.058 INFO:tasks.workunit.client.1.smithi203.stdout:3/489: mknod d0/c8c 0 2024-04-05T03:02:45.060 INFO:tasks.workunit.client.1.smithi203.stdout:1/567: getdents d7/d24/d33/d48 0 2024-04-05T03:02:45.060 INFO:tasks.workunit.client.1.smithi203.stdout:5/580: chown d3/d1f/d59/l66 1 1 2024-04-05T03:02:45.060 INFO:tasks.workunit.client.0.smithi153.stdout:8/226: unlink d7/d13/c3b 0 2024-04-05T03:02:45.060 INFO:tasks.workunit.client.1.smithi203.stdout:0/587: getdents dd/d17/d1d/d26/d34/d56 0 2024-04-05T03:02:45.062 INFO:tasks.workunit.client.1.smithi203.stdout:8/598: symlink d1/d12/d1a/la5 0 2024-04-05T03:02:45.062 INFO:tasks.workunit.client.1.smithi203.stdout:0/588: fdatasync dd/d17/d1d/f1e 0 2024-04-05T03:02:45.063 INFO:tasks.workunit.client.0.smithi153.stdout:9/171: mknod d1/d10/c41 0 2024-04-05T03:02:45.063 INFO:tasks.workunit.client.0.smithi153.stdout:5/209: dwrite d0/f27 [0,4194304] 0 2024-04-05T03:02:45.063 INFO:tasks.workunit.client.0.smithi153.stdout:9/172: chown d1/d9/d1b/f30 0 1 2024-04-05T03:02:45.064 INFO:tasks.workunit.client.1.smithi203.stdout:8/599: chown d1/d6/d2e/d92/da4 4530693 1 2024-04-05T03:02:45.064 INFO:tasks.workunit.client.0.smithi153.stdout:5/210: read d0/d25/f31 [565454,104000] 0 2024-04-05T03:02:45.065 INFO:tasks.workunit.client.0.smithi153.stdout:5/211: chown d0/de/c26 100717745 1 2024-04-05T03:02:45.066 INFO:tasks.workunit.client.0.smithi153.stdout:5/212: stat d0/de/f1a 0 2024-04-05T03:02:45.068 INFO:tasks.workunit.client.1.smithi203.stdout:5/581: symlink d3/d4/dc/d67/dac/lbf 0 2024-04-05T03:02:45.070 INFO:tasks.workunit.client.1.smithi203.stdout:1/568: creat d7/d24/d33/d48/d4b/d51/fb2 x:0 0 0 2024-04-05T03:02:45.071 INFO:tasks.workunit.client.1.smithi203.stdout:8/600: rmdir d1/d12/d17 39 2024-04-05T03:02:45.072 INFO:tasks.workunit.client.0.smithi153.stdout:5/213: mknod d0/de/c51 0 2024-04-05T03:02:45.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:44 smithi153 ceph-mon[29381]: pgmap v7: 65 pgs: 65 active+clean; 1.4 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 57 MiB/s rd, 179 MiB/s wr, 351 op/s 2024-04-05T03:02:45.072 INFO:tasks.workunit.client.1.smithi203.stdout:1/569: stat d7/d14/d2b/d58 0 2024-04-05T03:02:45.076 INFO:tasks.workunit.client.0.smithi153.stdout:9/173: dwrite d1/d9/d1b/f3f [0,4194304] 0 2024-04-05T03:02:45.076 INFO:tasks.workunit.client.1.smithi203.stdout:5/582: dwrite d3/d4/dc/d67/dac/fb6 [0,4194304] 0 2024-04-05T03:02:45.077 INFO:tasks.workunit.client.1.smithi203.stdout:4/401: getdents da/d29/d2e/d53 0 2024-04-05T03:02:45.079 INFO:tasks.workunit.client.1.smithi203.stdout:0/589: creat dd/d17/d1d/d22/fac x:0 0 0 2024-04-05T03:02:45.080 INFO:tasks.workunit.client.0.smithi153.stdout:9/174: symlink d1/d1d/l42 0 2024-04-05T03:02:45.085 INFO:tasks.workunit.client.0.smithi153.stdout:5/214: dread d0/d2c/f3a [0,4194304] 0 2024-04-05T03:02:45.086 INFO:tasks.workunit.client.1.smithi203.stdout:3/490: write d0/dc/d1a/d76/d35/f47 [1966878,77541] 0 2024-04-05T03:02:45.087 INFO:tasks.workunit.client.1.smithi203.stdout:7/749: write d8/da/fd [1834619,89939] 0 2024-04-05T03:02:45.091 INFO:tasks.workunit.client.0.smithi153.stdout:5/215: mknod d0/de/c52 0 2024-04-05T03:02:45.093 INFO:tasks.workunit.client.1.smithi203.stdout:8/601: link d1/d4b/d68/l9e d1/d4b/d68/la6 0 2024-04-05T03:02:45.095 INFO:tasks.workunit.client.0.smithi153.stdout:7/220: dread - d2/da/d3a/d41/d31/f45 zero size 2024-04-05T03:02:45.095 INFO:tasks.workunit.client.1.smithi203.stdout:7/750: unlink d8/d32/d16/d1a/f5c 0 2024-04-05T03:02:45.096 INFO:tasks.workunit.client.0.smithi153.stdout:7/221: readlink d2/da/l23 0 2024-04-05T03:02:45.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/227: dread f0 [0,4194304] 0 2024-04-05T03:02:45.097 INFO:tasks.workunit.client.0.smithi153.stdout:9/175: rename d1/d10/c3b to d1/d10/c43 0 2024-04-05T03:02:45.098 INFO:tasks.workunit.client.1.smithi203.stdout:7/751: fsync d8/d32/d5d/d6d/dca/d3f/d5e/d77/fb0 0 2024-04-05T03:02:45.100 INFO:tasks.workunit.client.1.smithi203.stdout:1/570: creat d7/d24/d33/d48/d4b/d6b/fb3 x:0 0 0 2024-04-05T03:02:45.100 INFO:tasks.workunit.client.1.smithi203.stdout:3/491: mkdir d0/dc/d1a/d8d 0 2024-04-05T03:02:45.100 INFO:tasks.workunit.client.1.smithi203.stdout:5/583: creat d3/d1f/fc0 x:0 0 0 2024-04-05T03:02:45.104 INFO:tasks.workunit.client.0.smithi153.stdout:7/222: creat d2/da/d3a/d41/d31/f4d x:0 0 0 2024-04-05T03:02:45.105 INFO:tasks.workunit.client.0.smithi153.stdout:5/216: write d0/d25/f31 [459494,107602] 0 2024-04-05T03:02:45.105 INFO:tasks.workunit.client.1.smithi203.stdout:8/602: dwrite d1/d4b/d68/f95 [0,4194304] 0 2024-04-05T03:02:45.106 INFO:tasks.workunit.client.0.smithi153.stdout:9/176: symlink d1/db/l44 0 2024-04-05T03:02:45.106 INFO:tasks.workunit.client.1.smithi203.stdout:5/584: mkdir d3/d1f/d59/d60/dc1 0 2024-04-05T03:02:45.106 INFO:tasks.workunit.client.1.smithi203.stdout:7/752: write d8/d32/d5d/d6d/dca/d3f/d73/fc2 [3511513,24132] 0 2024-04-05T03:02:45.109 INFO:tasks.workunit.client.1.smithi203.stdout:6/603: sync 2024-04-05T03:02:45.110 INFO:tasks.workunit.client.1.smithi203.stdout:2/763: sync 2024-04-05T03:02:45.110 INFO:tasks.workunit.client.1.smithi203.stdout:9/529: sync 2024-04-05T03:02:45.110 INFO:tasks.workunit.client.0.smithi153.stdout:8/228: creat d7/d13/d19/d1b/d33/f4c x:0 0 0 2024-04-05T03:02:45.113 INFO:tasks.workunit.client.0.smithi153.stdout:9/177: creat d1/db/f45 x:0 0 0 2024-04-05T03:02:45.113 INFO:tasks.workunit.client.0.smithi153.stdout:5/217: dwrite d0/d25/f35 [4194304,4194304] 0 2024-04-05T03:02:45.114 INFO:tasks.workunit.client.0.smithi153.stdout:7/223: dread d2/da/d3a/f46 [0,4194304] 0 2024-04-05T03:02:45.116 INFO:tasks.workunit.client.1.smithi203.stdout:0/590: truncate f2 1142206 0 2024-04-05T03:02:45.119 INFO:tasks.workunit.client.1.smithi203.stdout:3/492: dwrite d0/dc/d1a/d1d/d43/f6a [4194304,4194304] 0 2024-04-05T03:02:45.119 INFO:tasks.workunit.client.0.smithi153.stdout:9/178: dread d1/d9/d1b/f3f [0,4194304] 0 2024-04-05T03:02:45.120 INFO:tasks.workunit.client.1.smithi203.stdout:7/753: stat d8/d32/d5d/d6d/dca/d3f/d5e/d63/dc5 0 2024-04-05T03:02:45.121 INFO:tasks.workunit.client.0.smithi153.stdout:9/179: fsync d1/f2a 0 2024-04-05T03:02:45.124 INFO:tasks.workunit.client.1.smithi203.stdout:8/603: creat d1/d12/d2f/d5b/fa7 x:0 0 0 2024-04-05T03:02:45.129 INFO:tasks.workunit.client.1.smithi203.stdout:1/571: dwrite d7/f2a [0,4194304] 0 2024-04-05T03:02:45.130 INFO:tasks.workunit.client.0.smithi153.stdout:8/229: dread d7/d2a/f44 [0,4194304] 0 2024-04-05T03:02:45.130 INFO:tasks.workunit.client.1.smithi203.stdout:0/591: creat dd/d17/d1d/d26/d28/fad x:0 0 0 2024-04-05T03:02:45.132 INFO:tasks.workunit.client.1.smithi203.stdout:3/493: truncate d0/dc/d1a/d1d/d53/f73 329868 0 2024-04-05T03:02:45.135 INFO:tasks.workunit.client.0.smithi153.stdout:7/224: rename d2/da/d11/f3b to d2/da/d11/d49/f4e 0 2024-04-05T03:02:45.135 INFO:tasks.workunit.client.0.smithi153.stdout:9/180: creat d1/d10/d16/d22/d2c/f46 x:0 0 0 2024-04-05T03:02:45.136 INFO:tasks.workunit.client.1.smithi203.stdout:8/604: mkdir d1/d12/da8 0 2024-04-05T03:02:45.137 INFO:tasks.workunit.client.1.smithi203.stdout:7/754: chown d8/d32/d5d/d6d/dca/d3f/d73/fc2 6892714 1 2024-04-05T03:02:45.138 INFO:tasks.workunit.client.1.smithi203.stdout:6/604: stat d0/da/d18/c81 0 2024-04-05T03:02:45.140 INFO:tasks.workunit.client.1.smithi203.stdout:0/592: write dd/d17/d1d/d22/d80/f4b [156614,30725] 0 2024-04-05T03:02:45.141 INFO:tasks.workunit.client.0.smithi153.stdout:5/218: rmdir d0/d9/d22/d49 0 2024-04-05T03:02:45.142 INFO:tasks.workunit.client.0.smithi153.stdout:5/219: chown d0/d9/d22/f41 0 1 2024-04-05T03:02:45.142 INFO:tasks.workunit.client.0.smithi153.stdout:9/181: mknod d1/d10/d16/c47 0 2024-04-05T03:02:45.145 INFO:tasks.workunit.client.0.smithi153.stdout:7/225: dwrite d2/f5 [0,4194304] 0 2024-04-05T03:02:45.146 INFO:tasks.workunit.client.1.smithi203.stdout:2/764: write d5/d10/d35/d73/f95 [1236484,5498] 0 2024-04-05T03:02:45.146 INFO:tasks.workunit.client.1.smithi203.stdout:1/572: creat d7/d14/d83/fb4 x:0 0 0 2024-04-05T03:02:45.148 INFO:tasks.workunit.client.0.smithi153.stdout:7/226: write d2/da/d11/d49/f3e [658498,45909] 0 2024-04-05T03:02:45.149 INFO:tasks.workunit.client.1.smithi203.stdout:4/402: dread da/d29/d2e/d57/f66 [0,4194304] 0 2024-04-05T03:02:45.150 INFO:tasks.workunit.client.1.smithi203.stdout:9/530: dwrite d13/d1b/d26/f6f [0,4194304] 0 2024-04-05T03:02:45.153 INFO:tasks.workunit.client.0.smithi153.stdout:7/227: mknod d2/da/d11/d39/c4f 0 2024-04-05T03:02:45.156 INFO:tasks.workunit.client.0.smithi153.stdout:3/197: sync 2024-04-05T03:02:45.157 INFO:tasks.workunit.client.0.smithi153.stdout:5/220: link d0/de/d37/d39/f3b d0/d2c/f53 0 2024-04-05T03:02:45.157 INFO:tasks.workunit.client.1.smithi203.stdout:8/605: mkdir d1/d12/d2f/da9 0 2024-04-05T03:02:45.157 INFO:tasks.workunit.client.1.smithi203.stdout:5/585: dwrite d3/d1f/d59/d64/f44 [0,4194304] 0 2024-04-05T03:02:45.157 INFO:tasks.workunit.client.0.smithi153.stdout:1/204: sync 2024-04-05T03:02:45.158 INFO:tasks.workunit.client.0.smithi153.stdout:2/233: sync 2024-04-05T03:02:45.160 INFO:tasks.workunit.client.1.smithi203.stdout:7/755: mknod d8/d32/d16/d1a/ce6 0 2024-04-05T03:02:45.160 INFO:tasks.workunit.client.0.smithi153.stdout:7/228: mkdir d2/da/d42/d50 0 2024-04-05T03:02:45.161 INFO:tasks.workunit.client.1.smithi203.stdout:6/605: creat d0/da/d18/d38/d86/fb8 x:0 0 0 2024-04-05T03:02:45.161 INFO:tasks.workunit.client.0.smithi153.stdout:3/198: creat d1d/d32/f49 x:0 0 0 2024-04-05T03:02:45.162 INFO:tasks.workunit.client.1.smithi203.stdout:0/593: mknod dd/d17/d1d/d26/d34/d56/cae 0 2024-04-05T03:02:45.164 INFO:tasks.workunit.client.0.smithi153.stdout:2/234: dread d9/f11 [0,4194304] 0 2024-04-05T03:02:45.165 INFO:tasks.workunit.client.1.smithi203.stdout:2/765: rename d5/d10/db8/f56 to d5/d10/d35/d36/d41/df8/f107 0 2024-04-05T03:02:45.167 INFO:tasks.workunit.client.0.smithi153.stdout:7/229: dread - d2/da/d3a/d41/d31/f45 zero size 2024-04-05T03:02:45.169 INFO:tasks.workunit.client.1.smithi203.stdout:8/606: mkdir d1/d6/d2e/d92/daa 0 2024-04-05T03:02:45.170 INFO:tasks.workunit.client.0.smithi153.stdout:5/221: unlink d0/l12 0 2024-04-05T03:02:45.171 INFO:tasks.workunit.client.1.smithi203.stdout:8/607: write d1/d6/f56 [889546,54578] 0 2024-04-05T03:02:45.174 INFO:tasks.workunit.client.1.smithi203.stdout:1/573: rename d7/d24/d33/d48/c84 to d7/d14/d83/cb5 0 2024-04-05T03:02:45.174 INFO:tasks.workunit.client.0.smithi153.stdout:3/199: dwrite d1d/f28 [0,4194304] 0 2024-04-05T03:02:45.174 INFO:tasks.workunit.client.0.smithi153.stdout:5/222: symlink d0/de/l54 0 2024-04-05T03:02:45.174 INFO:tasks.workunit.client.0.smithi153.stdout:9/182: write d1/d9/d1b/f28 [3552461,72411] 0 2024-04-05T03:02:45.175 INFO:tasks.workunit.client.1.smithi203.stdout:1/574: chown d7/d24/d33/d48/d4b/d51/l9b 126457 1 2024-04-05T03:02:45.179 INFO:tasks.workunit.client.0.smithi153.stdout:5/223: symlink d0/de/l55 0 2024-04-05T03:02:45.181 INFO:tasks.workunit.client.1.smithi203.stdout:6/606: link d0/da/d18/d38/d2a/d2c/f4f d0/da/d18/d38/d2a/d2c/fb9 0 2024-04-05T03:02:45.182 INFO:tasks.workunit.client.0.smithi153.stdout:5/224: truncate d0/d9/d22/f28 588530 0 2024-04-05T03:02:45.182 INFO:tasks.workunit.client.1.smithi203.stdout:6/607: dread - d0/da/f96 zero size 2024-04-05T03:02:45.182 INFO:tasks.workunit.client.0.smithi153.stdout:3/200: creat d1d/d32/f4a x:0 0 0 2024-04-05T03:02:45.183 INFO:tasks.workunit.client.0.smithi153.stdout:9/183: creat d1/d10/d36/f48 x:0 0 0 2024-04-05T03:02:45.185 INFO:tasks.workunit.client.1.smithi203.stdout:9/531: rename d13/d1b/d33/d42/c70 to d13/d1b/d33/d42/d67/ca4 0 2024-04-05T03:02:45.186 INFO:tasks.workunit.client.1.smithi203.stdout:6/608: dread d0/da/f43 [0,4194304] 0 2024-04-05T03:02:45.187 INFO:tasks.workunit.client.0.smithi153.stdout:5/225: link d0/d25/f31 d0/de/f56 0 2024-04-05T03:02:45.187 INFO:tasks.workunit.client.0.smithi153.stdout:5/226: readlink d0/de/l55 0 2024-04-05T03:02:45.188 INFO:tasks.workunit.client.0.smithi153.stdout:9/184: creat d1/d9/f49 x:0 0 0 2024-04-05T03:02:45.190 INFO:tasks.workunit.client.1.smithi203.stdout:5/586: link d3/d5f/l6a d3/d4/dc/lc2 0 2024-04-05T03:02:45.190 INFO:tasks.workunit.client.0.smithi153.stdout:9/185: fdatasync d1/d10/d16/f1a 0 2024-04-05T03:02:45.192 INFO:tasks.workunit.client.0.smithi153.stdout:3/201: rename d1d/d38/c42 to d1d/d26/d3c/c4b 0 2024-04-05T03:02:45.193 INFO:tasks.workunit.client.1.smithi203.stdout:0/594: rmdir dd/d17/d1d/d22/d45/d51/d69 39 2024-04-05T03:02:45.195 INFO:tasks.workunit.client.1.smithi203.stdout:0/595: write dd/d17/d1d/d26/d28/f87 [1367653,2714] 0 2024-04-05T03:02:45.197 INFO:tasks.workunit.client.1.smithi203.stdout:8/608: unlink d1/d12/d17/d47/d53/f96 0 2024-04-05T03:02:45.197 INFO:tasks.workunit.client.0.smithi153.stdout:3/202: write f18 [2994417,45085] 0 2024-04-05T03:02:45.198 INFO:tasks.workunit.client.1.smithi203.stdout:4/403: dwrite da/f27 [4194304,4194304] 0 2024-04-05T03:02:45.199 INFO:tasks.workunit.client.1.smithi203.stdout:7/756: write d8/d32/d5d/d6d/dca/d3f/d5e/d77/f80 [854407,66813] 0 2024-04-05T03:02:45.200 INFO:tasks.workunit.client.0.smithi153.stdout:5/227: dwrite d0/de/d37/f4f [0,4194304] 0 2024-04-05T03:02:45.201 INFO:tasks.workunit.client.1.smithi203.stdout:6/609: symlink d0/da/d18/d38/d2a/d44/d78/lba 0 2024-04-05T03:02:45.203 INFO:tasks.workunit.client.0.smithi153.stdout:3/203: dread d1d/f3e [0,4194304] 0 2024-04-05T03:02:45.209 INFO:tasks.workunit.client.0.smithi153.stdout:5/228: unlink d0/f5 0 2024-04-05T03:02:45.213 INFO:tasks.workunit.client.0.smithi153.stdout:9/186: dwrite d1/d10/f19 [0,4194304] 0 2024-04-05T03:02:45.214 INFO:tasks.workunit.client.1.smithi203.stdout:2/766: truncate d5/d10/d2f/f88 38623 0 2024-04-05T03:02:45.216 INFO:tasks.workunit.client.0.smithi153.stdout:1/205: dread d5/f9 [0,4194304] 0 2024-04-05T03:02:45.218 INFO:tasks.workunit.client.1.smithi203.stdout:9/532: link d13/d1b/d26/f54 d13/d1b/d26/d51/d81/fa5 0 2024-04-05T03:02:45.218 INFO:tasks.workunit.client.0.smithi153.stdout:2/235: dread d9/da/f15 [0,4194304] 0 2024-04-05T03:02:45.218 INFO:tasks.workunit.client.0.smithi153.stdout:9/187: creat d1/db/f4a x:0 0 0 2024-04-05T03:02:45.219 INFO:tasks.workunit.client.1.smithi203.stdout:0/596: rename dd/d17/d1d/c39 to dd/d17/d1d/d22/d53/d7f/caf 0 2024-04-05T03:02:45.220 INFO:tasks.workunit.client.0.smithi153.stdout:5/229: symlink d0/de/d37/l57 0 2024-04-05T03:02:45.220 INFO:tasks.workunit.client.0.smithi153.stdout:1/206: dread d5/dd/d21/d18/f1c [0,4194304] 0 2024-04-05T03:02:45.221 INFO:tasks.workunit.client.1.smithi203.stdout:4/404: readlink da/d3c/l4d 0 2024-04-05T03:02:45.222 INFO:tasks.workunit.client.1.smithi203.stdout:9/533: fdatasync d13/f2a 0 2024-04-05T03:02:45.223 INFO:tasks.workunit.client.0.smithi153.stdout:5/230: dread d0/d9/d22/f41 [0,4194304] 0 2024-04-05T03:02:45.223 INFO:tasks.workunit.client.0.smithi153.stdout:5/231: readlink d0/de/l54 0 2024-04-05T03:02:45.224 INFO:tasks.workunit.client.0.smithi153.stdout:3/204: link d1d/d27/d2b/f3d d1d/d26/f4c 0 2024-04-05T03:02:45.225 INFO:tasks.workunit.client.1.smithi203.stdout:0/597: fsync dd/d17/d1d/d22/f3b 0 2024-04-05T03:02:45.226 INFO:tasks.workunit.client.1.smithi203.stdout:7/757: rename d8/d32/d5d/d6d/dca/d3f to d8/d32/d16/d1a/d3d/d59/de7 0 2024-04-05T03:02:45.227 INFO:tasks.workunit.client.0.smithi153.stdout:2/236: mknod d9/da/d37/c4a 0 2024-04-05T03:02:45.228 INFO:tasks.workunit.client.0.smithi153.stdout:5/232: mknod d0/d2c/c58 0 2024-04-05T03:02:45.228 INFO:tasks.workunit.client.1.smithi203.stdout:1/575: dwrite d7/d14/d2b/d58/f60 [0,4194304] 0 2024-04-05T03:02:45.229 INFO:tasks.workunit.client.0.smithi153.stdout:5/233: dread - d0/d9/f1f zero size 2024-04-05T03:02:45.230 INFO:tasks.workunit.client.1.smithi203.stdout:6/610: symlink d0/da/d54/lbb 0 2024-04-05T03:02:45.231 INFO:tasks.workunit.client.0.smithi153.stdout:3/205: creat d1d/d27/f4d x:0 0 0 2024-04-05T03:02:45.233 INFO:tasks.workunit.client.1.smithi203.stdout:6/611: chown d0/f3 483387 1 2024-04-05T03:02:45.235 INFO:tasks.workunit.client.1.smithi203.stdout:2/767: dwrite d5/d63/f75 [0,4194304] 0 2024-04-05T03:02:45.242 INFO:tasks.workunit.client.1.smithi203.stdout:6/612: creat d0/da/d54/d4c/da1/fbc x:0 0 0 2024-04-05T03:02:45.242 INFO:tasks.workunit.client.1.smithi203.stdout:5/587: truncate d3/d4/dc/f95 287454 0 2024-04-05T03:02:45.247 INFO:tasks.workunit.client.1.smithi203.stdout:7/758: symlink d8/d32/d16/d1a/d3d/d59/de7/d52/le8 0 2024-04-05T03:02:45.250 INFO:tasks.workunit.client.1.smithi203.stdout:9/534: dwrite d13/f4a [0,4194304] 0 2024-04-05T03:02:45.250 INFO:tasks.workunit.client.1.smithi203.stdout:4/405: dwrite da/f3a [0,4194304] 0 2024-04-05T03:02:45.250 INFO:tasks.workunit.client.1.smithi203.stdout:0/598: rename dd/d17/d29 to dd/d17/d1d/d22/d53/db0 0 2024-04-05T03:02:45.254 INFO:tasks.workunit.client.1.smithi203.stdout:0/599: chown dd/d17/d1d/d26/d8c/d6d 70389736 1 2024-04-05T03:02:45.254 INFO:tasks.workunit.client.1.smithi203.stdout:5/588: dread d3/d5f/f75 [0,4194304] 0 2024-04-05T03:02:45.255 INFO:tasks.workunit.client.1.smithi203.stdout:8/609: truncate d1/d6/f41 600583 0 2024-04-05T03:02:45.256 INFO:tasks.workunit.client.1.smithi203.stdout:6/613: write d0/da/d54/d4c/da1/db7/f7a [273721,21985] 0 2024-04-05T03:02:45.256 INFO:tasks.workunit.client.1.smithi203.stdout:7/759: rmdir d8/d32/d5d/d6d/dca 39 2024-04-05T03:02:45.256 INFO:tasks.workunit.client.1.smithi203.stdout:2/768: dread d5/d10/d35/d2d/d7c/f25 [4194304,4194304] 0 2024-04-05T03:02:45.258 INFO:tasks.workunit.client.1.smithi203.stdout:9/535: mknod d13/d1b/d33/d64/ca6 0 2024-04-05T03:02:45.259 INFO:tasks.workunit.client.1.smithi203.stdout:4/406: symlink da/d29/d2e/l7e 0 2024-04-05T03:02:45.261 INFO:tasks.workunit.client.1.smithi203.stdout:9/536: chown d13/d1b/d26/d51 46470532 1 2024-04-05T03:02:45.261 INFO:tasks.workunit.client.1.smithi203.stdout:6/614: chown d0/da/d18/d38/d2a 1 1 2024-04-05T03:02:45.263 INFO:tasks.workunit.client.1.smithi203.stdout:7/760: write d8/d32/d16/d1a/d30/f98 [5038918,112053] 0 2024-04-05T03:02:45.263 INFO:tasks.workunit.client.1.smithi203.stdout:0/600: fsync dd/d17/d1d/d26/d34/d56/fab 0 2024-04-05T03:02:45.266 INFO:tasks.workunit.client.1.smithi203.stdout:5/589: dwrite d3/f32 [0,4194304] 0 2024-04-05T03:02:45.270 INFO:tasks.workunit.client.1.smithi203.stdout:8/610: rename d1/d6/d2e/f94 to d1/d6/fab 0 2024-04-05T03:02:45.271 INFO:tasks.workunit.client.1.smithi203.stdout:5/590: chown d3/d1f/d59/d60/fad 108388604 1 2024-04-05T03:02:45.273 INFO:tasks.workunit.client.1.smithi203.stdout:4/407: creat da/d29/d2e/d57/f7f x:0 0 0 2024-04-05T03:02:45.274 INFO:tasks.workunit.client.1.smithi203.stdout:7/761: dwrite d8/d32/f44 [4194304,4194304] 0 2024-04-05T03:02:45.279 INFO:tasks.workunit.client.1.smithi203.stdout:2/769: symlink d5/d10/d35/d2d/d7c/d29/db1/l108 0 2024-04-05T03:02:45.282 INFO:tasks.workunit.client.1.smithi203.stdout:9/537: creat d13/d1b/d6b/d9e/d77/fa7 x:0 0 0 2024-04-05T03:02:45.285 INFO:tasks.workunit.client.1.smithi203.stdout:5/591: symlink d3/d5f/d92/lc3 0 2024-04-05T03:02:45.288 INFO:tasks.workunit.client.1.smithi203.stdout:8/611: rename d1/d6/d2e/d92 to d1/d4b/d68/d75/dac 0 2024-04-05T03:02:45.289 INFO:tasks.workunit.client.1.smithi203.stdout:4/408: mkdir da/d29/d6a/d80 0 2024-04-05T03:02:45.290 INFO:tasks.workunit.client.1.smithi203.stdout:7/762: creat d8/d32/dc3/dc8/fe9 x:0 0 0 2024-04-05T03:02:45.292 INFO:tasks.workunit.client.1.smithi203.stdout:2/770: mkdir d5/d10/d35/d36/d109 0 2024-04-05T03:02:45.296 INFO:tasks.workunit.client.1.smithi203.stdout:7/763: dwrite d8/d32/d16/fa6 [0,4194304] 0 2024-04-05T03:02:45.296 INFO:tasks.workunit.client.0.smithi153.stdout:9/188: rename d1/d10/d16/l21 to d1/d1d/l4b 0 2024-04-05T03:02:45.296 INFO:tasks.workunit.client.1.smithi203.stdout:7/764: dread - d8/d32/dc3/dc8/fe9 zero size 2024-04-05T03:02:45.297 INFO:tasks.workunit.client.0.smithi153.stdout:5/234: dread d0/d9/f19 [0,4194304] 0 2024-04-05T03:02:45.298 INFO:tasks.workunit.client.0.smithi153.stdout:7/230: dread d2/f6 [0,4194304] 0 2024-04-05T03:02:45.300 INFO:tasks.workunit.client.0.smithi153.stdout:7/231: chown d2/da/d11/d49/f3d 115 1 2024-04-05T03:02:45.300 INFO:tasks.workunit.client.1.smithi203.stdout:9/538: mknod d13/d5c/d9d/ca8 0 2024-04-05T03:02:45.300 INFO:tasks.workunit.client.0.smithi153.stdout:2/237: mkdir d9/da/d36/d38/d4b 0 2024-04-05T03:02:45.303 INFO:tasks.workunit.client.0.smithi153.stdout:7/232: write d2/f5 [5109756,76372] 0 2024-04-05T03:02:45.303 INFO:tasks.workunit.client.0.smithi153.stdout:9/189: chown d1/d9/d1b/c3d 1009865 1 2024-04-05T03:02:45.303 INFO:tasks.workunit.client.0.smithi153.stdout:2/238: write d9/da/d36/d38/d39/f45 [59555,41961] 0 2024-04-05T03:02:45.304 INFO:tasks.workunit.client.1.smithi203.stdout:2/771: creat d5/d66/f10a x:0 0 0 2024-04-05T03:02:45.304 INFO:tasks.workunit.client.0.smithi153.stdout:5/235: dread d0/f16 [0,4194304] 0 2024-04-05T03:02:45.305 INFO:tasks.workunit.client.1.smithi203.stdout:2/772: readlink d5/d10/d2f/ld9 0 2024-04-05T03:02:45.306 INFO:tasks.workunit.client.0.smithi153.stdout:3/206: rename d1d/f25 to d1d/d27/d2b/f4e 0 2024-04-05T03:02:45.306 INFO:tasks.workunit.client.0.smithi153.stdout:7/233: stat d2/da/d11/f2d 0 2024-04-05T03:02:45.306 INFO:tasks.workunit.client.1.smithi203.stdout:8/612: mknod d1/d90/da2/cad 0 2024-04-05T03:02:45.307 INFO:tasks.workunit.client.1.smithi203.stdout:4/409: creat da/d71/f81 x:0 0 0 2024-04-05T03:02:45.309 INFO:tasks.workunit.client.1.smithi203.stdout:6/615: write d0/da/f13 [344530,114584] 0 2024-04-05T03:02:45.311 INFO:tasks.workunit.client.0.smithi153.stdout:9/190: dread d1/f1f [0,4194304] 0 2024-04-05T03:02:45.311 INFO:tasks.workunit.client.1.smithi203.stdout:6/616: truncate d0/da/f13 4076421 0 2024-04-05T03:02:45.312 INFO:tasks.workunit.client.0.smithi153.stdout:9/191: dread - d1/f25 zero size 2024-04-05T03:02:45.313 INFO:tasks.workunit.client.1.smithi203.stdout:6/617: fsync d0/da/d18/d38/d86/f5a 0 2024-04-05T03:02:45.314 INFO:tasks.workunit.client.1.smithi203.stdout:0/601: truncate dd/d17/d1d/d22/f65 1908230 0 2024-04-05T03:02:45.315 INFO:tasks.workunit.client.1.smithi203.stdout:6/618: truncate d0/da/f96 716324 0 2024-04-05T03:02:45.315 INFO:tasks.workunit.client.0.smithi153.stdout:2/239: dwrite d9/da/d19/d2e/d48/f33 [0,4194304] 0 2024-04-05T03:02:45.316 INFO:tasks.workunit.client.1.smithi203.stdout:7/765: symlink d8/d32/d5d/d6d/lea 0 2024-04-05T03:02:45.317 INFO:tasks.workunit.client.1.smithi203.stdout:2/773: mkdir d5/d10/d35/d2d/d7c/d72/d10b 0 2024-04-05T03:02:45.318 INFO:tasks.workunit.client.1.smithi203.stdout:6/619: write d0/da/d54/d53/f8d [269450,97813] 0 2024-04-05T03:02:45.318 INFO:tasks.workunit.client.1.smithi203.stdout:9/539: dread d13/d1b/d26/f4c [0,4194304] 0 2024-04-05T03:02:45.319 INFO:tasks.workunit.client.1.smithi203.stdout:4/410: mknod da/d29/c82 0 2024-04-05T03:02:45.322 INFO:tasks.workunit.client.0.smithi153.stdout:9/192: dwrite d1/db/f45 [0,4194304] 0 2024-04-05T03:02:45.324 INFO:tasks.workunit.client.0.smithi153.stdout:5/236: getdents d0 0 2024-04-05T03:02:45.326 INFO:tasks.workunit.client.1.smithi203.stdout:5/592: dread d3/d4/dc/f54 [0,4194304] 0 2024-04-05T03:02:45.328 INFO:tasks.workunit.client.1.smithi203.stdout:5/593: chown d3/d4 3 1 2024-04-05T03:02:45.328 INFO:tasks.workunit.client.0.smithi153.stdout:5/237: dread d0/d25/f31 [0,4194304] 0 2024-04-05T03:02:45.328 INFO:tasks.workunit.client.1.smithi203.stdout:5/594: stat d3 0 2024-04-05T03:02:45.330 INFO:tasks.workunit.client.0.smithi153.stdout:5/238: write d0/d9/f20 [893933,18803] 0 2024-04-05T03:02:45.330 INFO:tasks.workunit.client.0.smithi153.stdout:9/193: unlink d1/l12 0 2024-04-05T03:02:45.331 INFO:tasks.workunit.client.1.smithi203.stdout:9/540: mkdir d13/d1b/d6b/d9e/d77/da9 0 2024-04-05T03:02:45.331 INFO:tasks.workunit.client.1.smithi203.stdout:2/774: dwrite d5/d10/d35/d2d/d7c/d72/d8f/fb9 [0,4194304] 0 2024-04-05T03:02:45.331 INFO:tasks.workunit.client.1.smithi203.stdout:0/602: dread dd/d17/d1d/d22/d45/d51/f86 [0,4194304] 0 2024-04-05T03:02:45.333 INFO:tasks.workunit.client.0.smithi153.stdout:2/240: dwrite d9/f11 [0,4194304] 0 2024-04-05T03:02:45.333 INFO:tasks.workunit.client.1.smithi203.stdout:2/775: chown d5/d10/d1a/f34 636456190 1 2024-04-05T03:02:45.335 INFO:tasks.workunit.client.1.smithi203.stdout:7/766: mkdir d8/d32/d5d/deb 0 2024-04-05T03:02:45.336 INFO:tasks.workunit.client.0.smithi153.stdout:9/194: write d1/d10/d16/f38 [590652,36169] 0 2024-04-05T03:02:45.336 INFO:tasks.workunit.client.0.smithi153.stdout:9/195: stat d1/d10/c43 0 2024-04-05T03:02:45.337 INFO:tasks.workunit.client.1.smithi203.stdout:4/411: symlink da/l83 0 2024-04-05T03:02:45.340 INFO:tasks.workunit.client.1.smithi203.stdout:9/541: dwrite d13/f20 [4194304,4194304] 0 2024-04-05T03:02:45.340 INFO:tasks.workunit.client.0.smithi153.stdout:5/239: rename d0/d25/f35 to d0/d2c/d3f/f59 0 2024-04-05T03:02:45.343 INFO:tasks.workunit.client.1.smithi203.stdout:8/613: truncate d1/d12/d1a/f42 3418380 0 2024-04-05T03:02:45.344 INFO:tasks.workunit.client.0.smithi153.stdout:2/241: mknod d9/da/d36/d38/d39/c4c 0 2024-04-05T03:02:45.346 INFO:tasks.workunit.client.0.smithi153.stdout:9/196: dwrite d1/d10/d16/f27 [0,4194304] 0 2024-04-05T03:02:45.350 INFO:tasks.workunit.client.1.smithi203.stdout:5/595: fsync d3/d4/dc/f37 0 2024-04-05T03:02:45.351 INFO:tasks.workunit.client.1.smithi203.stdout:5/596: readlink d3/d1f/d59/d64/d12/l97 0 2024-04-05T03:02:45.353 INFO:tasks.workunit.client.0.smithi153.stdout:3/207: truncate f18 1492375 0 2024-04-05T03:02:45.353 INFO:tasks.workunit.client.0.smithi153.stdout:2/242: rmdir d9/da/d36/d38 39 2024-04-05T03:02:45.354 INFO:tasks.workunit.client.1.smithi203.stdout:7/767: truncate d8/da/f14 983606 0 2024-04-05T03:02:45.355 INFO:tasks.workunit.client.1.smithi203.stdout:9/542: dwrite d13/d1b/d33/d63/f6e [0,4194304] 0 2024-04-05T03:02:45.355 INFO:tasks.workunit.client.1.smithi203.stdout:5/597: readlink d3/d1f/d59/l86 0 2024-04-05T03:02:45.356 INFO:tasks.workunit.client.1.smithi203.stdout:4/412: chown da/d29/l72 204275 1 2024-04-05T03:02:45.357 INFO:tasks.workunit.client.1.smithi203.stdout:5/598: stat d3/d1f/d59/d64/f3d 0 2024-04-05T03:02:45.357 INFO:tasks.workunit.client.0.smithi153.stdout:7/234: write d2/da/d11/f13 [1594297,123824] 0 2024-04-05T03:02:45.359 INFO:tasks.workunit.client.0.smithi153.stdout:2/243: fdatasync d9/da/d19/d2e/d48/f1a 0 2024-04-05T03:02:45.360 INFO:tasks.workunit.client.1.smithi203.stdout:0/603: mkdir dd/db1 0 2024-04-05T03:02:45.360 INFO:tasks.workunit.client.1.smithi203.stdout:9/543: write d13/d1b/d26/f6f [1865455,59843] 0 2024-04-05T03:02:45.360 INFO:tasks.workunit.client.0.smithi153.stdout:2/244: readlink l3 0 2024-04-05T03:02:45.362 INFO:tasks.workunit.client.1.smithi203.stdout:0/604: chown dd/d17/d1d/d26/d28/f8b 167418862 1 2024-04-05T03:02:45.362 INFO:tasks.workunit.client.0.smithi153.stdout:3/208: creat d1d/d26/d3c/d3f/f4f x:0 0 0 2024-04-05T03:02:45.362 INFO:tasks.workunit.client.0.smithi153.stdout:5/240: dwrite d0/d25/f2f [0,4194304] 0 2024-04-05T03:02:45.363 INFO:tasks.workunit.client.1.smithi203.stdout:0/605: chown dd/d17/d1d/d22/d53/d61/f92 21 1 2024-04-05T03:02:45.364 INFO:tasks.workunit.client.1.smithi203.stdout:4/413: rename f5 to da/d29/d6a/d80/f84 0 2024-04-05T03:02:45.365 INFO:tasks.workunit.client.1.smithi203.stdout:0/606: write dd/d17/d1d/d22/f9c [286990,110593] 0 2024-04-05T03:02:45.365 INFO:tasks.workunit.client.0.smithi153.stdout:9/197: dwrite d1/d9/d1b/f26 [0,4194304] 0 2024-04-05T03:02:45.367 INFO:tasks.workunit.client.0.smithi153.stdout:7/235: write d2/f1d [2996830,117387] 0 2024-04-05T03:02:45.371 INFO:tasks.workunit.client.0.smithi153.stdout:9/198: write d1/d10/d16/f17 [4051412,110921] 0 2024-04-05T03:02:45.371 INFO:tasks.workunit.client.0.smithi153.stdout:5/241: creat d0/d25/d4d/f5a x:0 0 0 2024-04-05T03:02:45.372 INFO:tasks.workunit.client.1.smithi203.stdout:4/414: dread da/d3c/f43 [4194304,4194304] 0 2024-04-05T03:02:45.373 INFO:tasks.workunit.client.1.smithi203.stdout:5/599: creat d3/d1f/fc4 x:0 0 0 2024-04-05T03:02:45.374 INFO:tasks.workunit.client.1.smithi203.stdout:4/415: stat da/d3c/d4c/f73 0 2024-04-05T03:02:45.376 INFO:tasks.workunit.client.1.smithi203.stdout:6/620: truncate d0/da/d54/d53/f59 856758 0 2024-04-05T03:02:45.377 INFO:tasks.workunit.client.0.smithi153.stdout:2/245: creat d9/da/d36/f4d x:0 0 0 2024-04-05T03:02:45.382 INFO:tasks.workunit.client.0.smithi153.stdout:7/236: mkdir d2/d51 0 2024-04-05T03:02:45.382 INFO:tasks.workunit.client.0.smithi153.stdout:3/209: dwrite d1d/d26/f47 [0,4194304] 0 2024-04-05T03:02:45.383 INFO:tasks.workunit.client.0.smithi153.stdout:5/242: chown d0/d2c/d3f/l47 8335173 1 2024-04-05T03:02:45.384 INFO:tasks.workunit.client.1.smithi203.stdout:5/600: dread - d3/d4/dc/f79 zero size 2024-04-05T03:02:45.385 INFO:tasks.workunit.client.1.smithi203.stdout:4/416: mknod da/d29/d6a/d80/c85 0 2024-04-05T03:02:45.386 INFO:tasks.workunit.client.1.smithi203.stdout:2/776: dwrite d5/d10/d35/d2d/d7c/d29/d67/f77 [0,4194304] 0 2024-04-05T03:02:45.387 INFO:tasks.workunit.client.1.smithi203.stdout:4/417: chown da/f74 7440 1 2024-04-05T03:02:45.388 INFO:tasks.workunit.client.1.smithi203.stdout:0/607: creat dd/d17/d1d/d95/fb2 x:0 0 0 2024-04-05T03:02:45.389 INFO:tasks.workunit.client.1.smithi203.stdout:6/621: creat d0/da/d54/d53/fbd x:0 0 0 2024-04-05T03:02:45.390 INFO:tasks.workunit.client.0.smithi153.stdout:2/246: mknod d9/da/d36/c4e 0 2024-04-05T03:02:45.390 INFO:tasks.workunit.client.1.smithi203.stdout:8/614: write d1/d12/d17/d10/f15 [158139,116321] 0 2024-04-05T03:02:45.394 INFO:tasks.workunit.client.0.smithi153.stdout:3/210: rmdir d1d/d41 0 2024-04-05T03:02:45.396 INFO:tasks.workunit.client.0.smithi153.stdout:9/199: dwrite d1/d10/d16/f1a [4194304,4194304] 0 2024-04-05T03:02:45.397 INFO:tasks.workunit.client.1.smithi203.stdout:5/601: rmdir d3/d4/d94 39 2024-04-05T03:02:45.397 INFO:tasks.workunit.client.1.smithi203.stdout:2/777: mknod d5/d10/d1a/d57/c10c 0 2024-04-05T03:02:45.400 INFO:tasks.workunit.client.1.smithi203.stdout:4/418: creat da/d29/d2e/d57/f86 x:0 0 0 2024-04-05T03:02:45.400 INFO:tasks.workunit.client.1.smithi203.stdout:0/608: dread dd/d17/f40 [0,4194304] 0 2024-04-05T03:02:45.401 INFO:tasks.workunit.client.1.smithi203.stdout:6/622: dwrite d0/da/d18/d38/d2a/d44/d78/f89 [0,4194304] 0 2024-04-05T03:02:45.403 INFO:tasks.workunit.client.1.smithi203.stdout:8/615: creat d1/d12/d2f/d1f/fae x:0 0 0 2024-04-05T03:02:45.403 INFO:tasks.workunit.client.0.smithi153.stdout:2/247: dwrite d9/da/f49 [0,4194304] 0 2024-04-05T03:02:45.404 INFO:tasks.workunit.client.0.smithi153.stdout:7/237: creat d2/da/d11/f52 x:0 0 0 2024-04-05T03:02:45.404 INFO:tasks.workunit.client.0.smithi153.stdout:5/243: link d0/d2c/c50 d0/de/c5b 0 2024-04-05T03:02:45.405 INFO:tasks.workunit.client.1.smithi203.stdout:5/602: mknod d3/d4/dc/d67/dac/cc5 0 2024-04-05T03:02:45.406 INFO:tasks.workunit.client.0.smithi153.stdout:5/244: chown d0/de/f10 28157 1 2024-04-05T03:02:45.407 INFO:tasks.workunit.client.1.smithi203.stdout:4/419: write da/d29/d6a/f6c [927749,63548] 0 2024-04-05T03:02:45.408 INFO:tasks.workunit.client.0.smithi153.stdout:3/211: rename c1 to d1d/d26/d3c/c50 0 2024-04-05T03:02:45.409 INFO:tasks.workunit.client.1.smithi203.stdout:0/609: truncate dd/d17/d1d/d26/f3d 3216864 0 2024-04-05T03:02:45.410 INFO:tasks.workunit.client.0.smithi153.stdout:2/248: mkdir d9/da/d19/d2e/d4f 0 2024-04-05T03:02:45.411 INFO:tasks.workunit.client.1.smithi203.stdout:0/610: write dd/f44 [1073216,54559] 0 2024-04-05T03:02:45.411 INFO:tasks.workunit.client.0.smithi153.stdout:6/277: sync 2024-04-05T03:02:45.411 INFO:tasks.workunit.client.0.smithi153.stdout:4/294: sync 2024-04-05T03:02:45.411 INFO:tasks.workunit.client.0.smithi153.stdout:8/230: sync 2024-04-05T03:02:45.412 INFO:tasks.workunit.client.0.smithi153.stdout:0/323: sync 2024-04-05T03:02:45.412 INFO:tasks.workunit.client.1.smithi203.stdout:4/420: creat da/d3c/d4c/f87 x:0 0 0 2024-04-05T03:02:45.413 INFO:tasks.workunit.client.0.smithi153.stdout:2/249: fsync d9/da/d19/d2e/d48/f33 0 2024-04-05T03:02:45.414 INFO:tasks.workunit.client.1.smithi203.stdout:5/603: symlink d3/d4/lc6 0 2024-04-05T03:02:45.415 INFO:tasks.workunit.client.0.smithi153.stdout:9/200: creat d1/d10/f4c x:0 0 0 2024-04-05T03:02:45.416 INFO:tasks.workunit.client.1.smithi203.stdout:5/604: readlink d3/d4/dc/l90 0 2024-04-05T03:02:45.417 INFO:tasks.workunit.client.0.smithi153.stdout:3/212: dread d1d/d27/d2b/f2d [0,4194304] 0 2024-04-05T03:02:45.417 INFO:tasks.workunit.client.1.smithi203.stdout:8/616: truncate d1/d12/d17/d10/f15 2387492 0 2024-04-05T03:02:45.417 INFO:tasks.workunit.client.1.smithi203.stdout:6/623: creat d0/da/d18/d38/fbe x:0 0 0 2024-04-05T03:02:45.418 INFO:tasks.workunit.client.0.smithi153.stdout:7/238: fsync d2/da/d11/f18 0 2024-04-05T03:02:45.418 INFO:tasks.workunit.client.0.smithi153.stdout:8/231: mknod d7/d46/c4d 0 2024-04-05T03:02:45.418 INFO:tasks.workunit.client.1.smithi203.stdout:4/421: mknod da/d3c/d4c/c88 0 2024-04-05T03:02:45.419 INFO:tasks.workunit.client.1.smithi203.stdout:8/617: chown d1/f34 0 1 2024-04-05T03:02:45.420 INFO:tasks.workunit.client.1.smithi203.stdout:4/422: chown da/d29/f2b 232 1 2024-04-05T03:02:45.422 INFO:tasks.workunit.client.0.smithi153.stdout:4/295: mknod da/db/d20/d35/c5d 0 2024-04-05T03:02:45.423 INFO:tasks.workunit.client.0.smithi153.stdout:0/324: mknod d9/df/d1d/c5d 0 2024-04-05T03:02:45.424 INFO:tasks.workunit.client.1.smithi203.stdout:9/544: truncate d13/d1b/d26/f4c 786802 0 2024-04-05T03:02:45.424 INFO:tasks.workunit.client.0.smithi153.stdout:8/232: dread d7/d13/d19/d31/f42 [0,4194304] 0 2024-04-05T03:02:45.425 INFO:tasks.workunit.client.1.smithi203.stdout:9/545: write d13/f24 [2623581,86795] 0 2024-04-05T03:02:45.427 INFO:tasks.workunit.client.1.smithi203.stdout:5/605: creat d3/fc7 x:0 0 0 2024-04-05T03:02:45.429 INFO:tasks.workunit.client.1.smithi203.stdout:4/423: dread da/d29/d2e/f45 [0,4194304] 0 2024-04-05T03:02:45.430 INFO:tasks.workunit.client.0.smithi153.stdout:5/245: link d0/de/l15 d0/de/l5c 0 2024-04-05T03:02:45.431 INFO:tasks.workunit.client.1.smithi203.stdout:4/424: dread - da/d3c/f44 zero size 2024-04-05T03:02:45.431 INFO:tasks.workunit.client.0.smithi153.stdout:8/233: dwrite d7/d13/f15 [0,4194304] 0 2024-04-05T03:02:45.432 INFO:tasks.workunit.client.0.smithi153.stdout:4/296: mknod da/de/d28/c5e 0 2024-04-05T03:02:45.435 INFO:tasks.workunit.client.1.smithi203.stdout:6/624: link d0/da/f10 d0/da/d54/d4c/da1/db7/fbf 0 2024-04-05T03:02:45.435 INFO:tasks.workunit.client.1.smithi203.stdout:5/606: read d3/d4/dc/f25 [3544326,43313] 0 2024-04-05T03:02:45.435 INFO:tasks.workunit.client.1.smithi203.stdout:4/425: fdatasync da/fc 0 2024-04-05T03:02:45.438 INFO:tasks.workunit.client.0.smithi153.stdout:0/325: mknod d9/df/d1d/c5e 0 2024-04-05T03:02:45.440 INFO:tasks.workunit.client.1.smithi203.stdout:5/607: chown d3/d1f/d59/d64/c2f 8128 1 2024-04-05T03:02:45.441 INFO:tasks.workunit.client.1.smithi203.stdout:4/426: unlink da/f3a 0 2024-04-05T03:02:45.444 INFO:tasks.workunit.client.0.smithi153.stdout:5/246: mknod d0/c5d 0 2024-04-05T03:02:45.446 INFO:tasks.workunit.client.0.smithi153.stdout:4/297: mknod da/db/d20/d35/d38/d3e/c5f 0 2024-04-05T03:02:45.446 INFO:tasks.workunit.client.1.smithi203.stdout:6/625: dwrite d0/da/d54/d4c/da1/db7/f5f [0,4194304] 0 2024-04-05T03:02:45.448 INFO:tasks.workunit.client.0.smithi153.stdout:4/298: dread da/db/d20/f21 [0,4194304] 0 2024-04-05T03:02:45.448 INFO:tasks.workunit.client.1.smithi203.stdout:2/778: dread d5/d10/d35/d2d/d7c/f25 [0,4194304] 0 2024-04-05T03:02:45.449 INFO:tasks.workunit.client.1.smithi203.stdout:9/546: truncate d13/d1b/d26/d31/f62 376571 0 2024-04-05T03:02:45.450 INFO:tasks.workunit.client.1.smithi203.stdout:4/427: read - da/d29/f50 zero size 2024-04-05T03:02:45.452 INFO:tasks.workunit.client.1.smithi203.stdout:3/494: sync 2024-04-05T03:02:45.454 INFO:tasks.workunit.client.1.smithi203.stdout:5/608: fdatasync d3/f36 0 2024-04-05T03:02:45.454 INFO:tasks.workunit.client.0.smithi153.stdout:7/239: rename d2/da/d42/d50 to d2/da/d53 0 2024-04-05T03:02:45.456 INFO:tasks.workunit.client.0.smithi153.stdout:6/278: getdents d4/dc/df/d16/d24/d4e 0 2024-04-05T03:02:45.457 INFO:tasks.workunit.client.1.smithi203.stdout:9/547: symlink d13/d5c/d9d/laa 0 2024-04-05T03:02:45.457 INFO:tasks.workunit.client.1.smithi203.stdout:2/779: write d5/d10/db8/dab/fa8 [3996359,46108] 0 2024-04-05T03:02:45.458 INFO:tasks.workunit.client.0.smithi153.stdout:0/326: mkdir d9/df/d5f 0 2024-04-05T03:02:45.459 INFO:tasks.workunit.client.0.smithi153.stdout:0/327: fdatasync d9/df/f17 0 2024-04-05T03:02:45.460 INFO:tasks.workunit.client.1.smithi203.stdout:8/618: dread d1/d12/d2f/d1f/f50 [4194304,4194304] 0 2024-04-05T03:02:45.461 INFO:tasks.workunit.client.0.smithi153.stdout:2/250: write d9/da/f24 [591233,75654] 0 2024-04-05T03:02:45.461 INFO:tasks.workunit.client.1.smithi203.stdout:6/626: rename d0/da/d18/f8e to d0/da/d18/d38/d2a/d44/fc0 0 2024-04-05T03:02:45.461 INFO:tasks.workunit.client.0.smithi153.stdout:4/299: mknod da/db/d20/c60 0 2024-04-05T03:02:45.462 INFO:tasks.workunit.client.0.smithi153.stdout:2/251: write d9/ff [9347911,82633] 0 2024-04-05T03:02:45.463 INFO:tasks.workunit.client.1.smithi203.stdout:5/609: mknod d3/d1f/d49/cc8 0 2024-04-05T03:02:45.466 INFO:tasks.workunit.client.1.smithi203.stdout:2/780: readlink d5/l11 0 2024-04-05T03:02:45.466 INFO:tasks.workunit.client.1.smithi203.stdout:9/548: creat d13/d1b/d26/d51/fab x:0 0 0 2024-04-05T03:02:45.466 INFO:tasks.workunit.client.0.smithi153.stdout:3/213: getdents d1d/d26/d3c/d3f 0 2024-04-05T03:02:45.466 INFO:tasks.workunit.client.0.smithi153.stdout:3/214: dread - d1d/d26/d3c/f44 zero size 2024-04-05T03:02:45.467 INFO:tasks.workunit.client.1.smithi203.stdout:3/495: fsync d0/dc/d1a/d1d/d43/f4c 0 2024-04-05T03:02:45.468 INFO:tasks.workunit.client.0.smithi153.stdout:7/240: mkdir d2/da/d11/d49/d54 0 2024-04-05T03:02:45.468 INFO:tasks.workunit.client.1.smithi203.stdout:8/619: fsync d1/d6/d2e/f30 0 2024-04-05T03:02:45.469 INFO:tasks.workunit.client.0.smithi153.stdout:4/300: dwrite f7 [0,4194304] 0 2024-04-05T03:02:45.471 INFO:tasks.workunit.client.0.smithi153.stdout:8/234: creat d7/d13/d19/f4e x:0 0 0 2024-04-05T03:02:45.472 INFO:tasks.workunit.client.1.smithi203.stdout:4/428: rename da/d29/l46 to da/d3c/l89 0 2024-04-05T03:02:45.475 INFO:tasks.workunit.client.0.smithi153.stdout:3/215: readlink lf 0 2024-04-05T03:02:45.477 INFO:tasks.workunit.client.0.smithi153.stdout:7/241: mkdir d2/da/d11/d49/d55 0 2024-04-05T03:02:45.478 INFO:tasks.workunit.client.1.smithi203.stdout:3/496: rmdir d0/d21 39 2024-04-05T03:02:45.479 INFO:tasks.workunit.client.1.smithi203.stdout:8/620: dread d1/d6/d2e/f6b [0,4194304] 0 2024-04-05T03:02:45.481 INFO:tasks.workunit.client.1.smithi203.stdout:9/549: symlink d13/d1b/lac 0 2024-04-05T03:02:45.483 INFO:tasks.workunit.client.0.smithi153.stdout:8/235: dwrite d7/d13/d19/f4e [0,4194304] 0 2024-04-05T03:02:45.483 INFO:tasks.workunit.client.0.smithi153.stdout:5/247: mkdir d0/d2c/d5e 0 2024-04-05T03:02:45.483 INFO:tasks.workunit.client.1.smithi203.stdout:5/610: dwrite d3/d4/fb [12582912,4194304] 0 2024-04-05T03:02:45.487 INFO:tasks.workunit.client.0.smithi153.stdout:6/279: dwrite d4/fb [0,4194304] 0 2024-04-05T03:02:45.488 INFO:tasks.workunit.client.1.smithi203.stdout:4/429: symlink da/d29/l8a 0 2024-04-05T03:02:45.488 INFO:tasks.workunit.client.0.smithi153.stdout:2/252: dread d9/da/f20 [0,4194304] 0 2024-04-05T03:02:45.490 INFO:tasks.workunit.client.1.smithi203.stdout:4/430: truncate da/d29/d2e/d53/f6f 576513 0 2024-04-05T03:02:45.490 INFO:tasks.workunit.client.0.smithi153.stdout:7/242: readlink d2/da/d47/l4a 0 2024-04-05T03:02:45.490 INFO:tasks.workunit.client.0.smithi153.stdout:0/328: creat d9/db/d42/f60 x:0 0 0 2024-04-05T03:02:45.491 INFO:tasks.workunit.client.1.smithi203.stdout:4/431: chown da/d29/d6a/d80/c85 0 1 2024-04-05T03:02:45.492 INFO:tasks.workunit.client.0.smithi153.stdout:4/301: creat da/de/d56/f61 x:0 0 0 2024-04-05T03:02:45.492 INFO:tasks.workunit.client.0.smithi153.stdout:8/236: rmdir d7 39 2024-04-05T03:02:45.493 INFO:tasks.workunit.client.1.smithi203.stdout:4/432: write da/d29/d2e/d57/f86 [768649,107059] 0 2024-04-05T03:02:45.495 INFO:tasks.workunit.client.1.smithi203.stdout:2/781: getdents d5/d10/db8/dab/de1/dbd 0 2024-04-05T03:02:45.499 INFO:tasks.workunit.client.1.smithi203.stdout:5/611: creat d3/d5f/d92/fc9 x:0 0 0 2024-04-05T03:02:45.500 INFO:tasks.workunit.client.0.smithi153.stdout:0/329: chown d9/df/d1e/d45/c51 0 1 2024-04-05T03:02:45.501 INFO:tasks.workunit.client.0.smithi153.stdout:6/280: mkdir d4/dc/df/d16/d36/d4d/d64 0 2024-04-05T03:02:45.502 INFO:tasks.workunit.client.1.smithi203.stdout:6/627: unlink d0/da/d18/d38/d86/f7d 0 2024-04-05T03:02:45.502 INFO:tasks.workunit.client.0.smithi153.stdout:7/243: creat d2/da/d11/d49/f56 x:0 0 0 2024-04-05T03:02:45.504 INFO:tasks.workunit.client.0.smithi153.stdout:4/302: creat da/db/d20/d35/d41/f62 x:0 0 0 2024-04-05T03:02:45.504 INFO:tasks.workunit.client.1.smithi203.stdout:4/433: mkdir da/d29/d2e/d8b 0 2024-04-05T03:02:45.505 INFO:tasks.workunit.client.0.smithi153.stdout:4/303: stat da/d2a 0 2024-04-05T03:02:45.506 INFO:tasks.workunit.client.0.smithi153.stdout:5/248: mkdir d0/de/d5f 0 2024-04-05T03:02:45.506 INFO:tasks.workunit.client.1.smithi203.stdout:2/782: creat d5/d10/d2f/d5d/f10d x:0 0 0 2024-04-05T03:02:45.506 INFO:tasks.workunit.client.1.smithi203.stdout:5/612: mkdir d3/d1f/d59/d60/dca 0 2024-04-05T03:02:45.507 INFO:tasks.workunit.client.1.smithi203.stdout:8/621: dwrite d1/d12/f6c [0,4194304] 0 2024-04-05T03:02:45.508 INFO:tasks.workunit.client.1.smithi203.stdout:3/497: dwrite d0/d21/f6c [4194304,4194304] 0 2024-04-05T03:02:45.509 INFO:tasks.workunit.client.1.smithi203.stdout:6/628: mknod d0/da/d18/d38/d2a/d2c/cc1 0 2024-04-05T03:02:45.509 INFO:tasks.workunit.client.0.smithi153.stdout:3/216: dwrite d1d/d26/f4c [0,4194304] 0 2024-04-05T03:02:45.511 INFO:tasks.workunit.client.0.smithi153.stdout:6/281: dread - d4/f7 zero size 2024-04-05T03:02:45.512 INFO:tasks.workunit.client.0.smithi153.stdout:0/330: read f7 [1193961,72870] 0 2024-04-05T03:02:45.513 INFO:tasks.workunit.client.0.smithi153.stdout:6/282: stat d4/dc/f46 0 2024-04-05T03:02:45.513 INFO:tasks.workunit.client.1.smithi203.stdout:4/434: dread - da/d29/d2e/f7a zero size 2024-04-05T03:02:45.514 INFO:tasks.workunit.client.1.smithi203.stdout:8/622: chown d1/d12/d2f/l4c 127848915 1 2024-04-05T03:02:45.514 INFO:tasks.workunit.client.0.smithi153.stdout:2/253: dwrite d9/da/d36/d38/d39/f45 [0,4194304] 0 2024-04-05T03:02:45.514 INFO:tasks.workunit.client.0.smithi153.stdout:8/237: fdatasync d7/f16 0 2024-04-05T03:02:45.515 INFO:tasks.workunit.client.1.smithi203.stdout:4/435: chown da/d29/c62 14084 1 2024-04-05T03:02:45.516 INFO:tasks.workunit.client.0.smithi153.stdout:5/249: unlink d0/d9/f19 0 2024-04-05T03:02:45.516 INFO:tasks.workunit.client.1.smithi203.stdout:2/783: unlink d5/d10/d35/d36/d41/df8/f107 0 2024-04-05T03:02:45.517 INFO:tasks.workunit.client.0.smithi153.stdout:3/217: mknod d1d/d27/d2b/c51 0 2024-04-05T03:02:45.519 INFO:tasks.workunit.client.0.smithi153.stdout:3/218: stat d1d/f21 0 2024-04-05T03:02:45.522 INFO:tasks.workunit.client.1.smithi203.stdout:8/623: mkdir d1/d4b/d5f/daf 0 2024-04-05T03:02:45.523 INFO:tasks.workunit.client.0.smithi153.stdout:8/238: rmdir d7/d13 39 2024-04-05T03:02:45.523 INFO:tasks.workunit.client.0.smithi153.stdout:4/304: dread f4 [0,4194304] 0 2024-04-05T03:02:45.524 INFO:tasks.workunit.client.1.smithi203.stdout:3/498: rmdir d0/d21 39 2024-04-05T03:02:45.524 INFO:tasks.workunit.client.1.smithi203.stdout:1/576: sync 2024-04-05T03:02:45.524 INFO:tasks.workunit.client.0.smithi153.stdout:3/219: write d1d/d32/f49 [132029,88363] 0 2024-04-05T03:02:45.525 INFO:tasks.workunit.client.1.smithi203.stdout:4/436: dread - da/d71/f81 zero size 2024-04-05T03:02:45.528 INFO:tasks.workunit.client.1.smithi203.stdout:4/437: chown da/fc 558 1 2024-04-05T03:02:45.528 INFO:tasks.workunit.client.1.smithi203.stdout:6/629: dwrite d0/da/d54/d53/fbd [0,4194304] 0 2024-04-05T03:02:45.530 INFO:tasks.workunit.client.1.smithi203.stdout:2/784: creat d5/d66/d7d/f10e x:0 0 0 2024-04-05T03:02:45.530 INFO:tasks.workunit.client.1.smithi203.stdout:9/550: truncate d13/d1b/d33/d64/f75 720881 0 2024-04-05T03:02:45.530 INFO:tasks.workunit.client.1.smithi203.stdout:5/613: creat d3/d4/dc/d67/db5/fcb x:0 0 0 2024-04-05T03:02:45.532 INFO:tasks.workunit.client.0.smithi153.stdout:3/220: symlink d1d/d26/d3c/l52 0 2024-04-05T03:02:45.533 INFO:tasks.workunit.client.1.smithi203.stdout:8/624: read d1/d4b/d68/f95 [2317044,93332] 0 2024-04-05T03:02:45.534 INFO:tasks.workunit.client.0.smithi153.stdout:8/239: mkdir d7/d13/d19/d1b/d4f 0 2024-04-05T03:02:45.535 INFO:tasks.workunit.client.1.smithi203.stdout:3/499: creat d0/d41/f8e x:0 0 0 2024-04-05T03:02:45.593 INFO:tasks.workunit.client.1.smithi203.stdout:4/438: rmdir da/d29/d6a/d80 39 2024-04-05T03:02:45.593 INFO:tasks.workunit.client.1.smithi203.stdout:1/577: dwrite d7/f9c [0,4194304] 0 2024-04-05T03:02:45.593 INFO:tasks.workunit.client.1.smithi203.stdout:2/785: chown d5/d10/d35/d36/d41/l48 19 1 2024-04-05T03:02:45.593 INFO:tasks.workunit.client.1.smithi203.stdout:5/614: unlink d3/d1f/d59/d64/c42 0 2024-04-05T03:02:45.593 INFO:tasks.workunit.client.1.smithi203.stdout:0/611: sync 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/768: sync 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:9/551: dwrite d13/d1b/d33/d42/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:9/552: stat d13/d1b/d33/f3f 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:1/578: symlink d7/d79/lb6 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/769: creat d8/d32/d5d/d6d/daf/fec x:0 0 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/770: write d8/d32/d16/d1a/d3d/d59/f61 [2292945,14381] 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:5/615: creat d3/d1f/d59/d60/dca/fcc x:0 0 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:2/786: rename d5/d10/db8/dab/c7b to d5/d10/d1a/c10f 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/771: symlink d8/d32/d5d/d6d/daf/led 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/772: chown d8/f4c 53 1 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:1/579: mknod d7/d14/d2b/cb7 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/773: write d8/d32/d16/d92/fdc [440527,109055] 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/774: stat d8/d32/d5d/dbe/dc9 0 2024-04-05T03:02:45.594 INFO:tasks.workunit.client.1.smithi203.stdout:6/630: dread d0/da/f10 [4194304,4194304] 0 2024-04-05T03:02:45.595 INFO:tasks.workunit.client.1.smithi203.stdout:1/580: dwrite d7/d14/f1e [0,4194304] 0 2024-04-05T03:02:45.595 INFO:tasks.workunit.client.1.smithi203.stdout:2/787: unlink d5/d10/d1a/d57/c10c 0 2024-04-05T03:02:45.595 INFO:tasks.workunit.client.1.smithi203.stdout:7/775: readlink d8/d32/d16/d1a/l2b 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:6/283: dwrite d4/dc/f3c [0,4194304] 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:8/240: stat d7/d2a/f44 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:4/305: dread da/db/d20/d45/f4e [0,4194304] 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:6/284: chown d4/dc/df/d16/d36/d4d/d64 30814 1 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:2/254: read d9/da/f24 [517158,2164] 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:3/221: creat d1d/d32/f53 x:0 0 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:6/285: fdatasync d4/dc/df/d16/d24/f29 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:8/241: mkdir d7/d2a/d50 0 2024-04-05T03:02:45.596 INFO:tasks.workunit.client.0.smithi153.stdout:2/255: fdatasync d9/da/fe 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:4/306: write da/db/d20/d35/d38/d3e/f50 [1119594,40857] 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/222: creat d1d/d27/f54 x:0 0 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:8/242: creat d7/d13/d19/d1b/d33/f51 x:0 0 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:6/286: mknod d4/dc/df/d16/d5f/c65 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:4/307: fsync da/de/f14 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:6/287: dread - d4/dc/f46 zero size 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:8/243: dread d7/f2b [0,4194304] 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/223: creat d1d/d27/f55 x:0 0 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:2/256: creat d9/da/d36/f50 x:0 0 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:4/308: dread da/de/f1e [0,4194304] 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:6/288: dread d4/dc/df/d16/d33/f47 [0,4194304] 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:8/244: mkdir d7/d2a/d52 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/224: creat d1d/d32/f56 x:0 0 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:2/257: mknod d9/da/d36/d38/c51 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:4/309: fdatasync da/de/f19 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:0/331: dread d9/df/f15 [0,4194304] 0 2024-04-05T03:02:45.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/225: truncate d1d/d27/d2b/f2d 585465 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:0/332: readlink d9/d14/l28 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:6/289: truncate d4/dc/df/d16/d24/f32 2899929 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:2/258: rename l3 to d9/l52 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:0/333: unlink d9/df/l38 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:4/310: mknod da/db/d20/d45/c63 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:7/244: truncate d2/f9 937890 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.0.smithi153.stdout:5/250: dread d0/d9/f1e [0,4194304] 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.1.smithi203.stdout:8/625: write d1/d6/f45 [4372718,73764] 0 2024-04-05T03:02:45.598 INFO:tasks.workunit.client.1.smithi203.stdout:9/553: dread d13/d1b/d26/f90 [0,4194304] 0 2024-04-05T03:02:45.599 INFO:tasks.workunit.client.1.smithi203.stdout:6/631: symlink d0/da/d54/lc2 0 2024-04-05T03:02:45.599 INFO:tasks.workunit.client.1.smithi203.stdout:0/612: truncate dd/d17/d1d/d26/d28/f3f 6831555 0 2024-04-05T03:02:45.602 INFO:tasks.workunit.client.1.smithi203.stdout:7/776: dread - d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/fbb zero size 2024-04-05T03:02:45.604 INFO:tasks.workunit.client.1.smithi203.stdout:8/626: truncate d1/d6/fab 594850 0 2024-04-05T03:02:45.606 INFO:tasks.workunit.client.1.smithi203.stdout:0/613: symlink dd/db1/lb3 0 2024-04-05T03:02:45.607 INFO:tasks.workunit.client.1.smithi203.stdout:3/500: dread d0/dc/d1a/d1d/d43/f51 [0,4194304] 0 2024-04-05T03:02:45.609 INFO:tasks.workunit.client.1.smithi203.stdout:5/616: creat d3/d1f/d59/d64/fcd x:0 0 0 2024-04-05T03:02:45.610 INFO:tasks.workunit.client.1.smithi203.stdout:4/439: dwrite da/d29/d2e/f48 [0,4194304] 0 2024-04-05T03:02:45.613 INFO:tasks.workunit.client.0.smithi153.stdout:0/334: readlink d9/df/l20 0 2024-04-05T03:02:45.614 INFO:tasks.workunit.client.1.smithi203.stdout:8/627: readlink d1/d12/d17/d47/d53/d69/l8c 0 2024-04-05T03:02:45.615 INFO:tasks.workunit.client.0.smithi153.stdout:8/245: rename d7/d13/l3d to d7/d13/d19/d1b/l53 0 2024-04-05T03:02:45.615 INFO:tasks.workunit.client.0.smithi153.stdout:4/311: fsync da/de/d28/d2d/f4a 0 2024-04-05T03:02:45.617 INFO:tasks.workunit.client.0.smithi153.stdout:6/290: dwrite d4/dc/df/d16/f21 [0,4194304] 0 2024-04-05T03:02:45.617 INFO:tasks.workunit.client.1.smithi203.stdout:9/554: chown d13/d1b/d33/d64/f75 100972884 1 2024-04-05T03:02:45.620 INFO:tasks.workunit.client.1.smithi203.stdout:6/632: rename d0/da/d54/l6f to d0/db4/lc3 0 2024-04-05T03:02:45.620 INFO:tasks.workunit.client.1.smithi203.stdout:8/628: chown d1/d4b/d68/d75/dac/da4 1279 1 2024-04-05T03:02:45.621 INFO:tasks.workunit.client.0.smithi153.stdout:2/259: write d9/da/d19/f1c [4937539,8981] 0 2024-04-05T03:02:45.621 INFO:tasks.workunit.client.0.smithi153.stdout:7/245: rmdir d2 39 2024-04-05T03:02:45.621 INFO:tasks.workunit.client.1.smithi203.stdout:9/555: chown d13/l2c 3537 1 2024-04-05T03:02:45.621 INFO:tasks.workunit.client.0.smithi153.stdout:0/335: fdatasync d9/db/f13 0 2024-04-05T03:02:45.623 INFO:tasks.workunit.client.0.smithi153.stdout:0/336: read - d9/db/d42/f60 zero size 2024-04-05T03:02:45.626 INFO:tasks.workunit.client.0.smithi153.stdout:2/260: dread d9/da/f15 [0,4194304] 0 2024-04-05T03:02:45.630 INFO:tasks.workunit.client.0.smithi153.stdout:6/291: creat d4/dc/df/d16/d36/d4d/f66 x:0 0 0 2024-04-05T03:02:45.631 INFO:tasks.workunit.client.0.smithi153.stdout:5/251: mknod d0/d25/c60 0 2024-04-05T03:02:45.631 INFO:tasks.workunit.client.0.smithi153.stdout:8/246: mkdir d7/d2a/d52/d54 0 2024-04-05T03:02:45.632 INFO:tasks.workunit.client.0.smithi153.stdout:8/247: dread - d7/d2a/f2d zero size 2024-04-05T03:02:45.634 INFO:tasks.workunit.client.0.smithi153.stdout:2/261: readlink d9/ld 0 2024-04-05T03:02:45.635 INFO:tasks.workunit.client.0.smithi153.stdout:2/262: chown d9/da/d19/d2e/d48/l3f 216851734 1 2024-04-05T03:02:45.635 INFO:tasks.workunit.client.0.smithi153.stdout:6/292: unlink d4/dc/df/d16/d44/c4f 0 2024-04-05T03:02:45.636 INFO:tasks.workunit.client.1.smithi203.stdout:4/440: dwrite da/f11 [0,4194304] 0 2024-04-05T03:02:45.636 INFO:tasks.workunit.client.1.smithi203.stdout:1/581: dread d7/d24/d33/d48/d4b/d51/f7c [0,4194304] 0 2024-04-05T03:02:45.636 INFO:tasks.workunit.client.1.smithi203.stdout:0/614: dwrite dd/d17/d1d/d22/d80/f4b [4194304,4194304] 0 2024-04-05T03:02:45.637 INFO:tasks.workunit.client.0.smithi153.stdout:4/312: link da/db/d20/d3c/c3f da/db/d20/d35/d41/c64 0 2024-04-05T03:02:45.638 INFO:tasks.workunit.client.0.smithi153.stdout:6/293: chown d4/dc/df/d16/d36 11068993 1 2024-04-05T03:02:45.638 INFO:tasks.workunit.client.0.smithi153.stdout:2/263: dread - d9/da/d19/d2e/f3a zero size 2024-04-05T03:02:45.639 INFO:tasks.workunit.client.0.smithi153.stdout:8/248: rmdir d7/d13/d19/d1b/d33 39 2024-04-05T03:02:45.639 INFO:tasks.workunit.client.1.smithi203.stdout:5/617: symlink d3/d1f/d59/d64/d12/lce 0 2024-04-05T03:02:45.640 INFO:tasks.workunit.client.0.smithi153.stdout:4/313: fdatasync da/db/fd 0 2024-04-05T03:02:45.641 INFO:tasks.workunit.client.0.smithi153.stdout:5/252: mkdir d0/de/d5f/d61 0 2024-04-05T03:02:45.642 INFO:tasks.workunit.client.1.smithi203.stdout:6/633: creat d0/da/d54/d4c/da1/db7/fc4 x:0 0 0 2024-04-05T03:02:45.644 INFO:tasks.workunit.client.1.smithi203.stdout:4/441: dread da/f5a [0,4194304] 0 2024-04-05T03:02:45.645 INFO:tasks.workunit.client.0.smithi153.stdout:7/246: dwrite d2/da/d11/d49/f4e [4194304,4194304] 0 2024-04-05T03:02:45.645 INFO:tasks.workunit.client.0.smithi153.stdout:8/249: mknod d7/d2a/c55 0 2024-04-05T03:02:45.647 INFO:tasks.workunit.client.0.smithi153.stdout:7/247: readlink l1 0 2024-04-05T03:02:45.647 INFO:tasks.workunit.client.1.smithi203.stdout:6/634: truncate d0/da/d54/d53/f8d 4659677 0 2024-04-05T03:02:45.648 INFO:tasks.workunit.client.0.smithi153.stdout:2/264: write d9/da/fb [4301309,116586] 0 2024-04-05T03:02:45.648 INFO:tasks.workunit.client.1.smithi203.stdout:6/635: fdatasync d0/da/f13 0 2024-04-05T03:02:45.649 INFO:tasks.workunit.client.0.smithi153.stdout:4/314: mknod da/db/d20/d35/c65 0 2024-04-05T03:02:45.650 INFO:tasks.workunit.client.1.smithi203.stdout:8/629: creat d1/d6/d8f/fb0 x:0 0 0 2024-04-05T03:02:45.650 INFO:tasks.workunit.client.0.smithi153.stdout:7/248: write d2/da/d3a/d41/d31/f43 [3685322,87244] 0 2024-04-05T03:02:45.651 INFO:tasks.workunit.client.0.smithi153.stdout:0/337: dread f7 [0,4194304] 0 2024-04-05T03:02:45.652 INFO:tasks.workunit.client.0.smithi153.stdout:1/207: sync 2024-04-05T03:02:45.652 INFO:tasks.workunit.client.0.smithi153.stdout:6/294: rename d4/d38/d4c/c63 to d4/dc/df/d16/d36/d4d/d64/c67 0 2024-04-05T03:02:45.654 INFO:tasks.workunit.client.0.smithi153.stdout:0/338: write d9/df/d12/f2b [3962878,17234] 0 2024-04-05T03:02:45.654 INFO:tasks.workunit.client.1.smithi203.stdout:4/442: mknod da/d71/c8c 0 2024-04-05T03:02:45.654 INFO:tasks.workunit.client.0.smithi153.stdout:8/250: fdatasync d7/f2b 0 2024-04-05T03:02:45.655 INFO:tasks.workunit.client.1.smithi203.stdout:5/618: creat d3/d1f/d59/d60/dca/fcf x:0 0 0 2024-04-05T03:02:45.655 INFO:tasks.workunit.client.1.smithi203.stdout:0/615: symlink dd/d17/lb4 0 2024-04-05T03:02:45.655 INFO:tasks.workunit.client.0.smithi153.stdout:0/339: write d9/df/d12/f2b [2470655,48594] 0 2024-04-05T03:02:45.655 INFO:tasks.workunit.client.0.smithi153.stdout:1/208: chown d5/dd/d21/d18/d16/f22 0 1 2024-04-05T03:02:45.656 INFO:tasks.workunit.client.1.smithi203.stdout:4/443: chown da/d29/d2e/l2f 141 1 2024-04-05T03:02:45.658 INFO:tasks.workunit.client.0.smithi153.stdout:8/251: write d7/f16 [648145,52550] 0 2024-04-05T03:02:45.659 INFO:tasks.workunit.client.1.smithi203.stdout:1/582: mknod d7/d24/d33/d48/d4b/d6b/cb8 0 2024-04-05T03:02:45.659 INFO:tasks.workunit.client.0.smithi153.stdout:8/252: stat d7/f2b 0 2024-04-05T03:02:45.660 INFO:tasks.workunit.client.1.smithi203.stdout:9/556: creat d13/d1b/fad x:0 0 0 2024-04-05T03:02:45.660 INFO:tasks.workunit.client.0.smithi153.stdout:8/253: chown d7/d13/l34 7956 1 2024-04-05T03:02:45.661 INFO:tasks.workunit.client.1.smithi203.stdout:9/557: readlink d13/l61 0 2024-04-05T03:02:45.664 INFO:tasks.workunit.client.0.smithi153.stdout:4/315: mkdir da/de/d66 0 2024-04-05T03:02:45.665 INFO:tasks.workunit.client.0.smithi153.stdout:2/265: write d9/da/f24 [204514,56924] 0 2024-04-05T03:02:45.666 INFO:tasks.workunit.client.0.smithi153.stdout:1/209: dwrite d5/dd/d1b/f2b [0,4194304] 0 2024-04-05T03:02:45.666 INFO:tasks.workunit.client.1.smithi203.stdout:9/558: readlink d13/d1b/d26/l28 0 2024-04-05T03:02:45.666 INFO:tasks.workunit.client.1.smithi203.stdout:6/636: rename d0/da/d54/d4c/da1/db7/f7a to d0/da/d18/d38/d2a/d44/d78/da7/fc5 0 2024-04-05T03:02:45.671 INFO:tasks.workunit.client.1.smithi203.stdout:0/616: symlink dd/d17/d1d/d26/d8c/lb5 0 2024-04-05T03:02:45.671 INFO:tasks.workunit.client.0.smithi153.stdout:6/295: symlink d4/dc/df/d16/d44/l68 0 2024-04-05T03:02:45.671 INFO:tasks.workunit.client.1.smithi203.stdout:4/444: mknod da/d29/d2e/d8b/c8d 0 2024-04-05T03:02:45.673 INFO:tasks.workunit.client.1.smithi203.stdout:9/559: symlink d13/d1b/d26/d51/d81/lae 0 2024-04-05T03:02:45.674 INFO:tasks.workunit.client.1.smithi203.stdout:1/583: read d7/d24/f6e [204253,30970] 0 2024-04-05T03:02:45.674 INFO:tasks.workunit.client.0.smithi153.stdout:8/254: dwrite f0 [8388608,4194304] 0 2024-04-05T03:02:45.676 INFO:tasks.workunit.client.0.smithi153.stdout:4/316: mkdir da/db/d20/d35/d41/d67 0 2024-04-05T03:02:45.676 INFO:tasks.workunit.client.1.smithi203.stdout:5/619: dwrite d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:45.676 INFO:tasks.workunit.client.0.smithi153.stdout:6/296: dread - d4/dc/df/d16/d36/d52/f55 zero size 2024-04-05T03:02:45.677 INFO:tasks.workunit.client.1.smithi203.stdout:3/501: dread d0/dc/d34/f61 [0,4194304] 0 2024-04-05T03:02:45.679 INFO:tasks.workunit.client.0.smithi153.stdout:8/255: read d7/d13/d19/d1b/f35 [1441753,17930] 0 2024-04-05T03:02:45.679 INFO:tasks.workunit.client.0.smithi153.stdout:1/210: mkdir d5/d8/d20/d3f 0 2024-04-05T03:02:45.680 INFO:tasks.workunit.client.0.smithi153.stdout:2/266: dwrite d9/da/d19/f1f [8388608,4194304] 0 2024-04-05T03:02:45.681 INFO:tasks.workunit.client.0.smithi153.stdout:1/211: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:45.683 INFO:tasks.workunit.client.0.smithi153.stdout:4/317: mkdir da/db/d20/d45/d3b/d68 0 2024-04-05T03:02:45.684 INFO:tasks.workunit.client.1.smithi203.stdout:4/445: mknod da/d29/d6a/d80/c8e 0 2024-04-05T03:02:45.687 INFO:tasks.workunit.client.0.smithi153.stdout:2/267: dread d9/da/f20 [0,4194304] 0 2024-04-05T03:02:45.688 INFO:tasks.workunit.client.0.smithi153.stdout:8/256: write d7/d13/d19/f22 [1195649,70356] 0 2024-04-05T03:02:45.688 INFO:tasks.workunit.client.0.smithi153.stdout:0/340: getdents d9/d14 0 2024-04-05T03:02:45.692 INFO:tasks.workunit.client.1.smithi203.stdout:1/584: dwrite d7/d14/d2b/f95 [0,4194304] 0 2024-04-05T03:02:45.692 INFO:tasks.workunit.client.1.smithi203.stdout:4/446: truncate da/d29/d2e/d57/f86 1244159 0 2024-04-05T03:02:45.693 INFO:tasks.workunit.client.1.smithi203.stdout:3/502: creat d0/dc/d38/f8f x:0 0 0 2024-04-05T03:02:45.695 INFO:tasks.workunit.client.0.smithi153.stdout:5/253: rename d0/de to d0/d25/d4d/d62 0 2024-04-05T03:02:45.695 INFO:tasks.workunit.client.0.smithi153.stdout:8/257: chown d7/d13/d19/d31/f41 4723105 1 2024-04-05T03:02:45.697 INFO:tasks.workunit.client.1.smithi203.stdout:1/585: write d7/fa [374513,8510] 0 2024-04-05T03:02:45.697 INFO:tasks.workunit.client.0.smithi153.stdout:1/212: dwrite d5/dd/d21/d18/d16/d33/f3e [0,4194304] 0 2024-04-05T03:02:45.698 INFO:tasks.workunit.client.0.smithi153.stdout:1/213: dread - d5/d8/d20/f38 zero size 2024-04-05T03:02:45.699 INFO:tasks.workunit.client.1.smithi203.stdout:9/560: creat d13/d1b/d33/d42/d67/faf x:0 0 0 2024-04-05T03:02:45.699 INFO:tasks.workunit.client.1.smithi203.stdout:6/637: mknod d0/da/d54/d4c/daf/cc6 0 2024-04-05T03:02:45.707 INFO:tasks.workunit.client.1.smithi203.stdout:4/447: mknod da/d3c/c8f 0 2024-04-05T03:02:45.708 INFO:tasks.workunit.client.0.smithi153.stdout:8/258: dwrite d7/d13/d19/d31/f4a [0,4194304] 0 2024-04-05T03:02:45.708 INFO:tasks.workunit.client.0.smithi153.stdout:2/268: mkdir d9/da/d36/d38/d4b/d53 0 2024-04-05T03:02:45.710 INFO:tasks.workunit.client.0.smithi153.stdout:6/297: rename d4/dc/df/d16/d17/c5c to d4/dc/df/d16/d17/d54/d45/c69 0 2024-04-05T03:02:45.710 INFO:tasks.workunit.client.1.smithi203.stdout:9/561: dread d13/d1b/d33/d42/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:45.711 INFO:tasks.workunit.client.0.smithi153.stdout:1/214: dwrite d5/f7 [4194304,4194304] 0 2024-04-05T03:02:45.711 INFO:tasks.workunit.client.0.smithi153.stdout:8/259: chown d7/d2a/l3c 25735 1 2024-04-05T03:02:45.712 INFO:tasks.workunit.client.1.smithi203.stdout:6/638: mknod d0/da/d18/d38/d2a/d2c/cc7 0 2024-04-05T03:02:45.713 INFO:tasks.workunit.client.0.smithi153.stdout:2/269: fdatasync d9/da/fc 0 2024-04-05T03:02:45.715 INFO:tasks.workunit.client.1.smithi203.stdout:0/617: creat dd/d17/d1d/d26/d34/fb6 x:0 0 0 2024-04-05T03:02:45.716 INFO:tasks.workunit.client.0.smithi153.stdout:0/341: rename d9/df/d12 to d9/df/d1d/d61 0 2024-04-05T03:02:45.717 INFO:tasks.workunit.client.0.smithi153.stdout:8/260: write d7/d13/f18 [1624439,84981] 0 2024-04-05T03:02:45.717 INFO:tasks.workunit.client.0.smithi153.stdout:1/215: chown d5/dd/c3b 9 1 2024-04-05T03:02:45.718 INFO:tasks.workunit.client.1.smithi203.stdout:1/586: mkdir d7/d24/d33/d48/d4b/d6b/db9 0 2024-04-05T03:02:45.720 INFO:tasks.workunit.client.1.smithi203.stdout:3/503: write d0/d21/f45 [2466451,63335] 0 2024-04-05T03:02:45.720 INFO:tasks.workunit.client.0.smithi153.stdout:6/298: creat d4/d38/d4c/f6a x:0 0 0 2024-04-05T03:02:45.722 INFO:tasks.workunit.client.0.smithi153.stdout:6/299: rename d4 to d4/dc/df/d16/d36/d4d/d6b 22 2024-04-05T03:02:45.722 INFO:tasks.workunit.client.0.smithi153.stdout:0/342: mkdir d9/df/d1e/d45/d62 0 2024-04-05T03:02:45.723 INFO:tasks.workunit.client.0.smithi153.stdout:6/300: dread - d4/d38/d4c/f6a zero size 2024-04-05T03:02:45.724 INFO:tasks.workunit.client.1.smithi203.stdout:8/630: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:45.724 INFO:tasks.workunit.client.1.smithi203.stdout:4/448: dwrite da/d29/d2e/d57/f7f [0,4194304] 0 2024-04-05T03:02:45.725 INFO:tasks.workunit.client.0.smithi153.stdout:1/216: mkdir d5/d8/d20/d2f/d40 0 2024-04-05T03:02:45.725 INFO:tasks.workunit.client.0.smithi153.stdout:1/217: rename d5/dd to d5/dd/d41 22 2024-04-05T03:02:45.726 INFO:tasks.workunit.client.1.smithi203.stdout:4/449: fdatasync da/d29/f42 0 2024-04-05T03:02:45.727 INFO:tasks.workunit.client.0.smithi153.stdout:2/270: mkdir d9/d44/d54 0 2024-04-05T03:02:45.727 INFO:tasks.workunit.client.1.smithi203.stdout:9/562: rename f8 to d13/d1b/d33/d42/fb0 0 2024-04-05T03:02:45.729 INFO:tasks.workunit.client.1.smithi203.stdout:0/618: mkdir dd/db7 0 2024-04-05T03:02:45.730 INFO:tasks.workunit.client.0.smithi153.stdout:8/261: getdents d7/d2a/d50 0 2024-04-05T03:02:45.731 INFO:tasks.workunit.client.0.smithi153.stdout:7/249: dwrite d2/f9 [0,4194304] 0 2024-04-05T03:02:45.732 INFO:tasks.workunit.client.0.smithi153.stdout:1/218: fdatasync d5/dd/f13 0 2024-04-05T03:02:45.733 INFO:tasks.workunit.client.1.smithi203.stdout:3/504: fsync d0/dc/d1a/d1d/d43/f6a 0 2024-04-05T03:02:45.736 INFO:tasks.workunit.client.0.smithi153.stdout:6/301: link d4/dc/df/f13 d4/dc/df/d16/d36/d4d/d64/f6c 0 2024-04-05T03:02:45.738 INFO:tasks.workunit.client.0.smithi153.stdout:2/271: dwrite d9/da/d19/f1f [4194304,4194304] 0 2024-04-05T03:02:45.741 INFO:tasks.workunit.client.1.smithi203.stdout:0/619: dwrite dd/d17/d1d/d22/fac [0,4194304] 0 2024-04-05T03:02:45.742 INFO:tasks.workunit.client.0.smithi153.stdout:1/219: symlink d5/d8/d20/l42 0 2024-04-05T03:02:45.742 INFO:tasks.workunit.client.0.smithi153.stdout:7/250: mkdir d2/da/d42/d57 0 2024-04-05T03:02:45.743 INFO:tasks.workunit.client.0.smithi153.stdout:6/302: symlink d4/dc/df/d16/d5f/l6d 0 2024-04-05T03:02:45.744 INFO:tasks.workunit.client.1.smithi203.stdout:1/587: rename d7/d24/d33/f56 to d7/d14/d2b/d58/fba 0 2024-04-05T03:02:45.745 INFO:tasks.workunit.client.0.smithi153.stdout:1/220: write d5/dd/d1b/f2b [5156080,15017] 0 2024-04-05T03:02:45.746 INFO:tasks.workunit.client.1.smithi203.stdout:9/563: mkdir d13/d1b/d26/d51/da1/db1 0 2024-04-05T03:02:45.747 INFO:tasks.workunit.client.0.smithi153.stdout:0/343: truncate d9/df/d1d/f24 1897233 0 2024-04-05T03:02:45.750 INFO:tasks.workunit.client.1.smithi203.stdout:1/588: write d7/d24/f85 [6393803,116494] 0 2024-04-05T03:02:45.751 INFO:tasks.workunit.client.1.smithi203.stdout:4/450: dwrite da/d29/d2e/f61 [0,4194304] 0 2024-04-05T03:02:45.751 INFO:tasks.workunit.client.1.smithi203.stdout:8/631: symlink d1/d12/lb1 0 2024-04-05T03:02:45.752 INFO:tasks.workunit.client.0.smithi153.stdout:2/272: dread d9/da/d19/d2e/d48/f33 [0,4194304] 0 2024-04-05T03:02:45.756 INFO:tasks.workunit.client.1.smithi203.stdout:1/589: write d7/d24/d33/d48/d4b/d6b/f98 [779918,18417] 0 2024-04-05T03:02:45.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/505: rename d0/ff to d0/dc/d1a/d1d/d43/f90 0 2024-04-05T03:02:45.762 INFO:tasks.workunit.client.1.smithi203.stdout:8/632: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:45.762 INFO:tasks.workunit.client.0.smithi153.stdout:5/254: chown d0/c5d 193 1 2024-04-05T03:02:45.763 INFO:tasks.workunit.client.1.smithi203.stdout:0/620: fsync dd/d17/d1d/d26/f67 0 2024-04-05T03:02:45.763 INFO:tasks.workunit.client.1.smithi203.stdout:3/506: chown d0/dc/d1a/d1d/f5c 157782 1 2024-04-05T03:02:45.765 INFO:tasks.workunit.client.1.smithi203.stdout:0/621: stat dd/d17/d1d/d22/f9c 0 2024-04-05T03:02:45.770 INFO:tasks.workunit.client.0.smithi153.stdout:4/318: dwrite da/db/d20/d35/d38/f59 [0,4194304] 0 2024-04-05T03:02:45.772 INFO:tasks.workunit.client.1.smithi203.stdout:5/620: dwrite d3/d4/dc/f95 [0,4194304] 0 2024-04-05T03:02:45.774 INFO:tasks.workunit.client.0.smithi153.stdout:6/303: rename d4/dc/df/d16/c1e to d4/dc/df/c6e 0 2024-04-05T03:02:45.775 INFO:tasks.workunit.client.1.smithi203.stdout:4/451: mknod da/d71/c90 0 2024-04-05T03:02:45.775 INFO:tasks.workunit.client.0.smithi153.stdout:2/273: dwrite d9/da/f24 [0,4194304] 0 2024-04-05T03:02:45.778 INFO:tasks.workunit.client.1.smithi203.stdout:4/452: write da/d3c/d4c/f7c [218239,125642] 0 2024-04-05T03:02:45.778 INFO:tasks.workunit.client.0.smithi153.stdout:7/251: getdents d2/da/d11/d39 0 2024-04-05T03:02:45.778 INFO:tasks.workunit.client.0.smithi153.stdout:4/319: readlink da/de/d28/l2b 0 2024-04-05T03:02:45.779 INFO:tasks.workunit.client.1.smithi203.stdout:4/453: fdatasync da/f22 0 2024-04-05T03:02:45.781 INFO:tasks.workunit.client.0.smithi153.stdout:2/274: stat d9/da/d36/d38/d39/c4c 0 2024-04-05T03:02:45.782 INFO:tasks.workunit.client.1.smithi203.stdout:8/633: creat d1/d4b/fb2 x:0 0 0 2024-04-05T03:02:45.783 INFO:tasks.workunit.client.1.smithi203.stdout:0/622: fdatasync dd/d17/d1d/d22/f77 0 2024-04-05T03:02:45.785 INFO:tasks.workunit.client.0.smithi153.stdout:2/275: readlink d9/da/d19/l2a 0 2024-04-05T03:02:45.787 INFO:tasks.workunit.client.0.smithi153.stdout:2/276: write d9/f11 [272252,36633] 0 2024-04-05T03:02:45.787 INFO:tasks.workunit.client.0.smithi153.stdout:4/320: chown da/f26 11167 1 2024-04-05T03:02:45.788 INFO:tasks.workunit.client.1.smithi203.stdout:9/564: rmdir d13/d1b/d6b/d9e/d77/da9 0 2024-04-05T03:02:45.789 INFO:tasks.workunit.client.0.smithi153.stdout:7/252: dwrite d2/da/d11/d49/f56 [0,4194304] 0 2024-04-05T03:02:45.790 INFO:tasks.workunit.client.0.smithi153.stdout:2/277: chown d9/da/d19/d2e/f43 323981774 1 2024-04-05T03:02:45.791 INFO:tasks.workunit.client.1.smithi203.stdout:0/623: dread - dd/d17/d1d/d26/d34/d56/fa7 zero size 2024-04-05T03:02:45.791 INFO:tasks.workunit.client.1.smithi203.stdout:3/507: dwrite d0/f52 [0,4194304] 0 2024-04-05T03:02:45.791 INFO:tasks.workunit.client.1.smithi203.stdout:8/634: creat d1/d12/d2f/d5b/fb3 x:0 0 0 2024-04-05T03:02:45.794 INFO:tasks.workunit.client.1.smithi203.stdout:4/454: symlink da/d29/l91 0 2024-04-05T03:02:45.795 INFO:tasks.workunit.client.1.smithi203.stdout:5/621: mknod d3/d4/d94/cd0 0 2024-04-05T03:02:45.796 INFO:tasks.workunit.client.0.smithi153.stdout:7/253: mknod d2/da/d42/d57/c58 0 2024-04-05T03:02:45.797 INFO:tasks.workunit.client.1.smithi203.stdout:9/565: write d13/d1b/d33/d64/f6d [730107,45949] 0 2024-04-05T03:02:45.800 INFO:tasks.workunit.client.0.smithi153.stdout:6/304: rename d4/c40 to d4/dc/df/c6f 0 2024-04-05T03:02:45.802 INFO:tasks.workunit.client.0.smithi153.stdout:7/254: chown d2/c26 817876925 1 2024-04-05T03:02:45.802 INFO:tasks.workunit.client.0.smithi153.stdout:4/321: creat da/db/d20/d45/f69 x:0 0 0 2024-04-05T03:02:45.802 INFO:tasks.workunit.client.1.smithi203.stdout:5/622: fsync d3/d4/dc/f4b 0 2024-04-05T03:02:45.803 INFO:tasks.workunit.client.0.smithi153.stdout:1/221: dread d5/dd/d21/d18/d16/f2d [4194304,4194304] 0 2024-04-05T03:02:45.805 INFO:tasks.workunit.client.1.smithi203.stdout:0/624: write dd/d17/d1d/d22/d53/d61/f9d [186456,76783] 0 2024-04-05T03:02:45.806 INFO:tasks.workunit.client.0.smithi153.stdout:6/305: fsync d4/dc/df/d16/f21 0 2024-04-05T03:02:45.806 INFO:tasks.workunit.client.0.smithi153.stdout:4/322: readlink da/db/d20/d35/l3a 0 2024-04-05T03:02:45.806 INFO:tasks.workunit.client.1.smithi203.stdout:4/455: unlink da/d3c/f6d 0 2024-04-05T03:02:45.806 INFO:tasks.workunit.client.1.smithi203.stdout:1/590: getdents d7 0 2024-04-05T03:02:45.807 INFO:tasks.workunit.client.0.smithi153.stdout:4/323: chown da/db/d20/d35/c65 0 1 2024-04-05T03:02:45.809 INFO:tasks.workunit.client.0.smithi153.stdout:4/324: write da/db/d20/d35/f58 [422121,14193] 0 2024-04-05T03:02:45.810 INFO:tasks.workunit.client.0.smithi153.stdout:2/278: rename d9/da/f20 to d9/da/d36/f55 0 2024-04-05T03:02:45.811 INFO:tasks.workunit.client.0.smithi153.stdout:2/279: write d9/da/d19/f2f [646168,127771] 0 2024-04-05T03:02:45.812 INFO:tasks.workunit.client.0.smithi153.stdout:6/306: mknod d4/dc/c70 0 2024-04-05T03:02:45.815 INFO:tasks.workunit.client.0.smithi153.stdout:4/325: rename da/de/d28 to da/de/d56/d6a 0 2024-04-05T03:02:45.816 INFO:tasks.workunit.client.1.smithi203.stdout:4/456: chown da/d29/c4b 7504 1 2024-04-05T03:02:45.816 INFO:tasks.workunit.client.1.smithi203.stdout:9/566: link d13/d1b/d6b/d9e/d77/f7d d13/d1b/d6b/d9e/d59/fb2 0 2024-04-05T03:02:45.816 INFO:tasks.workunit.client.1.smithi203.stdout:3/508: dwrite d0/dc/d1a/d1d/f5c [0,4194304] 0 2024-04-05T03:02:45.817 INFO:tasks.workunit.client.1.smithi203.stdout:8/635: dwrite d1/d12/d17/d10/f15 [0,4194304] 0 2024-04-05T03:02:45.819 INFO:tasks.workunit.client.1.smithi203.stdout:3/509: chown d0/dc/d1a/d1f 2452332 1 2024-04-05T03:02:45.820 INFO:tasks.workunit.client.1.smithi203.stdout:4/457: rmdir da/d29 39 2024-04-05T03:02:45.821 INFO:tasks.workunit.client.1.smithi203.stdout:8/636: stat d1/d6/d2e/d5c/l6a 0 2024-04-05T03:02:45.822 INFO:tasks.workunit.client.0.smithi153.stdout:6/307: dwrite d4/dc/df/f39 [0,4194304] 0 2024-04-05T03:02:45.823 INFO:tasks.workunit.client.1.smithi203.stdout:1/591: getdents d7/d24/d33/d48 0 2024-04-05T03:02:45.825 INFO:tasks.workunit.client.0.smithi153.stdout:2/280: dwrite d9/da/d19/d2e/d48/f33 [0,4194304] 0 2024-04-05T03:02:45.825 INFO:tasks.workunit.client.0.smithi153.stdout:4/326: truncate da/de/d56/d6a/f44 748590 0 2024-04-05T03:02:45.829 INFO:tasks.workunit.client.1.smithi203.stdout:0/625: dwrite dd/d17/d1d/d26/d8c/d6d/fa9 [0,4194304] 0 2024-04-05T03:02:45.834 INFO:tasks.workunit.client.0.smithi153.stdout:2/281: mknod d9/da/d36/d38/d4b/c56 0 2024-04-05T03:02:45.834 INFO:tasks.workunit.client.0.smithi153.stdout:6/308: dread d4/fb [0,4194304] 0 2024-04-05T03:02:45.836 INFO:tasks.workunit.client.1.smithi203.stdout:0/626: chown dd/d17/d1d/d22/d53/db0/c36 792 1 2024-04-05T03:02:45.836 INFO:tasks.workunit.client.1.smithi203.stdout:9/567: mknod d13/d1b/d26/cb3 0 2024-04-05T03:02:45.843 INFO:tasks.workunit.client.1.smithi203.stdout:8/637: creat d1/d12/d2f/d5b/fb4 x:0 0 0 2024-04-05T03:02:45.844 INFO:tasks.workunit.client.0.smithi153.stdout:4/327: creat da/db/d20/d45/f6b x:0 0 0 2024-04-05T03:02:45.845 INFO:tasks.workunit.client.0.smithi153.stdout:4/328: chown da/de/l54 2125 1 2024-04-05T03:02:45.846 INFO:tasks.workunit.client.1.smithi203.stdout:9/568: creat d13/d1b/d6b/fb4 x:0 0 0 2024-04-05T03:02:45.846 INFO:tasks.workunit.client.1.smithi203.stdout:0/627: fsync dd/d17/d1d/d22/d53/d61/f9d 0 2024-04-05T03:02:45.846 INFO:tasks.workunit.client.1.smithi203.stdout:4/458: dwrite da/d29/f42 [4194304,4194304] 0 2024-04-05T03:02:45.848 INFO:tasks.workunit.client.0.smithi153.stdout:5/255: dwrite d0/f14 [4194304,4194304] 0 2024-04-05T03:02:45.849 INFO:tasks.workunit.client.1.smithi203.stdout:9/569: truncate d13/d5c/f9f 106044 0 2024-04-05T03:02:45.850 INFO:tasks.workunit.client.1.smithi203.stdout:9/570: chown d13/d1b/d33 10325722 1 2024-04-05T03:02:45.852 INFO:tasks.workunit.client.0.smithi153.stdout:2/282: creat d9/d44/d54/f57 x:0 0 0 2024-04-05T03:02:45.852 INFO:tasks.workunit.client.0.smithi153.stdout:1/222: write d5/f9 [1311779,118772] 0 2024-04-05T03:02:45.855 INFO:tasks.workunit.client.1.smithi203.stdout:0/628: write dd/d17/d1d/d26/d34/d56/fab [1666011,35248] 0 2024-04-05T03:02:45.855 INFO:tasks.workunit.client.1.smithi203.stdout:5/623: dread d3/d4/f87 [0,4194304] 0 2024-04-05T03:02:45.855 INFO:tasks.workunit.client.0.smithi153.stdout:5/256: chown d0/d25/f2f 3060 1 2024-04-05T03:02:45.860 INFO:tasks.workunit.client.1.smithi203.stdout:5/624: dread - d3/d1f/fc0 zero size 2024-04-05T03:02:45.862 INFO:tasks.workunit.client.1.smithi203.stdout:9/571: fdatasync d13/d1b/d33/d42/f49 0 2024-04-05T03:02:45.864 INFO:tasks.workunit.client.0.smithi153.stdout:2/283: rename d9/da/d36/f50 to d9/da/d36/d38/d39/f58 0 2024-04-05T03:02:45.864 INFO:tasks.workunit.client.1.smithi203.stdout:9/572: chown d13/d1b/d6b/d9e/d59/d7b 901031 1 2024-04-05T03:02:45.864 INFO:tasks.workunit.client.1.smithi203.stdout:3/510: rename d0/dc/d1a/d76/c4f to d0/dc/d38/d5d/c91 0 2024-04-05T03:02:45.866 INFO:tasks.workunit.client.0.smithi153.stdout:5/257: mknod d0/d9/c63 0 2024-04-05T03:02:45.866 INFO:tasks.workunit.client.1.smithi203.stdout:0/629: symlink dd/d17/d1d/d26/d8c/d6d/lb8 0 2024-04-05T03:02:45.866 INFO:tasks.workunit.client.0.smithi153.stdout:4/329: getdents da/de/d56/d6a/d2d 0 2024-04-05T03:02:45.870 INFO:tasks.workunit.client.1.smithi203.stdout:0/630: chown dd/d17/d1d/d26/f67 7265786 1 2024-04-05T03:02:45.870 INFO:tasks.workunit.client.0.smithi153.stdout:5/258: creat d0/d9/d22/f64 x:0 0 0 2024-04-05T03:02:45.871 INFO:tasks.workunit.client.1.smithi203.stdout:0/631: stat dd/d17/d1d/d75 0 2024-04-05T03:02:45.871 INFO:tasks.workunit.client.0.smithi153.stdout:4/330: creat da/de/f6c x:0 0 0 2024-04-05T03:02:45.872 INFO:tasks.workunit.client.0.smithi153.stdout:5/259: write d0/f14 [7386477,102334] 0 2024-04-05T03:02:45.873 INFO:tasks.workunit.client.1.smithi203.stdout:9/573: link d13/d1b/d6b/d9e/c74 d13/d1b/d33/d42/d67/d82/cb5 0 2024-04-05T03:02:45.874 INFO:tasks.workunit.client.1.smithi203.stdout:3/511: dwrite d0/d21/f45 [0,4194304] 0 2024-04-05T03:02:45.878 INFO:tasks.workunit.client.0.smithi153.stdout:4/331: creat da/db/d20/d35/d38/f6d x:0 0 0 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:8/638: dread d1/f58 [0,4194304] 0 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:2/788: sync 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:8/639: fdatasync d1/d12/d17/d47/d53/f82 0 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:9/574: unlink d13/d1b/d26/f6f 0 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:9/575: write d13/f1d [1826823,114147] 0 2024-04-05T03:02:45.879 INFO:tasks.workunit.client.1.smithi203.stdout:7/777: sync 2024-04-05T03:02:45.880 INFO:tasks.workunit.client.1.smithi203.stdout:6/639: sync 2024-04-05T03:02:45.880 INFO:tasks.workunit.client.1.smithi203.stdout:8/640: dread d1/f58 [0,4194304] 0 2024-04-05T03:02:45.880 INFO:tasks.workunit.client.0.smithi153.stdout:4/332: dread f7 [0,4194304] 0 2024-04-05T03:02:45.880 INFO:tasks.workunit.client.1.smithi203.stdout:2/789: fsync d5/d10/d35/d2d/f33 0 2024-04-05T03:02:45.882 INFO:tasks.workunit.client.1.smithi203.stdout:8/641: stat d1/d4b/d68/d75/d77/f87 0 2024-04-05T03:02:45.883 INFO:tasks.workunit.client.0.smithi153.stdout:5/260: link d0/c29 d0/d9/c65 0 2024-04-05T03:02:45.884 INFO:tasks.workunit.client.1.smithi203.stdout:2/790: fsync d5/d10/d35/d2d/d7c/d72/ffc 0 2024-04-05T03:02:45.884 INFO:tasks.workunit.client.1.smithi203.stdout:9/576: creat d13/d1b/d6b/d7a/fb6 x:0 0 0 2024-04-05T03:02:45.885 INFO:tasks.workunit.client.1.smithi203.stdout:8/642: dread - d1/d12/d2f/d5b/fb4 zero size 2024-04-05T03:02:45.885 INFO:tasks.workunit.client.1.smithi203.stdout:6/640: readlink d0/da/d54/d4c/da1/db7/l98 0 2024-04-05T03:02:45.888 INFO:tasks.workunit.client.0.smithi153.stdout:5/261: creat d0/d25/d4d/f66 x:0 0 0 2024-04-05T03:02:45.889 INFO:tasks.workunit.client.0.smithi153.stdout:9/201: sync 2024-04-05T03:02:45.889 INFO:tasks.workunit.client.0.smithi153.stdout:4/333: getdents da/db/d20/d35 0 2024-04-05T03:02:45.892 INFO:tasks.workunit.client.0.smithi153.stdout:6/309: truncate d4/dc/df/d16/d17/f3a 3778913 0 2024-04-05T03:02:45.893 INFO:tasks.workunit.client.0.smithi153.stdout:4/334: creat da/db/d20/f6e x:0 0 0 2024-04-05T03:02:45.894 INFO:tasks.workunit.client.0.smithi153.stdout:5/262: fdatasync d0/d2c/f3a 0 2024-04-05T03:02:45.901 INFO:tasks.workunit.client.1.smithi203.stdout:9/577: symlink d13/d1b/d6b/d9e/d59/d7b/lb7 0 2024-04-05T03:02:45.903 INFO:tasks.workunit.client.1.smithi203.stdout:6/641: stat d0/da/d18/f30 0 2024-04-05T03:02:45.904 INFO:tasks.workunit.client.0.smithi153.stdout:2/284: dread d9/da/d19/f46 [0,4194304] 0 2024-04-05T03:02:45.905 INFO:tasks.workunit.client.0.smithi153.stdout:6/310: dwrite d4/f51 [0,4194304] 0 2024-04-05T03:02:45.905 INFO:tasks.workunit.client.1.smithi203.stdout:9/578: write d13/d1b/d33/d42/f66 [486183,12991] 0 2024-04-05T03:02:45.907 INFO:tasks.workunit.client.0.smithi153.stdout:6/311: readlink d4/dc/df/d16/d33/l49 0 2024-04-05T03:02:45.907 INFO:tasks.workunit.client.0.smithi153.stdout:9/202: mkdir d1/d10/d4d 0 2024-04-05T03:02:45.909 INFO:tasks.workunit.client.1.smithi203.stdout:2/791: link d5/d10/db8/dbe/fe0 d5/d10/d35/d73/f110 0 2024-04-05T03:02:45.909 INFO:tasks.workunit.client.1.smithi203.stdout:4/459: dwrite da/d3c/f5f [0,4194304] 0 2024-04-05T03:02:45.910 INFO:tasks.workunit.client.0.smithi153.stdout:4/335: mknod da/db/d20/d35/d41/d67/c6f 0 2024-04-05T03:02:45.911 INFO:tasks.workunit.client.1.smithi203.stdout:0/632: dread dd/f15 [0,4194304] 0 2024-04-05T03:02:45.911 INFO:tasks.workunit.client.1.smithi203.stdout:5/625: write d3/d4/dc/d67/f70 [1178670,36297] 0 2024-04-05T03:02:45.914 INFO:tasks.workunit.client.0.smithi153.stdout:2/285: mkdir d9/d44/d54/d59 0 2024-04-05T03:02:45.915 INFO:tasks.workunit.client.1.smithi203.stdout:3/512: dread d0/dc/d1a/d1d/d43/f4c [0,4194304] 0 2024-04-05T03:02:45.915 INFO:tasks.workunit.client.1.smithi203.stdout:0/633: write dd/d17/d1d/d22/f77 [2745732,8511] 0 2024-04-05T03:02:45.918 INFO:tasks.workunit.client.1.smithi203.stdout:1/592: dread d7/d24/f66 [0,4194304] 0 2024-04-05T03:02:45.919 INFO:tasks.workunit.client.0.smithi153.stdout:9/203: unlink d1/d10/d16/l34 0 2024-04-05T03:02:45.919 INFO:tasks.workunit.client.0.smithi153.stdout:5/263: creat d0/d25/d4d/d62/d5f/d61/f67 x:0 0 0 2024-04-05T03:02:45.922 INFO:tasks.workunit.client.0.smithi153.stdout:2/286: rename d9/da/d36/d38/d39/f45 to d9/da/d36/d38/d39/f5a 0 2024-04-05T03:02:45.924 INFO:tasks.workunit.client.0.smithi153.stdout:9/204: mknod d1/d10/d16/d22/d2c/c4e 0 2024-04-05T03:02:45.924 INFO:tasks.workunit.client.1.smithi203.stdout:6/642: chown d0/da/d18/f9f 5 1 2024-04-05T03:02:45.925 INFO:tasks.workunit.client.0.smithi153.stdout:5/264: mkdir d0/d25/d68 0 2024-04-05T03:02:45.927 INFO:tasks.workunit.client.0.smithi153.stdout:2/287: creat d9/da/d19/d2e/d48/f5b x:0 0 0 2024-04-05T03:02:45.928 INFO:tasks.workunit.client.1.smithi203.stdout:8/643: dwrite d1/d4b/d5f/f7d [0,4194304] 0 2024-04-05T03:02:45.928 INFO:tasks.workunit.client.1.smithi203.stdout:2/792: mknod d5/d63/d86/c111 0 2024-04-05T03:02:45.928 INFO:tasks.workunit.client.1.smithi203.stdout:4/460: creat da/d71/f92 x:0 0 0 2024-04-05T03:02:45.930 INFO:tasks.workunit.client.0.smithi153.stdout:2/288: rmdir d9/d44 39 2024-04-05T03:02:45.931 INFO:tasks.workunit.client.1.smithi203.stdout:4/461: write da/d29/d2e/f48 [3139961,15581] 0 2024-04-05T03:02:45.931 INFO:tasks.workunit.client.1.smithi203.stdout:9/579: symlink d13/d1b/lb8 0 2024-04-05T03:02:45.932 INFO:tasks.workunit.client.1.smithi203.stdout:4/462: fdatasync da/d71/f81 0 2024-04-05T03:02:45.934 INFO:tasks.workunit.client.1.smithi203.stdout:0/634: mknod dd/d17/d1d/d22/d80/cb9 0 2024-04-05T03:02:45.935 INFO:tasks.workunit.client.1.smithi203.stdout:2/793: chown d5/d10/d35/l74 1768188 1 2024-04-05T03:02:45.936 INFO:tasks.workunit.client.1.smithi203.stdout:4/463: write da/d29/d6a/f6c [1816228,19383] 0 2024-04-05T03:02:45.937 INFO:tasks.workunit.client.0.smithi153.stdout:1/223: dread d5/dd/d1b/f1d [0,4194304] 0 2024-04-05T03:02:45.938 INFO:tasks.workunit.client.0.smithi153.stdout:1/224: fsync d5/f7 0 2024-04-05T03:02:45.938 INFO:tasks.workunit.client.1.smithi203.stdout:9/580: rmdir d13/d1b/d26 39 2024-04-05T03:02:45.939 INFO:tasks.workunit.client.0.smithi153.stdout:2/289: rename d9/da/d19/c27 to d9/da/d19/d2e/d4f/c5c 0 2024-04-05T03:02:45.940 INFO:tasks.workunit.client.1.smithi203.stdout:2/794: chown d5/d10/db8/dab/de1/dbd/lcd 3205867 1 2024-04-05T03:02:45.941 INFO:tasks.workunit.client.0.smithi153.stdout:4/336: write da/f26 [1667537,36279] 0 2024-04-05T03:02:45.942 INFO:tasks.workunit.client.1.smithi203.stdout:0/635: rmdir dd/d17/d1d/d26/d8c 39 2024-04-05T03:02:45.943 INFO:tasks.workunit.client.1.smithi203.stdout:7/778: truncate d8/d32/d16/fa6 2767519 0 2024-04-05T03:02:45.943 INFO:tasks.workunit.client.0.smithi153.stdout:4/337: write da/db/d20/f32 [3880937,117770] 0 2024-04-05T03:02:45.944 INFO:tasks.workunit.client.0.smithi153.stdout:2/290: unlink d9/da/d19/d2e/d48/l14 0 2024-04-05T03:02:45.948 INFO:tasks.workunit.client.0.smithi153.stdout:4/338: fdatasync da/db/d20/f29 0 2024-04-05T03:02:45.948 INFO:tasks.workunit.client.1.smithi203.stdout:9/581: unlink d13/d1b/f21 0 2024-04-05T03:02:45.949 INFO:tasks.workunit.client.0.smithi153.stdout:6/312: dwrite d4/f20 [0,4194304] 0 2024-04-05T03:02:45.950 INFO:tasks.workunit.client.0.smithi153.stdout:1/225: link d5/dd/d1b/f2b d5/dd/d21/d1e/f43 0 2024-04-05T03:02:45.951 INFO:tasks.workunit.client.1.smithi203.stdout:2/795: mknod d5/d10/d35/d2d/d7c/d29/db1/c112 0 2024-04-05T03:02:45.952 INFO:tasks.workunit.client.0.smithi153.stdout:6/313: truncate d4/d38/d4c/f6a 221854 0 2024-04-05T03:02:45.952 INFO:tasks.workunit.client.0.smithi153.stdout:5/265: truncate d0/d25/f2f 2323832 0 2024-04-05T03:02:45.953 INFO:tasks.workunit.client.1.smithi203.stdout:7/779: creat d8/da/d72/fee x:0 0 0 2024-04-05T03:02:45.953 INFO:tasks.workunit.client.0.smithi153.stdout:4/339: creat da/de/d56/f70 x:0 0 0 2024-04-05T03:02:45.953 INFO:tasks.workunit.client.0.smithi153.stdout:5/266: write d0/d2c/f48 [663059,85007] 0 2024-04-05T03:02:45.954 INFO:tasks.workunit.client.0.smithi153.stdout:4/340: readlink da/db/d20/d35/d41/l5a 0 2024-04-05T03:02:45.955 INFO:tasks.workunit.client.1.smithi203.stdout:5/626: write d3/d4/dc/f25 [4854558,31877] 0 2024-04-05T03:02:45.955 INFO:tasks.workunit.client.1.smithi203.stdout:7/780: write d8/d32/d5d/dbe/dc9/fe0 [818829,60353] 0 2024-04-05T03:02:45.957 INFO:tasks.workunit.client.0.smithi153.stdout:6/314: mknod d4/dc/df/d16/d17/d54/c71 0 2024-04-05T03:02:45.958 INFO:tasks.workunit.client.1.smithi203.stdout:1/593: link d7/c82 d7/d14/d2b/cbb 0 2024-04-05T03:02:45.958 INFO:tasks.workunit.client.1.smithi203.stdout:7/781: write d8/d32/d16/d1a/d21/f27 [5147721,48930] 0 2024-04-05T03:02:45.959 INFO:tasks.workunit.client.0.smithi153.stdout:6/315: readlink d4/dc/df/d16/d5f/l6d 0 2024-04-05T03:02:45.960 INFO:tasks.workunit.client.0.smithi153.stdout:6/316: chown d4/dc/df/d16/d33/f47 3 1 2024-04-05T03:02:45.963 INFO:tasks.workunit.client.0.smithi153.stdout:5/267: mknod d0/c69 0 2024-04-05T03:02:45.963 INFO:tasks.workunit.client.1.smithi203.stdout:6/643: write d0/da/d18/d38/f47 [1409052,31364] 0 2024-04-05T03:02:45.963 INFO:tasks.workunit.client.1.smithi203.stdout:4/464: write da/d29/f2d [2616925,58375] 0 2024-04-05T03:02:45.963 INFO:tasks.workunit.client.1.smithi203.stdout:2/796: creat d5/d10/d44/df6/f113 x:0 0 0 2024-04-05T03:02:45.965 INFO:tasks.workunit.client.1.smithi203.stdout:8/644: dread d1/d12/d2f/d1f/f50 [0,4194304] 0 2024-04-05T03:02:45.966 INFO:tasks.workunit.client.1.smithi203.stdout:1/594: truncate d7/d14/d2b/d58/d7a/f8a 820185 0 2024-04-05T03:02:45.967 INFO:tasks.workunit.client.1.smithi203.stdout:9/582: link d13/d5c/f89 d13/d1b/d6b/d9e/d59/d5f/fb9 0 2024-04-05T03:02:45.967 INFO:tasks.workunit.client.0.smithi153.stdout:4/341: truncate da/fc 3069374 0 2024-04-05T03:02:45.967 INFO:tasks.workunit.client.1.smithi203.stdout:4/465: truncate da/d29/d2e/f7a 312432 0 2024-04-05T03:02:45.968 INFO:tasks.workunit.client.1.smithi203.stdout:7/782: mknod d8/d32/d16/d1a/d3d/cef 0 2024-04-05T03:02:45.969 INFO:tasks.workunit.client.1.smithi203.stdout:2/797: read d5/d63/f6e [68255,33274] 0 2024-04-05T03:02:45.969 INFO:tasks.workunit.client.1.smithi203.stdout:8/645: chown d1/d12/d2f/d1f/fae 1500 1 2024-04-05T03:02:45.971 INFO:tasks.workunit.client.1.smithi203.stdout:5/627: creat d3/d4/fd1 x:0 0 0 2024-04-05T03:02:45.972 INFO:tasks.workunit.client.0.smithi153.stdout:5/268: fsync d0/d25/d4d/d62/f10 0 2024-04-05T03:02:45.972 INFO:tasks.workunit.client.1.smithi203.stdout:2/798: chown d5/d63/d86/fb0 803 1 2024-04-05T03:02:45.974 INFO:tasks.workunit.client.0.smithi153.stdout:4/342: creat da/db/d20/d35/d38/d3e/f71 x:0 0 0 2024-04-05T03:02:45.981 INFO:tasks.workunit.client.1.smithi203.stdout:9/583: mkdir d13/d1b/d26/d51/da1/dba 0 2024-04-05T03:02:45.983 INFO:tasks.workunit.client.1.smithi203.stdout:5/628: mknod d3/cd2 0 2024-04-05T03:02:45.984 INFO:tasks.workunit.client.0.smithi153.stdout:2/291: dread d9/da/d37/f3d [0,4194304] 0 2024-04-05T03:02:45.984 INFO:tasks.workunit.client.0.smithi153.stdout:2/292: stat l2 0 2024-04-05T03:02:45.986 INFO:tasks.workunit.client.0.smithi153.stdout:6/317: getdents d4/dc/df/d16/d24 0 2024-04-05T03:02:45.989 INFO:tasks.workunit.client.1.smithi203.stdout:2/799: fdatasync d5/d10/d1a/d57/f6b 0 2024-04-05T03:02:45.989 INFO:tasks.workunit.client.0.smithi153.stdout:5/269: creat d0/d25/d4d/d62/d37/f6a x:0 0 0 2024-04-05T03:02:45.990 INFO:tasks.workunit.client.1.smithi203.stdout:4/466: dread da/d29/f2d [0,4194304] 0 2024-04-05T03:02:45.990 INFO:tasks.workunit.client.1.smithi203.stdout:9/584: mknod d13/d5c/d9d/cbb 0 2024-04-05T03:02:45.991 INFO:tasks.workunit.client.0.smithi153.stdout:4/343: creat da/db/d20/d45/f72 x:0 0 0 2024-04-05T03:02:45.991 INFO:tasks.workunit.client.1.smithi203.stdout:8/646: write d1/d6/f41 [1462019,29077] 0 2024-04-05T03:02:45.992 INFO:tasks.workunit.client.0.smithi153.stdout:2/293: dwrite d9/da/d19/f1f [8388608,4194304] 0 2024-04-05T03:02:45.992 INFO:tasks.workunit.client.1.smithi203.stdout:5/629: symlink d3/d4/dc/d67/dac/ld3 0 2024-04-05T03:02:45.992 INFO:tasks.workunit.client.1.smithi203.stdout:5/630: chown d3/d4/d9e 122 1 2024-04-05T03:02:45.993 INFO:tasks.workunit.client.0.smithi153.stdout:6/318: symlink d4/dc/df/d16/l72 0 2024-04-05T03:02:45.993 INFO:tasks.workunit.client.0.smithi153.stdout:5/270: truncate d0/d25/d4d/f5a 22955 0 2024-04-05T03:02:45.996 INFO:tasks.workunit.client.1.smithi203.stdout:6/644: truncate d0/da/d18/d38/d2a/d44/d78/f89 81880 0 2024-04-05T03:02:46.000 INFO:tasks.workunit.client.1.smithi203.stdout:2/800: rmdir d5/d10/d35/d36/db6 39 2024-04-05T03:02:46.000 INFO:tasks.workunit.client.0.smithi153.stdout:6/319: mkdir d4/dc/df/d16/d5f/d73 0 2024-04-05T03:02:46.000 INFO:tasks.workunit.client.0.smithi153.stdout:4/344: getdents da/de/d66 0 2024-04-05T03:02:46.002 INFO:tasks.workunit.client.1.smithi203.stdout:2/801: truncate d5/d10/d35/d36/f3b 4388697 0 2024-04-05T03:02:46.004 INFO:tasks.workunit.client.1.smithi203.stdout:8/647: rename d1/d4b/d5f/f7d to d1/d4b/d5f/d74/fb5 0 2024-04-05T03:02:46.005 INFO:tasks.workunit.client.0.smithi153.stdout:5/271: link d0/d25/d4d/d62/c51 d0/d25/c6b 0 2024-04-05T03:02:46.006 INFO:tasks.workunit.client.0.smithi153.stdout:6/320: creat d4/dc/df/d16/d5f/f74 x:0 0 0 2024-04-05T03:02:46.006 INFO:tasks.workunit.client.1.smithi203.stdout:7/783: getdents d8/d32/d16/d35/d64 0 2024-04-05T03:02:46.011 INFO:tasks.workunit.client.0.smithi153.stdout:6/321: unlink d4/dc/df/d16/f21 0 2024-04-05T03:02:46.012 INFO:tasks.workunit.client.1.smithi203.stdout:9/585: mkdir d13/d1b/d6b/d9e/d59/dbc 0 2024-04-05T03:02:46.013 INFO:tasks.workunit.client.1.smithi203.stdout:7/784: dwrite d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:46.013 INFO:tasks.workunit.client.0.smithi153.stdout:4/345: dwrite da/db/d20/f21 [0,4194304] 0 2024-04-05T03:02:46.013 INFO:tasks.workunit.client.0.smithi153.stdout:5/272: rename d0/d2c/f48 to d0/d2c/d3f/f6c 0 2024-04-05T03:02:46.014 INFO:tasks.workunit.client.0.smithi153.stdout:4/346: chown da/db/d20/d45/d3b/d68 5224 1 2024-04-05T03:02:46.014 INFO:tasks.workunit.client.0.smithi153.stdout:5/273: chown d0/d9/d22 45190000 1 2024-04-05T03:02:46.015 INFO:tasks.workunit.client.1.smithi203.stdout:5/631: dread d3/d1f/d59/d60/f99 [0,4194304] 0 2024-04-05T03:02:46.015 INFO:tasks.workunit.client.1.smithi203.stdout:7/785: chown d8/d32/d5d/dbe/dc9/ddd 242490990 1 2024-04-05T03:02:46.015 INFO:tasks.workunit.client.0.smithi153.stdout:6/322: creat d4/d38/f75 x:0 0 0 2024-04-05T03:02:46.016 INFO:tasks.workunit.client.1.smithi203.stdout:8/648: write d1/d6/fab [963585,18991] 0 2024-04-05T03:02:46.021 INFO:tasks.workunit.client.0.smithi153.stdout:6/323: symlink d4/dc/df/d16/d44/l76 0 2024-04-05T03:02:46.021 INFO:tasks.workunit.client.1.smithi203.stdout:1/595: getdents d7 0 2024-04-05T03:02:46.021 INFO:tasks.workunit.client.1.smithi203.stdout:2/802: fsync d5/d10/d35/d2d/d7c/d72/d8f/fb9 0 2024-04-05T03:02:46.022 INFO:tasks.workunit.client.0.smithi153.stdout:6/324: readlink d4/dc/df/d16/d44/l76 0 2024-04-05T03:02:46.022 INFO:tasks.workunit.client.1.smithi203.stdout:7/786: symlink d8/d32/d16/d1a/d3d/d59/de7/d73/da8/lf0 0 2024-04-05T03:02:46.023 INFO:tasks.workunit.client.0.smithi153.stdout:4/347: rename da/db/d20/d35/d38/d3e/c52 to da/db/d20/d3c/c73 0 2024-04-05T03:02:46.024 INFO:tasks.workunit.client.0.smithi153.stdout:4/348: chown da/db/d20/d35/c65 550634481 1 2024-04-05T03:02:46.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/649: mknod d1/d4b/d68/d75/dac/cb6 0 2024-04-05T03:02:46.026 INFO:tasks.workunit.client.1.smithi203.stdout:8/650: readlink d1/d12/d2f/d1f/l2d 0 2024-04-05T03:02:46.026 INFO:tasks.workunit.client.0.smithi153.stdout:4/349: truncate da/de/f11 1094398 0 2024-04-05T03:02:46.029 INFO:tasks.workunit.client.1.smithi203.stdout:9/586: creat d13/d1b/fbd x:0 0 0 2024-04-05T03:02:46.029 INFO:tasks.workunit.client.1.smithi203.stdout:5/632: creat d3/d4/fd4 x:0 0 0 2024-04-05T03:02:46.031 INFO:tasks.workunit.client.1.smithi203.stdout:8/651: write d1/d12/d2f/d1f/f50 [4461890,22034] 0 2024-04-05T03:02:46.034 INFO:tasks.workunit.client.1.smithi203.stdout:9/587: creat d13/d1b/d6b/d7a/fbe x:0 0 0 2024-04-05T03:02:46.034 INFO:tasks.workunit.client.1.smithi203.stdout:6/645: write d0/da/d54/fb0 [3534971,73529] 0 2024-04-05T03:02:46.034 INFO:tasks.workunit.client.1.smithi203.stdout:1/596: dwrite d7/d24/d33/d48/d4b/d51/f87 [0,4194304] 0 2024-04-05T03:02:46.035 INFO:tasks.workunit.client.1.smithi203.stdout:2/803: link d5/d10/d35/d36/f6c d5/d10/d7f/f114 0 2024-04-05T03:02:46.036 INFO:tasks.workunit.client.1.smithi203.stdout:7/787: getdents d8/d32/d5d/d95 0 2024-04-05T03:02:46.038 INFO:tasks.workunit.client.1.smithi203.stdout:5/633: rename d3/d4/dc/l61 to d3/d5f/ld5 0 2024-04-05T03:02:46.039 INFO:tasks.workunit.client.1.smithi203.stdout:1/597: stat d7/d79/l8d 0 2024-04-05T03:02:46.041 INFO:tasks.workunit.client.1.smithi203.stdout:6/646: creat d0/db4/fc8 x:0 0 0 2024-04-05T03:02:46.045 INFO:tasks.workunit.client.1.smithi203.stdout:4/467: truncate da/d3c/f5f 944590 0 2024-04-05T03:02:46.045 INFO:tasks.workunit.client.0.smithi153.stdout:5/274: unlink d0/d2c/d3f/f6c 0 2024-04-05T03:02:46.046 INFO:tasks.workunit.client.1.smithi203.stdout:7/788: rename d8/f20 to d8/d32/d16/d1a/d3d/d59/de3/ff1 0 2024-04-05T03:02:46.046 INFO:tasks.workunit.client.0.smithi153.stdout:5/275: stat d0/d25/d4d/d62/l54 0 2024-04-05T03:02:46.047 INFO:tasks.workunit.client.0.smithi153.stdout:5/276: write d0/d9/d22/f33 [263181,30804] 0 2024-04-05T03:02:46.050 INFO:tasks.workunit.client.1.smithi203.stdout:8/652: dwrite d1/d12/d2f/d1f/f50 [0,4194304] 0 2024-04-05T03:02:46.053 INFO:tasks.workunit.client.0.smithi153.stdout:5/277: rename d0/d25/c60 to d0/d25/d4d/d62/d37/d38/c6d 0 2024-04-05T03:02:46.055 INFO:tasks.workunit.client.0.smithi153.stdout:6/325: dwrite d4/dc/df/f13 [0,4194304] 0 2024-04-05T03:02:46.056 INFO:tasks.workunit.client.1.smithi203.stdout:9/588: dwrite d13/d1b/d6b/d9e/d59/d5f/f71 [0,4194304] 0 2024-04-05T03:02:46.057 INFO:tasks.workunit.client.0.smithi153.stdout:4/350: dwrite da/fc [0,4194304] 0 2024-04-05T03:02:46.058 INFO:tasks.workunit.client.1.smithi203.stdout:8/653: write d1/d6/d8f/fb0 [778344,108592] 0 2024-04-05T03:02:46.059 INFO:tasks.workunit.client.0.smithi153.stdout:6/326: chown d4/dc/df/d16/d33/l59 73 1 2024-04-05T03:02:46.063 INFO:tasks.workunit.client.1.smithi203.stdout:2/804: dwrite d5/d10/d35/d36/db6/f37 [0,4194304] 0 2024-04-05T03:02:46.064 INFO:tasks.workunit.client.1.smithi203.stdout:1/598: mknod d7/d24/d33/d48/d4b/d6b/db9/cbc 0 2024-04-05T03:02:46.067 INFO:tasks.workunit.client.1.smithi203.stdout:2/805: chown d5/d10/d35/d36/db6/d87 18318705 1 2024-04-05T03:02:46.069 INFO:tasks.workunit.client.1.smithi203.stdout:7/789: mknod d8/d32/d16/d1a/d3d/d59/de7/d52/d65/d82/cf2 0 2024-04-05T03:02:46.069 INFO:tasks.workunit.client.1.smithi203.stdout:6/647: dwrite d0/da/d18/d38/d86/f7c [4194304,4194304] 0 2024-04-05T03:02:46.071 INFO:tasks.workunit.client.0.smithi153.stdout:5/278: rename d0/d25/d4d/d62/d5f/d61/f67 to d0/d25/f6e 0 2024-04-05T03:02:46.075 INFO:tasks.workunit.client.1.smithi203.stdout:8/654: dwrite d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:46.075 INFO:tasks.workunit.client.0.smithi153.stdout:4/351: dwrite da/de/d56/d6a/d2d/f57 [0,4194304] 0 2024-04-05T03:02:46.076 INFO:tasks.workunit.client.0.smithi153.stdout:4/352: fsync da/de/d56/f70 0 2024-04-05T03:02:46.082 INFO:tasks.workunit.client.0.smithi153.stdout:4/353: dread f7 [0,4194304] 0 2024-04-05T03:02:46.082 INFO:tasks.workunit.client.0.smithi153.stdout:5/279: symlink d0/d25/d4d/d62/d37/l6f 0 2024-04-05T03:02:46.085 INFO:tasks.workunit.client.1.smithi203.stdout:2/806: mkdir d5/d10/db8/dbe/d115 0 2024-04-05T03:02:46.085 INFO:tasks.workunit.client.1.smithi203.stdout:7/790: write d8/d32/d5d/d6d/dca/f8c [4662573,32358] 0 2024-04-05T03:02:46.089 INFO:tasks.workunit.client.1.smithi203.stdout:6/648: creat d0/da/d54/d4c/daf/fc9 x:0 0 0 2024-04-05T03:02:46.090 INFO:tasks.workunit.client.0.smithi153.stdout:3/226: sync 2024-04-05T03:02:46.090 INFO:tasks.workunit.client.0.smithi153.stdout:4/354: dwrite da/db/d20/f29 [0,4194304] 0 2024-04-05T03:02:46.090 INFO:tasks.workunit.client.0.smithi153.stdout:3/227: readlink d1d/l22 0 2024-04-05T03:02:46.094 INFO:tasks.workunit.client.0.smithi153.stdout:3/228: fsync d1d/f3e 0 2024-04-05T03:02:46.095 INFO:tasks.workunit.client.1.smithi203.stdout:2/807: creat d5/d66/d7d/f116 x:0 0 0 2024-04-05T03:02:46.095 INFO:tasks.workunit.client.1.smithi203.stdout:6/649: read d0/da/f61 [236463,87433] 0 2024-04-05T03:02:46.097 INFO:tasks.workunit.client.1.smithi203.stdout:1/599: link d7/d24/caf d7/d24/d33/d48/cbd 0 2024-04-05T03:02:46.099 INFO:tasks.workunit.client.1.smithi203.stdout:8/655: truncate d1/d6/f56 2024985 0 2024-04-05T03:02:46.099 INFO:tasks.workunit.client.0.smithi153.stdout:3/229: rename d1d/d38 to d1d/d26/d57 0 2024-04-05T03:02:46.100 INFO:tasks.workunit.client.1.smithi203.stdout:5/634: dwrite d3/d1f/d59/d60/fb4 [4194304,4194304] 0 2024-04-05T03:02:46.101 INFO:tasks.workunit.client.0.smithi153.stdout:4/355: dwrite da/db/d20/d35/d38/d3e/f71 [0,4194304] 0 2024-04-05T03:02:46.102 INFO:tasks.workunit.client.0.smithi153.stdout:4/356: readlink da/l4c 0 2024-04-05T03:02:46.102 INFO:tasks.workunit.client.1.smithi203.stdout:5/635: dread - d3/d4/fd1 zero size 2024-04-05T03:02:46.103 INFO:tasks.workunit.client.1.smithi203.stdout:7/791: getdents d8/d32/d16/d1a/d3d/d59/de7/d52 0 2024-04-05T03:02:46.107 INFO:tasks.workunit.client.1.smithi203.stdout:6/650: rmdir d0/da/d54/d4c/da1/db7/d71 39 2024-04-05T03:02:46.107 INFO:tasks.workunit.client.1.smithi203.stdout:8/656: mknod d1/d12/cb7 0 2024-04-05T03:02:46.107 INFO:tasks.workunit.client.0.smithi153.stdout:3/230: creat d1d/d26/d3c/d3f/f58 x:0 0 0 2024-04-05T03:02:46.108 INFO:tasks.workunit.client.1.smithi203.stdout:4/468: write da/d29/d2e/d53/f68 [34756,40140] 0 2024-04-05T03:02:46.109 INFO:tasks.workunit.client.0.smithi153.stdout:3/231: write d1d/d26/d3c/f46 [29639,82850] 0 2024-04-05T03:02:46.109 INFO:tasks.workunit.client.1.smithi203.stdout:7/792: stat d8/d32/d16/d1a/d30/f81 0 2024-04-05T03:02:46.109 INFO:tasks.workunit.client.0.smithi153.stdout:4/357: rmdir da/db 39 2024-04-05T03:02:46.112 INFO:tasks.workunit.client.0.smithi153.stdout:3/232: readlink l13 0 2024-04-05T03:02:46.113 INFO:tasks.workunit.client.1.smithi203.stdout:4/469: truncate da/d3c/d4c/f79 644396 0 2024-04-05T03:02:46.113 INFO:tasks.workunit.client.1.smithi203.stdout:6/651: creat d0/da/d54/d4c/fca x:0 0 0 2024-04-05T03:02:46.114 INFO:tasks.workunit.client.0.smithi153.stdout:3/233: creat d1d/d26/d57/f59 x:0 0 0 2024-04-05T03:02:46.115 INFO:tasks.workunit.client.1.smithi203.stdout:8/657: mkdir d1/d12/d17/db8 0 2024-04-05T03:02:46.115 INFO:tasks.workunit.client.1.smithi203.stdout:1/600: dwrite d7/d24/d33/d48/d4b/d51/f67 [0,4194304] 0 2024-04-05T03:02:46.116 INFO:tasks.workunit.client.0.smithi153.stdout:6/327: dread d4/f15 [0,4194304] 0 2024-04-05T03:02:46.116 INFO:tasks.workunit.client.0.smithi153.stdout:3/234: dread - d1d/d27/f34 zero size 2024-04-05T03:02:46.117 INFO:tasks.workunit.client.0.smithi153.stdout:6/328: write d4/dc/f46 [159771,33774] 0 2024-04-05T03:02:46.118 INFO:tasks.workunit.client.1.smithi203.stdout:7/793: mknod d8/d32/d5d/d6d/dca/cf3 0 2024-04-05T03:02:46.118 INFO:tasks.workunit.client.1.smithi203.stdout:1/601: read d7/d24/d33/f64 [116373,122819] 0 2024-04-05T03:02:46.119 INFO:tasks.workunit.client.0.smithi153.stdout:6/329: read d4/dc/df/f39 [1936766,54138] 0 2024-04-05T03:02:46.122 INFO:tasks.workunit.client.0.smithi153.stdout:3/235: link d1d/d26/c48 d1d/d27/c5a 0 2024-04-05T03:02:46.123 INFO:tasks.workunit.client.1.smithi203.stdout:6/652: creat d0/da/d54/d4c/da1/db7/fcb x:0 0 0 2024-04-05T03:02:46.125 INFO:tasks.workunit.client.1.smithi203.stdout:1/602: write d7/d14/d2b/d58/d86/d91/f97 [340877,75163] 0 2024-04-05T03:02:46.126 INFO:tasks.workunit.client.0.smithi153.stdout:3/236: read d1d/d32/f49 [127200,30486] 0 2024-04-05T03:02:46.128 INFO:tasks.workunit.client.1.smithi203.stdout:8/658: dwrite d1/d12/d2f/d5b/fb4 [0,4194304] 0 2024-04-05T03:02:46.128 INFO:tasks.workunit.client.0.smithi153.stdout:3/237: creat d1d/d26/d3c/f5b x:0 0 0 2024-04-05T03:02:46.129 INFO:tasks.workunit.client.1.smithi203.stdout:6/653: chown d0/da/d54/d4c/da1/db7/l76 4391217 1 2024-04-05T03:02:46.129 INFO:tasks.workunit.client.1.smithi203.stdout:8/659: stat d1/d6/f93 0 2024-04-05T03:02:46.131 INFO:tasks.workunit.client.1.smithi203.stdout:1/603: chown d7/d24/d33/d48/d4b/d6b/f98 185813862 1 2024-04-05T03:02:46.133 INFO:tasks.workunit.client.1.smithi203.stdout:8/660: fsync d1/d4b/d68/d75/d77/f87 0 2024-04-05T03:02:46.134 INFO:tasks.workunit.client.1.smithi203.stdout:1/604: mknod d7/cbe 0 2024-04-05T03:02:46.135 INFO:tasks.workunit.client.1.smithi203.stdout:7/794: dwrite d8/d32/d5d/d6d/daf/fda [0,4194304] 0 2024-04-05T03:02:46.139 INFO:tasks.workunit.client.1.smithi203.stdout:8/661: truncate d1/d12/d2f/d5b/fa7 450509 0 2024-04-05T03:02:46.140 INFO:tasks.workunit.client.1.smithi203.stdout:1/605: mkdir d7/d14/d9d/dbf 0 2024-04-05T03:02:46.146 INFO:tasks.workunit.client.1.smithi203.stdout:1/606: rename d7/d14/c25 to d7/d14/cc0 0 2024-04-05T03:02:46.148 INFO:tasks.workunit.client.0.smithi153.stdout:5/280: dwrite d0/d25/f2f [0,4194304] 0 2024-04-05T03:02:46.149 INFO:tasks.workunit.client.0.smithi153.stdout:6/330: truncate d4/dc/df/d16/d17/f3b 591762 0 2024-04-05T03:02:46.149 INFO:tasks.workunit.client.0.smithi153.stdout:5/281: fsync d0/d25/d4d/f66 0 2024-04-05T03:02:46.150 INFO:tasks.workunit.client.1.smithi203.stdout:9/589: dwrite d13/d1b/d6b/d9e/d77/f7d [0,4194304] 0 2024-04-05T03:02:46.151 INFO:tasks.workunit.client.0.smithi153.stdout:4/358: dread da/de/f1c [0,4194304] 0 2024-04-05T03:02:46.151 INFO:tasks.workunit.client.1.smithi203.stdout:8/662: dwrite d1/d12/d2f/d1f/fae [0,4194304] 0 2024-04-05T03:02:46.151 INFO:tasks.workunit.client.0.smithi153.stdout:6/331: fdatasync d4/dc/f46 0 2024-04-05T03:02:46.153 INFO:tasks.workunit.client.0.smithi153.stdout:5/282: write d0/d2c/f2d [300255,119819] 0 2024-04-05T03:02:46.154 INFO:tasks.workunit.client.1.smithi203.stdout:8/663: write d1/d12/d2f/f2a [3345547,94986] 0 2024-04-05T03:02:46.154 INFO:tasks.workunit.client.1.smithi203.stdout:9/590: readlink d13/d1b/l4d 0 2024-04-05T03:02:46.154 INFO:tasks.workunit.client.0.smithi153.stdout:6/332: truncate d4/dc/f2e 941905 0 2024-04-05T03:02:46.157 INFO:tasks.workunit.client.0.smithi153.stdout:4/359: rmdir da/d2a 39 2024-04-05T03:02:46.157 INFO:tasks.workunit.client.1.smithi203.stdout:9/591: stat d13/f1d 0 2024-04-05T03:02:46.158 INFO:tasks.workunit.client.1.smithi203.stdout:7/795: link d8/d32/d16/d1a/d3d/d59/de7/d5e/l7c d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/lf4 0 2024-04-05T03:02:46.158 INFO:tasks.workunit.client.0.smithi153.stdout:3/238: dwrite fd [0,4194304] 0 2024-04-05T03:02:46.159 INFO:tasks.workunit.client.0.smithi153.stdout:5/283: creat d0/d25/f70 x:0 0 0 2024-04-05T03:02:46.163 INFO:tasks.workunit.client.0.smithi153.stdout:5/284: fsync d0/f27 0 2024-04-05T03:02:46.164 INFO:tasks.workunit.client.0.smithi153.stdout:3/239: symlink d1d/d32/l5c 0 2024-04-05T03:02:46.167 INFO:tasks.workunit.client.0.smithi153.stdout:5/285: write d0/d2c/f3d [731110,61222] 0 2024-04-05T03:02:46.169 INFO:tasks.workunit.client.0.smithi153.stdout:4/360: creat da/db/d20/d45/d3b/d68/f74 x:0 0 0 2024-04-05T03:02:46.169 INFO:tasks.workunit.client.1.smithi203.stdout:7/796: mknod d8/d32/d16/d1a/d21/cf5 0 2024-04-05T03:02:46.169 INFO:tasks.workunit.client.1.smithi203.stdout:5/636: write d3/d1f/d59/d64/f2e [4420239,80013] 0 2024-04-05T03:02:46.172 INFO:tasks.workunit.client.1.smithi203.stdout:1/607: dread d7/d24/d33/d48/d4b/d6b/f98 [0,4194304] 0 2024-04-05T03:02:46.173 INFO:tasks.workunit.client.0.smithi153.stdout:4/361: mknod da/de/d56/d6a/c75 0 2024-04-05T03:02:46.174 INFO:tasks.workunit.client.0.smithi153.stdout:5/286: write d0/f3c [3265048,11805] 0 2024-04-05T03:02:46.175 INFO:tasks.workunit.client.0.smithi153.stdout:4/362: write da/db/d20/d45/f72 [465673,24470] 0 2024-04-05T03:02:46.176 INFO:tasks.workunit.client.1.smithi203.stdout:8/664: dwrite d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:46.177 INFO:tasks.workunit.client.0.smithi153.stdout:4/363: dread - da/db/d20/d45/d3b/d68/f74 zero size 2024-04-05T03:02:46.178 INFO:tasks.workunit.client.0.smithi153.stdout:4/364: dread - da/db/d20/f4f zero size 2024-04-05T03:02:46.179 INFO:tasks.workunit.client.1.smithi203.stdout:5/637: mkdir d3/d4/dd6 0 2024-04-05T03:02:46.179 INFO:tasks.workunit.client.1.smithi203.stdout:1/608: chown d7/d24/l32 1598 1 2024-04-05T03:02:46.180 INFO:tasks.workunit.client.1.smithi203.stdout:7/797: symlink d8/d32/d16/d1a/d3d/d59/de7/d52/d65/lf6 0 2024-04-05T03:02:46.182 INFO:tasks.workunit.client.1.smithi203.stdout:8/665: symlink d1/d6/d2e/d5c/lb9 0 2024-04-05T03:02:46.183 INFO:tasks.workunit.client.1.smithi203.stdout:6/654: dwrite d0/da/d18/f26 [0,4194304] 0 2024-04-05T03:02:46.184 INFO:tasks.workunit.client.0.smithi153.stdout:4/365: mkdir da/db/d20/d35/d38/d76 0 2024-04-05T03:02:46.188 INFO:tasks.workunit.client.0.smithi153.stdout:5/287: rename d0/c45 to d0/d9/c71 0 2024-04-05T03:02:46.193 INFO:tasks.workunit.client.1.smithi203.stdout:1/609: link d7/f3d d7/d24/fc1 0 2024-04-05T03:02:46.194 INFO:tasks.workunit.client.1.smithi203.stdout:1/610: write d7/d14/d2b/d58/d86/d91/f97 [130357,29598] 0 2024-04-05T03:02:46.196 INFO:tasks.workunit.client.1.smithi203.stdout:1/611: write d7/f9c [3446160,5090] 0 2024-04-05T03:02:46.196 INFO:tasks.workunit.client.0.smithi153.stdout:5/288: dread d0/f14 [4194304,4194304] 0 2024-04-05T03:02:46.196 INFO:tasks.workunit.client.0.smithi153.stdout:3/240: truncate d1d/d26/f47 3879281 0 2024-04-05T03:02:46.197 INFO:tasks.workunit.client.0.smithi153.stdout:5/289: fdatasync d0/d9/f1f 0 2024-04-05T03:02:46.201 INFO:tasks.workunit.client.1.smithi203.stdout:9/592: dread d13/d1b/d33/d64/f6d [0,4194304] 0 2024-04-05T03:02:46.201 INFO:tasks.workunit.client.1.smithi203.stdout:1/612: unlink d7/f2d 0 2024-04-05T03:02:46.202 INFO:tasks.workunit.client.1.smithi203.stdout:6/655: getdents d0/da/d18/d38/d2a/d44/d78/da7 0 2024-04-05T03:02:46.203 INFO:tasks.workunit.client.0.smithi153.stdout:3/241: dwrite d1d/d27/f34 [0,4194304] 0 2024-04-05T03:02:46.206 INFO:tasks.workunit.client.1.smithi203.stdout:9/593: creat d13/d1b/d6b/fbf x:0 0 0 2024-04-05T03:02:46.207 INFO:tasks.workunit.client.0.smithi153.stdout:3/242: unlink d1d/d27/f55 0 2024-04-05T03:02:46.208 INFO:tasks.workunit.client.0.smithi153.stdout:3/243: chown d1d/d26/d3c 1011 1 2024-04-05T03:02:46.208 INFO:tasks.workunit.client.1.smithi203.stdout:1/613: truncate d7/d24/d33/d48/d4b/d51/f7f 675666 0 2024-04-05T03:02:46.210 INFO:tasks.workunit.client.1.smithi203.stdout:5/638: write d3/d1f/d59/d64/f3a [520116,90360] 0 2024-04-05T03:02:46.213 INFO:tasks.workunit.client.1.smithi203.stdout:6/656: rename d0/da/d54/d4c/da1/db7/d71/d8c to d0/da/d54/d4c/d7f/dcc 0 2024-04-05T03:02:46.214 INFO:tasks.workunit.client.0.smithi153.stdout:3/244: dwrite d1d/d26/d3c/d3f/f58 [0,4194304] 0 2024-04-05T03:02:46.217 INFO:tasks.workunit.client.1.smithi203.stdout:1/614: rmdir d7/d24/d33/d48/d4b 39 2024-04-05T03:02:46.217 INFO:tasks.workunit.client.1.smithi203.stdout:7/798: dread d8/d32/d16/d1a/d5a/f85 [0,4194304] 0 2024-04-05T03:02:46.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/366: dwrite da/de/f19 [0,4194304] 0 2024-04-05T03:02:46.221 INFO:tasks.workunit.client.0.smithi153.stdout:3/245: unlink d1d/d26/d3c/d3f/f58 0 2024-04-05T03:02:46.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/367: chown l6 121106784 1 2024-04-05T03:02:46.223 INFO:tasks.workunit.client.0.smithi153.stdout:3/246: write d1d/d27/d2b/f3d [2190813,11234] 0 2024-04-05T03:02:46.223 INFO:tasks.workunit.client.1.smithi203.stdout:7/799: creat d8/d32/d16/d1a/d3d/d59/de7/d73/ff7 x:0 0 0 2024-04-05T03:02:46.225 INFO:tasks.workunit.client.0.smithi153.stdout:4/368: symlink da/db/l77 0 2024-04-05T03:02:46.226 INFO:tasks.workunit.client.0.smithi153.stdout:3/247: write d1d/d32/f49 [1100844,55886] 0 2024-04-05T03:02:46.229 INFO:tasks.workunit.client.1.smithi203.stdout:6/657: dwrite d0/da/f43 [0,4194304] 0 2024-04-05T03:02:46.229 INFO:tasks.workunit.client.0.smithi153.stdout:4/369: mkdir da/de/d56/d78 0 2024-04-05T03:02:46.230 INFO:tasks.workunit.client.0.smithi153.stdout:4/370: readlink da/de/l10 0 2024-04-05T03:02:46.232 INFO:tasks.workunit.client.0.smithi153.stdout:3/248: unlink d1d/d27/d2b/f3d 0 2024-04-05T03:02:46.232 INFO:tasks.workunit.client.1.smithi203.stdout:1/615: dwrite d7/d79/f7d [0,4194304] 0 2024-04-05T03:02:46.233 INFO:tasks.workunit.client.0.smithi153.stdout:3/249: chown d1d/d32/f49 20134 1 2024-04-05T03:02:46.233 INFO:tasks.workunit.client.1.smithi203.stdout:9/594: truncate d13/f24 1935893 0 2024-04-05T03:02:46.234 INFO:tasks.workunit.client.0.smithi153.stdout:3/250: write d1d/d32/f53 [311578,63630] 0 2024-04-05T03:02:46.236 INFO:tasks.workunit.client.1.smithi203.stdout:7/800: symlink d8/da/d71/lf8 0 2024-04-05T03:02:46.236 INFO:tasks.workunit.client.1.smithi203.stdout:5/639: truncate d3/d4/dc/f37 536629 0 2024-04-05T03:02:46.237 INFO:tasks.workunit.client.0.smithi153.stdout:4/371: creat da/db/f79 x:0 0 0 2024-04-05T03:02:46.240 INFO:tasks.workunit.client.1.smithi203.stdout:5/640: symlink d3/d1f/d59/d60/ld7 0 2024-04-05T03:02:46.242 INFO:tasks.workunit.client.0.smithi153.stdout:3/251: mknod d1d/d26/d3c/d3f/c5d 0 2024-04-05T03:02:46.245 INFO:tasks.workunit.client.0.smithi153.stdout:4/372: rename da/db/d20/d45/d3b/d68/f74 to da/de/d56/d6a/d2d/f7a 0 2024-04-05T03:02:46.246 INFO:tasks.workunit.client.1.smithi203.stdout:5/641: write d3/d1f/d59/d60/f99 [2714517,30787] 0 2024-04-05T03:02:46.246 INFO:tasks.workunit.client.0.smithi153.stdout:3/252: mkdir d1d/d5e 0 2024-04-05T03:02:46.247 INFO:tasks.workunit.client.1.smithi203.stdout:1/616: creat d7/d24/fc2 x:0 0 0 2024-04-05T03:02:46.247 INFO:tasks.workunit.client.0.smithi153.stdout:4/373: creat da/db/f7b x:0 0 0 2024-04-05T03:02:46.248 INFO:tasks.workunit.client.0.smithi153.stdout:3/253: mkdir d1d/d32/d5f 0 2024-04-05T03:02:46.249 INFO:tasks.workunit.client.0.smithi153.stdout:4/374: truncate da/f26 1997547 0 2024-04-05T03:02:46.250 INFO:tasks.workunit.client.1.smithi203.stdout:9/595: dread d13/d1b/d26/f93 [0,4194304] 0 2024-04-05T03:02:46.251 INFO:tasks.workunit.client.1.smithi203.stdout:7/801: dwrite d8/d32/dc3/dc8/fe9 [0,4194304] 0 2024-04-05T03:02:46.258 INFO:tasks.workunit.client.0.smithi153.stdout:3/254: link d1d/d27/c5a d1d/d27/d2b/c60 0 2024-04-05T03:02:46.258 INFO:tasks.workunit.client.0.smithi153.stdout:4/375: dwrite da/de/f11 [0,4194304] 0 2024-04-05T03:02:46.260 INFO:tasks.workunit.client.0.smithi153.stdout:4/376: chown da/db/d20/d35/d41 1011025 1 2024-04-05T03:02:46.261 INFO:tasks.workunit.client.0.smithi153.stdout:3/255: mknod d1d/d27/d2b/c61 0 2024-04-05T03:02:46.261 INFO:tasks.workunit.client.1.smithi203.stdout:5/642: dwrite d3/d4/dc/d67/db5/fcb [0,4194304] 0 2024-04-05T03:02:46.262 INFO:tasks.workunit.client.0.smithi153.stdout:3/256: stat d1d/d32/c37 0 2024-04-05T03:02:46.265 INFO:tasks.workunit.client.0.smithi153.stdout:3/257: creat d1d/d27/d2b/f62 x:0 0 0 2024-04-05T03:02:46.271 INFO:tasks.workunit.client.1.smithi203.stdout:5/643: creat d3/d1f/d59/d60/dc1/fd8 x:0 0 0 2024-04-05T03:02:46.275 INFO:tasks.workunit.client.1.smithi203.stdout:1/617: truncate d7/d24/f66 2792011 0 2024-04-05T03:02:46.275 INFO:tasks.workunit.client.0.smithi153.stdout:8/262: sync 2024-04-05T03:02:46.276 INFO:tasks.workunit.client.1.smithi203.stdout:5/644: symlink d3/d4/dc/ld9 0 2024-04-05T03:02:46.277 INFO:tasks.workunit.client.1.smithi203.stdout:7/802: write d8/d32/d16/d1a/d3d/d59/de7/d52/f2e [4082274,32464] 0 2024-04-05T03:02:46.277 INFO:tasks.workunit.client.0.smithi153.stdout:3/258: mknod d1d/d32/d5f/c63 0 2024-04-05T03:02:46.278 INFO:tasks.workunit.client.1.smithi203.stdout:1/618: fsync d7/d14/d2b/fae 0 2024-04-05T03:02:46.279 INFO:tasks.workunit.client.1.smithi203.stdout:5/645: mkdir d3/d4/dc/d67/db5/dda 0 2024-04-05T03:02:46.280 INFO:tasks.workunit.client.1.smithi203.stdout:7/803: fsync d8/d32/d16/d1a/d30/f98 0 2024-04-05T03:02:46.281 INFO:tasks.workunit.client.0.smithi153.stdout:8/263: mknod d7/d13/d19/d1b/d4f/c56 0 2024-04-05T03:02:46.283 INFO:tasks.workunit.client.1.smithi203.stdout:6/658: dread d0/da/d54/d4c/f5b [0,4194304] 0 2024-04-05T03:02:46.285 INFO:tasks.workunit.client.1.smithi203.stdout:5/646: mknod d3/d1f/cdb 0 2024-04-05T03:02:46.286 INFO:tasks.workunit.client.1.smithi203.stdout:9/596: dwrite d13/d1b/d26/d31/f60 [0,4194304] 0 2024-04-05T03:02:46.286 INFO:tasks.workunit.client.1.smithi203.stdout:6/659: truncate d0/f9c 696532 0 2024-04-05T03:02:46.286 INFO:tasks.workunit.client.1.smithi203.stdout:9/597: write f2 [2611029,20897] 0 2024-04-05T03:02:46.289 INFO:tasks.workunit.client.1.smithi203.stdout:9/598: chown d13/d1b/d6b/d9e/f5e 0 1 2024-04-05T03:02:46.290 INFO:tasks.workunit.client.1.smithi203.stdout:1/619: rename d7/d14/d83/cb5 to d7/d14/d9d/cc3 0 2024-04-05T03:02:46.291 INFO:tasks.workunit.client.1.smithi203.stdout:1/620: dread - d7/d14/d83/fb4 zero size 2024-04-05T03:02:46.293 INFO:tasks.workunit.client.1.smithi203.stdout:1/621: write d7/d14/f52 [5115891,35730] 0 2024-04-05T03:02:46.293 INFO:tasks.workunit.client.0.smithi153.stdout:3/259: fsync d1d/d26/d3c/f44 0 2024-04-05T03:02:46.294 INFO:tasks.workunit.client.1.smithi203.stdout:7/804: mkdir d8/d32/d5d/dbe/dc9/ddd/df9 0 2024-04-05T03:02:46.295 INFO:tasks.workunit.client.1.smithi203.stdout:7/805: chown d8/d32/d16/d1a/d3d/d59/de7/d52/d65 76 1 2024-04-05T03:02:46.296 INFO:tasks.workunit.client.1.smithi203.stdout:7/806: write d8/d32/d5d/d6d/fbf [7268377,48319] 0 2024-04-05T03:02:46.297 INFO:tasks.workunit.client.0.smithi153.stdout:3/260: read d1d/d27/d2b/f2d [344194,95631] 0 2024-04-05T03:02:46.299 INFO:tasks.workunit.client.0.smithi153.stdout:3/261: fsync d1d/f29 0 2024-04-05T03:02:46.299 INFO:tasks.workunit.client.1.smithi203.stdout:1/622: creat d7/d14/d9d/fc4 x:0 0 0 2024-04-05T03:02:46.301 INFO:tasks.workunit.client.1.smithi203.stdout:5/647: unlink d3/d1f/d59/d64/c73 0 2024-04-05T03:02:46.302 INFO:tasks.workunit.client.0.smithi153.stdout:3/262: rename d1d/d26/f47 to d1d/d27/f64 0 2024-04-05T03:02:46.303 INFO:tasks.workunit.client.1.smithi203.stdout:6/660: getdents d0/da/d18/d38/d2a/d44/d78/da7 0 2024-04-05T03:02:46.305 INFO:tasks.workunit.client.0.smithi153.stdout:3/263: symlink d1d/d27/d2b/l65 0 2024-04-05T03:02:46.306 INFO:tasks.workunit.client.0.smithi153.stdout:3/264: symlink d1d/d27/d2b/l66 0 2024-04-05T03:02:46.307 INFO:tasks.workunit.client.1.smithi203.stdout:5/648: dwrite d3/f7a [0,4194304] 0 2024-04-05T03:02:46.308 INFO:tasks.workunit.client.0.smithi153.stdout:3/265: write d1d/d32/f4a [1016398,723] 0 2024-04-05T03:02:46.309 INFO:tasks.workunit.client.1.smithi203.stdout:6/661: mkdir d0/da/dcd 0 2024-04-05T03:02:46.310 INFO:tasks.workunit.client.1.smithi203.stdout:6/662: dread - d0/da/d54/d82/f83 zero size 2024-04-05T03:02:46.312 INFO:tasks.workunit.client.1.smithi203.stdout:6/663: chown d0/da/d54/d4c/daf/cc6 473205165 1 2024-04-05T03:02:46.314 INFO:tasks.workunit.client.1.smithi203.stdout:5/649: dwrite d3/d4/dc/f25 [0,4194304] 0 2024-04-05T03:02:46.316 INFO:tasks.workunit.client.1.smithi203.stdout:9/599: link d13/l56 d13/d1b/d6b/d9e/d59/dbc/lc0 0 2024-04-05T03:02:46.316 INFO:tasks.workunit.client.1.smithi203.stdout:1/623: creat d7/d24/fc5 x:0 0 0 2024-04-05T03:02:46.318 INFO:tasks.workunit.client.1.smithi203.stdout:1/624: fsync d7/d14/f52 0 2024-04-05T03:02:46.320 INFO:tasks.workunit.client.1.smithi203.stdout:9/600: unlink d13/d1b/d6b/fb4 0 2024-04-05T03:02:46.321 INFO:tasks.workunit.client.1.smithi203.stdout:9/601: stat d13/d1b/d6b/d9e/d59/d7b/f7f 0 2024-04-05T03:02:46.325 INFO:tasks.workunit.client.1.smithi203.stdout:1/625: mknod d7/d14/d2b/d58/cc6 0 2024-04-05T03:02:46.326 INFO:tasks.workunit.client.1.smithi203.stdout:6/664: creat d0/da/d18/d38/d2a/fce x:0 0 0 2024-04-05T03:02:46.328 INFO:tasks.workunit.client.1.smithi203.stdout:9/602: dwrite d13/d1b/d6b/d9e/d59/f85 [0,4194304] 0 2024-04-05T03:02:46.329 INFO:tasks.workunit.client.1.smithi203.stdout:6/665: readlink d0/da/d54/d4c/da1/db7/d71/l72 0 2024-04-05T03:02:46.338 INFO:tasks.workunit.client.0.smithi153.stdout:3/266: dread d1d/d27/f64 [0,4194304] 0 2024-04-05T03:02:46.339 INFO:tasks.workunit.client.1.smithi203.stdout:1/626: dwrite d7/d24/d33/d48/d4b/d51/fb2 [0,4194304] 0 2024-04-05T03:02:46.339 INFO:tasks.workunit.client.1.smithi203.stdout:9/603: rename f10 to d13/d1b/d6b/d9e/d59/fc1 0 2024-04-05T03:02:46.339 INFO:tasks.workunit.client.1.smithi203.stdout:6/666: dwrite d0/da/d18/d38/d2a/fce [0,4194304] 0 2024-04-05T03:02:46.339 INFO:tasks.workunit.client.1.smithi203.stdout:6/667: truncate d0/da/f94 752770 0 2024-04-05T03:02:46.346 INFO:tasks.workunit.client.0.smithi153.stdout:3/267: dwrite d1d/d26/d3c/f5b [0,4194304] 0 2024-04-05T03:02:46.349 INFO:tasks.workunit.client.1.smithi203.stdout:6/668: dwrite d0/f9c [0,4194304] 0 2024-04-05T03:02:46.353 INFO:tasks.workunit.client.1.smithi203.stdout:1/627: dread d7/d24/f3e [0,4194304] 0 2024-04-05T03:02:46.354 INFO:tasks.workunit.client.0.smithi153.stdout:3/268: link d1d/d26/d3c/f5b d1d/d32/d5f/f67 0 2024-04-05T03:02:46.355 INFO:tasks.workunit.client.1.smithi203.stdout:9/604: dwrite d13/d1b/d26/d31/f3e [0,4194304] 0 2024-04-05T03:02:46.358 INFO:tasks.workunit.client.1.smithi203.stdout:6/669: unlink d0/da/d18/d38/d2a/d2c/f3a 0 2024-04-05T03:02:46.363 INFO:tasks.workunit.client.1.smithi203.stdout:1/628: dread d7/d24/f6e [0,4194304] 0 2024-04-05T03:02:46.365 INFO:tasks.workunit.client.1.smithi203.stdout:1/629: write d7/d14/d2b/f95 [206955,104553] 0 2024-04-05T03:02:46.369 INFO:tasks.workunit.client.1.smithi203.stdout:1/630: mknod d7/d24/d33/d48/d4b/cc7 0 2024-04-05T03:02:46.369 INFO:tasks.workunit.client.1.smithi203.stdout:1/631: readlink d7/d24/l32 0 2024-04-05T03:02:46.372 INFO:tasks.workunit.client.1.smithi203.stdout:1/632: mknod d7/d14/d9d/dbf/cc8 0 2024-04-05T03:02:46.375 INFO:tasks.workunit.client.1.smithi203.stdout:1/633: link d7/d24/d33/d48/d4b/l6f d7/d14/d83/lc9 0 2024-04-05T03:02:46.377 INFO:tasks.workunit.client.1.smithi203.stdout:1/634: chown d7/d24/l92 4257525 1 2024-04-05T03:02:46.379 INFO:tasks.workunit.client.1.smithi203.stdout:1/635: unlink d7/d14/d2b/fae 0 2024-04-05T03:02:46.381 INFO:tasks.workunit.client.1.smithi203.stdout:1/636: mknod d7/d24/d33/cca 0 2024-04-05T03:02:46.384 INFO:tasks.workunit.client.1.smithi203.stdout:1/637: read d7/d24/d33/d48/d4b/d6b/f98 [634267,46284] 0 2024-04-05T03:02:46.388 INFO:tasks.workunit.client.1.smithi203.stdout:9/605: dwrite d13/d1b/d26/d31/f41 [0,4194304] 0 2024-04-05T03:02:46.388 INFO:tasks.workunit.client.1.smithi203.stdout:6/670: dwrite d0/da/d54/d53/f59 [0,4194304] 0 2024-04-05T03:02:46.391 INFO:tasks.workunit.client.1.smithi203.stdout:6/671: chown d0/da/d54/d4c/da1/db7/l76 1750779 1 2024-04-05T03:02:46.392 INFO:tasks.workunit.client.1.smithi203.stdout:1/638: dread d7/d24/d33/f45 [0,4194304] 0 2024-04-05T03:02:46.393 INFO:tasks.workunit.client.1.smithi203.stdout:9/606: symlink d13/d1b/d33/d64/lc2 0 2024-04-05T03:02:46.393 INFO:tasks.workunit.client.1.smithi203.stdout:1/639: dread - d7/d24/fc5 zero size 2024-04-05T03:02:46.394 INFO:tasks.workunit.client.1.smithi203.stdout:6/672: truncate d0/da/d18/d38/fbe 117153 0 2024-04-05T03:02:46.397 INFO:tasks.workunit.client.1.smithi203.stdout:9/607: creat d13/d1b/d33/d42/fc3 x:0 0 0 2024-04-05T03:02:46.399 INFO:tasks.workunit.client.1.smithi203.stdout:1/640: mkdir d7/d14/d2b/d58/d86/d91/dcb 0 2024-04-05T03:02:46.400 INFO:tasks.workunit.client.1.smithi203.stdout:1/641: readlink d7/l55 0 2024-04-05T03:02:46.400 INFO:tasks.workunit.client.1.smithi203.stdout:6/673: symlink d0/da/d18/lcf 0 2024-04-05T03:02:46.402 INFO:tasks.workunit.client.1.smithi203.stdout:9/608: creat d13/d1b/d6b/d9e/d59/dbc/fc4 x:0 0 0 2024-04-05T03:02:46.404 INFO:tasks.workunit.client.1.smithi203.stdout:1/642: mknod d7/d14/d2b/d58/ccc 0 2024-04-05T03:02:46.406 INFO:tasks.workunit.client.1.smithi203.stdout:9/609: unlink d13/d1b/d26/d51/f97 0 2024-04-05T03:02:46.406 INFO:tasks.workunit.client.1.smithi203.stdout:9/610: dread - d13/d1b/d33/d64/f79 zero size 2024-04-05T03:02:46.408 INFO:tasks.workunit.client.1.smithi203.stdout:9/611: write d13/d1b/d6b/d9e/f68 [3253590,58613] 0 2024-04-05T03:02:46.408 INFO:tasks.workunit.client.1.smithi203.stdout:1/643: creat d7/d14/d9d/dbf/fcd x:0 0 0 2024-04-05T03:02:46.408 INFO:tasks.workunit.client.1.smithi203.stdout:6/674: getdents d0/da/d18/d38/d2a/d44/d78/da7 0 2024-04-05T03:02:46.408 INFO:tasks.workunit.client.1.smithi203.stdout:1/644: stat d7/f99 0 2024-04-05T03:02:46.410 INFO:tasks.workunit.client.1.smithi203.stdout:6/675: fdatasync d0/da/d18/d38/f6d 0 2024-04-05T03:02:46.412 INFO:tasks.workunit.client.1.smithi203.stdout:9/612: creat d13/d1b/d6b/d9e/d59/dbc/fc5 x:0 0 0 2024-04-05T03:02:46.415 INFO:tasks.workunit.client.1.smithi203.stdout:1/645: link d7/d24/f35 d7/d24/d33/d48/d4b/d6b/db9/fce 0 2024-04-05T03:02:46.418 INFO:tasks.workunit.client.1.smithi203.stdout:1/646: chown d7/d24/cb0 953287922 1 2024-04-05T03:02:46.418 INFO:tasks.workunit.client.1.smithi203.stdout:1/647: readlink d7/d14/lab 0 2024-04-05T03:02:46.422 INFO:tasks.workunit.client.1.smithi203.stdout:1/648: rename d7/d14/d2b/c3a to d7/d24/d33/ccf 0 2024-04-05T03:02:46.424 INFO:tasks.workunit.client.1.smithi203.stdout:1/649: unlink d7/d24/d33/f64 0 2024-04-05T03:02:46.425 INFO:tasks.workunit.client.1.smithi203.stdout:9/613: write d13/d1b/d26/f54 [3596642,99872] 0 2024-04-05T03:02:46.425 INFO:tasks.workunit.client.1.smithi203.stdout:6/676: truncate d0/da/d54/d53/f63 2452418 0 2024-04-05T03:02:46.427 INFO:tasks.workunit.client.1.smithi203.stdout:1/650: symlink d7/d79/ld0 0 2024-04-05T03:02:46.427 INFO:tasks.workunit.client.1.smithi203.stdout:6/677: symlink d0/da/d18/d38/d2a/d44/d78/ld0 0 2024-04-05T03:02:46.429 INFO:tasks.workunit.client.1.smithi203.stdout:9/614: link d13/d1b/d33/d64/f6d d13/d1b/d33/d64/fc6 0 2024-04-05T03:02:46.430 INFO:tasks.workunit.client.1.smithi203.stdout:6/678: unlink d0/f37 0 2024-04-05T03:02:46.432 INFO:tasks.workunit.client.1.smithi203.stdout:9/615: symlink d13/d1b/d26/lc7 0 2024-04-05T03:02:46.433 INFO:tasks.workunit.client.1.smithi203.stdout:1/651: getdents d7/d24/d33/d48/d4b/d51 0 2024-04-05T03:02:46.433 INFO:tasks.workunit.client.1.smithi203.stdout:9/616: mkdir d13/dc8 0 2024-04-05T03:02:46.436 INFO:tasks.workunit.client.1.smithi203.stdout:9/617: symlink d13/d1b/d26/d51/lc9 0 2024-04-05T03:02:46.437 INFO:tasks.workunit.client.1.smithi203.stdout:6/679: dwrite d0/da/d18/f9f [0,4194304] 0 2024-04-05T03:02:46.440 INFO:tasks.workunit.client.1.smithi203.stdout:9/618: read d13/d1b/d26/d51/f91 [1204463,89323] 0 2024-04-05T03:02:46.441 INFO:tasks.workunit.client.1.smithi203.stdout:6/680: rename d0 to d0/da/d18/d38/d2a/d44/d78/da7/dd1 22 2024-04-05T03:02:46.441 INFO:tasks.workunit.client.1.smithi203.stdout:1/652: dwrite d7/d14/d83/fa4 [0,4194304] 0 2024-04-05T03:02:46.442 INFO:tasks.workunit.client.0.smithi153.stdout:0/344: sync 2024-04-05T03:02:46.448 INFO:tasks.workunit.client.1.smithi203.stdout:1/653: dwrite d7/d14/f76 [0,4194304] 0 2024-04-05T03:02:46.455 INFO:tasks.workunit.client.0.smithi153.stdout:0/345: dwrite d9/df/d1d/f24 [0,4194304] 0 2024-04-05T03:02:46.457 INFO:tasks.workunit.client.0.smithi153.stdout:0/346: write d9/d14/f40 [92715,80934] 0 2024-04-05T03:02:46.457 INFO:tasks.workunit.client.1.smithi203.stdout:9/619: creat d13/d1b/d33/fca x:0 0 0 2024-04-05T03:02:46.463 INFO:tasks.workunit.client.1.smithi203.stdout:9/620: mkdir d13/d1b/d26/d51/dcb 0 2024-04-05T03:02:46.463 INFO:tasks.workunit.client.1.smithi203.stdout:9/621: stat f2 0 2024-04-05T03:02:46.464 INFO:tasks.workunit.client.1.smithi203.stdout:6/681: getdents d0/da/d18/d38 0 2024-04-05T03:02:46.465 INFO:tasks.workunit.client.0.smithi153.stdout:0/347: read d9/df/d1d/d61/f1b [690825,113606] 0 2024-04-05T03:02:46.466 INFO:tasks.workunit.client.0.smithi153.stdout:0/348: fsync d9/df/d1d/f3b 0 2024-04-05T03:02:46.466 INFO:tasks.workunit.client.1.smithi203.stdout:9/622: creat d13/d5c/fcc x:0 0 0 2024-04-05T03:02:46.468 INFO:tasks.workunit.client.1.smithi203.stdout:6/682: getdents d0/da/d18/d92 0 2024-04-05T03:02:46.468 INFO:tasks.workunit.client.1.smithi203.stdout:6/683: fdatasync d0/da/d18/d38/fbe 0 2024-04-05T03:02:46.470 INFO:tasks.workunit.client.1.smithi203.stdout:6/684: dread d0/da/f96 [0,4194304] 0 2024-04-05T03:02:46.474 INFO:tasks.workunit.client.1.smithi203.stdout:6/685: link d0/da/d54/d4c/l55 d0/da/d18/d38/d2a/d44/ld2 0 2024-04-05T03:02:46.479 INFO:tasks.workunit.client.1.smithi203.stdout:6/686: dwrite d0/da/d18/d38/d2a/d2c/f46 [0,4194304] 0 2024-04-05T03:02:46.484 INFO:tasks.workunit.client.1.smithi203.stdout:9/623: dwrite d13/d1b/f47 [0,4194304] 0 2024-04-05T03:02:46.489 INFO:tasks.workunit.client.0.smithi153.stdout:0/349: dread d9/df/f39 [0,4194304] 0 2024-04-05T03:02:46.489 INFO:tasks.workunit.client.1.smithi203.stdout:9/624: creat d13/d5c/d9d/fcd x:0 0 0 2024-04-05T03:02:46.501 INFO:tasks.workunit.client.1.smithi203.stdout:6/687: dread d0/da/f13 [0,4194304] 0 2024-04-05T03:02:46.503 INFO:tasks.workunit.client.1.smithi203.stdout:6/688: creat d0/da/d54/d82/fd3 x:0 0 0 2024-04-05T03:02:46.506 INFO:tasks.workunit.client.0.smithi153.stdout:0/350: dread d9/db/f11 [0,4194304] 0 2024-04-05T03:02:46.508 INFO:tasks.workunit.client.1.smithi203.stdout:6/689: dwrite d0/da/d54/f4b [0,4194304] 0 2024-04-05T03:02:46.509 INFO:tasks.workunit.client.1.smithi203.stdout:9/625: truncate d13/d1b/d33/d63/f6e 1735509 0 2024-04-05T03:02:46.512 INFO:tasks.workunit.client.1.smithi203.stdout:9/626: truncate d13/d1b/d26/d51/f91 3616517 0 2024-04-05T03:02:46.513 INFO:tasks.workunit.client.0.smithi153.stdout:0/351: dwrite d9/db/d42/f60 [0,4194304] 0 2024-04-05T03:02:46.513 INFO:tasks.workunit.client.1.smithi203.stdout:6/690: truncate d0/da/d54/d53/f63 814242 0 2024-04-05T03:02:46.515 INFO:tasks.workunit.client.1.smithi203.stdout:6/691: chown d0/da/l75 607 1 2024-04-05T03:02:46.515 INFO:tasks.workunit.client.1.smithi203.stdout:9/627: symlink d13/d1b/d6b/d7a/lce 0 2024-04-05T03:02:46.519 INFO:tasks.workunit.client.1.smithi203.stdout:6/692: write d0/da/d18/d38/d2a/d44/d78/da7/fc5 [883789,83991] 0 2024-04-05T03:02:46.519 INFO:tasks.workunit.client.0.smithi153.stdout:0/352: getdents d9/db/d42/d43 0 2024-04-05T03:02:46.522 INFO:tasks.workunit.client.1.smithi203.stdout:6/693: write d0/da/d54/d53/f59 [5093101,86049] 0 2024-04-05T03:02:46.524 INFO:tasks.workunit.client.1.smithi203.stdout:9/628: dwrite d13/d1b/fad [0,4194304] 0 2024-04-05T03:02:46.526 INFO:tasks.workunit.client.0.smithi153.stdout:0/353: dwrite d9/d14/f3f [0,4194304] 0 2024-04-05T03:02:46.527 INFO:tasks.workunit.client.1.smithi203.stdout:9/629: write d13/d1b/d33/d42/f58 [792532,117896] 0 2024-04-05T03:02:46.527 INFO:tasks.workunit.client.0.smithi153.stdout:0/354: chown d9/df/d1d 1 1 2024-04-05T03:02:46.529 INFO:tasks.workunit.client.1.smithi203.stdout:6/694: link d0/da/d18/d38/d86/c5c d0/da/d54/d4c/cd4 0 2024-04-05T03:02:46.530 INFO:tasks.workunit.client.1.smithi203.stdout:9/630: mknod d13/d1b/d33/d64/ccf 0 2024-04-05T03:02:46.531 INFO:tasks.workunit.client.1.smithi203.stdout:6/695: write d0/da/d18/d38/d2a/fce [3416856,109878] 0 2024-04-05T03:02:46.532 INFO:tasks.workunit.client.1.smithi203.stdout:6/696: dread - d0/f9d zero size 2024-04-05T03:02:46.533 INFO:tasks.workunit.client.0.smithi153.stdout:0/355: rename d9/df/d1e/l26 to d9/df/d1d/l63 0 2024-04-05T03:02:46.533 INFO:tasks.workunit.client.1.smithi203.stdout:9/631: write d13/f4a [54129,64478] 0 2024-04-05T03:02:46.534 INFO:tasks.workunit.client.1.smithi203.stdout:9/632: fsync d13/d1b/f47 0 2024-04-05T03:02:46.539 INFO:tasks.workunit.client.1.smithi203.stdout:9/633: creat d13/d1b/d26/d51/dcb/fd0 x:0 0 0 2024-04-05T03:02:46.556 INFO:tasks.workunit.client.0.smithi153.stdout:9/205: sync 2024-04-05T03:02:46.556 INFO:tasks.workunit.client.0.smithi153.stdout:7/255: sync 2024-04-05T03:02:46.556 INFO:tasks.workunit.client.0.smithi153.stdout:1/226: sync 2024-04-05T03:02:46.557 INFO:tasks.workunit.client.0.smithi153.stdout:9/206: dread - d1/db/f4a zero size 2024-04-05T03:02:46.557 INFO:tasks.workunit.client.0.smithi153.stdout:9/207: dread - d1/f2e zero size 2024-04-05T03:02:46.558 INFO:tasks.workunit.client.0.smithi153.stdout:1/227: mknod d5/dd/d1b/c44 0 2024-04-05T03:02:46.558 INFO:tasks.workunit.client.1.smithi203.stdout:8/666: sync 2024-04-05T03:02:46.559 INFO:tasks.workunit.client.1.smithi203.stdout:4/470: sync 2024-04-05T03:02:46.559 INFO:tasks.workunit.client.1.smithi203.stdout:2/808: sync 2024-04-05T03:02:46.559 INFO:tasks.workunit.client.1.smithi203.stdout:0/636: sync 2024-04-05T03:02:46.559 INFO:tasks.workunit.client.1.smithi203.stdout:3/513: sync 2024-04-05T03:02:46.559 INFO:tasks.workunit.client.0.smithi153.stdout:9/208: chown d1/db/c13 462336714 1 2024-04-05T03:02:46.560 INFO:tasks.workunit.client.1.smithi203.stdout:2/809: fdatasync d5/d10/d35/d73/f95 0 2024-04-05T03:02:46.560 INFO:tasks.workunit.client.0.smithi153.stdout:1/228: fdatasync d5/dd/d21/d1e/f43 0 2024-04-05T03:02:46.562 INFO:tasks.workunit.client.0.smithi153.stdout:7/256: dwrite d2/da/d11/f52 [0,4194304] 0 2024-04-05T03:02:46.563 INFO:tasks.workunit.client.0.smithi153.stdout:9/209: mknod d1/d10/d36/c4f 0 2024-04-05T03:02:46.565 INFO:tasks.workunit.client.1.smithi203.stdout:4/471: dwrite da/d29/d2e/f48 [0,4194304] 0 2024-04-05T03:02:46.566 INFO:tasks.workunit.client.1.smithi203.stdout:4/472: dread - da/d3c/f49 zero size 2024-04-05T03:02:46.567 INFO:tasks.workunit.client.0.smithi153.stdout:1/229: dread d5/dd/d21/d18/d16/f2d [0,4194304] 0 2024-04-05T03:02:46.567 INFO:tasks.workunit.client.0.smithi153.stdout:7/257: dread d2/da/d11/d49/f20 [0,4194304] 0 2024-04-05T03:02:46.567 INFO:tasks.workunit.client.0.smithi153.stdout:7/258: chown d2/d51 1 1 2024-04-05T03:02:46.567 INFO:tasks.workunit.client.1.smithi203.stdout:4/473: chown da/d29/d2e/d57/f66 1667053976 1 2024-04-05T03:02:46.571 INFO:tasks.workunit.client.0.smithi153.stdout:1/230: mkdir d5/dd/d21/d18/d16/d45 0 2024-04-05T03:02:46.573 INFO:tasks.workunit.client.0.smithi153.stdout:7/259: rename d2/da/d42/d57 to d2/d51/d59 0 2024-04-05T03:02:46.576 INFO:tasks.workunit.client.0.smithi153.stdout:7/260: getdents d2/da/d11/d49/d54 0 2024-04-05T03:02:46.577 INFO:tasks.workunit.client.0.smithi153.stdout:9/210: getdents d1/d10/d16 0 2024-04-05T03:02:46.578 INFO:tasks.workunit.client.0.smithi153.stdout:9/211: readlink d1/d1d/l42 0 2024-04-05T03:02:46.580 INFO:tasks.workunit.client.0.smithi153.stdout:7/261: creat d2/f5a x:0 0 0 2024-04-05T03:02:46.581 INFO:tasks.workunit.client.1.smithi203.stdout:3/514: mkdir d0/dc/d38/d92 0 2024-04-05T03:02:46.582 INFO:tasks.workunit.client.0.smithi153.stdout:1/231: dwrite d5/dd/d21/d18/f1a [0,4194304] 0 2024-04-05T03:02:46.584 INFO:tasks.workunit.client.1.smithi203.stdout:8/667: fdatasync d1/d4b/d5f/d74/fb5 0 2024-04-05T03:02:46.584 INFO:tasks.workunit.client.1.smithi203.stdout:8/668: read - d1/d6/f93 zero size 2024-04-05T03:02:46.585 INFO:tasks.workunit.client.0.smithi153.stdout:1/232: write d5/dd/d21/d18/d16/d33/f3c [456381,63144] 0 2024-04-05T03:02:46.588 INFO:tasks.workunit.client.1.smithi203.stdout:8/669: dread d1/d12/d2f/d1f/fae [0,4194304] 0 2024-04-05T03:02:46.588 INFO:tasks.workunit.client.1.smithi203.stdout:0/637: mkdir dd/d17/d1d/d22/dba 0 2024-04-05T03:02:46.589 INFO:tasks.workunit.client.0.smithi153.stdout:7/262: dwrite d2/da/d11/f18 [0,4194304] 0 2024-04-05T03:02:46.590 INFO:tasks.workunit.client.0.smithi153.stdout:1/233: fdatasync d5/dd/d21/d18/d16/f22 0 2024-04-05T03:02:46.590 INFO:tasks.workunit.client.1.smithi203.stdout:3/515: creat d0/dc/d38/d5d/d66/f93 x:0 0 0 2024-04-05T03:02:46.591 INFO:tasks.workunit.client.0.smithi153.stdout:7/263: dread - d2/da/d11/f3f zero size 2024-04-05T03:02:46.591 INFO:tasks.workunit.client.1.smithi203.stdout:8/670: write d1/d6/d2e/f30 [4160470,8733] 0 2024-04-05T03:02:46.592 INFO:tasks.workunit.client.1.smithi203.stdout:8/671: dread - d1/d12/d2f/d5b/fb3 zero size 2024-04-05T03:02:46.593 INFO:tasks.workunit.client.0.smithi153.stdout:7/264: stat d2/da 0 2024-04-05T03:02:46.593 INFO:tasks.workunit.client.0.smithi153.stdout:7/265: stat d2/da/d11/d49/c17 0 2024-04-05T03:02:46.595 INFO:tasks.workunit.client.1.smithi203.stdout:3/516: creat d0/dc/d1a/d76/f94 x:0 0 0 2024-04-05T03:02:46.596 INFO:tasks.workunit.client.1.smithi203.stdout:3/517: dread - d0/dc/d1a/d1d/f72 zero size 2024-04-05T03:02:46.598 INFO:tasks.workunit.client.1.smithi203.stdout:9/634: rename d13/d1b/d26 to d13/d5c/dd1 0 2024-04-05T03:02:46.601 INFO:tasks.workunit.client.1.smithi203.stdout:0/638: mkdir dd/d17/d1d/d22/d45/d51/dbb 0 2024-04-05T03:02:46.601 INFO:tasks.workunit.client.0.smithi153.stdout:7/266: creat d2/da/d53/f5b x:0 0 0 2024-04-05T03:02:46.602 INFO:tasks.workunit.client.1.smithi203.stdout:3/518: readlink d0/dc/d1a/d1f/l6f 0 2024-04-05T03:02:46.605 INFO:tasks.workunit.client.0.smithi153.stdout:1/234: rmdir d5/dd/d21/d18/d16/d3a 0 2024-04-05T03:02:46.606 INFO:tasks.workunit.client.0.smithi153.stdout:7/267: rename d2/da/d11/d39/c4f to d2/da/d11/d49/d55/c5c 0 2024-04-05T03:02:46.607 INFO:tasks.workunit.client.1.smithi203.stdout:2/810: rename d5/d10/d35/d2d/d7c/d72/ffc to d5/d10/d35/d36/db6/d87/f117 0 2024-04-05T03:02:46.607 INFO:tasks.workunit.client.0.smithi153.stdout:1/235: fdatasync d5/dd/d1b/f2b 0 2024-04-05T03:02:46.608 INFO:tasks.workunit.client.1.smithi203.stdout:4/474: write da/f5a [148140,105763] 0 2024-04-05T03:02:46.609 INFO:tasks.workunit.client.1.smithi203.stdout:9/635: mkdir d13/dc8/dd2 0 2024-04-05T03:02:46.609 INFO:tasks.workunit.client.0.smithi153.stdout:7/268: read d2/da/d11/d49/f3e [334341,62402] 0 2024-04-05T03:02:46.609 INFO:tasks.workunit.client.0.smithi153.stdout:7/269: readlink l1 0 2024-04-05T03:02:46.610 INFO:tasks.workunit.client.0.smithi153.stdout:9/212: dread d1/f1f [4194304,4194304] 0 2024-04-05T03:02:46.610 INFO:tasks.workunit.client.0.smithi153.stdout:1/236: unlink d5/dd/c34 0 2024-04-05T03:02:46.611 INFO:tasks.workunit.client.0.smithi153.stdout:7/270: fdatasync d2/da/d11/f2b 0 2024-04-05T03:02:46.612 INFO:tasks.workunit.client.1.smithi203.stdout:0/639: rename dd/d17/fa5 to dd/d93/fbc 0 2024-04-05T03:02:46.612 INFO:tasks.workunit.client.0.smithi153.stdout:7/271: dread - d2/da/d3a/d41/d31/f37 zero size 2024-04-05T03:02:46.613 INFO:tasks.workunit.client.1.smithi203.stdout:9/636: dread d13/d1b/fad [0,4194304] 0 2024-04-05T03:02:46.613 INFO:tasks.workunit.client.0.smithi153.stdout:9/213: read d1/d9/d1b/f1c [2068386,101465] 0 2024-04-05T03:02:46.614 INFO:tasks.workunit.client.0.smithi153.stdout:1/237: rmdir d5/d8/d20 39 2024-04-05T03:02:46.614 INFO:tasks.workunit.client.1.smithi203.stdout:9/637: write d13/d5c/d9d/fcd [33707,77462] 0 2024-04-05T03:02:46.615 INFO:tasks.workunit.client.0.smithi153.stdout:9/214: truncate d1/f4 5930037 0 2024-04-05T03:02:46.616 INFO:tasks.workunit.client.0.smithi153.stdout:9/215: write d1/d10/d16/d22/f3e [51051,58848] 0 2024-04-05T03:02:46.617 INFO:tasks.workunit.client.1.smithi203.stdout:3/519: rename d0/d21/d81/l44 to d0/d21/d83/l95 0 2024-04-05T03:02:46.618 INFO:tasks.workunit.client.1.smithi203.stdout:3/520: write d0/d41/f8e [350632,123707] 0 2024-04-05T03:02:46.618 INFO:tasks.workunit.client.0.smithi153.stdout:7/272: getdents d2/da/d47 0 2024-04-05T03:02:46.620 INFO:tasks.workunit.client.1.smithi203.stdout:0/640: mkdir dd/dbd 0 2024-04-05T03:02:46.621 INFO:tasks.workunit.client.0.smithi153.stdout:7/273: readlink d2/da/l23 0 2024-04-05T03:02:46.621 INFO:tasks.workunit.client.1.smithi203.stdout:0/641: chown dd/d17/l1a 0 1 2024-04-05T03:02:46.622 INFO:tasks.workunit.client.1.smithi203.stdout:9/638: creat d13/d1b/d6b/d9e/d77/fd3 x:0 0 0 2024-04-05T03:02:46.624 INFO:tasks.workunit.client.0.smithi153.stdout:1/238: dwrite d5/dd/d21/d18/f37 [0,4194304] 0 2024-04-05T03:02:46.625 INFO:tasks.workunit.client.0.smithi153.stdout:7/274: mkdir d2/da/d47/d5d 0 2024-04-05T03:02:46.626 INFO:tasks.workunit.client.1.smithi203.stdout:3/521: rename d0/dc/d1a/d1d/d43 to d0/dc/d1a/d8d/d96 0 2024-04-05T03:02:46.627 INFO:tasks.workunit.client.1.smithi203.stdout:3/522: dread - d0/d21/d81/f7b zero size 2024-04-05T03:02:46.627 INFO:tasks.workunit.client.1.smithi203.stdout:4/475: getdents da/d29/d2e/d57 0 2024-04-05T03:02:46.628 INFO:tasks.workunit.client.0.smithi153.stdout:9/216: write d1/d9/d1b/f3f [4804458,110006] 0 2024-04-05T03:02:46.629 INFO:tasks.workunit.client.0.smithi153.stdout:1/239: symlink d5/d8/d20/d2f/d40/l46 0 2024-04-05T03:02:46.632 INFO:tasks.workunit.client.0.smithi153.stdout:9/217: symlink d1/d1d/l50 0 2024-04-05T03:02:46.633 INFO:tasks.workunit.client.0.smithi153.stdout:1/240: fdatasync d5/dd/d21/d18/d16/f2d 0 2024-04-05T03:02:46.634 INFO:tasks.workunit.client.0.smithi153.stdout:7/275: getdents d2/da/d11/d49/d55 0 2024-04-05T03:02:46.636 INFO:tasks.workunit.client.1.smithi203.stdout:0/642: mkdir dd/d17/d1d/d22/d45/dbe 0 2024-04-05T03:02:46.638 INFO:tasks.workunit.client.0.smithi153.stdout:7/276: mknod d2/da/d47/d5d/c5e 0 2024-04-05T03:02:46.638 INFO:tasks.workunit.client.0.smithi153.stdout:7/277: stat d2/da/d11/d49/d54 0 2024-04-05T03:02:46.639 INFO:tasks.workunit.client.0.smithi153.stdout:7/278: fsync d2/da/d11/f2d 0 2024-04-05T03:02:46.639 INFO:tasks.workunit.client.0.smithi153.stdout:1/241: link d5/dd/d1b/l29 d5/dd/l47 0 2024-04-05T03:02:46.640 INFO:tasks.workunit.client.1.smithi203.stdout:2/811: dwrite d5/d10/d35/d36/db6/f27 [4194304,4194304] 0 2024-04-05T03:02:46.642 INFO:tasks.workunit.client.0.smithi153.stdout:7/279: rmdir d2/da 39 2024-04-05T03:02:46.645 INFO:tasks.workunit.client.0.smithi153.stdout:7/280: dread - d2/da/d11/f2b zero size 2024-04-05T03:02:46.648 INFO:tasks.workunit.client.0.smithi153.stdout:1/242: dwrite d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:46.651 INFO:tasks.workunit.client.0.smithi153.stdout:7/281: creat d2/da/d3a/d41/d31/f5f x:0 0 0 2024-04-05T03:02:46.654 INFO:tasks.workunit.client.0.smithi153.stdout:7/282: write d2/da/d3a/d41/d31/f45 [484932,12196] 0 2024-04-05T03:02:46.657 INFO:tasks.workunit.client.0.smithi153.stdout:9/218: dread d1/d9/d1b/f30 [4194304,4194304] 0 2024-04-05T03:02:46.658 INFO:tasks.workunit.client.0.smithi153.stdout:9/219: write d1/db/f4a [967849,46527] 0 2024-04-05T03:02:46.660 INFO:tasks.workunit.client.1.smithi203.stdout:9/639: creat d13/d5c/dd1/d51/da1/dba/fd4 x:0 0 0 2024-04-05T03:02:46.661 INFO:tasks.workunit.client.0.smithi153.stdout:9/220: unlink d1/d10/d16/c47 0 2024-04-05T03:02:46.662 INFO:tasks.workunit.client.1.smithi203.stdout:4/476: unlink da/d29/f40 0 2024-04-05T03:02:46.662 INFO:tasks.workunit.client.1.smithi203.stdout:9/640: chown d13/d5c/d9d/cbb 274432 1 2024-04-05T03:02:46.663 INFO:tasks.workunit.client.1.smithi203.stdout:0/643: fdatasync dd/d17/d1d/d26/d28/f2b 0 2024-04-05T03:02:46.663 INFO:tasks.workunit.client.1.smithi203.stdout:2/812: read - d5/fb3 zero size 2024-04-05T03:02:46.664 INFO:tasks.workunit.client.1.smithi203.stdout:4/477: write da/d3c/d4c/f79 [512480,33911] 0 2024-04-05T03:02:46.665 INFO:tasks.workunit.client.1.smithi203.stdout:9/641: mknod d13/d5c/dd1/d51/dcb/cd5 0 2024-04-05T03:02:46.666 INFO:tasks.workunit.client.0.smithi153.stdout:9/221: creat d1/d10/f51 x:0 0 0 2024-04-05T03:02:46.666 INFO:tasks.workunit.client.1.smithi203.stdout:3/523: dwrite d0/dc/d34/f6e [0,4194304] 0 2024-04-05T03:02:46.667 INFO:tasks.workunit.client.0.smithi153.stdout:1/243: write d5/dd/d21/d18/f1c [514619,125774] 0 2024-04-05T03:02:46.668 INFO:tasks.workunit.client.0.smithi153.stdout:7/283: rmdir d2/da/d3a/d41/d31 39 2024-04-05T03:02:46.671 INFO:tasks.workunit.client.0.smithi153.stdout:7/284: chown d2/da/d11/f18 106485779 1 2024-04-05T03:02:46.672 INFO:tasks.workunit.client.1.smithi203.stdout:2/813: dwrite d5/d66/f10a [0,4194304] 0 2024-04-05T03:02:46.677 INFO:tasks.workunit.client.0.smithi153.stdout:9/222: dwrite d1/db/f45 [4194304,4194304] 0 2024-04-05T03:02:46.677 INFO:tasks.workunit.client.0.smithi153.stdout:7/285: unlink d2/da/d11/d49/f3d 0 2024-04-05T03:02:46.678 INFO:tasks.workunit.client.1.smithi203.stdout:4/478: creat da/f93 x:0 0 0 2024-04-05T03:02:46.679 INFO:tasks.workunit.client.1.smithi203.stdout:4/479: chown da/d29/f42 32519796 1 2024-04-05T03:02:46.679 INFO:tasks.workunit.client.0.smithi153.stdout:1/244: dwrite d5/d8/d20/f38 [0,4194304] 0 2024-04-05T03:02:46.680 INFO:tasks.workunit.client.1.smithi203.stdout:4/480: stat da/d29/d6a/d80/c85 0 2024-04-05T03:02:46.680 INFO:tasks.workunit.client.1.smithi203.stdout:4/481: stat da/f93 0 2024-04-05T03:02:46.681 INFO:tasks.workunit.client.1.smithi203.stdout:3/524: link d0/f52 d0/dc/d38/d5d/d66/d80/f97 0 2024-04-05T03:02:46.683 INFO:tasks.workunit.client.1.smithi203.stdout:3/525: write d0/dc/d1a/d8d/d96/f5e [1529837,108549] 0 2024-04-05T03:02:46.684 INFO:tasks.workunit.client.0.smithi153.stdout:1/245: rmdir d5/dd/d21 39 2024-04-05T03:02:46.685 INFO:tasks.workunit.client.1.smithi203.stdout:3/526: fsync d0/dc/d1a/d8d/d96/f51 0 2024-04-05T03:02:46.685 INFO:tasks.workunit.client.1.smithi203.stdout:3/527: stat d0/dc/d38/f3b 0 2024-04-05T03:02:46.687 INFO:tasks.workunit.client.0.smithi153.stdout:9/223: write d1/d9/d1b/f28 [4465583,34905] 0 2024-04-05T03:02:46.689 INFO:tasks.workunit.client.0.smithi153.stdout:1/246: mkdir d5/dd/d21/d18/d16/d48 0 2024-04-05T03:02:46.689 INFO:tasks.workunit.client.0.smithi153.stdout:9/224: write d1/d9/d1b/f1c [5310122,122863] 0 2024-04-05T03:02:46.690 INFO:tasks.workunit.client.1.smithi203.stdout:3/528: creat d0/dc/d1a/d1d/d53/f98 x:0 0 0 2024-04-05T03:02:46.693 INFO:tasks.workunit.client.0.smithi153.stdout:9/225: chown d1/d10/d16/d22/c29 124731765 1 2024-04-05T03:02:46.693 INFO:tasks.workunit.client.0.smithi153.stdout:9/226: truncate d1/d10/d16/f23 1604576 0 2024-04-05T03:02:46.694 INFO:tasks.workunit.client.1.smithi203.stdout:4/482: rename da/c19 to da/d3c/d4c/c94 0 2024-04-05T03:02:46.694 INFO:tasks.workunit.client.1.smithi203.stdout:4/483: fdatasync da/f93 0 2024-04-05T03:02:46.694 INFO:tasks.workunit.client.1.smithi203.stdout:4/484: fsync da/d29/d2e/d53/f68 0 2024-04-05T03:02:46.696 INFO:tasks.workunit.client.1.smithi203.stdout:1/654: sync 2024-04-05T03:02:46.697 INFO:tasks.workunit.client.1.smithi203.stdout:7/807: sync 2024-04-05T03:02:46.700 INFO:tasks.workunit.client.1.smithi203.stdout:4/485: dwrite da/d29/d2e/f48 [0,4194304] 0 2024-04-05T03:02:46.701 INFO:tasks.workunit.client.1.smithi203.stdout:5/650: sync 2024-04-05T03:02:46.703 INFO:tasks.workunit.client.1.smithi203.stdout:7/808: mknod d8/d32/cfa 0 2024-04-05T03:02:46.704 INFO:tasks.workunit.client.0.smithi153.stdout:1/247: dwrite d5/d8/d20/f38 [4194304,4194304] 0 2024-04-05T03:02:46.704 INFO:tasks.workunit.client.0.smithi153.stdout:7/286: dread d2/da/d42/f4b [0,4194304] 0 2024-04-05T03:02:46.705 INFO:tasks.workunit.client.1.smithi203.stdout:4/486: unlink da/d29/d2e/l7e 0 2024-04-05T03:02:46.706 INFO:tasks.workunit.client.0.smithi153.stdout:7/287: chown d2/da/d3a/d41/f19 21924101 1 2024-04-05T03:02:46.707 INFO:tasks.workunit.client.1.smithi203.stdout:5/651: truncate d3/f7 3446395 0 2024-04-05T03:02:46.708 INFO:tasks.workunit.client.1.smithi203.stdout:4/487: chown da/d29/d2e/d53/d78/f7d 132695315 1 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:3/269: sync 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:4/377: sync 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:6/333: sync 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:2/294: sync 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:5/290: sync 2024-04-05T03:02:46.709 INFO:tasks.workunit.client.0.smithi153.stdout:8/264: sync 2024-04-05T03:02:46.711 INFO:tasks.workunit.client.0.smithi153.stdout:1/248: dread d5/dd/d21/d18/f1a [0,4194304] 0 2024-04-05T03:02:46.712 INFO:tasks.workunit.client.0.smithi153.stdout:2/295: stat d9/da/d19/c30 0 2024-04-05T03:02:46.713 INFO:tasks.workunit.client.1.smithi203.stdout:4/488: symlink da/d3c/d4c/l95 0 2024-04-05T03:02:46.713 INFO:tasks.workunit.client.0.smithi153.stdout:6/334: chown d4/dc/df/d16/d24/d5e/f60 4 1 2024-04-05T03:02:46.714 INFO:tasks.workunit.client.0.smithi153.stdout:7/288: rename d2/da/d3a/d41/f22 to d2/da/d47/f60 0 2024-04-05T03:02:46.715 INFO:tasks.workunit.client.0.smithi153.stdout:7/289: chown d2/da/d11/f24 50282 1 2024-04-05T03:02:46.716 INFO:tasks.workunit.client.0.smithi153.stdout:8/265: dread - d7/d13/d19/d1b/d33/f51 zero size 2024-04-05T03:02:46.717 INFO:tasks.workunit.client.0.smithi153.stdout:5/291: fdatasync d0/d9/f21 0 2024-04-05T03:02:46.719 INFO:tasks.workunit.client.0.smithi153.stdout:2/296: symlink d9/da/d37/l5d 0 2024-04-05T03:02:46.720 INFO:tasks.workunit.client.1.smithi203.stdout:0/644: dwrite dd/d17/d1d/d22/f65 [0,4194304] 0 2024-04-05T03:02:46.720 INFO:tasks.workunit.client.0.smithi153.stdout:2/297: fdatasync d9/da/d19/f1c 0 2024-04-05T03:02:46.720 INFO:tasks.workunit.client.1.smithi203.stdout:5/652: symlink d3/d4/dc/d67/dac/ldc 0 2024-04-05T03:02:46.721 INFO:tasks.workunit.client.0.smithi153.stdout:2/298: readlink d9/da/d19/d2e/l31 0 2024-04-05T03:02:46.723 INFO:tasks.workunit.client.0.smithi153.stdout:4/378: rename da/fc to da/de/d56/d6a/f7c 0 2024-04-05T03:02:46.723 INFO:tasks.workunit.client.0.smithi153.stdout:6/335: truncate d4/dc/df/d16/d24/f32 3200963 0 2024-04-05T03:02:46.726 INFO:tasks.workunit.client.1.smithi203.stdout:1/655: dwrite d7/d14/d2b/f72 [0,4194304] 0 2024-04-05T03:02:46.726 INFO:tasks.workunit.client.1.smithi203.stdout:4/489: read da/f22 [2351505,122089] 0 2024-04-05T03:02:46.727 INFO:tasks.workunit.client.0.smithi153.stdout:4/379: write da/de/d56/f61 [24369,82518] 0 2024-04-05T03:02:46.729 INFO:tasks.workunit.client.1.smithi203.stdout:7/809: dwrite d8/d32/d16/d1a/d3d/d59/de7/d52/f5b [4194304,4194304] 0 2024-04-05T03:02:46.730 INFO:tasks.workunit.client.0.smithi153.stdout:2/299: rename d9/f11 to d9/da/d36/d38/d4b/f5e 0 2024-04-05T03:02:46.730 INFO:tasks.workunit.client.1.smithi203.stdout:3/529: dread d0/dc/f14 [0,4194304] 0 2024-04-05T03:02:46.735 INFO:tasks.workunit.client.0.smithi153.stdout:8/266: mknod d7/d2a/d52/d54/c57 0 2024-04-05T03:02:46.736 INFO:tasks.workunit.client.1.smithi203.stdout:5/653: unlink d3/d4/caf 0 2024-04-05T03:02:46.739 INFO:tasks.workunit.client.0.smithi153.stdout:2/300: dwrite d9/da/d19/d2e/f43 [0,4194304] 0 2024-04-05T03:02:46.739 INFO:tasks.workunit.client.0.smithi153.stdout:5/292: truncate d0/f14 4299648 0 2024-04-05T03:02:46.740 INFO:tasks.workunit.client.0.smithi153.stdout:5/293: fsync d0/d9/d22/f64 0 2024-04-05T03:02:46.743 INFO:tasks.workunit.client.0.smithi153.stdout:2/301: read d9/da/d19/f1f [7094699,79959] 0 2024-04-05T03:02:46.746 INFO:tasks.workunit.client.0.smithi153.stdout:6/336: symlink d4/d38/d4c/l77 0 2024-04-05T03:02:46.747 INFO:tasks.workunit.client.0.smithi153.stdout:1/249: write d5/dd/d1b/f1d [3080429,82687] 0 2024-04-05T03:02:46.747 INFO:tasks.workunit.client.0.smithi153.stdout:4/380: fsync da/de/d56/d6a/d2d/f7a 0 2024-04-05T03:02:46.749 INFO:tasks.workunit.client.1.smithi203.stdout:2/814: dread d5/d10/f1d [0,4194304] 0 2024-04-05T03:02:46.749 INFO:tasks.workunit.client.1.smithi203.stdout:4/490: dwrite da/f1b [4194304,4194304] 0 2024-04-05T03:02:46.752 INFO:tasks.workunit.client.0.smithi153.stdout:2/302: write d9/da/d19/f46 [4448296,94621] 0 2024-04-05T03:02:46.753 INFO:tasks.workunit.client.1.smithi203.stdout:7/810: mknod d8/d32/dc3/cfb 0 2024-04-05T03:02:46.753 INFO:tasks.workunit.client.0.smithi153.stdout:4/381: symlink da/db/d20/d35/d41/l7d 0 2024-04-05T03:02:46.754 INFO:tasks.workunit.client.1.smithi203.stdout:3/530: dwrite d0/dc/d1a/d1d/d53/f73 [0,4194304] 0 2024-04-05T03:02:46.756 INFO:tasks.workunit.client.0.smithi153.stdout:5/294: dwrite d0/d25/d4d/d62/d37/f6a [0,4194304] 0 2024-04-05T03:02:46.758 INFO:tasks.workunit.client.1.smithi203.stdout:0/645: write dd/d17/d1d/d26/d28/f2b [2426468,53586] 0 2024-04-05T03:02:46.760 INFO:tasks.workunit.client.1.smithi203.stdout:4/491: truncate da/d29/d2e/f35 2462906 0 2024-04-05T03:02:46.760 INFO:tasks.workunit.client.1.smithi203.stdout:4/492: chown da/d29/d6a 187829 1 2024-04-05T03:02:46.762 INFO:tasks.workunit.client.1.smithi203.stdout:3/531: chown d0/c55 7622179 1 2024-04-05T03:02:46.763 INFO:tasks.workunit.client.0.smithi153.stdout:2/303: readlink d9/da/l25 0 2024-04-05T03:02:46.764 INFO:tasks.workunit.client.0.smithi153.stdout:3/270: dread d1d/f29 [0,4194304] 0 2024-04-05T03:02:46.765 INFO:tasks.workunit.client.1.smithi203.stdout:7/811: symlink d8/d32/d5d/deb/lfc 0 2024-04-05T03:02:46.768 INFO:tasks.workunit.client.0.smithi153.stdout:8/267: rename d7/d13/d19/d1b/d21/c30 to d7/d13/d19/c58 0 2024-04-05T03:02:46.769 INFO:tasks.workunit.client.1.smithi203.stdout:4/493: mkdir da/d29/d2e/d57/d96 0 2024-04-05T03:02:46.769 INFO:tasks.workunit.client.1.smithi203.stdout:2/815: mkdir d5/d10/d35/d2d/d7c/d29/d67/d118 0 2024-04-05T03:02:46.770 INFO:tasks.workunit.client.0.smithi153.stdout:8/268: write d7/d13/d19/d31/f32 [4242432,115624] 0 2024-04-05T03:02:46.771 INFO:tasks.workunit.client.1.smithi203.stdout:3/532: rename d0/dc/d38/f5f to d0/dc/d1a/d1d/d53/f99 0 2024-04-05T03:02:46.771 INFO:tasks.workunit.client.1.smithi203.stdout:5/654: link d3/d4/dc/f37 d3/d1f/d59/d64/fdd 0 2024-04-05T03:02:46.771 INFO:tasks.workunit.client.1.smithi203.stdout:4/494: fdatasync da/d29/d2e/f7a 0 2024-04-05T03:02:46.772 INFO:tasks.workunit.client.0.smithi153.stdout:6/337: mknod d4/dc/df/d16/c78 0 2024-04-05T03:02:46.775 INFO:tasks.workunit.client.0.smithi153.stdout:4/382: creat da/de/d56/d78/f7e x:0 0 0 2024-04-05T03:02:46.776 INFO:tasks.workunit.client.1.smithi203.stdout:7/812: rename d8/lb1 to d8/d32/d16/d1a/d5a/lfd 0 2024-04-05T03:02:46.776 INFO:tasks.workunit.client.1.smithi203.stdout:5/655: mkdir d3/d4/d9e/dde 0 2024-04-05T03:02:46.777 INFO:tasks.workunit.client.0.smithi153.stdout:1/250: getdents d5/dd/d21/d18/d16/d33 0 2024-04-05T03:02:46.777 INFO:tasks.workunit.client.0.smithi153.stdout:4/383: chown da/db/d20/d3c/c4b 0 1 2024-04-05T03:02:46.777 INFO:tasks.workunit.client.1.smithi203.stdout:4/495: creat da/d29/d2e/d53/f97 x:0 0 0 2024-04-05T03:02:46.778 INFO:tasks.workunit.client.1.smithi203.stdout:7/813: truncate d8/d32/d16/d1a/d5a/fa3 1222500 0 2024-04-05T03:02:46.778 INFO:tasks.workunit.client.1.smithi203.stdout:3/533: truncate d0/dc/d1a/d1d/d53/f75 1266789 0 2024-04-05T03:02:46.779 INFO:tasks.workunit.client.0.smithi153.stdout:8/269: creat d7/d13/d19/d1b/d33/f59 x:0 0 0 2024-04-05T03:02:46.779 INFO:tasks.workunit.client.0.smithi153.stdout:6/338: truncate d4/f15 1249386 0 2024-04-05T03:02:46.780 INFO:tasks.workunit.client.1.smithi203.stdout:2/816: creat d5/d10/d35/f119 x:0 0 0 2024-04-05T03:02:46.780 INFO:tasks.workunit.client.1.smithi203.stdout:4/496: readlink da/l1f 0 2024-04-05T03:02:46.782 INFO:tasks.workunit.client.1.smithi203.stdout:7/814: mknod d8/d6c/cfe 0 2024-04-05T03:02:46.782 INFO:tasks.workunit.client.1.smithi203.stdout:4/497: dread - da/d29/d2e/d53/d78/f7d zero size 2024-04-05T03:02:46.782 INFO:tasks.workunit.client.1.smithi203.stdout:5/656: mknod d3/d1f/d59/cdf 0 2024-04-05T03:02:46.783 INFO:tasks.workunit.client.0.smithi153.stdout:8/270: write d7/d13/d19/d31/f4a [732673,44497] 0 2024-04-05T03:02:46.783 INFO:tasks.workunit.client.1.smithi203.stdout:5/657: fsync d3/d4/f71 0 2024-04-05T03:02:46.784 INFO:tasks.workunit.client.0.smithi153.stdout:4/384: rename da/de/f1c to da/de/d56/d6a/f7f 0 2024-04-05T03:02:46.784 INFO:tasks.workunit.client.0.smithi153.stdout:2/304: creat d9/da/f5f x:0 0 0 2024-04-05T03:02:46.784 INFO:tasks.workunit.client.0.smithi153.stdout:6/339: chown d4/dc/df/d16/d17/d54/c71 17305 1 2024-04-05T03:02:46.785 INFO:tasks.workunit.client.0.smithi153.stdout:1/251: dwrite d5/dd/d21/fe [4194304,4194304] 0 2024-04-05T03:02:46.786 INFO:tasks.workunit.client.0.smithi153.stdout:8/271: dread - d7/d13/d19/d1b/d33/f51 zero size 2024-04-05T03:02:46.787 INFO:tasks.workunit.client.1.smithi203.stdout:5/658: mkdir d3/d1f/d59/d60/dc1/de0 0 2024-04-05T03:02:46.790 INFO:tasks.workunit.client.0.smithi153.stdout:6/340: rmdir d4/dc/df/d16/d17/d54 39 2024-04-05T03:02:46.791 INFO:tasks.workunit.client.1.smithi203.stdout:4/498: dwrite da/d29/d2e/d53/d78/f7d [0,4194304] 0 2024-04-05T03:02:46.792 INFO:tasks.workunit.client.0.smithi153.stdout:1/252: mknod d5/dd/d21/d1e/c49 0 2024-04-05T03:02:46.795 INFO:tasks.workunit.client.1.smithi203.stdout:5/659: rename d3/d4/dc/c2c to d3/d4/dd6/ce1 0 2024-04-05T03:02:46.796 INFO:tasks.workunit.client.0.smithi153.stdout:2/305: rename d9/d44 to d9/da/d19/d2e/d60 0 2024-04-05T03:02:46.796 INFO:tasks.workunit.client.1.smithi203.stdout:1/656: dread d7/fa [0,4194304] 0 2024-04-05T03:02:46.798 INFO:tasks.workunit.client.0.smithi153.stdout:6/341: unlink d4/dc/df/l58 0 2024-04-05T03:02:46.801 INFO:tasks.workunit.client.1.smithi203.stdout:7/815: dread d8/d32/f43 [0,4194304] 0 2024-04-05T03:02:46.803 INFO:tasks.workunit.client.0.smithi153.stdout:7/290: truncate d2/da/d11/d49/f56 2493319 0 2024-04-05T03:02:46.804 INFO:tasks.workunit.client.1.smithi203.stdout:4/499: rename da/cb to da/d29/d2e/d53/c98 0 2024-04-05T03:02:46.805 INFO:tasks.workunit.client.0.smithi153.stdout:4/385: link da/ff da/db/d20/d35/d41/d67/f80 0 2024-04-05T03:02:46.805 INFO:tasks.workunit.client.1.smithi203.stdout:7/816: fdatasync d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/f80 0 2024-04-05T03:02:46.807 INFO:tasks.workunit.client.1.smithi203.stdout:1/657: dwrite d7/d24/d33/d48/d4b/d51/fb2 [0,4194304] 0 2024-04-05T03:02:46.807 INFO:tasks.workunit.client.0.smithi153.stdout:2/306: dwrite d9/da/f5f [0,4194304] 0 2024-04-05T03:02:46.809 INFO:tasks.workunit.client.1.smithi203.stdout:4/500: write da/d3c/f43 [3284580,50892] 0 2024-04-05T03:02:46.809 INFO:tasks.workunit.client.1.smithi203.stdout:0/646: write dd/d17/d1d/d26/d34/f47 [7698511,117396] 0 2024-04-05T03:02:46.816 INFO:tasks.workunit.client.0.smithi153.stdout:4/386: mkdir da/db/d20/d3c/d81 0 2024-04-05T03:02:46.816 INFO:tasks.workunit.client.0.smithi153.stdout:2/307: mknod d9/da/d37/c61 0 2024-04-05T03:02:46.817 INFO:tasks.workunit.client.0.smithi153.stdout:4/387: write da/f48 [1026988,117034] 0 2024-04-05T03:02:46.822 INFO:tasks.workunit.client.0.smithi153.stdout:8/272: link d7/l1d d7/d13/d19/d1b/l5a 0 2024-04-05T03:02:46.826 INFO:tasks.workunit.client.0.smithi153.stdout:7/291: getdents d2/da/d47 0 2024-04-05T03:02:46.827 INFO:tasks.workunit.client.0.smithi153.stdout:8/273: stat d7/d13/d19/d1b/d33/f3a 0 2024-04-05T03:02:46.827 INFO:tasks.workunit.client.0.smithi153.stdout:5/295: dread d0/d25/d4d/d62/d37/f4f [0,4194304] 0 2024-04-05T03:02:46.829 INFO:tasks.workunit.client.0.smithi153.stdout:4/388: creat da/db/d20/d45/f82 x:0 0 0 2024-04-05T03:02:46.830 INFO:tasks.workunit.client.0.smithi153.stdout:7/292: write d2/f9 [4084402,40976] 0 2024-04-05T03:02:46.830 INFO:tasks.workunit.client.0.smithi153.stdout:2/308: dwrite d9/da/d19/d2e/f43 [0,4194304] 0 2024-04-05T03:02:46.835 INFO:tasks.workunit.client.0.smithi153.stdout:3/271: dwrite d1d/f21 [0,4194304] 0 2024-04-05T03:02:46.835 INFO:tasks.workunit.client.0.smithi153.stdout:5/296: mknod d0/d25/d4d/c72 0 2024-04-05T03:02:46.835 INFO:tasks.workunit.client.0.smithi153.stdout:2/309: readlink d9/da/d19/l2a 0 2024-04-05T03:02:46.836 INFO:tasks.workunit.client.0.smithi153.stdout:0/356: sync 2024-04-05T03:02:46.836 INFO:tasks.workunit.client.0.smithi153.stdout:1/253: dread d5/dd/d21/d18/d16/d33/f3e [0,4194304] 0 2024-04-05T03:02:46.839 INFO:tasks.workunit.client.0.smithi153.stdout:8/274: mkdir d7/d2a/d52/d54/d5b 0 2024-04-05T03:02:46.839 INFO:tasks.workunit.client.0.smithi153.stdout:0/357: chown d9/df/d1e/f54 113 1 2024-04-05T03:02:46.845 INFO:tasks.workunit.client.0.smithi153.stdout:5/297: chown d0/d25/d4d/d62/d5f/d61 913 1 2024-04-05T03:02:46.846 INFO:tasks.workunit.client.0.smithi153.stdout:1/254: unlink d5/dd/c2a 0 2024-04-05T03:02:46.847 INFO:tasks.workunit.client.0.smithi153.stdout:4/389: dwrite da/db/d20/f29 [0,4194304] 0 2024-04-05T03:02:46.850 INFO:tasks.workunit.client.0.smithi153.stdout:0/358: mkdir d9/df/d1e/d45/d62/d64 0 2024-04-05T03:02:46.853 INFO:tasks.workunit.client.0.smithi153.stdout:2/310: dread d9/ff [4194304,4194304] 0 2024-04-05T03:02:46.856 INFO:tasks.workunit.client.0.smithi153.stdout:5/298: dwrite d0/f3c [0,4194304] 0 2024-04-05T03:02:46.856 INFO:tasks.workunit.client.0.smithi153.stdout:6/342: dread d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:46.857 INFO:tasks.workunit.client.0.smithi153.stdout:0/359: mkdir d9/d65 0 2024-04-05T03:02:46.859 INFO:tasks.workunit.client.0.smithi153.stdout:1/255: dwrite d5/dd/d21/d18/f1c [0,4194304] 0 2024-04-05T03:02:46.859 INFO:tasks.workunit.client.0.smithi153.stdout:5/299: chown d0/c69 6 1 2024-04-05T03:02:46.861 INFO:tasks.workunit.client.0.smithi153.stdout:2/311: mkdir d9/d62 0 2024-04-05T03:02:46.862 INFO:tasks.workunit.client.0.smithi153.stdout:4/390: link da/db/d20/d35/d41/c5b da/d2a/c83 0 2024-04-05T03:02:46.864 INFO:tasks.workunit.client.0.smithi153.stdout:5/300: mknod d0/d2c/d3f/c73 0 2024-04-05T03:02:46.867 INFO:tasks.workunit.client.1.smithi203.stdout:6/697: sync 2024-04-05T03:02:46.867 INFO:tasks.workunit.client.1.smithi203.stdout:8/672: sync 2024-04-05T03:02:46.870 INFO:tasks.workunit.client.0.smithi153.stdout:5/301: creat d0/d25/d4d/d62/d5f/d61/f74 x:0 0 0 2024-04-05T03:02:46.870 INFO:tasks.workunit.client.0.smithi153.stdout:6/343: dread d4/dc/df/d16/d24/d4e/f5a [0,4194304] 0 2024-04-05T03:02:46.870 INFO:tasks.workunit.client.0.smithi153.stdout:4/391: mknod da/db/d20/d45/c84 0 2024-04-05T03:02:46.871 INFO:tasks.workunit.client.0.smithi153.stdout:1/256: creat d5/dd/d21/d18/d16/d45/f4a x:0 0 0 2024-04-05T03:02:46.872 INFO:tasks.workunit.client.0.smithi153.stdout:5/302: dread - d0/d25/f6e zero size 2024-04-05T03:02:46.875 INFO:tasks.workunit.client.1.smithi203.stdout:0/647: read - dd/d17/d1d/d26/d28/fad zero size 2024-04-05T03:02:46.876 INFO:tasks.workunit.client.1.smithi203.stdout:7/817: creat d8/d32/d16/fff x:0 0 0 2024-04-05T03:02:46.877 INFO:tasks.workunit.client.1.smithi203.stdout:4/501: mkdir da/d3c/d4c/d99 0 2024-04-05T03:02:46.877 INFO:tasks.workunit.client.0.smithi153.stdout:4/392: symlink da/db/d20/d45/d3b/d68/l85 0 2024-04-05T03:02:46.879 INFO:tasks.workunit.client.0.smithi153.stdout:6/344: readlink d4/dc/df/d16/d33/l59 0 2024-04-05T03:02:46.880 INFO:tasks.workunit.client.0.smithi153.stdout:7/293: dread d2/da/d3a/d41/d31/f43 [0,4194304] 0 2024-04-05T03:02:46.880 INFO:tasks.workunit.client.0.smithi153.stdout:4/393: mkdir da/d2a/d86 0 2024-04-05T03:02:46.884 INFO:tasks.workunit.client.0.smithi153.stdout:6/345: creat d4/dc/df/d16/d36/d4d/f79 x:0 0 0 2024-04-05T03:02:46.884 INFO:tasks.workunit.client.0.smithi153.stdout:5/303: read d0/f27 [4053846,22713] 0 2024-04-05T03:02:46.895 INFO:tasks.workunit.client.0.smithi153.stdout:1/257: rename d5/dd/d21 to d5/d8/d20/d4b 0 2024-04-05T03:02:46.897 INFO:tasks.workunit.client.0.smithi153.stdout:3/272: dread d1d/d27/d2b/f4e [0,4194304] 0 2024-04-05T03:02:46.897 INFO:tasks.workunit.client.0.smithi153.stdout:6/346: mkdir d4/dc/d7a 0 2024-04-05T03:02:46.897 INFO:tasks.workunit.client.0.smithi153.stdout:4/394: dwrite da/de/d56/f70 [0,4194304] 0 2024-04-05T03:02:46.901 INFO:tasks.workunit.client.0.smithi153.stdout:5/304: dwrite d0/d25/f2b [0,4194304] 0 2024-04-05T03:02:46.902 INFO:tasks.workunit.client.0.smithi153.stdout:4/395: chown da/db/d20/f32 1 1 2024-04-05T03:02:46.902 INFO:tasks.workunit.client.0.smithi153.stdout:6/347: chown d4/dc/c43 408604 1 2024-04-05T03:02:46.904 INFO:tasks.workunit.client.0.smithi153.stdout:7/294: rename d2/da/d11/d49/f10 to d2/da/d11/d49/d55/f61 0 2024-04-05T03:02:46.906 INFO:tasks.workunit.client.0.smithi153.stdout:1/258: fdatasync d5/d8/d20/d4b/d1e/f43 0 2024-04-05T03:02:46.907 INFO:tasks.workunit.client.0.smithi153.stdout:7/295: read - d2/da/d11/d49/f38 zero size 2024-04-05T03:02:46.907 INFO:tasks.workunit.client.0.smithi153.stdout:2/312: dread d9/da/d36/d38/d4b/f5e [0,4194304] 0 2024-04-05T03:02:46.912 INFO:tasks.workunit.client.0.smithi153.stdout:7/296: dread d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:46.963 INFO:tasks.workunit.client.0.smithi153.stdout:9/227: sync 2024-04-05T03:02:46.964 INFO:tasks.workunit.client.0.smithi153.stdout:3/273: rename d1d/l22 to d1d/d32/d5f/l68 0 2024-04-05T03:02:46.965 INFO:tasks.workunit.client.0.smithi153.stdout:4/396: dread da/db/d20/d35/d38/d3e/f50 [0,4194304] 0 2024-04-05T03:02:46.965 INFO:tasks.workunit.client.0.smithi153.stdout:5/305: link d0/d25/f2b d0/f75 0 2024-04-05T03:02:46.967 INFO:tasks.workunit.client.0.smithi153.stdout:7/297: mkdir d2/d51/d59/d62 0 2024-04-05T03:02:46.968 INFO:tasks.workunit.client.0.smithi153.stdout:9/228: write d1/d10/d16/f40 [542808,81602] 0 2024-04-05T03:02:46.968 INFO:tasks.workunit.client.1.smithi203.stdout:6/698: rmdir d0/da/d54/d4c/d7f/dcc 39 2024-04-05T03:02:46.969 INFO:tasks.workunit.client.0.smithi153.stdout:6/348: dwrite d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:46.969 INFO:tasks.workunit.client.0.smithi153.stdout:2/313: creat d9/da/d36/f63 x:0 0 0 2024-04-05T03:02:46.969 INFO:tasks.workunit.client.1.smithi203.stdout:8/673: creat d1/d4b/d5f/daf/fba x:0 0 0 2024-04-05T03:02:46.970 INFO:tasks.workunit.client.0.smithi153.stdout:7/298: chown d2/da/d11/d49/d54 1441600 1 2024-04-05T03:02:46.972 INFO:tasks.workunit.client.0.smithi153.stdout:3/274: chown d1d/f36 6437986 1 2024-04-05T03:02:46.972 INFO:tasks.workunit.client.0.smithi153.stdout:4/397: creat da/de/d56/d6a/f87 x:0 0 0 2024-04-05T03:02:46.973 INFO:tasks.workunit.client.1.smithi203.stdout:7/818: dread - d8/d32/d16/fff zero size 2024-04-05T03:02:46.973 INFO:tasks.workunit.client.1.smithi203.stdout:1/658: dwrite d7/d24/d33/d48/d4b/d51/f87 [0,4194304] 0 2024-04-05T03:02:46.974 INFO:tasks.workunit.client.0.smithi153.stdout:9/229: write d1/d9/d1b/f26 [1615104,47800] 0 2024-04-05T03:02:46.974 INFO:tasks.workunit.client.1.smithi203.stdout:2/817: dread d5/d63/f93 [0,4194304] 0 2024-04-05T03:02:46.974 INFO:tasks.workunit.client.1.smithi203.stdout:6/699: fsync d0/da/d54/f45 0 2024-04-05T03:02:46.975 INFO:tasks.workunit.client.0.smithi153.stdout:1/259: dread d5/d8/d20/d4b/d18/d16/d33/f3e [0,4194304] 0 2024-04-05T03:02:46.975 INFO:tasks.workunit.client.0.smithi153.stdout:2/314: dread - d9/da/d36/f63 zero size 2024-04-05T03:02:46.976 INFO:tasks.workunit.client.1.smithi203.stdout:4/502: read da/d3c/d4c/f7c [192205,71708] 0 2024-04-05T03:02:46.976 INFO:tasks.workunit.client.0.smithi153.stdout:3/275: chown d1d/d27/f4d 19 1 2024-04-05T03:02:46.976 INFO:tasks.workunit.client.1.smithi203.stdout:0/648: mkdir dd/d17/d1d/d22/d53/d7f/dbf 0 2024-04-05T03:02:46.978 INFO:tasks.workunit.client.0.smithi153.stdout:5/306: write d0/f16 [818554,21058] 0 2024-04-05T03:02:46.981 INFO:tasks.workunit.client.1.smithi203.stdout:7/819: symlink d8/d32/d5d/d6d/daf/l100 0 2024-04-05T03:02:46.982 INFO:tasks.workunit.client.0.smithi153.stdout:3/276: read - d1d/d32/f45 zero size 2024-04-05T03:02:46.984 INFO:tasks.workunit.client.1.smithi203.stdout:1/659: symlink d7/d14/d2b/d58/d86/ld1 0 2024-04-05T03:02:46.985 INFO:tasks.workunit.client.0.smithi153.stdout:3/277: write fd [1620623,90124] 0 2024-04-05T03:02:46.985 INFO:tasks.workunit.client.1.smithi203.stdout:2/818: rename d5/d10/d35 to d5/d10/d35/d2d/d7c/d29/d67/d9e/d11a 22 2024-04-05T03:02:46.987 INFO:tasks.workunit.client.0.smithi153.stdout:6/349: mknod d4/dc/df/d16/d17/d54/d45/c7b 0 2024-04-05T03:02:46.988 INFO:tasks.workunit.client.1.smithi203.stdout:1/660: read d7/d24/d33/f45 [51830,14243] 0 2024-04-05T03:02:46.988 INFO:tasks.workunit.client.1.smithi203.stdout:2/819: dread - d5/f98 zero size 2024-04-05T03:02:46.988 INFO:tasks.workunit.client.1.smithi203.stdout:6/700: dread d0/da/d18/d38/d2a/fce [0,4194304] 0 2024-04-05T03:02:46.991 INFO:tasks.workunit.client.0.smithi153.stdout:2/315: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:46.992 INFO:tasks.workunit.client.1.smithi203.stdout:7/820: read d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/f80 [1258576,72574] 0 2024-04-05T03:02:46.993 INFO:tasks.workunit.client.1.smithi203.stdout:1/661: read d7/d79/f7d [3184947,104142] 0 2024-04-05T03:02:46.993 INFO:tasks.workunit.client.0.smithi153.stdout:5/307: dwrite d0/f16 [4194304,4194304] 0 2024-04-05T03:02:46.995 INFO:tasks.workunit.client.0.smithi153.stdout:7/299: link d2/da/d53/f5b d2/da/d11/d39/f63 0 2024-04-05T03:02:46.996 INFO:tasks.workunit.client.1.smithi203.stdout:2/820: unlink d5/d10/d1a/d57/f6b 0 2024-04-05T03:02:46.996 INFO:tasks.workunit.client.1.smithi203.stdout:4/503: dwrite da/d29/d6a/f6c [0,4194304] 0 2024-04-05T03:02:46.998 INFO:tasks.workunit.client.0.smithi153.stdout:7/300: readlink l1 0 2024-04-05T03:02:46.998 INFO:tasks.workunit.client.0.smithi153.stdout:6/350: creat d4/dc/df/d16/d5f/f7c x:0 0 0 2024-04-05T03:02:46.998 INFO:tasks.workunit.client.0.smithi153.stdout:7/301: truncate d2/da/d11/f24 5168015 0 2024-04-05T03:02:46.999 INFO:tasks.workunit.client.1.smithi203.stdout:4/504: fsync da/d3c/d4c/f5c 0 2024-04-05T03:02:47.003 INFO:tasks.workunit.client.1.smithi203.stdout:4/505: write da/fc [3727248,27208] 0 2024-04-05T03:02:47.003 INFO:tasks.workunit.client.1.smithi203.stdout:6/701: readlink d0/da/d54/d4c/da1/db7/la8 0 2024-04-05T03:02:47.005 INFO:tasks.workunit.client.1.smithi203.stdout:8/674: dwrite d1/d4b/d68/f95 [0,4194304] 0 2024-04-05T03:02:47.005 INFO:tasks.workunit.client.0.smithi153.stdout:5/308: truncate d0/f27 3669746 0 2024-04-05T03:02:47.006 INFO:tasks.workunit.client.0.smithi153.stdout:5/309: dread - d0/d25/f36 zero size 2024-04-05T03:02:47.006 INFO:tasks.workunit.client.1.smithi203.stdout:1/662: fdatasync d7/d24/fc1 0 2024-04-05T03:02:47.007 INFO:tasks.workunit.client.1.smithi203.stdout:4/506: fdatasync da/f22 0 2024-04-05T03:02:47.007 INFO:tasks.workunit.client.0.smithi153.stdout:7/302: symlink d2/da/d42/l64 0 2024-04-05T03:02:47.007 INFO:tasks.workunit.client.0.smithi153.stdout:5/310: readlink d0/d25/d4d/d62/l55 0 2024-04-05T03:02:47.008 INFO:tasks.workunit.client.0.smithi153.stdout:5/311: chown d0/d25/d4d/c72 843745383 1 2024-04-05T03:02:47.008 INFO:tasks.workunit.client.0.smithi153.stdout:6/351: symlink d4/dc/df/l7d 0 2024-04-05T03:02:47.010 INFO:tasks.workunit.client.0.smithi153.stdout:5/312: chown d0/d25/d4d/d62/d5f/d61/f74 0 1 2024-04-05T03:02:47.011 INFO:tasks.workunit.client.0.smithi153.stdout:2/316: unlink d9/da/d19/d2e/d48/l26 0 2024-04-05T03:02:47.013 INFO:tasks.workunit.client.1.smithi203.stdout:0/649: rename dd/d17/d1d/d22/d45/d51/d69/f91 to dd/d17/d1d/d26/fc0 0 2024-04-05T03:02:47.013 INFO:tasks.workunit.client.0.smithi153.stdout:6/352: creat d4/dc/df/d16/d5f/d73/f7e x:0 0 0 2024-04-05T03:02:47.014 INFO:tasks.workunit.client.1.smithi203.stdout:1/663: truncate d7/d79/f88 944414 0 2024-04-05T03:02:47.014 INFO:tasks.workunit.client.1.smithi203.stdout:4/507: chown da/d29/d2e/f63 963006 1 2024-04-05T03:02:47.015 INFO:tasks.workunit.client.0.smithi153.stdout:6/353: write d4/dc/f3c [2593576,40032] 0 2024-04-05T03:02:47.017 INFO:tasks.workunit.client.1.smithi203.stdout:2/821: creat d5/d10/d35/d2d/d7c/d72/f11b x:0 0 0 2024-04-05T03:02:47.017 INFO:tasks.workunit.client.0.smithi153.stdout:6/354: dread - d4/dc/df/d16/d5f/f7c zero size 2024-04-05T03:02:47.017 INFO:tasks.workunit.client.0.smithi153.stdout:7/303: rename d2/da/c4c to d2/da/d47/c65 0 2024-04-05T03:02:47.019 INFO:tasks.workunit.client.1.smithi203.stdout:2/822: write d5/d10/d35/d2d/d7c/d72/fe6 [905006,38086] 0 2024-04-05T03:02:47.019 INFO:tasks.workunit.client.1.smithi203.stdout:1/664: creat d7/d24/d33/d48/d4b/d6b/db9/fd2 x:0 0 0 2024-04-05T03:02:47.020 INFO:tasks.workunit.client.0.smithi153.stdout:5/313: creat d0/d9/f76 x:0 0 0 2024-04-05T03:02:47.021 INFO:tasks.workunit.client.1.smithi203.stdout:0/650: mkdir dd/d17/d1d/d22/dc1 0 2024-04-05T03:02:47.022 INFO:tasks.workunit.client.0.smithi153.stdout:2/317: creat d9/da/f64 x:0 0 0 2024-04-05T03:02:47.022 INFO:tasks.workunit.client.1.smithi203.stdout:6/702: mkdir d0/da/d18/dd5 0 2024-04-05T03:02:47.023 INFO:tasks.workunit.client.1.smithi203.stdout:0/651: chown dd/d17/d1d/d22/d53/d55/fa1 417 1 2024-04-05T03:02:47.023 INFO:tasks.workunit.client.0.smithi153.stdout:6/355: symlink d4/dc/df/d16/d5f/l7f 0 2024-04-05T03:02:47.024 INFO:tasks.workunit.client.1.smithi203.stdout:7/821: link d8/fe d8/d32/d5d/d6d/f101 0 2024-04-05T03:02:47.025 INFO:tasks.workunit.client.0.smithi153.stdout:5/314: creat d0/d25/d4d/d62/d5f/d61/f77 x:0 0 0 2024-04-05T03:02:47.026 INFO:tasks.workunit.client.1.smithi203.stdout:2/823: creat d5/d10/d35/d2d/d7c/d29/d67/d9e/f11c x:0 0 0 2024-04-05T03:02:47.027 INFO:tasks.workunit.client.1.smithi203.stdout:2/824: fsync d5/f8 0 2024-04-05T03:02:47.027 INFO:tasks.workunit.client.0.smithi153.stdout:7/304: dwrite d2/da/d11/f52 [0,4194304] 0 2024-04-05T03:02:47.028 INFO:tasks.workunit.client.0.smithi153.stdout:6/356: write d4/dc/df/d16/f23 [694424,109771] 0 2024-04-05T03:02:47.028 INFO:tasks.workunit.client.0.smithi153.stdout:2/318: link c5 d9/da/d36/d38/d39/c65 0 2024-04-05T03:02:47.028 INFO:tasks.workunit.client.1.smithi203.stdout:6/703: dread - d0/da/d18/d38/d86/f93 zero size 2024-04-05T03:02:47.028 INFO:tasks.workunit.client.1.smithi203.stdout:8/675: link d1/d4b/d68/d75/d77/c79 d1/d12/cbb 0 2024-04-05T03:02:47.029 INFO:tasks.workunit.client.1.smithi203.stdout:2/825: read - d5/d10/d44/df6/f113 zero size 2024-04-05T03:02:47.029 INFO:tasks.workunit.client.1.smithi203.stdout:6/704: write d0/da/d54/fb0 [7935475,49197] 0 2024-04-05T03:02:47.029 INFO:tasks.workunit.client.1.smithi203.stdout:7/822: creat d8/da/d72/f102 x:0 0 0 2024-04-05T03:02:47.030 INFO:tasks.workunit.client.0.smithi153.stdout:5/315: dread - d0/d25/d4d/d62/d37/d39/f3b zero size 2024-04-05T03:02:47.030 INFO:tasks.workunit.client.0.smithi153.stdout:6/357: chown d4/dc/df/d16 0 1 2024-04-05T03:02:47.031 INFO:tasks.workunit.client.1.smithi203.stdout:6/705: read - d0/da/d18/d38/d2a/fb1 zero size 2024-04-05T03:02:47.033 INFO:tasks.workunit.client.0.smithi153.stdout:7/305: write d2/f32 [796665,126374] 0 2024-04-05T03:02:47.038 INFO:tasks.workunit.client.1.smithi203.stdout:8/676: mkdir d1/d6/d2e/d5c/d99/dbc 0 2024-04-05T03:02:47.040 INFO:tasks.workunit.client.0.smithi153.stdout:2/319: dwrite d9/da/d36/f4d [0,4194304] 0 2024-04-05T03:02:47.040 INFO:tasks.workunit.client.1.smithi203.stdout:8/677: write d1/d12/d2f/d1f/f50 [1910326,56237] 0 2024-04-05T03:02:47.040 INFO:tasks.workunit.client.0.smithi153.stdout:9/230: dread d1/f18 [0,4194304] 0 2024-04-05T03:02:47.042 INFO:tasks.workunit.client.0.smithi153.stdout:5/316: symlink d0/d25/d4d/d62/d5f/l78 0 2024-04-05T03:02:47.042 INFO:tasks.workunit.client.0.smithi153.stdout:2/320: chown d9/da/d36/d38/d39 869772 1 2024-04-05T03:02:47.045 INFO:tasks.workunit.client.0.smithi153.stdout:6/358: rename d4/dc/c4a to d4/dc/df/d16/d17/d54/d45/c80 0 2024-04-05T03:02:47.046 INFO:tasks.workunit.client.1.smithi203.stdout:8/678: creat d1/d4b/d68/d75/dac/da4/fbd x:0 0 0 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: pgmap v8: 65 pgs: 65 active+clean; 1.4 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 44 MiB/s rd, 139 MiB/s wr, 273 op/s 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: Standby manager daemon smithi153.npzyqy started 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/crt"}]: dispatch 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/key"}]: dispatch 2024-04-05T03:02:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:46 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:02:47.050 INFO:tasks.workunit.client.0.smithi153.stdout:9/231: symlink d1/d10/d16/l52 0 2024-04-05T03:02:47.051 INFO:tasks.workunit.client.0.smithi153.stdout:6/359: dread d4/dc/f46 [0,4194304] 0 2024-04-05T03:02:47.052 INFO:tasks.workunit.client.0.smithi153.stdout:4/398: truncate da/db/d20/f29 1624955 0 2024-04-05T03:02:47.053 INFO:tasks.workunit.client.0.smithi153.stdout:7/306: rmdir d2/d51/d59/d62 0 2024-04-05T03:02:47.053 INFO:tasks.workunit.client.1.smithi203.stdout:0/652: write f2 [1992262,78125] 0 2024-04-05T03:02:47.053 INFO:tasks.workunit.client.0.smithi153.stdout:1/260: dwrite d5/d8/ff [0,4194304] 0 2024-04-05T03:02:47.054 INFO:tasks.workunit.client.1.smithi203.stdout:7/823: write d8/d32/d16/d35/fcf [3448493,26801] 0 2024-04-05T03:02:47.054 INFO:tasks.workunit.client.0.smithi153.stdout:2/321: symlink d9/da/d36/l66 0 2024-04-05T03:02:47.056 INFO:tasks.workunit.client.0.smithi153.stdout:5/317: dwrite d0/d2c/f3a [0,4194304] 0 2024-04-05T03:02:47.057 INFO:tasks.workunit.client.0.smithi153.stdout:3/278: write d1d/f28 [4798513,56941] 0 2024-04-05T03:02:47.058 INFO:tasks.workunit.client.0.smithi153.stdout:7/307: readlink d2/da/l29 0 2024-04-05T03:02:47.059 INFO:tasks.workunit.client.1.smithi203.stdout:0/653: stat dd/d17/d1d/d22/d53/db0/f3c 0 2024-04-05T03:02:47.059 INFO:tasks.workunit.client.0.smithi153.stdout:5/318: chown d0/d9/f1e 129528 1 2024-04-05T03:02:47.060 INFO:tasks.workunit.client.0.smithi153.stdout:7/308: stat d2/da/d11/d39/f63 0 2024-04-05T03:02:47.061 INFO:tasks.workunit.client.0.smithi153.stdout:2/322: chown d9/da/d36/d38/l3b 0 1 2024-04-05T03:02:47.062 INFO:tasks.workunit.client.0.smithi153.stdout:4/399: creat da/de/f88 x:0 0 0 2024-04-05T03:02:47.064 INFO:tasks.workunit.client.1.smithi203.stdout:2/826: dwrite d5/d63/f9c [0,4194304] 0 2024-04-05T03:02:47.064 INFO:tasks.workunit.client.1.smithi203.stdout:0/654: dread - dd/d17/d1d/d26/d28/fad zero size 2024-04-05T03:02:47.065 INFO:tasks.workunit.client.0.smithi153.stdout:6/360: unlink d4/dc/df/d16/d36/d4d/f79 0 2024-04-05T03:02:47.066 INFO:tasks.workunit.client.0.smithi153.stdout:4/400: write da/d2a/f47 [1003711,38734] 0 2024-04-05T03:02:47.068 INFO:tasks.workunit.client.1.smithi203.stdout:2/827: write d5/d10/d35/d36/ffa [500547,24500] 0 2024-04-05T03:02:47.068 INFO:tasks.workunit.client.0.smithi153.stdout:1/261: creat d5/d8/d20/d2f/d40/f4c x:0 0 0 2024-04-05T03:02:47.069 INFO:tasks.workunit.client.1.smithi203.stdout:7/824: link d8/d32/d5d/d95/c97 d8/d32/d16/d35/d64/c103 0 2024-04-05T03:02:47.069 INFO:tasks.workunit.client.0.smithi153.stdout:6/361: fsync d4/dc/df/d16/d5f/f74 0 2024-04-05T03:02:47.071 INFO:tasks.workunit.client.1.smithi203.stdout:2/828: symlink d5/d10/d7f/l11d 0 2024-04-05T03:02:47.071 INFO:tasks.workunit.client.1.smithi203.stdout:7/825: write d8/d32/d5d/d6d/dca/f7e [3067390,89675] 0 2024-04-05T03:02:47.072 INFO:tasks.workunit.client.0.smithi153.stdout:1/262: stat d5/d8/d20/d2f/d40/f4c 0 2024-04-05T03:02:47.072 INFO:tasks.workunit.client.0.smithi153.stdout:3/279: dwrite d1d/d27/f4d [0,4194304] 0 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: pgmap v8: 65 pgs: 65 active+clean; 1.4 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 44 MiB/s rd, 139 MiB/s wr, 273 op/s 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: Standby manager daemon smithi153.npzyqy started 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/crt"}]: dispatch 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/key"}]: dispatch 2024-04-05T03:02:47.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:46 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/449002479' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:02:47.075 INFO:tasks.workunit.client.0.smithi153.stdout:5/319: symlink d0/d25/l79 0 2024-04-05T03:02:47.075 INFO:tasks.workunit.client.0.smithi153.stdout:4/401: dread da/de/f14 [0,4194304] 0 2024-04-05T03:02:47.075 INFO:tasks.workunit.client.0.smithi153.stdout:6/362: dread d4/d38/d4c/f6a [0,4194304] 0 2024-04-05T03:02:47.076 INFO:tasks.workunit.client.1.smithi203.stdout:7/826: unlink f4 0 2024-04-05T03:02:47.076 INFO:tasks.workunit.client.1.smithi203.stdout:7/827: chown d8/c1c 0 1 2024-04-05T03:02:47.076 INFO:tasks.workunit.client.0.smithi153.stdout:1/263: write d5/dd/d1b/f1d [143888,124300] 0 2024-04-05T03:02:47.078 INFO:tasks.workunit.client.0.smithi153.stdout:9/232: rmdir d1/d10/d4d 0 2024-04-05T03:02:47.080 INFO:tasks.workunit.client.0.smithi153.stdout:4/402: truncate da/db/d20/d35/f58 744348 0 2024-04-05T03:02:47.084 INFO:tasks.workunit.client.0.smithi153.stdout:1/264: rename d5/d8/d20/d4b/d18/d16 to d5/d8/d20/d4b/d18/d16/d48/d4d 22 2024-04-05T03:02:47.084 INFO:tasks.workunit.client.0.smithi153.stdout:5/320: creat d0/d9/d22/f7a x:0 0 0 2024-04-05T03:02:47.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/403: write da/de/f88 [820240,104653] 0 2024-04-05T03:02:47.086 INFO:tasks.workunit.client.0.smithi153.stdout:2/323: mknod d9/da/d19/d2e/d48/c67 0 2024-04-05T03:02:47.087 INFO:tasks.workunit.client.1.smithi203.stdout:0/655: dread dd/d17/d1d/d22/d53/f63 [0,4194304] 0 2024-04-05T03:02:47.087 INFO:tasks.workunit.client.1.smithi203.stdout:0/656: fdatasync f5 0 2024-04-05T03:02:47.090 INFO:tasks.workunit.client.0.smithi153.stdout:2/324: dread - d9/da/d36/f63 zero size 2024-04-05T03:02:47.091 INFO:tasks.workunit.client.1.smithi203.stdout:0/657: write dd/d17/d1d/d22/f3b [1157428,100063] 0 2024-04-05T03:02:47.092 INFO:tasks.workunit.client.0.smithi153.stdout:9/233: dread d1/d10/d16/f27 [0,4194304] 0 2024-04-05T03:02:47.093 INFO:tasks.workunit.client.0.smithi153.stdout:3/280: link d1d/d27/f4d d1d/d26/f69 0 2024-04-05T03:02:47.094 INFO:tasks.workunit.client.1.smithi203.stdout:0/658: fdatasync dd/d17/d1d/d26/fc0 0 2024-04-05T03:02:47.094 INFO:tasks.workunit.client.0.smithi153.stdout:6/363: dwrite d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:47.094 INFO:tasks.workunit.client.1.smithi203.stdout:6/706: dread d0/f3 [4194304,4194304] 0 2024-04-05T03:02:47.095 INFO:tasks.workunit.client.0.smithi153.stdout:1/265: symlink d5/dd/d1b/l4e 0 2024-04-05T03:02:47.101 INFO:tasks.workunit.client.0.smithi153.stdout:4/404: creat da/de/d66/f89 x:0 0 0 2024-04-05T03:02:47.103 INFO:tasks.workunit.client.1.smithi203.stdout:0/659: symlink dd/d17/d1d/d75/lc2 0 2024-04-05T03:02:47.107 INFO:tasks.workunit.client.0.smithi153.stdout:4/405: fsync f7 0 2024-04-05T03:02:47.108 INFO:tasks.workunit.client.1.smithi203.stdout:2/829: dwrite d5/d10/d35/d2d/f33 [0,4194304] 0 2024-04-05T03:02:47.109 INFO:tasks.workunit.client.0.smithi153.stdout:9/234: dwrite d1/f25 [0,4194304] 0 2024-04-05T03:02:47.110 INFO:tasks.workunit.client.0.smithi153.stdout:3/281: creat d1d/f6a x:0 0 0 2024-04-05T03:02:47.111 INFO:tasks.workunit.client.0.smithi153.stdout:2/325: dwrite d9/da/f5f [0,4194304] 0 2024-04-05T03:02:47.111 INFO:tasks.workunit.client.1.smithi203.stdout:7/828: dwrite d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:47.116 INFO:tasks.workunit.client.0.smithi153.stdout:5/321: dread d0/d25/d4d/d62/f56 [0,4194304] 0 2024-04-05T03:02:47.118 INFO:tasks.workunit.client.0.smithi153.stdout:3/282: unlink d1d/d27/f34 0 2024-04-05T03:02:47.118 INFO:tasks.workunit.client.0.smithi153.stdout:6/364: dwrite d4/dc/df/d16/d5f/d73/f7e [0,4194304] 0 2024-04-05T03:02:47.119 INFO:tasks.workunit.client.0.smithi153.stdout:7/309: dread d2/da/d3a/f46 [0,4194304] 0 2024-04-05T03:02:47.121 INFO:tasks.workunit.client.0.smithi153.stdout:7/310: write d2/da/d11/d49/f4e [4966095,38550] 0 2024-04-05T03:02:47.125 INFO:tasks.workunit.client.1.smithi203.stdout:9/642: sync 2024-04-05T03:02:47.128 INFO:tasks.workunit.client.0.smithi153.stdout:9/235: creat d1/d10/d16/f53 x:0 0 0 2024-04-05T03:02:47.129 INFO:tasks.workunit.client.0.smithi153.stdout:5/322: fsync d0/d25/d4d/d62/d37/f4f 0 2024-04-05T03:02:47.130 INFO:tasks.workunit.client.0.smithi153.stdout:1/266: dread d5/d8/d20/d4b/fe [0,4194304] 0 2024-04-05T03:02:47.134 INFO:tasks.workunit.client.0.smithi153.stdout:1/267: chown d5/d8/d20/d4b/d18/f37 2 1 2024-04-05T03:02:47.134 INFO:tasks.workunit.client.0.smithi153.stdout:6/365: rmdir d4/dc/df/d16/d24 39 2024-04-05T03:02:47.134 INFO:tasks.workunit.client.1.smithi203.stdout:9/643: dwrite d13/d5c/dd1/d51/fab [0,4194304] 0 2024-04-05T03:02:47.135 INFO:tasks.workunit.client.0.smithi153.stdout:4/406: creat da/db/d20/d3c/d81/f8a x:0 0 0 2024-04-05T03:02:47.136 INFO:tasks.workunit.client.0.smithi153.stdout:4/407: fdatasync f8 0 2024-04-05T03:02:47.139 INFO:tasks.workunit.client.0.smithi153.stdout:5/323: truncate d0/d2c/f53 1019246 0 2024-04-05T03:02:47.139 INFO:tasks.workunit.client.1.smithi203.stdout:6/707: dread d0/da/d54/d53/fbd [0,4194304] 0 2024-04-05T03:02:47.141 INFO:tasks.workunit.client.0.smithi153.stdout:5/324: chown d0/d9/c23 163973238 1 2024-04-05T03:02:47.141 INFO:tasks.workunit.client.1.smithi203.stdout:9/644: mkdir d13/d1b/d33/d64/dd6 0 2024-04-05T03:02:47.142 INFO:tasks.workunit.client.0.smithi153.stdout:2/326: rename d9/da/d36/d38 to d9/d68 0 2024-04-05T03:02:47.143 INFO:tasks.workunit.client.0.smithi153.stdout:9/236: dwrite d1/db/f4a [0,4194304] 0 2024-04-05T03:02:47.144 INFO:tasks.workunit.client.0.smithi153.stdout:4/408: mkdir da/db/d8b 0 2024-04-05T03:02:47.144 INFO:tasks.workunit.client.1.smithi203.stdout:6/708: creat d0/da/d18/d38/d2a/d44/fd6 x:0 0 0 2024-04-05T03:02:47.145 INFO:tasks.workunit.client.1.smithi203.stdout:6/709: chown d0/da/d18/d38/d86/f7c 19621407 1 2024-04-05T03:02:47.147 INFO:tasks.workunit.client.0.smithi153.stdout:1/268: rename d5/d8/d20/d4b/d1e/f43 to d5/d8/d20/d4b/d1e/f4f 0 2024-04-05T03:02:47.148 INFO:tasks.workunit.client.0.smithi153.stdout:5/325: mknod d0/d25/d4d/d62/d37/d39/c7b 0 2024-04-05T03:02:47.150 INFO:tasks.workunit.client.0.smithi153.stdout:3/283: rename d1d/d32/d5f to d1d/d26/d57/d6b 0 2024-04-05T03:02:47.151 INFO:tasks.workunit.client.0.smithi153.stdout:8/275: sync 2024-04-05T03:02:47.151 INFO:tasks.workunit.client.0.smithi153.stdout:2/327: creat d9/d68/d39/f69 x:0 0 0 2024-04-05T03:02:47.151 INFO:tasks.workunit.client.0.smithi153.stdout:8/276: chown d7/d13/c1e 334882 1 2024-04-05T03:02:47.152 INFO:tasks.workunit.client.0.smithi153.stdout:8/277: chown d7/d46 0 1 2024-04-05T03:02:47.152 INFO:tasks.workunit.client.0.smithi153.stdout:4/409: unlink da/db/d20/d35/c5d 0 2024-04-05T03:02:47.154 INFO:tasks.workunit.client.0.smithi153.stdout:8/278: write d7/d13/d19/f4e [4596676,40073] 0 2024-04-05T03:02:47.154 INFO:tasks.workunit.client.0.smithi153.stdout:4/410: readlink da/db/d20/d35/d41/l7d 0 2024-04-05T03:02:47.154 INFO:tasks.workunit.client.0.smithi153.stdout:5/326: symlink d0/d25/d4d/d62/l7c 0 2024-04-05T03:02:47.155 INFO:tasks.workunit.client.0.smithi153.stdout:8/279: read - d7/d13/d19/d31/f41 zero size 2024-04-05T03:02:47.155 INFO:tasks.workunit.client.0.smithi153.stdout:1/269: creat d5/d8/d20/d4b/d18/d16/d48/f50 x:0 0 0 2024-04-05T03:02:47.158 INFO:tasks.workunit.client.1.smithi203.stdout:7/829: dread d8/f3e [0,4194304] 0 2024-04-05T03:02:47.159 INFO:tasks.workunit.client.1.smithi203.stdout:7/830: chown d8/da/cc 0 1 2024-04-05T03:02:47.159 INFO:tasks.workunit.client.1.smithi203.stdout:7/831: chown d8/d32/d16/d35 304 1 2024-04-05T03:02:47.159 INFO:tasks.workunit.client.0.smithi153.stdout:8/280: readlink d7/l1d 0 2024-04-05T03:02:47.160 INFO:tasks.workunit.client.0.smithi153.stdout:8/281: stat d7/d2a/d52 0 2024-04-05T03:02:47.163 INFO:tasks.workunit.client.0.smithi153.stdout:1/270: dwrite f4 [4194304,4194304] 0 2024-04-05T03:02:47.165 INFO:tasks.workunit.client.1.smithi203.stdout:0/660: dread dd/d17/d1d/d22/d80/d49/f7d [0,4194304] 0 2024-04-05T03:02:47.165 INFO:tasks.workunit.client.1.smithi203.stdout:7/832: symlink d8/d32/d16/d1a/d3d/d59/de7/d52/l104 0 2024-04-05T03:02:47.166 INFO:tasks.workunit.client.0.smithi153.stdout:7/311: dread d2/da/d11/f2d [0,4194304] 0 2024-04-05T03:02:47.166 INFO:tasks.workunit.client.1.smithi203.stdout:0/661: chown dd/d17/d1d/d22/d45 873446 1 2024-04-05T03:02:47.168 INFO:tasks.workunit.client.0.smithi153.stdout:8/282: creat d7/d13/d19/f5c x:0 0 0 2024-04-05T03:02:47.169 INFO:tasks.workunit.client.1.smithi203.stdout:0/662: truncate dd/d17/d1d/d22/d53/db0/f3c 2377299 0 2024-04-05T03:02:47.169 INFO:tasks.workunit.client.1.smithi203.stdout:7/833: dread d8/d32/d16/d92/fdc [0,4194304] 0 2024-04-05T03:02:47.171 INFO:tasks.workunit.client.1.smithi203.stdout:9/645: dread d13/d1b/d6b/d9e/d77/f95 [0,4194304] 0 2024-04-05T03:02:47.172 INFO:tasks.workunit.client.1.smithi203.stdout:9/646: chown d13/f20 20 1 2024-04-05T03:02:47.172 INFO:tasks.workunit.client.1.smithi203.stdout:0/663: truncate dd/d17/d1d/d26/f4f 907810 0 2024-04-05T03:02:47.174 INFO:tasks.workunit.client.0.smithi153.stdout:1/271: dread d5/d8/d20/d4b/d18/d16/f2d [0,4194304] 0 2024-04-05T03:02:47.174 INFO:tasks.workunit.client.1.smithi203.stdout:0/664: write dd/d17/d1d/d26/d34/fb6 [1009166,115379] 0 2024-04-05T03:02:47.176 INFO:tasks.workunit.client.1.smithi203.stdout:7/834: link d8/d32/d16/d1a/l2b d8/d32/d16/d1a/d30/d90/l105 0 2024-04-05T03:02:47.177 INFO:tasks.workunit.client.0.smithi153.stdout:8/283: dwrite d7/d13/f14 [0,4194304] 0 2024-04-05T03:02:47.178 INFO:tasks.workunit.client.0.smithi153.stdout:1/272: chown d5/d8/d20/d4b/d18/d16/d33 8969374 1 2024-04-05T03:02:47.178 INFO:tasks.workunit.client.1.smithi203.stdout:0/665: write dd/d17/f40 [5165288,78596] 0 2024-04-05T03:02:47.178 INFO:tasks.workunit.client.0.smithi153.stdout:5/327: dread d0/d25/d4d/d62/f10 [0,4194304] 0 2024-04-05T03:02:47.180 INFO:tasks.workunit.client.0.smithi153.stdout:1/273: stat d5/d8/d20/d4b/d18/d16/d33/f3c 0 2024-04-05T03:02:47.180 INFO:tasks.workunit.client.0.smithi153.stdout:8/284: rmdir d7/d2a 39 2024-04-05T03:02:47.180 INFO:tasks.workunit.client.1.smithi203.stdout:7/835: unlink d8/d32/d16/d92/cb9 0 2024-04-05T03:02:47.181 INFO:tasks.workunit.client.0.smithi153.stdout:8/285: chown d7/d13/d19/d1b/l5a 1 1 2024-04-05T03:02:47.184 INFO:tasks.workunit.client.1.smithi203.stdout:7/836: symlink d8/d32/d5d/d6d/daf/l106 0 2024-04-05T03:02:47.186 INFO:tasks.workunit.client.1.smithi203.stdout:0/666: dwrite dd/d17/f40 [0,4194304] 0 2024-04-05T03:02:47.188 INFO:tasks.workunit.client.0.smithi153.stdout:1/274: dwrite d5/d8/d20/d4b/d18/d16/f36 [0,4194304] 0 2024-04-05T03:02:47.189 INFO:tasks.workunit.client.1.smithi203.stdout:7/837: unlink d8/c9 0 2024-04-05T03:02:47.190 INFO:tasks.workunit.client.1.smithi203.stdout:7/838: chown d8/d32/d16/d1a/d3d/d59/de7/d73/da8 26 1 2024-04-05T03:02:47.192 INFO:tasks.workunit.client.1.smithi203.stdout:7/839: fsync d8/d32/d16/d1a/d3d/d59/de7/d73/ff7 0 2024-04-05T03:02:47.195 INFO:tasks.workunit.client.1.smithi203.stdout:7/840: mkdir d8/d32/d16/d1a/d3d/d59/d107 0 2024-04-05T03:02:47.196 INFO:tasks.workunit.client.0.smithi153.stdout:1/275: dwrite f4 [4194304,4194304] 0 2024-04-05T03:02:47.196 INFO:tasks.workunit.client.1.smithi203.stdout:7/841: dread - d8/d32/d5d/d95/fa2 zero size 2024-04-05T03:02:47.197 INFO:tasks.workunit.client.0.smithi153.stdout:1/276: write d5/dd/f13 [2239959,43469] 0 2024-04-05T03:02:47.197 INFO:tasks.workunit.client.1.smithi203.stdout:7/842: write d8/d32/d16/d1a/d5a/fbc [851163,42742] 0 2024-04-05T03:02:47.198 INFO:tasks.workunit.client.0.smithi153.stdout:1/277: fsync d5/d8/d20/d2f/d40/f4c 0 2024-04-05T03:02:47.200 INFO:tasks.workunit.client.1.smithi203.stdout:9/647: dread d13/d1b/d6b/d9e/d59/fb2 [0,4194304] 0 2024-04-05T03:02:47.200 INFO:tasks.workunit.client.0.smithi153.stdout:1/278: fsync d5/d8/d20/f25 0 2024-04-05T03:02:47.202 INFO:tasks.workunit.client.1.smithi203.stdout:9/648: creat d13/d1b/d33/d64/fd7 x:0 0 0 2024-04-05T03:02:47.204 INFO:tasks.workunit.client.0.smithi153.stdout:2/328: truncate d9/da/d19/f46 3557468 0 2024-04-05T03:02:47.205 INFO:tasks.workunit.client.0.smithi153.stdout:1/279: mkdir d5/d8/d20/d4b/d51 0 2024-04-05T03:02:47.207 INFO:tasks.workunit.client.0.smithi153.stdout:1/280: chown d5/dd/f26 9005 1 2024-04-05T03:02:47.207 INFO:tasks.workunit.client.1.smithi203.stdout:7/843: read d8/da/f1d [2305382,17191] 0 2024-04-05T03:02:47.207 INFO:tasks.workunit.client.1.smithi203.stdout:9/649: dread d13/d5c/f9f [0,4194304] 0 2024-04-05T03:02:47.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/312: truncate d2/da/d3a/f46 530465 0 2024-04-05T03:02:47.209 INFO:tasks.workunit.client.0.smithi153.stdout:2/329: fsync d9/d68/d4b/f5e 0 2024-04-05T03:02:47.210 INFO:tasks.workunit.client.1.smithi203.stdout:7/844: readlink d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/l93 0 2024-04-05T03:02:47.211 INFO:tasks.workunit.client.0.smithi153.stdout:1/281: chown d5/d8/d20/d4b/c17 973 1 2024-04-05T03:02:47.211 INFO:tasks.workunit.client.1.smithi203.stdout:9/650: rmdir d13/d1b/d6b/d9e/d59/d7b 39 2024-04-05T03:02:47.211 INFO:tasks.workunit.client.0.smithi153.stdout:8/286: write d7/d13/d19/d31/f42 [2270107,15719] 0 2024-04-05T03:02:47.212 INFO:tasks.workunit.client.0.smithi153.stdout:3/284: dwrite f18 [0,4194304] 0 2024-04-05T03:02:47.212 INFO:tasks.workunit.client.1.smithi203.stdout:9/651: dread - d13/d1b/d33/d64/fd7 zero size 2024-04-05T03:02:47.218 INFO:tasks.workunit.client.0.smithi153.stdout:7/313: dread d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:47.218 INFO:tasks.workunit.client.1.smithi203.stdout:9/652: mkdir d13/d1b/d6b/d9e/d59/d9b/dd8 0 2024-04-05T03:02:47.218 INFO:tasks.workunit.client.1.smithi203.stdout:7/845: dwrite d8/d32/d16/d1a/d5a/db5/fd7 [0,4194304] 0 2024-04-05T03:02:47.219 INFO:tasks.workunit.client.1.smithi203.stdout:9/653: chown d13/d1b/fbd 31 1 2024-04-05T03:02:47.219 INFO:tasks.workunit.client.0.smithi153.stdout:2/330: creat d9/da/d19/f6a x:0 0 0 2024-04-05T03:02:47.221 INFO:tasks.workunit.client.1.smithi203.stdout:0/667: dread dd/d17/d1d/d26/f42 [0,4194304] 0 2024-04-05T03:02:47.224 INFO:tasks.workunit.client.0.smithi153.stdout:8/287: creat d7/d13/d19/d1b/d4f/f5d x:0 0 0 2024-04-05T03:02:47.224 INFO:tasks.workunit.client.0.smithi153.stdout:8/288: fsync f0 0 2024-04-05T03:02:47.226 INFO:tasks.workunit.client.0.smithi153.stdout:5/328: dwrite d0/f14 [0,4194304] 0 2024-04-05T03:02:47.231 INFO:tasks.workunit.client.0.smithi153.stdout:5/329: readlink d0/d9/d22/l2e 0 2024-04-05T03:02:47.232 INFO:tasks.workunit.client.0.smithi153.stdout:8/289: read d7/d13/d19/d31/f42 [2591549,72162] 0 2024-04-05T03:02:47.233 INFO:tasks.workunit.client.1.smithi203.stdout:0/668: dread dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:47.234 INFO:tasks.workunit.client.0.smithi153.stdout:1/282: rename d5/dd/c27 to d5/d8/c52 0 2024-04-05T03:02:47.236 INFO:tasks.workunit.client.1.smithi203.stdout:7/846: creat d8/d32/d16/d1a/d3d/d59/de7/f108 x:0 0 0 2024-04-05T03:02:47.237 INFO:tasks.workunit.client.1.smithi203.stdout:7/847: fdatasync d8/d32/d16/d1a/d30/f98 0 2024-04-05T03:02:47.238 INFO:tasks.workunit.client.0.smithi153.stdout:8/290: dread - d7/d2a/f2d zero size 2024-04-05T03:02:47.239 INFO:tasks.workunit.client.0.smithi153.stdout:2/331: creat d9/da/d19/d2e/d48/f6b x:0 0 0 2024-04-05T03:02:47.239 INFO:tasks.workunit.client.1.smithi203.stdout:0/669: getdents dd/db1 0 2024-04-05T03:02:47.241 INFO:tasks.workunit.client.1.smithi203.stdout:7/848: creat d8/da/d71/f109 x:0 0 0 2024-04-05T03:02:47.242 INFO:tasks.workunit.client.0.smithi153.stdout:2/332: rename d9/da/d19/c30 to d9/da/d19/c6c 0 2024-04-05T03:02:47.243 INFO:tasks.workunit.client.1.smithi203.stdout:0/670: dread dd/d17/d1d/d22/d53/f63 [0,4194304] 0 2024-04-05T03:02:47.244 INFO:tasks.workunit.client.0.smithi153.stdout:8/291: mknod d7/d13/d19/c5e 0 2024-04-05T03:02:47.245 INFO:tasks.workunit.client.0.smithi153.stdout:8/292: write d7/d13/d19/f22 [811446,28106] 0 2024-04-05T03:02:47.245 INFO:tasks.workunit.client.1.smithi203.stdout:3/534: sync 2024-04-05T03:02:47.245 INFO:tasks.workunit.client.0.smithi153.stdout:8/293: chown d7/d46 99696183 1 2024-04-05T03:02:47.247 INFO:tasks.workunit.client.0.smithi153.stdout:2/333: fsync d9/da/d19/f6a 0 2024-04-05T03:02:47.247 INFO:tasks.workunit.client.1.smithi203.stdout:3/535: write d0/dc/d1a/d1d/d53/f98 [474495,53676] 0 2024-04-05T03:02:47.248 INFO:tasks.workunit.client.0.smithi153.stdout:8/294: chown d7/f23 28 1 2024-04-05T03:02:47.248 INFO:tasks.workunit.client.1.smithi203.stdout:7/849: dwrite d8/d32/f43 [0,4194304] 0 2024-04-05T03:02:47.249 INFO:tasks.workunit.client.1.smithi203.stdout:3/536: dread - d0/d69/f88 zero size 2024-04-05T03:02:47.250 INFO:tasks.workunit.client.0.smithi153.stdout:2/334: mkdir d9/da/d19/d2e/d60/d54/d6d 0 2024-04-05T03:02:47.254 INFO:tasks.workunit.client.0.smithi153.stdout:8/295: rename d7/d2a/d52 to d7/d13/d19/d1b/d4f/d5f 0 2024-04-05T03:02:47.259 INFO:tasks.workunit.client.0.smithi153.stdout:5/330: write d0/d9/f18 [2237806,51950] 0 2024-04-05T03:02:47.262 INFO:tasks.workunit.client.0.smithi153.stdout:7/314: dwrite d2/da/d53/f5b [0,4194304] 0 2024-04-05T03:02:47.270 INFO:tasks.workunit.client.0.smithi153.stdout:3/285: dwrite d1d/d27/d2b/f2d [0,4194304] 0 2024-04-05T03:02:47.270 INFO:tasks.workunit.client.0.smithi153.stdout:5/331: dread d0/d25/f31 [0,4194304] 0 2024-04-05T03:02:47.272 INFO:tasks.workunit.client.0.smithi153.stdout:7/315: symlink d2/da/d11/d39/l66 0 2024-04-05T03:02:47.273 INFO:tasks.workunit.client.0.smithi153.stdout:8/296: dwrite d7/d13/d19/f5c [0,4194304] 0 2024-04-05T03:02:47.275 INFO:tasks.workunit.client.0.smithi153.stdout:5/332: stat d0/d25/d4d/d62/l55 0 2024-04-05T03:02:47.275 INFO:tasks.workunit.client.0.smithi153.stdout:1/283: dwrite d5/dd/d1b/f2b [0,4194304] 0 2024-04-05T03:02:47.276 INFO:tasks.workunit.client.0.smithi153.stdout:3/286: chown d1d/d26/d3c/c50 52509593 1 2024-04-05T03:02:47.278 INFO:tasks.workunit.client.0.smithi153.stdout:5/333: readlink d0/d25/d4d/d62/l54 0 2024-04-05T03:02:47.278 INFO:tasks.workunit.client.0.smithi153.stdout:8/297: mknod d7/d13/d19/d1b/d4f/c60 0 2024-04-05T03:02:47.281 INFO:tasks.workunit.client.0.smithi153.stdout:8/298: read d7/f2b [61509,120010] 0 2024-04-05T03:02:47.282 INFO:tasks.workunit.client.0.smithi153.stdout:3/287: symlink d1d/d26/d57/l6c 0 2024-04-05T03:02:47.283 INFO:tasks.workunit.client.0.smithi153.stdout:8/299: stat d7/d13/d19/l27 0 2024-04-05T03:02:47.284 INFO:tasks.workunit.client.0.smithi153.stdout:3/288: read - d1d/d32/f56 zero size 2024-04-05T03:02:47.285 INFO:tasks.workunit.client.0.smithi153.stdout:1/284: dread d5/d8/d20/d4b/d1e/f4f [0,4194304] 0 2024-04-05T03:02:47.286 INFO:tasks.workunit.client.0.smithi153.stdout:2/335: dread d9/da/f15 [0,4194304] 0 2024-04-05T03:02:47.286 INFO:tasks.workunit.client.0.smithi153.stdout:7/316: dread d2/da/d11/d49/f3e [0,4194304] 0 2024-04-05T03:02:47.286 INFO:tasks.workunit.client.0.smithi153.stdout:1/285: readlink d5/d8/l2e 0 2024-04-05T03:02:47.286 INFO:tasks.workunit.client.0.smithi153.stdout:5/334: rename d0/d9/c65 to d0/d9/d22/c7d 0 2024-04-05T03:02:47.288 INFO:tasks.workunit.client.0.smithi153.stdout:8/300: symlink d7/d13/d19/d1b/d33/l61 0 2024-04-05T03:02:47.288 INFO:tasks.workunit.client.0.smithi153.stdout:5/335: readlink d0/d25/l79 0 2024-04-05T03:02:47.290 INFO:tasks.workunit.client.0.smithi153.stdout:7/317: symlink d2/da/d42/l67 0 2024-04-05T03:02:47.292 INFO:tasks.workunit.client.0.smithi153.stdout:8/301: chown d7/ff 180466460 1 2024-04-05T03:02:47.292 INFO:tasks.workunit.client.0.smithi153.stdout:3/289: dwrite d1d/f29 [0,4194304] 0 2024-04-05T03:02:47.293 INFO:tasks.workunit.client.0.smithi153.stdout:1/286: symlink d5/d8/d20/d4b/d51/l53 0 2024-04-05T03:02:47.294 INFO:tasks.workunit.client.0.smithi153.stdout:5/336: unlink d0/d25/d4d/d62/d37/d39/c7b 0 2024-04-05T03:02:47.296 INFO:tasks.workunit.client.0.smithi153.stdout:7/318: read d2/da/d11/f24 [3866484,69077] 0 2024-04-05T03:02:47.297 INFO:tasks.workunit.client.0.smithi153.stdout:2/336: creat d9/da/d19/d2e/d60/d54/d6d/f6e x:0 0 0 2024-04-05T03:02:47.297 INFO:tasks.workunit.client.0.smithi153.stdout:7/319: stat d2/da/d11/f13 0 2024-04-05T03:02:47.298 INFO:tasks.workunit.client.0.smithi153.stdout:3/290: dwrite f18 [0,4194304] 0 2024-04-05T03:02:47.300 INFO:tasks.workunit.client.0.smithi153.stdout:7/320: dread d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:47.301 INFO:tasks.workunit.client.0.smithi153.stdout:8/302: rename d7/d13/d19/d1b/d21/f39 to d7/d13/f62 0 2024-04-05T03:02:47.302 INFO:tasks.workunit.client.0.smithi153.stdout:1/287: creat d5/d8/d20/d4b/d18/d16/d45/f54 x:0 0 0 2024-04-05T03:02:47.304 INFO:tasks.workunit.client.0.smithi153.stdout:5/337: unlink d0/d2c/d3f/c73 0 2024-04-05T03:02:47.305 INFO:tasks.workunit.client.0.smithi153.stdout:3/291: mknod d1d/d32/c6d 0 2024-04-05T03:02:47.309 INFO:tasks.workunit.client.0.smithi153.stdout:1/288: mknod d5/d8/d20/d2f/c55 0 2024-04-05T03:02:47.309 INFO:tasks.workunit.client.0.smithi153.stdout:8/303: rename d7/d13/d19/d31/f42 to d7/d13/d19/d1b/d33/f63 0 2024-04-05T03:02:47.311 INFO:tasks.workunit.client.0.smithi153.stdout:5/338: write d0/f75 [98229,54654] 0 2024-04-05T03:02:47.312 INFO:tasks.workunit.client.0.smithi153.stdout:3/292: readlink d1d/l24 0 2024-04-05T03:02:47.312 INFO:tasks.workunit.client.0.smithi153.stdout:5/339: write d0/d25/f2b [1721852,5310] 0 2024-04-05T03:02:47.314 INFO:tasks.workunit.client.0.smithi153.stdout:1/289: creat d5/dd/d1b/f56 x:0 0 0 2024-04-05T03:02:47.314 INFO:tasks.workunit.client.0.smithi153.stdout:0/360: sync 2024-04-05T03:02:47.315 INFO:tasks.workunit.client.0.smithi153.stdout:8/304: creat d7/d13/f64 x:0 0 0 2024-04-05T03:02:47.316 INFO:tasks.workunit.client.0.smithi153.stdout:3/293: chown l1b 1 1 2024-04-05T03:02:47.317 INFO:tasks.workunit.client.0.smithi153.stdout:5/340: creat d0/d25/d4d/d62/d5f/f7e x:0 0 0 2024-04-05T03:02:47.318 INFO:tasks.workunit.client.0.smithi153.stdout:7/321: creat d2/da/f68 x:0 0 0 2024-04-05T03:02:47.318 INFO:tasks.workunit.client.0.smithi153.stdout:2/337: getdents d9/da/d19/d2e/d4f 0 2024-04-05T03:02:47.319 INFO:tasks.workunit.client.0.smithi153.stdout:7/322: write d2/da/f68 [6691,75851] 0 2024-04-05T03:02:47.320 INFO:tasks.workunit.client.0.smithi153.stdout:1/290: rmdir d5/d8/d20/d2f/d40 39 2024-04-05T03:02:47.323 INFO:tasks.workunit.client.0.smithi153.stdout:2/338: mknod d9/da/d19/d2e/d60/c6f 0 2024-04-05T03:02:47.323 INFO:tasks.workunit.client.0.smithi153.stdout:8/305: write d7/d13/d19/d1b/d33/f63 [476955,95023] 0 2024-04-05T03:02:47.323 INFO:tasks.workunit.client.0.smithi153.stdout:3/294: write d1d/d27/d2b/f35 [250078,70571] 0 2024-04-05T03:02:47.325 INFO:tasks.workunit.client.0.smithi153.stdout:1/291: mknod d5/d8/d20/c57 0 2024-04-05T03:02:47.326 INFO:tasks.workunit.client.0.smithi153.stdout:3/295: write d1d/f29 [2993596,28527] 0 2024-04-05T03:02:47.326 INFO:tasks.workunit.client.0.smithi153.stdout:2/339: chown d9/da/d37/c4a 0 1 2024-04-05T03:02:47.327 INFO:tasks.workunit.client.0.smithi153.stdout:0/361: creat d9/db/f66 x:0 0 0 2024-04-05T03:02:47.328 INFO:tasks.workunit.client.0.smithi153.stdout:5/341: dwrite d0/d25/d4d/d62/d37/f4b [0,4194304] 0 2024-04-05T03:02:47.331 INFO:tasks.workunit.client.0.smithi153.stdout:5/342: write d0/d9/d22/f28 [400177,28745] 0 2024-04-05T03:02:47.331 INFO:tasks.workunit.client.0.smithi153.stdout:2/340: truncate d9/d68/d39/f58 13744 0 2024-04-05T03:02:47.332 INFO:tasks.workunit.client.0.smithi153.stdout:8/306: creat d7/d2a/f65 x:0 0 0 2024-04-05T03:02:47.334 INFO:tasks.workunit.client.0.smithi153.stdout:0/362: mkdir d9/d14/d67 0 2024-04-05T03:02:47.335 INFO:tasks.workunit.client.1.smithi203.stdout:3/537: creat d0/d41/f9a x:0 0 0 2024-04-05T03:02:47.336 INFO:tasks.workunit.client.1.smithi203.stdout:0/671: link dd/c10 dd/d17/d1d/d26/d34/d56/cc3 0 2024-04-05T03:02:47.340 INFO:tasks.workunit.client.0.smithi153.stdout:3/296: dwrite d1d/d32/f49 [0,4194304] 0 2024-04-05T03:02:47.342 INFO:tasks.workunit.client.1.smithi203.stdout:3/538: rename d0/d21/d81/f58 to d0/dc/d1a/d1d/f9b 0 2024-04-05T03:02:47.343 INFO:tasks.workunit.client.0.smithi153.stdout:1/292: dwrite d5/d8/d20/d4b/d18/d16/d45/f4a [0,4194304] 0 2024-04-05T03:02:47.344 INFO:tasks.workunit.client.1.smithi203.stdout:0/672: mkdir dd/d17/d1d/d26/d8c/d6d/dc4 0 2024-04-05T03:02:47.346 INFO:tasks.workunit.client.0.smithi153.stdout:8/307: dwrite d7/d13/f14 [0,4194304] 0 2024-04-05T03:02:47.348 INFO:tasks.workunit.client.1.smithi203.stdout:9/654: dread d13/f1d [0,4194304] 0 2024-04-05T03:02:47.348 INFO:tasks.workunit.client.1.smithi203.stdout:3/539: creat d0/dc/d1a/d8d/f9c x:0 0 0 2024-04-05T03:02:47.349 INFO:tasks.workunit.client.0.smithi153.stdout:0/363: mkdir d9/d68 0 2024-04-05T03:02:47.350 INFO:tasks.workunit.client.0.smithi153.stdout:0/364: chown d9/df/d1e 4672 1 2024-04-05T03:02:47.350 INFO:tasks.workunit.client.1.smithi203.stdout:7/850: getdents d8/d32/d16/d1a/d3d/d59/de7/d73/da8 0 2024-04-05T03:02:47.351 INFO:tasks.workunit.client.0.smithi153.stdout:0/365: truncate d9/d14/f3f 4757707 0 2024-04-05T03:02:47.353 INFO:tasks.workunit.client.1.smithi203.stdout:0/673: rename dd/d17/d1d/d75 to dd/d17/d1d/d22/d53/d7f/dc5 0 2024-04-05T03:02:47.354 INFO:tasks.workunit.client.0.smithi153.stdout:0/366: chown d9/df/d1e/l2c 12347135 1 2024-04-05T03:02:47.355 INFO:tasks.workunit.client.1.smithi203.stdout:9/655: creat d13/d1b/d6b/d9e/d59/d9b/fd9 x:0 0 0 2024-04-05T03:02:47.356 INFO:tasks.workunit.client.0.smithi153.stdout:1/293: mknod d5/dd/c58 0 2024-04-05T03:02:47.358 INFO:tasks.workunit.client.1.smithi203.stdout:7/851: mkdir d8/da/d71/d10a 0 2024-04-05T03:02:47.358 INFO:tasks.workunit.client.0.smithi153.stdout:1/294: write d5/d8/d20/d4b/d18/f37 [4395703,46156] 0 2024-04-05T03:02:47.358 INFO:tasks.workunit.client.0.smithi153.stdout:2/341: link d9/d68/l47 d9/da/d19/d2e/d4f/l70 0 2024-04-05T03:02:47.360 INFO:tasks.workunit.client.0.smithi153.stdout:1/295: chown d5/d8/d20/d3f 23055588 1 2024-04-05T03:02:47.360 INFO:tasks.workunit.client.1.smithi203.stdout:0/674: rename dd/d17/d1d/d26/d34/l60 to dd/d17/d1d/d22/d53/d55/lc6 0 2024-04-05T03:02:47.362 INFO:tasks.workunit.client.0.smithi153.stdout:5/343: getdents d0/d9/d22 0 2024-04-05T03:02:47.362 INFO:tasks.workunit.client.0.smithi153.stdout:1/296: readlink d5/dd/d1b/l4e 0 2024-04-05T03:02:47.363 INFO:tasks.workunit.client.0.smithi153.stdout:1/297: stat d5/d8/d20/f38 0 2024-04-05T03:02:47.365 INFO:tasks.workunit.client.0.smithi153.stdout:2/342: read d9/da/d19/d2e/f42 [3231064,46196] 0 2024-04-05T03:02:47.367 INFO:tasks.workunit.client.0.smithi153.stdout:2/343: write d9/da/d19/f1c [4638979,51708] 0 2024-04-05T03:02:47.367 INFO:tasks.workunit.client.0.smithi153.stdout:7/323: dwrite d2/da/d11/d49/f20 [0,4194304] 0 2024-04-05T03:02:47.370 INFO:tasks.workunit.client.1.smithi203.stdout:5/660: sync 2024-04-05T03:02:47.371 INFO:tasks.workunit.client.1.smithi203.stdout:9/656: symlink d13/d5c/dd1/d31/lda 0 2024-04-05T03:02:47.374 INFO:tasks.workunit.client.1.smithi203.stdout:9/657: symlink d13/d5c/d9d/ldb 0 2024-04-05T03:02:47.375 INFO:tasks.workunit.client.0.smithi153.stdout:3/297: rmdir d1d/d5e 0 2024-04-05T03:02:47.376 INFO:tasks.workunit.client.1.smithi203.stdout:7/852: dwrite d8/d32/d16/d1a/d3d/fce [0,4194304] 0 2024-04-05T03:02:47.377 INFO:tasks.workunit.client.1.smithi203.stdout:9/658: read - d13/d1b/fbd zero size 2024-04-05T03:02:47.380 INFO:tasks.workunit.client.0.smithi153.stdout:5/344: symlink d0/d25/d4d/l7f 0 2024-04-05T03:02:47.380 INFO:tasks.workunit.client.1.smithi203.stdout:9/659: write f12 [2234479,99390] 0 2024-04-05T03:02:47.384 INFO:tasks.workunit.client.1.smithi203.stdout:9/660: mkdir d13/d1b/d6b/d7a/ddc 0 2024-04-05T03:02:47.386 INFO:tasks.workunit.client.0.smithi153.stdout:3/298: symlink d1d/d26/l6e 0 2024-04-05T03:02:47.387 INFO:tasks.workunit.client.0.smithi153.stdout:0/367: creat d9/f69 x:0 0 0 2024-04-05T03:02:47.389 INFO:tasks.workunit.client.0.smithi153.stdout:2/344: mkdir d9/d71 0 2024-04-05T03:02:47.390 INFO:tasks.workunit.client.1.smithi203.stdout:9/661: unlink d13/d1b/d33/d42/d67/d82/c92 0 2024-04-05T03:02:47.390 INFO:tasks.workunit.client.0.smithi153.stdout:3/299: creat d1d/d27/f6f x:0 0 0 2024-04-05T03:02:47.391 INFO:tasks.workunit.client.0.smithi153.stdout:3/300: stat f18 0 2024-04-05T03:02:47.393 INFO:tasks.workunit.client.0.smithi153.stdout:1/298: getdents d5 0 2024-04-05T03:02:47.394 INFO:tasks.workunit.client.0.smithi153.stdout:3/301: stat l13 0 2024-04-05T03:02:47.395 INFO:tasks.workunit.client.0.smithi153.stdout:1/299: fdatasync d5/f7 0 2024-04-05T03:02:47.395 INFO:tasks.workunit.client.1.smithi203.stdout:9/662: truncate d13/d1b/d6b/d9e/d59/f7e 486853 0 2024-04-05T03:02:47.398 INFO:tasks.workunit.client.1.smithi203.stdout:9/663: creat d13/d1b/d6b/d9e/d59/dbc/fdd x:0 0 0 2024-04-05T03:02:47.399 INFO:tasks.workunit.client.1.smithi203.stdout:3/540: write d0/dc/d38/d5d/f62 [63531,26406] 0 2024-04-05T03:02:47.399 INFO:tasks.workunit.client.1.smithi203.stdout:9/664: chown d13/d1b/d33/d64/f75 44 1 2024-04-05T03:02:47.399 INFO:tasks.workunit.client.0.smithi153.stdout:3/302: write d1d/d27/d2b/f2f [113676,3405] 0 2024-04-05T03:02:47.400 INFO:tasks.workunit.client.0.smithi153.stdout:7/324: dwrite d2/da/d11/d49/f3e [0,4194304] 0 2024-04-05T03:02:47.401 INFO:tasks.workunit.client.0.smithi153.stdout:0/368: link d9/d14/f40 d9/df/d1e/d45/f6a 0 2024-04-05T03:02:47.402 INFO:tasks.workunit.client.0.smithi153.stdout:5/345: dwrite d0/d2c/d3f/f59 [4194304,4194304] 0 2024-04-05T03:02:47.403 INFO:tasks.workunit.client.1.smithi203.stdout:5/661: write d3/d4/dc/d67/f7f [3016354,108833] 0 2024-04-05T03:02:47.405 INFO:tasks.workunit.client.0.smithi153.stdout:2/345: rename d9/da/d19/d2e/d4f/l70 to d9/da/d19/d2e/d4f/l72 0 2024-04-05T03:02:47.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/675: dread dd/d17/d1d/d22/fac [0,4194304] 0 2024-04-05T03:02:47.407 INFO:tasks.workunit.client.1.smithi203.stdout:7/853: dread d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:47.410 INFO:tasks.workunit.client.1.smithi203.stdout:9/665: dwrite d13/d5c/d9d/fcd [0,4194304] 0 2024-04-05T03:02:47.410 INFO:tasks.workunit.client.0.smithi153.stdout:7/325: creat d2/da/d47/d5d/f69 x:0 0 0 2024-04-05T03:02:47.411 INFO:tasks.workunit.client.1.smithi203.stdout:5/662: write d3/d1f/d59/d64/f63 [884359,57344] 0 2024-04-05T03:02:47.411 INFO:tasks.workunit.client.1.smithi203.stdout:3/541: link d0/dc/d38/d5d/d66/d80/f97 d0/dc/d1a/d1f/f9d 0 2024-04-05T03:02:47.411 INFO:tasks.workunit.client.0.smithi153.stdout:7/326: chown d2/da/d47/d5d/f69 318326925 1 2024-04-05T03:02:47.412 INFO:tasks.workunit.client.1.smithi203.stdout:0/676: write f2 [739205,43569] 0 2024-04-05T03:02:47.412 INFO:tasks.workunit.client.1.smithi203.stdout:7/854: dread - d8/d32/d16/d1a/d3d/d59/de7/d5e/d63/f75 zero size 2024-04-05T03:02:47.412 INFO:tasks.workunit.client.0.smithi153.stdout:1/300: dwrite d5/d8/d20/d4b/d18/f1a [0,4194304] 0 2024-04-05T03:02:47.413 INFO:tasks.workunit.client.0.smithi153.stdout:5/346: creat d0/d25/d4d/d62/f80 x:0 0 0 2024-04-05T03:02:47.414 INFO:tasks.workunit.client.1.smithi203.stdout:7/855: write d8/da/fd [3209456,92027] 0 2024-04-05T03:02:47.414 INFO:tasks.workunit.client.0.smithi153.stdout:5/347: write d0/f27 [1119599,75714] 0 2024-04-05T03:02:47.415 INFO:tasks.workunit.client.0.smithi153.stdout:2/346: mknod d9/d68/d4b/c73 0 2024-04-05T03:02:47.419 INFO:tasks.workunit.client.1.smithi203.stdout:5/663: stat d3/d4/dc/lba 0 2024-04-05T03:02:47.419 INFO:tasks.workunit.client.1.smithi203.stdout:9/666: dread d13/d1b/d33/d64/f9c [0,4194304] 0 2024-04-05T03:02:47.421 INFO:tasks.workunit.client.1.smithi203.stdout:7/856: chown d8/d32/d5d/d6d/f8f 16567 1 2024-04-05T03:02:47.425 INFO:tasks.workunit.client.1.smithi203.stdout:5/664: mknod d3/d1f/ce2 0 2024-04-05T03:02:47.425 INFO:tasks.workunit.client.0.smithi153.stdout:7/327: rename d2/da/d53 to d2/d51/d6a 0 2024-04-05T03:02:47.427 INFO:tasks.workunit.client.0.smithi153.stdout:3/303: link d1d/f6a d1d/f70 0 2024-04-05T03:02:47.428 INFO:tasks.workunit.client.1.smithi203.stdout:7/857: creat d8/d32/d16/d1a/d3d/d59/de3/f10b x:0 0 0 2024-04-05T03:02:47.430 INFO:tasks.workunit.client.1.smithi203.stdout:3/542: truncate d0/dc/d1a/d8d/d96/f6a 6363798 0 2024-04-05T03:02:47.430 INFO:tasks.workunit.client.1.smithi203.stdout:9/667: creat d13/d5c/d9d/fde x:0 0 0 2024-04-05T03:02:47.431 INFO:tasks.workunit.client.1.smithi203.stdout:3/543: read - d0/d21/d81/f7b zero size 2024-04-05T03:02:47.433 INFO:tasks.workunit.client.0.smithi153.stdout:7/328: mknod d2/da/d11/d49/d54/c6b 0 2024-04-05T03:02:47.433 INFO:tasks.workunit.client.1.smithi203.stdout:9/668: dread - d13/d1b/d6b/f72 zero size 2024-04-05T03:02:47.433 INFO:tasks.workunit.client.1.smithi203.stdout:7/858: write d8/d32/d5d/d6d/dca/d9b/fba [159934,73211] 0 2024-04-05T03:02:47.434 INFO:tasks.workunit.client.1.smithi203.stdout:5/665: dwrite d3/d4/f71 [0,4194304] 0 2024-04-05T03:02:47.435 INFO:tasks.workunit.client.0.smithi153.stdout:3/304: dwrite fd [4194304,4194304] 0 2024-04-05T03:02:47.435 INFO:tasks.workunit.client.0.smithi153.stdout:0/369: dread d9/db/f49 [0,4194304] 0 2024-04-05T03:02:47.436 INFO:tasks.workunit.client.1.smithi203.stdout:7/859: truncate d8/da/d72/f102 521813 0 2024-04-05T03:02:47.437 INFO:tasks.workunit.client.0.smithi153.stdout:0/370: dread - d9/d14/f53 zero size 2024-04-05T03:02:47.438 INFO:tasks.workunit.client.0.smithi153.stdout:7/329: unlink d2/c26 0 2024-04-05T03:02:47.438 INFO:tasks.workunit.client.0.smithi153.stdout:0/371: dread - d9/df/d1e/f50 zero size 2024-04-05T03:02:47.439 INFO:tasks.workunit.client.1.smithi203.stdout:5/666: symlink d3/d4/dc/d67/le3 0 2024-04-05T03:02:47.439 INFO:tasks.workunit.client.0.smithi153.stdout:5/348: link d0/c29 d0/d25/d4d/d62/d37/d39/c81 0 2024-04-05T03:02:47.441 INFO:tasks.workunit.client.1.smithi203.stdout:9/669: dread d13/d1b/d6b/d9e/d77/f7d [0,4194304] 0 2024-04-05T03:02:47.443 INFO:tasks.workunit.client.0.smithi153.stdout:3/305: rename d1d/d32/f4a to d1d/d27/d2b/f71 0 2024-04-05T03:02:47.444 INFO:tasks.workunit.client.0.smithi153.stdout:0/372: write d9/db/f11 [2547438,18443] 0 2024-04-05T03:02:47.445 INFO:tasks.workunit.client.1.smithi203.stdout:9/670: rmdir d13/d5c/dd1/d51/d81 39 2024-04-05T03:02:47.445 INFO:tasks.workunit.client.1.smithi203.stdout:9/671: chown c11 2365131 1 2024-04-05T03:02:47.447 INFO:tasks.workunit.client.1.smithi203.stdout:9/672: readlink d13/d5c/d9d/ldb 0 2024-04-05T03:02:47.447 INFO:tasks.workunit.client.0.smithi153.stdout:7/330: creat d2/da/d3a/d41/f6c x:0 0 0 2024-04-05T03:02:47.448 INFO:tasks.workunit.client.0.smithi153.stdout:5/349: mknod d0/d25/d4d/d62/d37/c82 0 2024-04-05T03:02:47.450 INFO:tasks.workunit.client.1.smithi203.stdout:7/860: link d8/d32/d16/d1a/d3d/d59/de7/d52/d65/cab d8/da/d72/c10c 0 2024-04-05T03:02:47.454 INFO:tasks.workunit.client.1.smithi203.stdout:7/861: symlink d8/d32/d16/d1a/d3d/d59/de7/d73/da8/l10d 0 2024-04-05T03:02:47.454 INFO:tasks.workunit.client.1.smithi203.stdout:9/673: dwrite d13/d1b/d6b/d9e/d59/d5f/f71 [0,4194304] 0 2024-04-05T03:02:47.455 INFO:tasks.workunit.client.0.smithi153.stdout:3/306: dwrite fd [0,4194304] 0 2024-04-05T03:02:47.456 INFO:tasks.workunit.client.1.smithi203.stdout:7/862: stat d8/d32/d16/d1a/d3d/d59/de7/d73/da8/fc0 0 2024-04-05T03:02:47.457 INFO:tasks.workunit.client.0.smithi153.stdout:3/307: dread - d1d/d32/f56 zero size 2024-04-05T03:02:47.459 INFO:tasks.workunit.client.0.smithi153.stdout:1/301: truncate d5/d8/ff 1961819 0 2024-04-05T03:02:47.459 INFO:tasks.workunit.client.1.smithi203.stdout:9/674: symlink d13/d5c/dd1/d51/d81/ldf 0 2024-04-05T03:02:47.466 INFO:tasks.workunit.client.0.smithi153.stdout:3/308: chown d1d/d32/f53 2001409 1 2024-04-05T03:02:47.466 INFO:tasks.workunit.client.0.smithi153.stdout:0/373: dwrite f0 [0,4194304] 0 2024-04-05T03:02:47.468 INFO:tasks.workunit.client.0.smithi153.stdout:7/331: rename d2/da/d11/d49 to d2/da/d42/d6d 0 2024-04-05T03:02:47.473 INFO:tasks.workunit.client.1.smithi203.stdout:9/675: truncate d13/d1b/d33/d63/f6e 1336076 0 2024-04-05T03:02:47.473 INFO:tasks.workunit.client.0.smithi153.stdout:1/302: write d5/d8/d20/d4b/d18/d16/d33/f3c [1150066,115188] 0 2024-04-05T03:02:47.474 INFO:tasks.workunit.client.0.smithi153.stdout:7/332: dread - d2/da/d42/d6d/f38 zero size 2024-04-05T03:02:47.476 INFO:tasks.workunit.client.0.smithi153.stdout:2/347: dwrite d9/ff [8388608,4194304] 0 2024-04-05T03:02:47.476 INFO:tasks.workunit.client.0.smithi153.stdout:7/333: chown d2/da/d42/d6d/f38 487500 1 2024-04-05T03:02:47.481 INFO:tasks.workunit.client.0.smithi153.stdout:5/350: dwrite d0/d2c/d3f/f59 [8388608,4194304] 0 2024-04-05T03:02:47.481 INFO:tasks.workunit.client.1.smithi203.stdout:0/677: dread dd/f44 [0,4194304] 0 2024-04-05T03:02:47.485 INFO:tasks.workunit.client.1.smithi203.stdout:5/667: dread d3/d1f/f3b [0,4194304] 0 2024-04-05T03:02:47.486 INFO:tasks.workunit.client.1.smithi203.stdout:7/863: write d8/fe [3404717,14231] 0 2024-04-05T03:02:47.487 INFO:tasks.workunit.client.1.smithi203.stdout:0/678: creat dd/d17/d1d/d22/d53/d7f/fc7 x:0 0 0 2024-04-05T03:02:47.487 INFO:tasks.workunit.client.0.smithi153.stdout:1/303: write d5/dd/d1b/f1d [8766,56767] 0 2024-04-05T03:02:47.488 INFO:tasks.workunit.client.0.smithi153.stdout:2/348: dread - d9/da/f64 zero size 2024-04-05T03:02:47.490 INFO:tasks.workunit.client.0.smithi153.stdout:0/374: dwrite d9/d14/f53 [0,4194304] 0 2024-04-05T03:02:47.494 INFO:tasks.workunit.client.1.smithi203.stdout:5/668: dwrite d3/d4/d9e/f9f [0,4194304] 0 2024-04-05T03:02:47.500 INFO:tasks.workunit.client.1.smithi203.stdout:5/669: chown d3/d1f/d59/d64/f23 31728 1 2024-04-05T03:02:47.500 INFO:tasks.workunit.client.1.smithi203.stdout:0/679: dwrite dd/f44 [0,4194304] 0 2024-04-05T03:02:47.502 INFO:tasks.workunit.client.0.smithi153.stdout:1/304: rename d5/dd/c1f to d5/d8/d20/d4b/d1e/c59 0 2024-04-05T03:02:47.507 INFO:tasks.workunit.client.1.smithi203.stdout:9/676: write d13/d1b/d6b/f72 [946194,88535] 0 2024-04-05T03:02:47.510 INFO:tasks.workunit.client.0.smithi153.stdout:1/305: readlink d5/d8/d20/d4b/d51/l53 0 2024-04-05T03:02:47.513 INFO:tasks.workunit.client.0.smithi153.stdout:5/351: dwrite d0/d25/d4d/d62/f10 [0,4194304] 0 2024-04-05T03:02:47.514 INFO:tasks.workunit.client.1.smithi203.stdout:0/680: rmdir dd/dbd 0 2024-04-05T03:02:47.515 INFO:tasks.workunit.client.0.smithi153.stdout:3/309: truncate f3 399404 0 2024-04-05T03:02:47.520 INFO:tasks.workunit.client.1.smithi203.stdout:0/681: symlink dd/d17/d1d/d22/d80/lc8 0 2024-04-05T03:02:47.523 INFO:tasks.workunit.client.0.smithi153.stdout:2/349: dwrite d9/da/d19/f1f [8388608,4194304] 0 2024-04-05T03:02:47.524 INFO:tasks.workunit.client.1.smithi203.stdout:0/682: creat dd/d17/d1d/d26/d8c/fc9 x:0 0 0 2024-04-05T03:02:47.527 INFO:tasks.workunit.client.1.smithi203.stdout:3/544: dread d0/dc/d1a/d8d/d96/f6a [4194304,4194304] 0 2024-04-05T03:02:47.530 INFO:tasks.workunit.client.1.smithi203.stdout:3/545: creat d0/dc/d38/f9e x:0 0 0 2024-04-05T03:02:47.533 INFO:tasks.workunit.client.1.smithi203.stdout:7/864: dread d8/d32/d16/d1a/d3d/f6b [0,4194304] 0 2024-04-05T03:02:47.533 INFO:tasks.workunit.client.1.smithi203.stdout:3/546: creat d0/dc/d1a/d76/f9f x:0 0 0 2024-04-05T03:02:47.534 INFO:tasks.workunit.client.0.smithi153.stdout:2/350: truncate d9/da/d19/d2e/f43 4421686 0 2024-04-05T03:02:47.535 INFO:tasks.workunit.client.1.smithi203.stdout:3/547: readlink d0/dc/d38/l79 0 2024-04-05T03:02:47.535 INFO:tasks.workunit.client.0.smithi153.stdout:1/306: link d5/d8/d20/c57 d5/d8/d20/d4b/d18/d16/d45/c5a 0 2024-04-05T03:02:47.535 INFO:tasks.workunit.client.0.smithi153.stdout:3/310: dwrite d1d/d26/d3c/d3f/f4f [0,4194304] 0 2024-04-05T03:02:47.536 INFO:tasks.workunit.client.1.smithi203.stdout:3/548: read - d0/dc/d1a/d1d/f72 zero size 2024-04-05T03:02:47.536 INFO:tasks.workunit.client.1.smithi203.stdout:9/677: write d13/f24 [946215,124090] 0 2024-04-05T03:02:47.538 INFO:tasks.workunit.client.0.smithi153.stdout:9/237: sync 2024-04-05T03:02:47.538 INFO:tasks.workunit.client.0.smithi153.stdout:6/366: sync 2024-04-05T03:02:47.538 INFO:tasks.workunit.client.0.smithi153.stdout:0/375: truncate f3 2941200 0 2024-04-05T03:02:47.538 INFO:tasks.workunit.client.0.smithi153.stdout:5/352: dwrite d0/d25/d4d/f66 [0,4194304] 0 2024-04-05T03:02:47.538 INFO:tasks.workunit.client.0.smithi153.stdout:7/334: dread d2/f9 [0,4194304] 0 2024-04-05T03:02:47.539 INFO:tasks.workunit.client.1.smithi203.stdout:7/865: creat d8/d32/d5d/d6d/dca/f10e x:0 0 0 2024-04-05T03:02:47.540 INFO:tasks.workunit.client.1.smithi203.stdout:7/866: write d8/d32/d16/d1a/d3d/d59/de7/d73/f7b [4416746,3371] 0 2024-04-05T03:02:47.542 INFO:tasks.workunit.client.1.smithi203.stdout:7/867: truncate d8/d32/d16/fe1 287704 0 2024-04-05T03:02:47.542 INFO:tasks.workunit.client.0.smithi153.stdout:5/353: readlink d0/d25/l79 0 2024-04-05T03:02:47.542 INFO:tasks.workunit.client.0.smithi153.stdout:2/351: chown d9/d68/d39 582626474 1 2024-04-05T03:02:47.542 INFO:tasks.workunit.client.0.smithi153.stdout:6/367: write d4/dc/df/d16/d17/f2a [2635240,24837] 0 2024-04-05T03:02:47.544 INFO:tasks.workunit.client.1.smithi203.stdout:7/868: write d8/d32/d16/d1a/d21/f62 [4397293,38689] 0 2024-04-05T03:02:47.546 INFO:tasks.workunit.client.0.smithi153.stdout:1/307: write d5/d8/d20/d4b/d18/f1c [2452905,89911] 0 2024-04-05T03:02:47.546 INFO:tasks.workunit.client.0.smithi153.stdout:1/308: read - d5/d8/d20/d4b/d18/d16/d45/f54 zero size 2024-04-05T03:02:47.547 INFO:tasks.workunit.client.1.smithi203.stdout:3/549: truncate d0/dc/d1a/d76/d35/f3d 1326978 0 2024-04-05T03:02:47.548 INFO:tasks.workunit.client.0.smithi153.stdout:5/354: unlink d0/d9/d22/f64 0 2024-04-05T03:02:47.549 INFO:tasks.workunit.client.0.smithi153.stdout:3/311: readlink d1d/l23 0 2024-04-05T03:02:47.549 INFO:tasks.workunit.client.0.smithi153.stdout:5/355: truncate d0/f14 4847214 0 2024-04-05T03:02:47.550 INFO:tasks.workunit.client.0.smithi153.stdout:3/312: dread - d1d/d27/d2b/f62 zero size 2024-04-05T03:02:47.552 INFO:tasks.workunit.client.0.smithi153.stdout:2/352: unlink d9/da/d19/f1c 0 2024-04-05T03:02:47.552 INFO:tasks.workunit.client.1.smithi203.stdout:3/550: symlink d0/dc/d38/d5d/la0 0 2024-04-05T03:02:47.554 INFO:tasks.workunit.client.0.smithi153.stdout:0/376: read d9/f30 [2882656,130016] 0 2024-04-05T03:02:47.556 INFO:tasks.workunit.client.1.smithi203.stdout:3/551: mkdir d0/dc/d1a/d1f/da1 0 2024-04-05T03:02:47.558 INFO:tasks.workunit.client.0.smithi153.stdout:3/313: dread f18 [0,4194304] 0 2024-04-05T03:02:47.559 INFO:tasks.workunit.client.0.smithi153.stdout:3/314: truncate d1d/f70 252312 0 2024-04-05T03:02:47.560 INFO:tasks.workunit.client.1.smithi203.stdout:7/869: dwrite d8/d32/d16/d1a/d3d/d59/de7/d73/f78 [0,4194304] 0 2024-04-05T03:02:47.561 INFO:tasks.workunit.client.1.smithi203.stdout:7/870: read - d8/d32/d5d/d95/fa2 zero size 2024-04-05T03:02:47.561 INFO:tasks.workunit.client.1.smithi203.stdout:3/552: dread d0/dc/d1a/d8d/d96/f6a [4194304,4194304] 0 2024-04-05T03:02:47.561 INFO:tasks.workunit.client.0.smithi153.stdout:5/356: symlink d0/d25/d4d/d62/d37/d39/l83 0 2024-04-05T03:02:47.562 INFO:tasks.workunit.client.1.smithi203.stdout:7/871: chown d8/d6c 97 1 2024-04-05T03:02:47.562 INFO:tasks.workunit.client.1.smithi203.stdout:7/872: stat d8/d32/cfa 0 2024-04-05T03:02:47.564 INFO:tasks.workunit.client.1.smithi203.stdout:3/553: mkdir d0/dc/d1a/d1f/da2 0 2024-04-05T03:02:47.565 INFO:tasks.workunit.client.0.smithi153.stdout:2/353: write d9/da/fb [2639738,20131] 0 2024-04-05T03:02:47.566 INFO:tasks.workunit.client.1.smithi203.stdout:7/873: mknod d8/d32/d16/d1a/d30/c10f 0 2024-04-05T03:02:47.569 INFO:tasks.workunit.client.0.smithi153.stdout:1/309: fsync f3 0 2024-04-05T03:02:47.573 INFO:tasks.workunit.client.1.smithi203.stdout:7/874: getdents d8/d32/d16/d35/d64 0 2024-04-05T03:02:47.574 INFO:tasks.workunit.client.0.smithi153.stdout:0/377: unlink d9/c10 0 2024-04-05T03:02:47.574 INFO:tasks.workunit.client.0.smithi153.stdout:2/354: dread - d9/da/d19/d2e/d48/f5b zero size 2024-04-05T03:02:47.574 INFO:tasks.workunit.client.0.smithi153.stdout:5/357: dread d0/f44 [0,4194304] 0 2024-04-05T03:02:47.576 INFO:tasks.workunit.client.0.smithi153.stdout:1/310: dread - d5/d8/d20/d2f/d40/f4c zero size 2024-04-05T03:02:47.577 INFO:tasks.workunit.client.1.smithi203.stdout:7/875: getdents d8/d32/d16/d1a/d3d 0 2024-04-05T03:02:47.578 INFO:tasks.workunit.client.0.smithi153.stdout:2/355: read d9/da/f24 [541972,35942] 0 2024-04-05T03:02:47.579 INFO:tasks.workunit.client.1.smithi203.stdout:7/876: mknod d8/d6c/c110 0 2024-04-05T03:02:47.580 INFO:tasks.workunit.client.1.smithi203.stdout:7/877: fdatasync d8/d32/d5d/d6d/f8f 0 2024-04-05T03:02:47.582 INFO:tasks.workunit.client.0.smithi153.stdout:3/315: creat d1d/f72 x:0 0 0 2024-04-05T03:02:47.583 INFO:tasks.workunit.client.1.smithi203.stdout:7/878: chown d8/d32/d16/d1a/d3d/d59/de7/d52/d65/d82/fc1 20755 1 2024-04-05T03:02:47.583 INFO:tasks.workunit.client.1.smithi203.stdout:9/678: dread d13/d1b/d6b/d9e/d59/f85 [0,4194304] 0 2024-04-05T03:02:47.583 INFO:tasks.workunit.client.0.smithi153.stdout:3/316: write d1d/d32/f56 [1045159,56314] 0 2024-04-05T03:02:47.585 INFO:tasks.workunit.client.1.smithi203.stdout:3/554: fsync d0/dc/d1a/d76/d35/f3d 0 2024-04-05T03:02:47.588 INFO:tasks.workunit.client.1.smithi203.stdout:3/555: read - d0/d21/d81/f5b zero size 2024-04-05T03:02:47.589 INFO:tasks.workunit.client.1.smithi203.stdout:3/556: fsync d0/d21/d81/f7b 0 2024-04-05T03:02:47.589 INFO:tasks.workunit.client.0.smithi153.stdout:2/356: mknod d9/da/d19/c74 0 2024-04-05T03:02:47.589 INFO:tasks.workunit.client.1.smithi203.stdout:3/557: readlink d0/l40 0 2024-04-05T03:02:47.590 INFO:tasks.workunit.client.0.smithi153.stdout:6/368: dwrite d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:47.590 INFO:tasks.workunit.client.1.smithi203.stdout:7/879: rename d8/d32/d16/d1a/d3d/d59/de3/ff1 to d8/d32/d16/d1a/d3d/d59/de3/f111 0 2024-04-05T03:02:47.591 INFO:tasks.workunit.client.1.smithi203.stdout:9/679: dwrite d13/d5c/d9d/fde [0,4194304] 0 2024-04-05T03:02:47.591 INFO:tasks.workunit.client.1.smithi203.stdout:3/558: chown d0/dc/d1a 47 1 2024-04-05T03:02:47.591 INFO:tasks.workunit.client.0.smithi153.stdout:3/317: creat d1d/d26/d3c/f73 x:0 0 0 2024-04-05T03:02:47.600 INFO:tasks.workunit.client.0.smithi153.stdout:5/358: write d0/d25/d4d/d62/f56 [1687714,73245] 0 2024-04-05T03:02:47.600 INFO:tasks.workunit.client.0.smithi153.stdout:6/369: rmdir d4/dc 39 2024-04-05T03:02:47.601 INFO:tasks.workunit.client.1.smithi203.stdout:7/880: dwrite d8/d32/d16/d35/d64/f8e [0,4194304] 0 2024-04-05T03:02:47.602 INFO:tasks.workunit.client.1.smithi203.stdout:3/559: getdents d0/d21/d81 0 2024-04-05T03:02:47.603 INFO:tasks.workunit.client.0.smithi153.stdout:2/357: unlink d9/da/d19/d2e/d48/c13 0 2024-04-05T03:02:47.605 INFO:tasks.workunit.client.1.smithi203.stdout:3/560: write d0/d21/d81/f5b [107473,2081] 0 2024-04-05T03:02:47.605 INFO:tasks.workunit.client.0.smithi153.stdout:6/370: chown d4/dc/df/c11 8695399 1 2024-04-05T03:02:47.608 INFO:tasks.workunit.client.0.smithi153.stdout:5/359: creat d0/d9/d22/f84 x:0 0 0 2024-04-05T03:02:47.611 INFO:tasks.workunit.client.0.smithi153.stdout:2/358: link d9/d68/d4b/c56 d9/da/d19/d2e/d4f/c75 0 2024-04-05T03:02:47.613 INFO:tasks.workunit.client.0.smithi153.stdout:1/311: dwrite d5/d8/d20/d4b/d18/d16/f22 [0,4194304] 0 2024-04-05T03:02:47.613 INFO:tasks.workunit.client.1.smithi203.stdout:7/881: dwrite d8/d32/d5d/d6d/dca/d9b/fba [0,4194304] 0 2024-04-05T03:02:47.615 INFO:tasks.workunit.client.0.smithi153.stdout:6/371: rename d4/dc/df/d16/c78 to d4/d38/d4c/c81 0 2024-04-05T03:02:47.618 INFO:tasks.workunit.client.0.smithi153.stdout:6/372: write d4/dc/df/d16/f23 [1726349,51815] 0 2024-04-05T03:02:47.618 INFO:tasks.workunit.client.1.smithi203.stdout:7/882: unlink d8/d32/cfa 0 2024-04-05T03:02:47.622 INFO:tasks.workunit.client.1.smithi203.stdout:7/883: dread d8/d32/d16/d1a/d21/f22 [0,4194304] 0 2024-04-05T03:02:47.624 INFO:tasks.workunit.client.1.smithi203.stdout:7/884: readlink d8/d32/d5d/d6d/daf/led 0 2024-04-05T03:02:47.624 INFO:tasks.workunit.client.0.smithi153.stdout:5/360: rename d0/d9/d22/f84 to d0/d25/d4d/d62/d5f/d61/f85 0 2024-04-05T03:02:47.625 INFO:tasks.workunit.client.1.smithi203.stdout:7/885: write d8/da/d72/fee [371090,99720] 0 2024-04-05T03:02:47.627 INFO:tasks.workunit.client.1.smithi203.stdout:3/561: write d0/dc/d1a/d76/d35/f3d [298331,62105] 0 2024-04-05T03:02:47.628 INFO:tasks.workunit.client.1.smithi203.stdout:3/562: stat d0/dc/d1a/d76/f94 0 2024-04-05T03:02:47.629 INFO:tasks.workunit.client.1.smithi203.stdout:3/563: readlink d0/dc/d1a/d1f/l6f 0 2024-04-05T03:02:47.629 INFO:tasks.workunit.client.1.smithi203.stdout:9/680: dread d13/d1b/d33/d42/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:47.630 INFO:tasks.workunit.client.0.smithi153.stdout:2/359: getdents d9/da/d19/d2e/d4f 0 2024-04-05T03:02:47.631 INFO:tasks.workunit.client.0.smithi153.stdout:2/360: truncate d9/da/f64 771504 0 2024-04-05T03:02:47.632 INFO:tasks.workunit.client.0.smithi153.stdout:3/318: dwrite d1d/d32/f49 [4194304,4194304] 0 2024-04-05T03:02:47.633 INFO:tasks.workunit.client.1.smithi203.stdout:3/564: rmdir d0/dc/d1a/d76 39 2024-04-05T03:02:47.634 INFO:tasks.workunit.client.0.smithi153.stdout:2/361: chown d9/da/d19/d2e/d60/d54/d6d/f6e 389722 1 2024-04-05T03:02:47.635 INFO:tasks.workunit.client.0.smithi153.stdout:3/319: creat d1d/d27/f74 x:0 0 0 2024-04-05T03:02:47.635 INFO:tasks.workunit.client.0.smithi153.stdout:6/373: link d4/ce d4/dc/df/d16/d36/c82 0 2024-04-05T03:02:47.641 INFO:tasks.workunit.client.1.smithi203.stdout:9/681: creat d13/d1b/d6b/fe0 x:0 0 0 2024-04-05T03:02:47.642 INFO:tasks.workunit.client.0.smithi153.stdout:1/312: fdatasync d5/d8/d20/d4b/d18/d16/f22 0 2024-04-05T03:02:47.643 INFO:tasks.workunit.client.0.smithi153.stdout:5/361: rename d0/d25/d4d/d62/d5f/d61 to d0/d25/d4d/d62/d5f/d86 0 2024-04-05T03:02:47.644 INFO:tasks.workunit.client.1.smithi203.stdout:7/886: dwrite d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:47.644 INFO:tasks.workunit.client.0.smithi153.stdout:6/374: dread d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:47.645 INFO:tasks.workunit.client.1.smithi203.stdout:7/887: chown d8/d6c/c110 766 1 2024-04-05T03:02:47.649 INFO:tasks.workunit.client.0.smithi153.stdout:6/375: chown d4/dc/df/f56 204666 1 2024-04-05T03:02:47.651 INFO:tasks.workunit.client.0.smithi153.stdout:3/320: dwrite d1d/d26/d3c/f44 [0,4194304] 0 2024-04-05T03:02:47.656 INFO:tasks.workunit.client.0.smithi153.stdout:2/362: dwrite d9/da/d19/d2e/d48/f1a [4194304,4194304] 0 2024-04-05T03:02:47.659 INFO:tasks.workunit.client.1.smithi203.stdout:3/565: truncate d0/dc/d1a/d1d/f29 3076526 0 2024-04-05T03:02:47.659 INFO:tasks.workunit.client.0.smithi153.stdout:5/362: dread d0/d2c/f3a [0,4194304] 0 2024-04-05T03:02:47.660 INFO:tasks.workunit.client.0.smithi153.stdout:3/321: truncate f3 1099247 0 2024-04-05T03:02:47.662 INFO:tasks.workunit.client.0.smithi153.stdout:1/313: dread d5/d8/d20/d4b/d18/f1a [0,4194304] 0 2024-04-05T03:02:47.663 INFO:tasks.workunit.client.0.smithi153.stdout:2/363: mknod d9/d62/c76 0 2024-04-05T03:02:47.664 INFO:tasks.workunit.client.0.smithi153.stdout:2/364: dread - d9/da/d19/d2e/d60/d54/d6d/f6e zero size 2024-04-05T03:02:47.666 INFO:tasks.workunit.client.0.smithi153.stdout:2/365: fdatasync d9/da/d36/f63 0 2024-04-05T03:02:47.666 INFO:tasks.workunit.client.0.smithi153.stdout:3/322: mkdir d1d/d75 0 2024-04-05T03:02:47.666 INFO:tasks.workunit.client.1.smithi203.stdout:7/888: creat d8/d32/d5d/d6d/f112 x:0 0 0 2024-04-05T03:02:47.668 INFO:tasks.workunit.client.1.smithi203.stdout:9/682: getdents d13/d1b/d33/d42 0 2024-04-05T03:02:47.670 INFO:tasks.workunit.client.1.smithi203.stdout:9/683: readlink d13/d1b/d6b/d9e/d59/d5f/l83 0 2024-04-05T03:02:47.671 INFO:tasks.workunit.client.1.smithi203.stdout:3/566: dwrite d0/dc/d38/d5d/f8b [0,4194304] 0 2024-04-05T03:02:47.672 INFO:tasks.workunit.client.0.smithi153.stdout:2/366: rename d9/d68/d39 to d9/d77 0 2024-04-05T03:02:47.674 INFO:tasks.workunit.client.0.smithi153.stdout:2/367: write d9/da/fc [4493775,30281] 0 2024-04-05T03:02:47.675 INFO:tasks.workunit.client.0.smithi153.stdout:3/323: dwrite d1d/d32/f45 [0,4194304] 0 2024-04-05T03:02:47.675 INFO:tasks.workunit.client.1.smithi203.stdout:7/889: rename d8/da/cc to d8/d32/d16/d1a/d3d/d59/de7/d52/c113 0 2024-04-05T03:02:47.678 INFO:tasks.workunit.client.1.smithi203.stdout:7/890: mknod d8/da/d72/c114 0 2024-04-05T03:02:47.679 INFO:tasks.workunit.client.0.smithi153.stdout:2/368: write d9/da/d36/f55 [1911288,119229] 0 2024-04-05T03:02:47.682 INFO:tasks.workunit.client.1.smithi203.stdout:3/567: rename d0/d21/d83/l84 to d0/dc/la3 0 2024-04-05T03:02:47.682 INFO:tasks.workunit.client.1.smithi203.stdout:3/568: chown d0/dc/d1a 716 1 2024-04-05T03:02:47.683 INFO:tasks.workunit.client.1.smithi203.stdout:7/891: symlink d8/d32/d16/d1a/d3d/d59/de7/d5e/l115 0 2024-04-05T03:02:47.685 INFO:tasks.workunit.client.1.smithi203.stdout:7/892: chown d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/la7 398775595 1 2024-04-05T03:02:47.686 INFO:tasks.workunit.client.1.smithi203.stdout:7/893: chown d8/d32/d5d/d6d/daf/led 0 1 2024-04-05T03:02:47.688 INFO:tasks.workunit.client.0.smithi153.stdout:2/369: dwrite d9/da/f49 [0,4194304] 0 2024-04-05T03:02:47.693 INFO:tasks.workunit.client.0.smithi153.stdout:2/370: mkdir d9/da/d78 0 2024-04-05T03:02:47.693 INFO:tasks.workunit.client.1.smithi203.stdout:3/569: dwrite d0/dc/d1a/d76/f94 [0,4194304] 0 2024-04-05T03:02:47.696 INFO:tasks.workunit.client.1.smithi203.stdout:7/894: dwrite d8/fb7 [0,4194304] 0 2024-04-05T03:02:47.700 INFO:tasks.workunit.client.1.smithi203.stdout:3/570: symlink d0/dc/d1a/d76/la4 0 2024-04-05T03:02:47.707 INFO:tasks.workunit.client.1.smithi203.stdout:3/571: creat d0/d69/fa5 x:0 0 0 2024-04-05T03:02:47.708 INFO:tasks.workunit.client.1.smithi203.stdout:7/895: dwrite d8/da/fd [0,4194304] 0 2024-04-05T03:02:47.710 INFO:tasks.workunit.client.1.smithi203.stdout:7/896: read - d8/d32/d16/d1a/d3d/d59/de7/d73/ff7 zero size 2024-04-05T03:02:47.710 INFO:tasks.workunit.client.1.smithi203.stdout:7/897: stat d8/d6c 0 2024-04-05T03:02:47.711 INFO:tasks.workunit.client.1.smithi203.stdout:7/898: chown d8/da/f1d 10911732 1 2024-04-05T03:02:47.714 INFO:tasks.workunit.client.1.smithi203.stdout:3/572: unlink d0/dc/d1a/d1d/d53/f75 0 2024-04-05T03:02:47.718 INFO:tasks.workunit.client.0.smithi153.stdout:2/371: dwrite d9/da/f24 [0,4194304] 0 2024-04-05T03:02:47.718 INFO:tasks.workunit.client.0.smithi153.stdout:3/324: dread d1d/d26/d3c/f5b [0,4194304] 0 2024-04-05T03:02:47.718 INFO:tasks.workunit.client.1.smithi203.stdout:7/899: readlink d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/lf4 0 2024-04-05T03:02:47.718 INFO:tasks.workunit.client.1.smithi203.stdout:3/573: read d0/dc/d38/d5d/f62 [80823,66619] 0 2024-04-05T03:02:47.719 INFO:tasks.workunit.client.0.smithi153.stdout:2/372: write d9/da/f49 [2013467,6917] 0 2024-04-05T03:02:47.719 INFO:tasks.workunit.client.0.smithi153.stdout:3/325: fsync d1d/f72 0 2024-04-05T03:02:47.720 INFO:tasks.workunit.client.1.smithi203.stdout:1/665: sync 2024-04-05T03:02:47.720 INFO:tasks.workunit.client.1.smithi203.stdout:4/508: sync 2024-04-05T03:02:47.720 INFO:tasks.workunit.client.1.smithi203.stdout:8/679: sync 2024-04-05T03:02:47.721 INFO:tasks.workunit.client.1.smithi203.stdout:4/509: chown da/d3c 1688283 1 2024-04-05T03:02:47.724 INFO:tasks.workunit.client.1.smithi203.stdout:8/680: write d1/d4b/d68/d75/d77/f87 [2402123,60588] 0 2024-04-05T03:02:47.730 INFO:tasks.workunit.client.0.smithi153.stdout:2/373: rmdir d9/d71 0 2024-04-05T03:02:47.732 INFO:tasks.workunit.client.0.smithi153.stdout:3/326: write d1d/f70 [383984,20955] 0 2024-04-05T03:02:47.734 INFO:tasks.workunit.client.0.smithi153.stdout:2/374: symlink d9/l79 0 2024-04-05T03:02:47.735 INFO:tasks.workunit.client.0.smithi153.stdout:2/375: stat d9/da 0 2024-04-05T03:02:47.738 INFO:tasks.workunit.client.0.smithi153.stdout:3/327: creat d1d/d26/d57/f76 x:0 0 0 2024-04-05T03:02:47.741 INFO:tasks.workunit.client.0.smithi153.stdout:3/328: creat d1d/d75/f77 x:0 0 0 2024-04-05T03:02:47.743 INFO:tasks.workunit.client.0.smithi153.stdout:2/376: getdents d9/da/d19/d2e/d60 0 2024-04-05T03:02:47.743 INFO:tasks.workunit.client.0.smithi153.stdout:4/411: sync 2024-04-05T03:02:47.743 INFO:tasks.workunit.client.0.smithi153.stdout:8/308: sync 2024-04-05T03:02:47.744 INFO:tasks.workunit.client.0.smithi153.stdout:4/412: chown da/d2a/f47 244660 1 2024-04-05T03:02:47.746 INFO:tasks.workunit.client.0.smithi153.stdout:4/413: read da/db/fd [1517995,47623] 0 2024-04-05T03:02:47.753 INFO:tasks.workunit.client.0.smithi153.stdout:3/329: dread f18 [0,4194304] 0 2024-04-05T03:02:47.755 INFO:tasks.workunit.client.0.smithi153.stdout:8/309: link d7/d13/d19/l38 d7/d13/d19/l66 0 2024-04-05T03:02:47.758 INFO:tasks.workunit.client.0.smithi153.stdout:4/414: truncate da/db/d20/d45/f6b 552859 0 2024-04-05T03:02:47.759 INFO:tasks.workunit.client.0.smithi153.stdout:3/330: write d1d/d26/d57/d6b/f67 [3663730,120619] 0 2024-04-05T03:02:47.762 INFO:tasks.workunit.client.0.smithi153.stdout:8/310: mknod d7/d13/d19/c67 0 2024-04-05T03:02:47.762 INFO:tasks.workunit.client.0.smithi153.stdout:4/415: write da/db/f7b [668195,77849] 0 2024-04-05T03:02:47.763 INFO:tasks.workunit.client.0.smithi153.stdout:2/377: dwrite d9/da/f5f [0,4194304] 0 2024-04-05T03:02:47.765 INFO:tasks.workunit.client.0.smithi153.stdout:3/331: rename d1d/l24 to d1d/d26/l78 0 2024-04-05T03:02:47.766 INFO:tasks.workunit.client.0.smithi153.stdout:8/311: read d7/d13/d19/f5c [193024,71604] 0 2024-04-05T03:02:47.766 INFO:tasks.workunit.client.0.smithi153.stdout:4/416: rmdir da/db/d20/d35 39 2024-04-05T03:02:47.768 INFO:tasks.workunit.client.0.smithi153.stdout:3/332: write d1d/d27/f6f [205550,46571] 0 2024-04-05T03:02:47.772 INFO:tasks.workunit.client.0.smithi153.stdout:3/333: dread d1d/d27/d2b/f2f [0,4194304] 0 2024-04-05T03:02:47.773 INFO:tasks.workunit.client.0.smithi153.stdout:2/378: rename d9/da/d19/d2e/d60/d54 to d9/d68/d4b/d7a 0 2024-04-05T03:02:47.773 INFO:tasks.workunit.client.0.smithi153.stdout:2/379: stat d9/d62 0 2024-04-05T03:02:47.778 INFO:tasks.workunit.client.0.smithi153.stdout:4/417: rmdir da/db/d20/d3c 39 2024-04-05T03:02:47.780 INFO:tasks.workunit.client.0.smithi153.stdout:3/334: unlink f18 0 2024-04-05T03:02:47.782 INFO:tasks.workunit.client.0.smithi153.stdout:2/380: creat d9/da/d37/f7b x:0 0 0 2024-04-05T03:02:47.785 INFO:tasks.workunit.client.0.smithi153.stdout:3/335: creat d1d/d75/f79 x:0 0 0 2024-04-05T03:02:47.788 INFO:tasks.workunit.client.0.smithi153.stdout:3/336: chown l13 22483 1 2024-04-05T03:02:47.790 INFO:tasks.workunit.client.0.smithi153.stdout:2/381: dwrite d9/da/fc [0,4194304] 0 2024-04-05T03:02:47.790 INFO:tasks.workunit.client.0.smithi153.stdout:8/312: write d7/d13/d19/d1b/f35 [2099427,8015] 0 2024-04-05T03:02:47.791 INFO:tasks.workunit.client.0.smithi153.stdout:8/313: chown d7/d13 0 1 2024-04-05T03:02:47.793 INFO:tasks.workunit.client.1.smithi203.stdout:1/666: unlink d7/d24/caf 0 2024-04-05T03:02:47.793 INFO:tasks.workunit.client.1.smithi203.stdout:1/667: chown d7/c42 1925222 1 2024-04-05T03:02:47.797 INFO:tasks.workunit.client.0.smithi153.stdout:8/314: mkdir d7/d13/d19/d1b/d4f/d68 0 2024-04-05T03:02:47.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/382: link d9/d62/c76 d9/da/d19/d2e/c7c 0 2024-04-05T03:02:47.801 INFO:tasks.workunit.client.0.smithi153.stdout:8/315: dread d7/f2b [0,4194304] 0 2024-04-05T03:02:47.802 INFO:tasks.workunit.client.1.smithi203.stdout:1/668: dwrite d7/d24/d33/d48/d4b/d6b/db9/fd2 [0,4194304] 0 2024-04-05T03:02:47.802 INFO:tasks.workunit.client.0.smithi153.stdout:2/383: write d9/da/fe [401791,60610] 0 2024-04-05T03:02:47.804 INFO:tasks.workunit.client.0.smithi153.stdout:3/337: dwrite d1d/d27/d2b/f2f [0,4194304] 0 2024-04-05T03:02:47.805 INFO:tasks.workunit.client.1.smithi203.stdout:8/681: symlink d1/d12/d1a/lbe 0 2024-04-05T03:02:47.806 INFO:tasks.workunit.client.1.smithi203.stdout:7/900: write d8/d32/d5d/d6d/dca/f42 [899077,91773] 0 2024-04-05T03:02:47.808 INFO:tasks.workunit.client.1.smithi203.stdout:1/669: dread - d7/d24/d33/d48/d4b/f94 zero size 2024-04-05T03:02:47.812 INFO:tasks.workunit.client.1.smithi203.stdout:3/574: dwrite d0/dc/f54 [0,4194304] 0 2024-04-05T03:02:47.814 INFO:tasks.workunit.client.0.smithi153.stdout:3/338: creat d1d/f7a x:0 0 0 2024-04-05T03:02:47.814 INFO:tasks.workunit.client.0.smithi153.stdout:2/384: dwrite d9/d68/d4b/d7a/d6d/f6e [0,4194304] 0 2024-04-05T03:02:47.818 INFO:tasks.workunit.client.0.smithi153.stdout:3/339: creat d1d/d26/d3c/f7b x:0 0 0 2024-04-05T03:02:47.819 INFO:tasks.workunit.client.1.smithi203.stdout:4/510: rename da/d3c/d4c/f7c to da/f9a 0 2024-04-05T03:02:47.821 INFO:tasks.workunit.client.1.smithi203.stdout:8/682: dwrite d1/d4b/d68/d75/dac/da4/fbd [0,4194304] 0 2024-04-05T03:02:47.824 INFO:tasks.workunit.client.1.smithi203.stdout:7/901: creat d8/d32/d16/d1a/d3d/d59/de7/d52/d65/d82/f116 x:0 0 0 2024-04-05T03:02:47.825 INFO:tasks.workunit.client.0.smithi153.stdout:3/340: mkdir d1d/d26/d3c/d3f/d7c 0 2024-04-05T03:02:47.828 INFO:tasks.workunit.client.1.smithi203.stdout:3/575: truncate d0/f5 1976359 0 2024-04-05T03:02:47.831 INFO:tasks.workunit.client.0.smithi153.stdout:4/418: dread da/db/f23 [0,4194304] 0 2024-04-05T03:02:47.834 INFO:tasks.workunit.client.1.smithi203.stdout:1/670: dread d7/d24/d33/d48/d4b/d6b/db9/fce [0,4194304] 0 2024-04-05T03:02:47.838 INFO:tasks.workunit.client.0.smithi153.stdout:4/419: creat da/db/d20/d35/d38/d3e/f8c x:0 0 0 2024-04-05T03:02:47.838 INFO:tasks.workunit.client.1.smithi203.stdout:7/902: mknod d8/d32/d16/d1a/d21/c117 0 2024-04-05T03:02:47.842 INFO:tasks.workunit.client.1.smithi203.stdout:1/671: write d7/d24/d33/d48/d4b/d6b/f98 [1542363,41206] 0 2024-04-05T03:02:47.847 INFO:tasks.workunit.client.0.smithi153.stdout:4/420: dread da/db/d20/d45/f4e [4194304,4194304] 0 2024-04-05T03:02:47.849 INFO:tasks.workunit.client.0.smithi153.stdout:4/421: unlink da/f17 0 2024-04-05T03:02:47.852 INFO:tasks.workunit.client.0.smithi153.stdout:4/422: write da/db/d20/d35/d38/d3e/f50 [737052,105288] 0 2024-04-05T03:02:47.855 INFO:tasks.workunit.client.0.smithi153.stdout:4/423: symlink da/db/d20/d35/d41/d67/l8d 0 2024-04-05T03:02:47.855 INFO:tasks.workunit.client.0.smithi153.stdout:4/424: stat da/de/d56 0 2024-04-05T03:02:47.857 INFO:tasks.workunit.client.0.smithi153.stdout:4/425: dread - da/db/d20/f6e zero size 2024-04-05T03:02:47.857 INFO:tasks.workunit.client.0.smithi153.stdout:3/341: rmdir d1d/d26/d3c/d3f 39 2024-04-05T03:02:47.857 INFO:tasks.workunit.client.0.smithi153.stdout:4/426: readlink da/l4c 0 2024-04-05T03:02:47.858 INFO:tasks.workunit.client.0.smithi153.stdout:3/342: fsync d1d/f3e 0 2024-04-05T03:02:47.860 INFO:tasks.workunit.client.0.smithi153.stdout:4/427: symlink da/l8e 0 2024-04-05T03:02:47.861 INFO:tasks.workunit.client.0.smithi153.stdout:3/343: mknod d1d/d32/c7d 0 2024-04-05T03:02:47.861 INFO:tasks.workunit.client.0.smithi153.stdout:4/428: write da/db/d20/f32 [704356,54355] 0 2024-04-05T03:02:47.862 INFO:tasks.workunit.client.1.smithi203.stdout:4/511: mknod da/d29/d2e/c9b 0 2024-04-05T03:02:47.865 INFO:tasks.workunit.client.1.smithi203.stdout:7/903: dread d8/d32/d5d/d6d/f101 [0,4194304] 0 2024-04-05T03:02:47.865 INFO:tasks.workunit.client.0.smithi153.stdout:3/344: chown c10 16 1 2024-04-05T03:02:47.866 INFO:tasks.workunit.client.0.smithi153.stdout:3/345: write d1d/d26/d3c/f46 [744275,66148] 0 2024-04-05T03:02:47.867 INFO:tasks.workunit.client.0.smithi153.stdout:4/429: unlink da/de/d56/d6a/f7c 0 2024-04-05T03:02:47.868 INFO:tasks.workunit.client.0.smithi153.stdout:4/430: dread - da/db/f46 zero size 2024-04-05T03:02:47.869 INFO:tasks.workunit.client.0.smithi153.stdout:3/346: dread d1d/d27/f6f [0,4194304] 0 2024-04-05T03:02:47.871 INFO:tasks.workunit.client.0.smithi153.stdout:3/347: dread - d1d/d27/d2b/f62 zero size 2024-04-05T03:02:47.875 INFO:tasks.workunit.client.0.smithi153.stdout:4/431: rename da/db/d20/d45/d3b to da/de/d56/d78/d8f 0 2024-04-05T03:02:47.876 INFO:tasks.workunit.client.0.smithi153.stdout:3/348: creat d1d/d27/d2b/f7e x:0 0 0 2024-04-05T03:02:47.879 INFO:tasks.workunit.client.1.smithi203.stdout:3/576: link d0/dc/d34/f61 d0/dc/d1a/d76/fa6 0 2024-04-05T03:02:47.880 INFO:tasks.workunit.client.1.smithi203.stdout:2/830: sync 2024-04-05T03:02:47.881 INFO:tasks.workunit.client.0.smithi153.stdout:4/432: dread da/f48 [0,4194304] 0 2024-04-05T03:02:47.881 INFO:tasks.workunit.client.1.smithi203.stdout:7/904: link d8/d32/d5d/d95/c97 d8/d32/d16/d1a/d3d/d59/c118 0 2024-04-05T03:02:47.882 INFO:tasks.workunit.client.1.smithi203.stdout:7/905: read d8/da/f4e [1989196,78104] 0 2024-04-05T03:02:47.882 INFO:tasks.workunit.client.1.smithi203.stdout:3/577: rmdir d0/dc/d1a/d8d/d96 39 2024-04-05T03:02:47.886 INFO:tasks.workunit.client.1.smithi203.stdout:2/831: mkdir d5/d10/d35/d2d/d7c/d29/d67/d118/d11e 0 2024-04-05T03:02:47.888 INFO:tasks.workunit.client.0.smithi153.stdout:4/433: mkdir da/db/d20/d35/d38/d76/d90 0 2024-04-05T03:02:47.889 INFO:tasks.workunit.client.0.smithi153.stdout:4/434: chown da/f48 8730483 1 2024-04-05T03:02:47.891 INFO:tasks.workunit.client.0.smithi153.stdout:4/435: truncate da/de/f6c 392001 0 2024-04-05T03:02:47.894 INFO:tasks.workunit.client.0.smithi153.stdout:4/436: unlink da/db/d20/f2c 0 2024-04-05T03:02:47.894 INFO:tasks.workunit.client.1.smithi203.stdout:8/683: rename d1/f34 to d1/d4b/d5f/daf/fbf 0 2024-04-05T03:02:47.897 INFO:tasks.workunit.client.0.smithi153.stdout:4/437: mkdir da/db/d20/d3c/d91 0 2024-04-05T03:02:47.898 INFO:tasks.workunit.client.1.smithi203.stdout:4/512: getdents da 0 2024-04-05T03:02:47.899 INFO:tasks.workunit.client.0.smithi153.stdout:4/438: mknod da/db/d20/c92 0 2024-04-05T03:02:47.899 INFO:tasks.workunit.client.1.smithi203.stdout:1/672: rename d7/d24/d33/ccf to d7/d14/d2b/d58/cd3 0 2024-04-05T03:02:47.900 INFO:tasks.workunit.client.1.smithi203.stdout:4/513: creat da/d29/d6a/d80/f9c x:0 0 0 2024-04-05T03:02:47.901 INFO:tasks.workunit.client.1.smithi203.stdout:8/684: dwrite d1/d12/d2f/f61 [0,4194304] 0 2024-04-05T03:02:47.901 INFO:tasks.workunit.client.0.smithi153.stdout:4/439: mkdir da/de/d56/d93 0 2024-04-05T03:02:47.903 INFO:tasks.workunit.client.1.smithi203.stdout:3/578: rename d0/dc/d1a/d8d/d96/c68 to d0/d41/ca7 0 2024-04-05T03:02:47.903 INFO:tasks.workunit.client.1.smithi203.stdout:1/673: creat d7/d14/d83/fd4 x:0 0 0 2024-04-05T03:02:47.904 INFO:tasks.workunit.client.0.smithi153.stdout:4/440: mknod da/db/d20/c94 0 2024-04-05T03:02:47.904 INFO:tasks.workunit.client.1.smithi203.stdout:4/514: mknod da/d29/d2e/d8b/c9d 0 2024-04-05T03:02:47.905 INFO:tasks.workunit.client.1.smithi203.stdout:1/674: dread - d7/d24/fc2 zero size 2024-04-05T03:02:47.908 INFO:tasks.workunit.client.1.smithi203.stdout:8/685: read d1/d12/d17/d10/f49 [4153387,129767] 0 2024-04-05T03:02:47.913 INFO:tasks.workunit.client.1.smithi203.stdout:1/675: creat d7/d14/d83/fd5 x:0 0 0 2024-04-05T03:02:47.915 INFO:tasks.workunit.client.1.smithi203.stdout:3/579: mknod d0/dc/d1a/ca8 0 2024-04-05T03:02:47.916 INFO:tasks.workunit.client.1.smithi203.stdout:3/580: write d0/dc/d1a/d8d/d96/f51 [550180,70980] 0 2024-04-05T03:02:47.916 INFO:tasks.workunit.client.1.smithi203.stdout:8/686: mkdir d1/d6/d2e/d5c/dc0 0 2024-04-05T03:02:47.916 INFO:tasks.workunit.client.1.smithi203.stdout:0/683: sync 2024-04-05T03:02:47.917 INFO:tasks.workunit.client.1.smithi203.stdout:9/684: sync 2024-04-05T03:02:47.917 INFO:tasks.workunit.client.1.smithi203.stdout:5/670: sync 2024-04-05T03:02:47.917 INFO:tasks.workunit.client.1.smithi203.stdout:6/710: sync 2024-04-05T03:02:47.918 INFO:tasks.workunit.client.1.smithi203.stdout:1/676: rmdir d7/d14/d2b 39 2024-04-05T03:02:47.922 INFO:tasks.workunit.client.1.smithi203.stdout:5/671: write d3/d1f/fc0 [526954,78082] 0 2024-04-05T03:02:47.922 INFO:tasks.workunit.client.1.smithi203.stdout:4/515: creat da/d29/f9e x:0 0 0 2024-04-05T03:02:47.922 INFO:tasks.workunit.client.1.smithi203.stdout:9/685: symlink d13/d1b/d33/d42/le1 0 2024-04-05T03:02:47.922 INFO:tasks.workunit.client.1.smithi203.stdout:6/711: chown d0/da/d18/d38/d2a/d44/d78/ld0 990307583 1 2024-04-05T03:02:47.924 INFO:tasks.workunit.client.1.smithi203.stdout:0/684: rmdir dd/d17/d1d/d22/d53/d7f 39 2024-04-05T03:02:47.924 INFO:tasks.workunit.client.1.smithi203.stdout:4/516: chown da/d29/d2e/f63 48 1 2024-04-05T03:02:47.924 INFO:tasks.workunit.client.1.smithi203.stdout:6/712: readlink d0/da/d54/lbb 0 2024-04-05T03:02:47.925 INFO:tasks.workunit.client.1.smithi203.stdout:5/672: mknod d3/d4/dd6/ce4 0 2024-04-05T03:02:47.926 INFO:tasks.workunit.client.0.smithi153.stdout:4/441: dread da/ff [0,4194304] 0 2024-04-05T03:02:47.926 INFO:tasks.workunit.client.1.smithi203.stdout:3/581: dwrite d0/dc/d1a/d8d/d96/f51 [0,4194304] 0 2024-04-05T03:02:47.927 INFO:tasks.workunit.client.0.smithi153.stdout:7/335: sync 2024-04-05T03:02:47.928 INFO:tasks.workunit.client.0.smithi153.stdout:7/336: chown d2/d51/d59/c58 5754 1 2024-04-05T03:02:47.928 INFO:tasks.workunit.client.1.smithi203.stdout:0/685: write dd/d17/d1d/d22/d53/d55/fa1 [135848,111623] 0 2024-04-05T03:02:47.928 INFO:tasks.workunit.client.1.smithi203.stdout:9/686: rename l4 to d13/d5c/dd1/d51/da1/dba/le2 0 2024-04-05T03:02:47.935 INFO:tasks.workunit.client.0.smithi153.stdout:4/442: dwrite da/db/d20/d35/d38/d3e/f8c [0,4194304] 0 2024-04-05T03:02:47.936 INFO:tasks.workunit.client.0.smithi153.stdout:4/443: readlink da/db/l77 0 2024-04-05T03:02:47.940 INFO:tasks.workunit.client.1.smithi203.stdout:4/517: dwrite da/d29/d2e/d57/f86 [0,4194304] 0 2024-04-05T03:02:47.941 INFO:tasks.workunit.client.1.smithi203.stdout:5/673: dwrite d3/d1f/f93 [8388608,4194304] 0 2024-04-05T03:02:47.942 INFO:tasks.workunit.client.1.smithi203.stdout:0/686: dwrite dd/d17/d1d/d26/d8c/fa8 [4194304,4194304] 0 2024-04-05T03:02:47.944 INFO:tasks.workunit.client.1.smithi203.stdout:0/687: readlink dd/d17/l1a 0 2024-04-05T03:02:47.946 INFO:tasks.workunit.client.1.smithi203.stdout:0/688: readlink dd/d17/d1d/d26/d34/d56/l73 0 2024-04-05T03:02:47.951 INFO:tasks.workunit.client.1.smithi203.stdout:5/674: dread d3/d4/f71 [0,4194304] 0 2024-04-05T03:02:47.963 INFO:tasks.workunit.client.1.smithi203.stdout:3/582: creat d0/dc/d1a/d8d/d96/fa9 x:0 0 0 2024-04-05T03:02:47.963 INFO:tasks.workunit.client.0.smithi153.stdout:7/337: dread d2/da/d3a/d41/f1e [0,4194304] 0 2024-04-05T03:02:47.967 INFO:tasks.workunit.client.0.smithi153.stdout:7/338: symlink d2/da/d47/l6e 0 2024-04-05T03:02:47.968 INFO:tasks.workunit.client.1.smithi203.stdout:3/583: dread d0/f23 [0,4194304] 0 2024-04-05T03:02:47.969 INFO:tasks.workunit.client.0.smithi153.stdout:7/339: rmdir d2/da/d42/d6d 39 2024-04-05T03:02:47.970 INFO:tasks.workunit.client.1.smithi203.stdout:1/677: symlink d7/d14/d2b/ld6 0 2024-04-05T03:02:47.972 INFO:tasks.workunit.client.1.smithi203.stdout:4/518: rename da/d29/d6a/f6c to da/f9f 0 2024-04-05T03:02:47.973 INFO:tasks.workunit.client.1.smithi203.stdout:8/687: mknod d1/d4b/d68/cc1 0 2024-04-05T03:02:47.974 INFO:tasks.workunit.client.1.smithi203.stdout:5/675: fdatasync d3/d1f/f3b 0 2024-04-05T03:02:47.976 INFO:tasks.workunit.client.1.smithi203.stdout:8/688: read d1/d6/d2e/f6b [55974,97606] 0 2024-04-05T03:02:47.976 INFO:tasks.workunit.client.1.smithi203.stdout:6/713: link d0/da/d54/d4c/da1/db7/l98 d0/da/d54/d4c/da1/db7/d95/ld7 0 2024-04-05T03:02:47.977 INFO:tasks.workunit.client.1.smithi203.stdout:3/584: creat d0/d41/faa x:0 0 0 2024-04-05T03:02:47.978 INFO:tasks.workunit.client.1.smithi203.stdout:3/585: chown d0/dc/d1a/d76/fa6 0 1 2024-04-05T03:02:47.980 INFO:tasks.workunit.client.1.smithi203.stdout:3/586: dread - d0/dc/d1a/d8d/d96/fa9 zero size 2024-04-05T03:02:47.980 INFO:tasks.workunit.client.1.smithi203.stdout:6/714: dread d0/da/d54/d53/fbd [0,4194304] 0 2024-04-05T03:02:47.981 INFO:tasks.workunit.client.1.smithi203.stdout:0/689: mknod dd/d17/d1d/d22/d53/d7f/dc5/cca 0 2024-04-05T03:02:47.985 INFO:tasks.workunit.client.1.smithi203.stdout:1/678: creat d7/d90/fd7 x:0 0 0 2024-04-05T03:02:47.985 INFO:tasks.workunit.client.0.smithi153.stdout:4/444: dread da/d2a/f47 [0,4194304] 0 2024-04-05T03:02:47.985 INFO:tasks.workunit.client.1.smithi203.stdout:3/587: dwrite d0/dc/d38/d5d/d66/f7e [0,4194304] 0 2024-04-05T03:02:47.986 INFO:tasks.workunit.client.1.smithi203.stdout:4/519: creat da/d29/d6a/d80/fa0 x:0 0 0 2024-04-05T03:02:47.988 INFO:tasks.workunit.client.1.smithi203.stdout:0/690: symlink dd/d17/d1d/d26/d28/lcb 0 2024-04-05T03:02:47.988 INFO:tasks.workunit.client.1.smithi203.stdout:6/715: chown d0/da/d54/d53/f63 10 1 2024-04-05T03:02:47.989 INFO:tasks.workunit.client.0.smithi153.stdout:4/445: creat da/de/d56/d78/d8f/d68/f95 x:0 0 0 2024-04-05T03:02:47.990 INFO:tasks.workunit.client.1.smithi203.stdout:8/689: unlink d1/d12/d17/l7f 0 2024-04-05T03:02:47.992 INFO:tasks.workunit.client.1.smithi203.stdout:2/832: write d5/d63/f6e [1085024,24793] 0 2024-04-05T03:02:47.992 INFO:tasks.workunit.client.0.smithi153.stdout:4/446: rename da/db/fd to da/db/f96 0 2024-04-05T03:02:47.994 INFO:tasks.workunit.client.1.smithi203.stdout:4/520: symlink da/d29/d2e/d8b/la1 0 2024-04-05T03:02:47.996 INFO:tasks.workunit.client.1.smithi203.stdout:7/906: truncate d8/d32/d16/d1a/d5a/db5/fd7 239043 0 2024-04-05T03:02:47.998 INFO:tasks.workunit.client.1.smithi203.stdout:1/679: dwrite d7/d24/f85 [0,4194304] 0 2024-04-05T03:02:48.001 INFO:tasks.workunit.client.1.smithi203.stdout:4/521: rmdir da/d29/d2e/d53 39 2024-04-05T03:02:48.004 INFO:tasks.workunit.client.1.smithi203.stdout:8/690: symlink d1/d12/d17/db8/lc2 0 2024-04-05T03:02:48.008 INFO:tasks.workunit.client.1.smithi203.stdout:1/680: unlink d7/d14/c1a 0 2024-04-05T03:02:48.011 INFO:tasks.workunit.client.1.smithi203.stdout:4/522: dwrite da/d29/f5b [0,4194304] 0 2024-04-05T03:02:48.011 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:47 smithi203 ceph-mon[39911]: mgrmap e25: smithi203.iakodz(active, since 11s), standbys: smithi153.npzyqy 2024-04-05T03:02:48.011 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:47 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:02:48.014 INFO:tasks.workunit.client.1.smithi203.stdout:6/716: rmdir d0/da/d18/dd5 0 2024-04-05T03:02:48.016 INFO:tasks.workunit.client.1.smithi203.stdout:7/907: symlink d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/l119 0 2024-04-05T03:02:48.017 INFO:tasks.workunit.client.1.smithi203.stdout:8/691: symlink d1/d12/d2f/d5b/lc3 0 2024-04-05T03:02:48.020 INFO:tasks.workunit.client.1.smithi203.stdout:0/691: link dd/d17/d1d/d26/d8c/d6d/la6 dd/d17/d1d/d22/d80/d49/lcc 0 2024-04-05T03:02:48.021 INFO:tasks.workunit.client.1.smithi203.stdout:1/681: dwrite d7/d14/f1e [8388608,4194304] 0 2024-04-05T03:02:48.021 INFO:tasks.workunit.client.1.smithi203.stdout:8/692: chown d1/d4b/d68/d75/d77/l8d 2142 1 2024-04-05T03:02:48.024 INFO:tasks.workunit.client.1.smithi203.stdout:4/523: unlink da/d29/d2e/f48 0 2024-04-05T03:02:48.024 INFO:tasks.workunit.client.1.smithi203.stdout:0/692: dread - dd/d17/d1d/d26/d8c/fc9 zero size 2024-04-05T03:02:48.028 INFO:tasks.workunit.client.1.smithi203.stdout:2/833: dread d5/d10/d35/d2d/d7c/d72/d8f/fb9 [0,4194304] 0 2024-04-05T03:02:48.033 INFO:tasks.workunit.client.1.smithi203.stdout:3/588: dread d0/dc/d1a/d1f/f9d [0,4194304] 0 2024-04-05T03:02:48.033 INFO:tasks.workunit.client.1.smithi203.stdout:6/717: creat d0/da/d18/d38/d2a/fd8 x:0 0 0 2024-04-05T03:02:48.034 INFO:tasks.workunit.client.1.smithi203.stdout:8/693: symlink d1/d12/d17/d47/d53/d69/d9c/lc4 0 2024-04-05T03:02:48.035 INFO:tasks.workunit.client.1.smithi203.stdout:2/834: symlink d5/d63/l11f 0 2024-04-05T03:02:48.037 INFO:tasks.workunit.client.1.smithi203.stdout:0/693: mknod dd/d17/d1d/d22/d80/d49/ccd 0 2024-04-05T03:02:48.038 INFO:tasks.workunit.client.1.smithi203.stdout:7/908: dwrite d8/d32/d16/d35/fcf [0,4194304] 0 2024-04-05T03:02:48.038 INFO:tasks.workunit.client.1.smithi203.stdout:5/676: write d3/d1f/d59/d60/f72 [550539,80577] 0 2024-04-05T03:02:48.038 INFO:tasks.workunit.client.1.smithi203.stdout:1/682: write d7/d24/d33/d48/d4b/d51/fb2 [4661409,112084] 0 2024-04-05T03:02:48.038 INFO:tasks.workunit.client.1.smithi203.stdout:3/589: unlink d0/d21/d81/c63 0 2024-04-05T03:02:48.041 INFO:tasks.workunit.client.1.smithi203.stdout:9/687: dwrite d13/d5c/f9f [0,4194304] 0 2024-04-05T03:02:48.045 INFO:tasks.workunit.client.1.smithi203.stdout:8/694: creat d1/d6/d2e/d5c/dc0/fc5 x:0 0 0 2024-04-05T03:02:48.053 INFO:tasks.workunit.client.0.smithi153.stdout:7/340: truncate d2/da/d11/d39/f63 1312378 0 2024-04-05T03:02:48.056 INFO:tasks.workunit.client.1.smithi203.stdout:9/688: chown d13/d1b/d6b/d9e/d59/d5f/f71 188 1 2024-04-05T03:02:48.057 INFO:tasks.workunit.client.0.smithi153.stdout:7/341: write d2/f6 [8869344,122671] 0 2024-04-05T03:02:48.057 INFO:tasks.workunit.client.1.smithi203.stdout:5/677: symlink d3/d1f/le5 0 2024-04-05T03:02:48.060 INFO:tasks.workunit.client.1.smithi203.stdout:6/718: dread d0/da/d54/fb0 [4194304,4194304] 0 2024-04-05T03:02:48.061 INFO:tasks.workunit.client.0.smithi153.stdout:4/447: dwrite da/db/d20/d35/d38/f59 [4194304,4194304] 0 2024-04-05T03:02:48.061 INFO:tasks.workunit.client.0.smithi153.stdout:7/342: mkdir d2/da/d42/d6d/d55/d6f 0 2024-04-05T03:02:48.062 INFO:tasks.workunit.client.1.smithi203.stdout:4/524: write da/d29/f50 [919661,31455] 0 2024-04-05T03:02:48.062 INFO:tasks.workunit.client.1.smithi203.stdout:5/678: write d3/d4/fd4 [547447,122823] 0 2024-04-05T03:02:48.063 INFO:tasks.workunit.client.1.smithi203.stdout:2/835: dwrite d5/d10/d35/d36/db6/f27 [4194304,4194304] 0 2024-04-05T03:02:48.063 INFO:tasks.workunit.client.1.smithi203.stdout:0/694: symlink dd/d17/d1d/d22/d45/d51/lce 0 2024-04-05T03:02:48.063 INFO:tasks.workunit.client.0.smithi153.stdout:7/343: chown d2/da/d42/l64 5992886 1 2024-04-05T03:02:48.064 INFO:tasks.workunit.client.0.smithi153.stdout:7/344: chown d2/da/d3a/d41/f1e 43 1 2024-04-05T03:02:48.066 INFO:tasks.workunit.client.1.smithi203.stdout:1/683: dwrite d7/d24/d33/d48/d4b/d51/f67 [0,4194304] 0 2024-04-05T03:02:48.069 INFO:tasks.workunit.client.1.smithi203.stdout:9/689: rename d13/c2b to d13/d1b/d33/d42/d67/d82/ce3 0 2024-04-05T03:02:48.071 INFO:tasks.workunit.client.0.smithi153.stdout:7/345: getdents d2/da/d3a 0 2024-04-05T03:02:48.071 INFO:tasks.workunit.client.1.smithi203.stdout:5/679: truncate d3/f18 5397819 0 2024-04-05T03:02:48.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:47 smithi153 ceph-mon[29381]: mgrmap e25: smithi203.iakodz(active, since 11s), standbys: smithi153.npzyqy 2024-04-05T03:02:48.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:47 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:02:48.074 INFO:tasks.workunit.client.1.smithi203.stdout:6/719: write d0/da/d18/d38/d2a/fce [1752902,43860] 0 2024-04-05T03:02:48.074 INFO:tasks.workunit.client.1.smithi203.stdout:8/695: dwrite d1/d12/d17/d10/f40 [0,4194304] 0 2024-04-05T03:02:48.075 INFO:tasks.workunit.client.0.smithi153.stdout:4/448: dwrite da/d2a/f47 [0,4194304] 0 2024-04-05T03:02:48.076 INFO:tasks.workunit.client.0.smithi153.stdout:7/346: dread d2/da/d42/d6d/f3c [0,4194304] 0 2024-04-05T03:02:48.079 INFO:tasks.workunit.client.0.smithi153.stdout:7/347: chown d2/da/d3a/d41/d31/f5f 943436 1 2024-04-05T03:02:48.081 INFO:tasks.workunit.client.0.smithi153.stdout:7/348: truncate d2/da/d3a/d41/d31/f5f 261247 0 2024-04-05T03:02:48.082 INFO:tasks.workunit.client.1.smithi203.stdout:9/690: unlink d13/d1b/d33/d42/f66 0 2024-04-05T03:02:48.083 INFO:tasks.workunit.client.1.smithi203.stdout:4/525: creat da/d29/d2e/d57/d96/fa2 x:0 0 0 2024-04-05T03:02:48.084 INFO:tasks.workunit.client.0.smithi153.stdout:4/449: creat da/f97 x:0 0 0 2024-04-05T03:02:48.085 INFO:tasks.workunit.client.0.smithi153.stdout:7/349: symlink d2/d51/l70 0 2024-04-05T03:02:48.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/450: chown da/db/d20/d35/d38 44450 1 2024-04-05T03:02:48.086 INFO:tasks.workunit.client.1.smithi203.stdout:8/696: dwrite d1/d12/d2f/d5b/fa7 [0,4194304] 0 2024-04-05T03:02:48.086 INFO:tasks.workunit.client.1.smithi203.stdout:9/691: truncate d13/d1b/fbd 707951 0 2024-04-05T03:02:48.089 INFO:tasks.workunit.client.0.smithi153.stdout:7/350: read d2/f5 [5040582,87224] 0 2024-04-05T03:02:48.091 INFO:tasks.workunit.client.1.smithi203.stdout:6/720: mknod d0/da/d54/d4c/daf/cd9 0 2024-04-05T03:02:48.095 INFO:tasks.workunit.client.1.smithi203.stdout:8/697: read d1/d12/d2f/d1f/f23 [2230374,65947] 0 2024-04-05T03:02:48.098 INFO:tasks.workunit.client.0.smithi153.stdout:4/451: dwrite da/db/f23 [0,4194304] 0 2024-04-05T03:02:48.099 INFO:tasks.workunit.client.0.smithi153.stdout:4/452: chown l6 25765 1 2024-04-05T03:02:48.100 INFO:tasks.workunit.client.0.smithi153.stdout:7/351: link d2/da/d47/f60 d2/da/d3a/f71 0 2024-04-05T03:02:48.103 INFO:tasks.workunit.client.1.smithi203.stdout:6/721: dwrite d0/da/d54/d4c/da1/db7/fc4 [0,4194304] 0 2024-04-05T03:02:48.103 INFO:tasks.workunit.client.1.smithi203.stdout:1/684: symlink d7/d24/d33/ld8 0 2024-04-05T03:02:48.104 INFO:tasks.workunit.client.1.smithi203.stdout:9/692: creat d13/d5c/d9d/fe4 x:0 0 0 2024-04-05T03:02:48.104 INFO:tasks.workunit.client.0.smithi153.stdout:4/453: mknod da/db/d20/d45/c98 0 2024-04-05T03:02:48.106 INFO:tasks.workunit.client.0.smithi153.stdout:7/352: creat d2/da/d11/f72 x:0 0 0 2024-04-05T03:02:48.108 INFO:tasks.workunit.client.1.smithi203.stdout:2/836: dread d5/d10/d35/d2d/f33 [0,4194304] 0 2024-04-05T03:02:48.108 INFO:tasks.workunit.client.1.smithi203.stdout:7/909: link d8/d32/d16/d35/d64/c103 d8/d32/d16/d1a/d3d/d59/de7/d52/c11a 0 2024-04-05T03:02:48.109 INFO:tasks.workunit.client.0.smithi153.stdout:4/454: rename da/db/d20/d35/f58 to da/de/d56/d78/d8f/f99 0 2024-04-05T03:02:48.110 INFO:tasks.workunit.client.0.smithi153.stdout:4/455: fsync da/de/d56/d78/f7e 0 2024-04-05T03:02:48.111 INFO:tasks.workunit.client.1.smithi203.stdout:7/910: write d8/da/d72/fee [962057,57715] 0 2024-04-05T03:02:48.112 INFO:tasks.workunit.client.1.smithi203.stdout:9/693: unlink d13/d5c/d9d/ca8 0 2024-04-05T03:02:48.112 INFO:tasks.workunit.client.0.smithi153.stdout:7/353: write d2/da/d42/f4b [1507322,65984] 0 2024-04-05T03:02:48.114 INFO:tasks.workunit.client.0.smithi153.stdout:4/456: rmdir da/de/d56/d78/d8f 39 2024-04-05T03:02:48.115 INFO:tasks.workunit.client.0.smithi153.stdout:4/457: chown da/ff 245560 1 2024-04-05T03:02:48.115 INFO:tasks.workunit.client.1.smithi203.stdout:0/695: getdents dd/d17/d1d/d26/d8c 0 2024-04-05T03:02:48.116 INFO:tasks.workunit.client.1.smithi203.stdout:8/698: mknod d1/d12/d1a/d6d/d97/cc6 0 2024-04-05T03:02:48.116 INFO:tasks.workunit.client.0.smithi153.stdout:4/458: write da/db/d20/d35/d38/f59 [5506001,103363] 0 2024-04-05T03:02:48.117 INFO:tasks.workunit.client.0.smithi153.stdout:4/459: stat da/de/d56/f61 0 2024-04-05T03:02:48.118 INFO:tasks.workunit.client.1.smithi203.stdout:2/837: creat d5/d10/db8/dab/de1/dbd/f120 x:0 0 0 2024-04-05T03:02:48.119 INFO:tasks.workunit.client.0.smithi153.stdout:7/354: fdatasync d2/da/d42/d6d/f56 0 2024-04-05T03:02:48.120 INFO:tasks.workunit.client.0.smithi153.stdout:7/355: fsync d2/da/d3a/d41/d31/f4d 0 2024-04-05T03:02:48.121 INFO:tasks.workunit.client.1.smithi203.stdout:9/694: dread d13/d1b/d33/d42/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:48.123 INFO:tasks.workunit.client.1.smithi203.stdout:8/699: mknod d1/d6/d8f/cc7 0 2024-04-05T03:02:48.126 INFO:tasks.workunit.client.1.smithi203.stdout:0/696: dwrite dd/d17/f40 [0,4194304] 0 2024-04-05T03:02:48.127 INFO:tasks.workunit.client.0.smithi153.stdout:4/460: dwrite da/db/d20/d45/f69 [0,4194304] 0 2024-04-05T03:02:48.127 INFO:tasks.workunit.client.1.smithi203.stdout:8/700: chown d1/d4b/d68/cc1 34503 1 2024-04-05T03:02:48.127 INFO:tasks.workunit.client.1.smithi203.stdout:7/911: creat d8/d32/dc3/dd8/f11b x:0 0 0 2024-04-05T03:02:48.130 INFO:tasks.workunit.client.1.smithi203.stdout:2/838: dwrite d5/d10/db8/dab/fa8 [0,4194304] 0 2024-04-05T03:02:48.130 INFO:tasks.workunit.client.0.smithi153.stdout:4/461: rename da/db to da/db/d20/d35/d41/d67/d9a 22 2024-04-05T03:02:48.131 INFO:tasks.workunit.client.0.smithi153.stdout:7/356: dwrite d2/da/d3a/d41/d31/f5f [0,4194304] 0 2024-04-05T03:02:48.132 INFO:tasks.workunit.client.1.smithi203.stdout:1/685: rename d7/d14/d2b/d58/fa8 to d7/d14/d2b/fd9 0 2024-04-05T03:02:48.134 INFO:tasks.workunit.client.0.smithi153.stdout:7/357: truncate d2/da/d42/d6d/f3c 2021175 0 2024-04-05T03:02:48.135 INFO:tasks.workunit.client.0.smithi153.stdout:7/358: write d2/da/d11/f52 [2485673,25676] 0 2024-04-05T03:02:48.135 INFO:tasks.workunit.client.1.smithi203.stdout:1/686: fdatasync d7/d14/d2b/d58/f60 0 2024-04-05T03:02:48.138 INFO:tasks.workunit.client.1.smithi203.stdout:8/701: rename d1/d12/d2f/d5b/lc3 to d1/d12/d17/db8/lc8 0 2024-04-05T03:02:48.140 INFO:tasks.workunit.client.1.smithi203.stdout:7/912: creat d8/d32/d5d/f11c x:0 0 0 2024-04-05T03:02:48.140 INFO:tasks.workunit.client.1.smithi203.stdout:8/702: fdatasync d1/d4b/d68/d75/dac/da4/fbd 0 2024-04-05T03:02:48.141 INFO:tasks.workunit.client.1.smithi203.stdout:7/913: stat d8/d32/d5d/d95 0 2024-04-05T03:02:48.142 INFO:tasks.workunit.client.1.smithi203.stdout:1/687: creat d7/d14/d2b/d58/d7a/fda x:0 0 0 2024-04-05T03:02:48.144 INFO:tasks.workunit.client.0.smithi153.stdout:7/359: dread d2/f9 [0,4194304] 0 2024-04-05T03:02:48.146 INFO:tasks.workunit.client.1.smithi203.stdout:2/839: rename d5/d10/d35/d2d/d7c/d29/d67/fd5 to d5/d10/db8/dab/f121 0 2024-04-05T03:02:48.147 INFO:tasks.workunit.client.1.smithi203.stdout:8/703: rmdir d1/d12/d17/d47/d53/d69/d9c 39 2024-04-05T03:02:48.150 INFO:tasks.workunit.client.0.smithi153.stdout:4/462: dwrite da/de/d56/d78/d8f/f99 [0,4194304] 0 2024-04-05T03:02:48.150 INFO:tasks.workunit.client.1.smithi203.stdout:7/914: rename d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/l93 to d8/d32/d16/d1a/d3d/d59/de7/d52/d65/d82/l11d 0 2024-04-05T03:02:48.152 INFO:tasks.workunit.client.0.smithi153.stdout:7/360: rename d2/da/d11/f18 to d2/da/d3a/d41/d31/f73 0 2024-04-05T03:02:48.153 INFO:tasks.workunit.client.0.smithi153.stdout:7/361: fdatasync d2/da/d11/f72 0 2024-04-05T03:02:48.154 INFO:tasks.workunit.client.1.smithi203.stdout:1/688: mknod d7/d24/d33/d48/d4b/d6b/cdb 0 2024-04-05T03:02:48.155 INFO:tasks.workunit.client.1.smithi203.stdout:2/840: dread d5/d10/d35/d2d/d7c/d72/d8f/fe4 [4194304,4194304] 0 2024-04-05T03:02:48.156 INFO:tasks.workunit.client.1.smithi203.stdout:5/680: dread d3/f7a [0,4194304] 0 2024-04-05T03:02:48.156 INFO:tasks.workunit.client.0.smithi153.stdout:7/362: fdatasync d2/da/d42/d6d/f3e 0 2024-04-05T03:02:48.156 INFO:tasks.workunit.client.1.smithi203.stdout:8/704: truncate d1/d12/d1a/f78 3282552 0 2024-04-05T03:02:48.161 INFO:tasks.workunit.client.1.smithi203.stdout:2/841: mknod d5/d66/c122 0 2024-04-05T03:02:48.162 INFO:tasks.workunit.client.1.smithi203.stdout:2/842: write d5/d10/d1a/fa6 [480559,117181] 0 2024-04-05T03:02:48.163 INFO:tasks.workunit.client.0.smithi153.stdout:4/463: dwrite da/db/d20/d45/f72 [0,4194304] 0 2024-04-05T03:02:48.166 INFO:tasks.workunit.client.0.smithi153.stdout:7/363: dwrite d2/f5a [0,4194304] 0 2024-04-05T03:02:48.167 INFO:tasks.workunit.client.1.smithi203.stdout:1/689: read d7/f77 [128079,110757] 0 2024-04-05T03:02:48.173 INFO:tasks.workunit.client.1.smithi203.stdout:5/681: mknod d3/d4/d9e/dde/ce6 0 2024-04-05T03:02:48.173 INFO:tasks.workunit.client.0.smithi153.stdout:7/364: creat d2/da/d42/f74 x:0 0 0 2024-04-05T03:02:48.175 INFO:tasks.workunit.client.0.smithi153.stdout:7/365: write d2/da/d3a/d41/d31/f5f [2381432,105902] 0 2024-04-05T03:02:48.175 INFO:tasks.workunit.client.1.smithi203.stdout:1/690: truncate d7/d24/f35 1726392 0 2024-04-05T03:02:48.176 INFO:tasks.workunit.client.0.smithi153.stdout:7/366: fdatasync d2/f5 0 2024-04-05T03:02:48.177 INFO:tasks.workunit.client.1.smithi203.stdout:4/526: dwrite da/f10 [0,4194304] 0 2024-04-05T03:02:48.181 INFO:tasks.workunit.client.1.smithi203.stdout:4/527: chown da/d29/l54 0 1 2024-04-05T03:02:48.181 INFO:tasks.workunit.client.0.smithi153.stdout:7/367: rename d2/da/d11/d39/l66 to d2/da/d42/d6d/d54/l75 0 2024-04-05T03:02:48.181 INFO:tasks.workunit.client.1.smithi203.stdout:8/705: getdents d1/d12/d2f/d1f 0 2024-04-05T03:02:48.182 INFO:tasks.workunit.client.0.smithi153.stdout:7/368: write d2/da/d3a/d41/d31/f5f [3358617,99350] 0 2024-04-05T03:02:48.183 INFO:tasks.workunit.client.0.smithi153.stdout:7/369: write d2/da/d11/f72 [715640,108728] 0 2024-04-05T03:02:48.184 INFO:tasks.workunit.client.1.smithi203.stdout:0/697: write dd/d17/d1d/d26/f3d [511800,25165] 0 2024-04-05T03:02:48.184 INFO:tasks.workunit.client.1.smithi203.stdout:5/682: creat d3/d4/dc/d67/dac/fe7 x:0 0 0 2024-04-05T03:02:48.186 INFO:tasks.workunit.client.1.smithi203.stdout:2/843: link d5/d10/d2f/d5d/ff9 d5/d10/db8/f123 0 2024-04-05T03:02:48.186 INFO:tasks.workunit.client.1.smithi203.stdout:6/722: dwrite d0/da/d18/d38/d2a/d2c/f4f [0,4194304] 0 2024-04-05T03:02:48.187 INFO:tasks.workunit.client.0.smithi153.stdout:7/370: creat d2/da/d11/d39/f76 x:0 0 0 2024-04-05T03:02:48.188 INFO:tasks.workunit.client.1.smithi203.stdout:8/706: creat d1/d4b/d68/d75/dac/da4/fc9 x:0 0 0 2024-04-05T03:02:48.189 INFO:tasks.workunit.client.0.smithi153.stdout:7/371: dread - d2/da/d42/f44 zero size 2024-04-05T03:02:48.189 INFO:tasks.workunit.client.1.smithi203.stdout:2/844: readlink d5/d10/d35/d2d/l3d 0 2024-04-05T03:02:48.191 INFO:tasks.workunit.client.0.smithi153.stdout:7/372: mkdir d2/d51/d77 0 2024-04-05T03:02:48.194 INFO:tasks.workunit.client.1.smithi203.stdout:1/691: rmdir d7/d14/d2b/d58/d86/d91/dcb 0 2024-04-05T03:02:48.195 INFO:tasks.workunit.client.1.smithi203.stdout:8/707: creat d1/d12/d2f/da9/fca x:0 0 0 2024-04-05T03:02:48.197 INFO:tasks.workunit.client.1.smithi203.stdout:0/698: rename dd/d17/d1d/d22/d45/d51/dbb to dd/d17/d1d/d26/d8c/dcf 0 2024-04-05T03:02:48.201 INFO:tasks.workunit.client.0.smithi153.stdout:7/373: dwrite d2/da/d42/d6d/f4e [4194304,4194304] 0 2024-04-05T03:02:48.204 INFO:tasks.workunit.client.1.smithi203.stdout:1/692: creat d7/d14/d9d/dbf/fdc x:0 0 0 2024-04-05T03:02:48.205 INFO:tasks.workunit.client.1.smithi203.stdout:1/693: fdatasync d7/d24/d33/d48/d4b/d6b/f98 0 2024-04-05T03:02:48.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/374: dwrite d2/da/d11/f2b [0,4194304] 0 2024-04-05T03:02:48.209 INFO:tasks.workunit.client.1.smithi203.stdout:8/708: creat d1/d4b/d68/fcb x:0 0 0 2024-04-05T03:02:48.210 INFO:tasks.workunit.client.1.smithi203.stdout:8/709: stat d1/d4b/d68/d75/dac/cb6 0 2024-04-05T03:02:48.211 INFO:tasks.workunit.client.1.smithi203.stdout:8/710: write d1/d6/f7c [1197129,116742] 0 2024-04-05T03:02:48.213 INFO:tasks.workunit.client.1.smithi203.stdout:5/683: rmdir d3/d1f/d59/d60/da2 0 2024-04-05T03:02:48.213 INFO:tasks.workunit.client.0.smithi153.stdout:7/375: creat d2/da/d42/d6d/d55/d6f/f78 x:0 0 0 2024-04-05T03:02:48.213 INFO:tasks.workunit.client.1.smithi203.stdout:8/711: read d1/d12/d2f/d5b/fa7 [4099546,77550] 0 2024-04-05T03:02:48.216 INFO:tasks.workunit.client.0.smithi153.stdout:7/376: creat d2/d51/f79 x:0 0 0 2024-04-05T03:02:48.217 INFO:tasks.workunit.client.1.smithi203.stdout:7/915: truncate d8/d32/d16/d1a/d3d/d59/de7/d5e/d77/f87 3608677 0 2024-04-05T03:02:48.218 INFO:tasks.workunit.client.1.smithi203.stdout:2/845: getdents d5/d10/db8/dbe/d115 0 2024-04-05T03:02:48.225 INFO:tasks.workunit.client.1.smithi203.stdout:7/916: rmdir d8/d32/d5d/d6d/daf 39 2024-04-05T03:02:48.229 INFO:tasks.workunit.client.1.smithi203.stdout:2/846: rename d5/d10/db8/dbe/le8 to d5/d10/db8/dab/de1/l124 0 2024-04-05T03:02:48.229 INFO:tasks.workunit.client.0.smithi153.stdout:7/377: rename d2/da/d42/d6d to d2/da/d3a/d41/d7a 0 2024-04-05T03:02:48.230 INFO:tasks.workunit.client.1.smithi203.stdout:0/699: creat dd/fd0 x:0 0 0 2024-04-05T03:02:48.231 INFO:tasks.workunit.client.1.smithi203.stdout:6/723: write d0/da/d54/d4c/da1/db7/fbf [5851165,65259] 0 2024-04-05T03:02:48.232 INFO:tasks.workunit.client.0.smithi153.stdout:7/378: creat d2/f7b x:0 0 0 2024-04-05T03:02:48.235 INFO:tasks.workunit.client.1.smithi203.stdout:1/694: creat d7/d14/fdd x:0 0 0 2024-04-05T03:02:48.237 INFO:tasks.workunit.client.1.smithi203.stdout:4/528: truncate da/d3c/f43 4591257 0 2024-04-05T03:02:48.237 INFO:tasks.workunit.client.1.smithi203.stdout:2/847: dread d5/d10/d2f/f88 [0,4194304] 0 2024-04-05T03:02:48.238 INFO:tasks.workunit.client.1.smithi203.stdout:4/529: readlink da/d29/d2e/d57/l70 0 2024-04-05T03:02:48.240 INFO:tasks.workunit.client.1.smithi203.stdout:7/917: mkdir d8/d11e 0 2024-04-05T03:02:48.241 INFO:tasks.workunit.client.1.smithi203.stdout:1/695: fdatasync d7/d24/f3e 0 2024-04-05T03:02:48.242 INFO:tasks.workunit.client.1.smithi203.stdout:1/696: stat d7/d14/d9d/dbf 0 2024-04-05T03:02:48.243 INFO:tasks.workunit.client.1.smithi203.stdout:5/684: rename d3/f18 to d3/d4/dc/d67/db5/fe8 0 2024-04-05T03:02:48.244 INFO:tasks.workunit.client.1.smithi203.stdout:5/685: readlink d3/d1f/d59/d64/d12/l97 0 2024-04-05T03:02:48.245 INFO:tasks.workunit.client.1.smithi203.stdout:6/724: symlink d0/da/d18/d92/lda 0 2024-04-05T03:02:48.247 INFO:tasks.workunit.client.1.smithi203.stdout:4/530: chown da/d3c/f5f 15 1 2024-04-05T03:02:48.249 INFO:tasks.workunit.client.1.smithi203.stdout:1/697: dwrite d7/d14/f1e [8388608,4194304] 0 2024-04-05T03:02:48.250 INFO:tasks.workunit.client.1.smithi203.stdout:4/531: readlink da/d3c/d4c/l4f 0 2024-04-05T03:02:48.251 INFO:tasks.workunit.client.1.smithi203.stdout:0/700: creat dd/d17/d1d/d26/fd1 x:0 0 0 2024-04-05T03:02:48.253 INFO:tasks.workunit.client.1.smithi203.stdout:1/698: write d7/f99 [633198,106135] 0 2024-04-05T03:02:48.254 INFO:tasks.workunit.client.1.smithi203.stdout:5/686: symlink d3/d1f/d59/d64/d12/le9 0 2024-04-05T03:02:48.258 INFO:tasks.workunit.client.1.smithi203.stdout:8/712: dwrite d1/d12/d1a/f42 [0,4194304] 0 2024-04-05T03:02:48.258 INFO:tasks.workunit.client.1.smithi203.stdout:4/532: mknod da/d29/d2e/d57/ca3 0 2024-04-05T03:02:48.260 INFO:tasks.workunit.client.1.smithi203.stdout:8/713: dread - d1/d12/d2f/da9/fca zero size 2024-04-05T03:02:48.260 INFO:tasks.workunit.client.1.smithi203.stdout:1/699: write d7/d24/d33/d48/d4b/d6b/fb3 [551966,53303] 0 2024-04-05T03:02:48.265 INFO:tasks.workunit.client.1.smithi203.stdout:0/701: creat dd/d17/d1d/d22/d45/dbe/fd2 x:0 0 0 2024-04-05T03:02:48.266 INFO:tasks.workunit.client.1.smithi203.stdout:1/700: read d7/d24/d33/d48/d4b/d51/f87 [3648515,42016] 0 2024-04-05T03:02:48.268 INFO:tasks.workunit.client.1.smithi203.stdout:6/725: write d0/da/d54/d4c/d7f/f91 [726256,99446] 0 2024-04-05T03:02:48.270 INFO:tasks.workunit.client.1.smithi203.stdout:8/714: rename d1/d12/d2f/d1f/fae to d1/d12/d1a/d6d/fcc 0 2024-04-05T03:02:48.270 INFO:tasks.workunit.client.1.smithi203.stdout:0/702: fdatasync dd/d17/d1d/d26/d28/fad 0 2024-04-05T03:02:48.274 INFO:tasks.workunit.client.1.smithi203.stdout:7/918: dwrite d8/da/f14 [0,4194304] 0 2024-04-05T03:02:48.274 INFO:tasks.workunit.client.1.smithi203.stdout:4/533: getdents da/d29/d2e/d57/d96 0 2024-04-05T03:02:48.276 INFO:tasks.workunit.client.1.smithi203.stdout:2/848: dread d5/d10/f5b [0,4194304] 0 2024-04-05T03:02:48.278 INFO:tasks.workunit.client.1.smithi203.stdout:6/726: mknod d0/da/d54/d4c/da1/cdb 0 2024-04-05T03:02:48.280 INFO:tasks.workunit.client.1.smithi203.stdout:4/534: fsync da/d29/d6a/d80/fa0 0 2024-04-05T03:02:48.281 INFO:tasks.workunit.client.1.smithi203.stdout:0/703: rename dd/d17/d1d/f1e to dd/d17/d1d/d22/dc1/fd3 0 2024-04-05T03:02:48.282 INFO:tasks.workunit.client.1.smithi203.stdout:6/727: write d0/da/d18/d38/d86/f7c [12138,96341] 0 2024-04-05T03:02:48.283 INFO:tasks.workunit.client.1.smithi203.stdout:7/919: creat d8/d32/d16/d1a/d3d/d59/de7/d52/d65/f11f x:0 0 0 2024-04-05T03:02:48.284 INFO:tasks.workunit.client.1.smithi203.stdout:0/704: write dd/d17/d1d/d26/d28/f87 [192433,71779] 0 2024-04-05T03:02:48.287 INFO:tasks.workunit.client.1.smithi203.stdout:4/535: creat da/d29/d2e/d53/fa4 x:0 0 0 2024-04-05T03:02:48.288 INFO:tasks.workunit.client.1.smithi203.stdout:2/849: dwrite d5/d66/f10a [0,4194304] 0 2024-04-05T03:02:48.288 INFO:tasks.workunit.client.1.smithi203.stdout:7/920: mkdir d8/d32/d16/d1a/d3d/d59/d120 0 2024-04-05T03:02:48.292 INFO:tasks.workunit.client.1.smithi203.stdout:6/728: fdatasync d0/da/d54/d53/fb3 0 2024-04-05T03:02:48.292 INFO:tasks.workunit.client.1.smithi203.stdout:8/715: fdatasync d1/d6/f56 0 2024-04-05T03:02:48.293 INFO:tasks.workunit.client.1.smithi203.stdout:4/536: readlink da/d3c/l3f 0 2024-04-05T03:02:48.294 INFO:tasks.workunit.client.1.smithi203.stdout:5/687: dread d3/d1f/d59/d64/f2e [0,4194304] 0 2024-04-05T03:02:48.295 INFO:tasks.workunit.client.1.smithi203.stdout:7/921: write d8/d32/d16/d1a/d5a/fa3 [2077399,114330] 0 2024-04-05T03:02:48.296 INFO:tasks.workunit.client.1.smithi203.stdout:0/705: link dd/d17/d1d/d22/f3b dd/d17/d1d/d22/d53/db0/fd4 0 2024-04-05T03:02:48.298 INFO:tasks.workunit.client.1.smithi203.stdout:0/706: dread - dd/d17/d1d/d26/d8c/fc9 zero size 2024-04-05T03:02:48.299 INFO:tasks.workunit.client.1.smithi203.stdout:7/922: truncate d8/d32/dc3/dd8/f11b 35804 0 2024-04-05T03:02:48.299 INFO:tasks.workunit.client.1.smithi203.stdout:2/850: truncate d5/d10/d44/fec 83821 0 2024-04-05T03:02:48.301 INFO:tasks.workunit.client.1.smithi203.stdout:8/716: mkdir d1/d90/dcd 0 2024-04-05T03:02:48.301 INFO:tasks.workunit.client.1.smithi203.stdout:6/729: write d0/da/d54/fb0 [3987815,11262] 0 2024-04-05T03:02:48.304 INFO:tasks.workunit.client.1.smithi203.stdout:0/707: symlink dd/db1/ld5 0 2024-04-05T03:02:48.304 INFO:tasks.workunit.client.1.smithi203.stdout:5/688: symlink d3/d4/dc/d67/db5/dda/lea 0 2024-04-05T03:02:48.307 INFO:tasks.workunit.client.1.smithi203.stdout:2/851: rename d5/d10/d35/d2d/d7c/d29/de5/lfb to d5/d10/d35/d36/db6/d87/l125 0 2024-04-05T03:02:48.307 INFO:tasks.workunit.client.1.smithi203.stdout:4/537: dwrite da/f37 [0,4194304] 0 2024-04-05T03:02:48.309 INFO:tasks.workunit.client.1.smithi203.stdout:6/730: creat d0/da/d18/fdc x:0 0 0 2024-04-05T03:02:48.310 INFO:tasks.workunit.client.1.smithi203.stdout:5/689: readlink d3/d1f/d59/d64/l55 0 2024-04-05T03:02:48.313 INFO:tasks.workunit.client.1.smithi203.stdout:5/690: dread - d3/d4/f8e zero size 2024-04-05T03:02:48.313 INFO:tasks.workunit.client.1.smithi203.stdout:6/731: read - d0/da/d54/d4c/da1/db7/fcb zero size 2024-04-05T03:02:48.319 INFO:tasks.workunit.client.1.smithi203.stdout:4/538: dwrite da/d29/d2e/d57/f7f [0,4194304] 0 2024-04-05T03:02:48.320 INFO:tasks.workunit.client.1.smithi203.stdout:2/852: dwrite d5/d66/f10a [4194304,4194304] 0 2024-04-05T03:02:48.321 INFO:tasks.workunit.client.1.smithi203.stdout:7/923: write d8/d32/d16/d1a/d3d/f76 [4676323,21416] 0 2024-04-05T03:02:48.325 INFO:tasks.workunit.client.1.smithi203.stdout:1/701: dread d7/d14/d2b/d58/f60 [0,4194304] 0 2024-04-05T03:02:48.328 INFO:tasks.workunit.client.1.smithi203.stdout:4/539: mkdir da/d29/d2e/d57/da5 0 2024-04-05T03:02:48.330 INFO:tasks.workunit.client.1.smithi203.stdout:8/717: truncate d1/d12/d1a/f1d 661342 0 2024-04-05T03:02:48.332 INFO:tasks.workunit.client.1.smithi203.stdout:6/732: dwrite d0/da/d54/d4c/f77 [0,4194304] 0 2024-04-05T03:02:48.337 INFO:tasks.workunit.client.1.smithi203.stdout:8/718: write d1/d4b/d5f/d74/fb5 [3658592,66622] 0 2024-04-05T03:02:48.339 INFO:tasks.workunit.client.1.smithi203.stdout:6/733: mknod d0/da/d54/cdd 0 2024-04-05T03:02:48.340 INFO:tasks.workunit.client.1.smithi203.stdout:7/924: dwrite d8/d32/d5d/dbe/dc9/fd6 [0,4194304] 0 2024-04-05T03:02:48.340 INFO:tasks.workunit.client.1.smithi203.stdout:8/719: mkdir d1/d12/d2f/da9/dce 0 2024-04-05T03:02:48.343 INFO:tasks.workunit.client.1.smithi203.stdout:6/734: write d0/da/d54/d4c/fca [787322,95704] 0 2024-04-05T03:02:48.344 INFO:tasks.workunit.client.1.smithi203.stdout:4/540: dread da/d29/d2e/d53/f6f [0,4194304] 0 2024-04-05T03:02:48.346 INFO:tasks.workunit.client.1.smithi203.stdout:8/720: creat d1/d12/d17/fcf x:0 0 0 2024-04-05T03:02:48.349 INFO:tasks.workunit.client.1.smithi203.stdout:6/735: unlink d0/da/d18/fdc 0 2024-04-05T03:02:48.349 INFO:tasks.workunit.client.1.smithi203.stdout:4/541: dread da/f10 [0,4194304] 0 2024-04-05T03:02:48.354 INFO:tasks.workunit.client.1.smithi203.stdout:7/925: rename d8/d32/d16/d1a/d3d/d59/de7/d52/d65 to d8/d32/dc3/d121 0 2024-04-05T03:02:48.356 INFO:tasks.workunit.client.0.smithi153.stdout:9/238: sync 2024-04-05T03:02:48.359 INFO:tasks.workunit.client.1.smithi203.stdout:8/721: mkdir d1/d12/d17/d47/d53/dd0 0 2024-04-05T03:02:48.359 INFO:tasks.workunit.client.0.smithi153.stdout:9/239: symlink d1/d10/d36/l54 0 2024-04-05T03:02:48.360 INFO:tasks.workunit.client.1.smithi203.stdout:6/736: rmdir d0/da/d18/d38/d2a/d44/d78 39 2024-04-05T03:02:48.362 INFO:tasks.workunit.client.1.smithi203.stdout:7/926: rmdir d8/d32/d16/d1a/d3d/d59/de7/d52 39 2024-04-05T03:02:48.362 INFO:tasks.workunit.client.0.smithi153.stdout:9/240: rename d1/d10/d16/l35 to d1/d10/d16/l55 0 2024-04-05T03:02:48.363 INFO:tasks.workunit.client.1.smithi203.stdout:5/691: dread d3/d1f/d59/d64/f1c [0,4194304] 0 2024-04-05T03:02:48.364 INFO:tasks.workunit.client.0.smithi153.stdout:9/241: creat d1/d10/d36/f56 x:0 0 0 2024-04-05T03:02:48.365 INFO:tasks.workunit.client.1.smithi203.stdout:8/722: mknod d1/d12/d17/d47/d53/cd1 0 2024-04-05T03:02:48.371 INFO:tasks.workunit.client.1.smithi203.stdout:1/702: dwrite d7/d24/d33/d48/d4b/d6b/db9/fce [0,4194304] 0 2024-04-05T03:02:48.371 INFO:tasks.workunit.client.1.smithi203.stdout:7/927: unlink d8/d32/d16/fe1 0 2024-04-05T03:02:48.371 INFO:tasks.workunit.client.0.smithi153.stdout:9/242: dwrite d1/d10/d16/f27 [4194304,4194304] 0 2024-04-05T03:02:48.378 INFO:tasks.workunit.client.1.smithi203.stdout:6/737: mknod d0/da/d18/d38/d2a/d44/d78/da7/cde 0 2024-04-05T03:02:48.378 INFO:tasks.workunit.client.1.smithi203.stdout:7/928: readlink d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/l119 0 2024-04-05T03:02:48.378 INFO:tasks.workunit.client.1.smithi203.stdout:8/723: unlink d1/d12/d2f/da9/fca 0 2024-04-05T03:02:48.379 INFO:tasks.workunit.client.1.smithi203.stdout:5/692: dwrite d3/d4/dc/d67/dac/fe7 [0,4194304] 0 2024-04-05T03:02:48.381 INFO:tasks.workunit.client.0.smithi153.stdout:9/243: dwrite d1/d10/d16/f1a [0,4194304] 0 2024-04-05T03:02:48.381 INFO:tasks.workunit.client.1.smithi203.stdout:1/703: mkdir d7/d24/d33/d48/d4b/d6b/db9/dde 0 2024-04-05T03:02:48.385 INFO:tasks.workunit.client.1.smithi203.stdout:1/704: mknod d7/d79/cdf 0 2024-04-05T03:02:48.392 INFO:tasks.workunit.client.0.smithi153.stdout:9/244: dwrite d1/d9/d1b/f30 [4194304,4194304] 0 2024-04-05T03:02:48.394 INFO:tasks.workunit.client.1.smithi203.stdout:6/738: dread d0/da/d18/d38/d2a/d44/f4a [0,4194304] 0 2024-04-05T03:02:48.395 INFO:tasks.workunit.client.0.smithi153.stdout:9/245: chown d1/d10/d16/l55 7158 1 2024-04-05T03:02:48.396 INFO:tasks.workunit.client.1.smithi203.stdout:1/705: truncate d7/d24/f66 414031 0 2024-04-05T03:02:48.396 INFO:tasks.workunit.client.1.smithi203.stdout:5/693: dwrite d3/d1f/d59/d64/f2e [0,4194304] 0 2024-04-05T03:02:48.397 INFO:tasks.workunit.client.1.smithi203.stdout:4/542: dread da/fc [0,4194304] 0 2024-04-05T03:02:48.399 INFO:tasks.workunit.client.1.smithi203.stdout:1/706: truncate d7/d79/f88 1005969 0 2024-04-05T03:02:48.399 INFO:tasks.workunit.client.1.smithi203.stdout:5/694: write d3/d1f/d59/d60/dc1/fd8 [789505,42345] 0 2024-04-05T03:02:48.400 INFO:tasks.workunit.client.0.smithi153.stdout:9/246: mkdir d1/d10/d36/d57 0 2024-04-05T03:02:48.404 INFO:tasks.workunit.client.0.smithi153.stdout:9/247: creat d1/d10/d16/d22/d2c/f58 x:0 0 0 2024-04-05T03:02:48.405 INFO:tasks.workunit.client.1.smithi203.stdout:4/543: rmdir da/d3c/d4c 39 2024-04-05T03:02:48.408 INFO:tasks.workunit.client.0.smithi153.stdout:9/248: unlink d1/f25 0 2024-04-05T03:02:48.409 INFO:tasks.workunit.client.1.smithi203.stdout:6/739: rename d0/da/d18/d38/d2a/d2c/l3b to d0/da/d54/d4c/ldf 0 2024-04-05T03:02:48.410 INFO:tasks.workunit.client.1.smithi203.stdout:1/707: mknod d7/d24/d33/d48/d4b/ce0 0 2024-04-05T03:02:48.411 INFO:tasks.workunit.client.0.smithi153.stdout:9/249: symlink d1/d10/d16/l59 0 2024-04-05T03:02:48.413 INFO:tasks.workunit.client.1.smithi203.stdout:7/929: truncate d8/da/d72/f102 447090 0 2024-04-05T03:02:48.413 INFO:tasks.workunit.client.1.smithi203.stdout:1/708: chown d7/c82 676 1 2024-04-05T03:02:48.414 INFO:tasks.workunit.client.0.smithi153.stdout:9/250: creat d1/d1d/f5a x:0 0 0 2024-04-05T03:02:48.414 INFO:tasks.workunit.client.1.smithi203.stdout:5/695: dwrite d3/d5f/d92/fbc [0,4194304] 0 2024-04-05T03:02:48.416 INFO:tasks.workunit.client.1.smithi203.stdout:7/930: write d8/d32/d16/d35/d64/f8e [5141159,66202] 0 2024-04-05T03:02:48.416 INFO:tasks.workunit.client.1.smithi203.stdout:8/724: truncate d1/d4b/d68/f95 1389730 0 2024-04-05T03:02:48.417 INFO:tasks.workunit.client.1.smithi203.stdout:8/725: stat d1/d12/d17/d47/d53/d69 0 2024-04-05T03:02:48.420 INFO:tasks.workunit.client.1.smithi203.stdout:8/726: write d1/d12/d2f/d5b/fb4 [4672105,121066] 0 2024-04-05T03:02:48.422 INFO:tasks.workunit.client.1.smithi203.stdout:5/696: chown d3/d4/dc/d67/f70 6138 1 2024-04-05T03:02:48.424 INFO:tasks.workunit.client.1.smithi203.stdout:7/931: creat d8/d32/dc3/d121/d82/f122 x:0 0 0 2024-04-05T03:02:48.427 INFO:tasks.workunit.client.1.smithi203.stdout:8/727: symlink d1/d12/d1a/d6d/ld2 0 2024-04-05T03:02:48.428 INFO:tasks.workunit.client.1.smithi203.stdout:7/932: dread d8/fe [0,4194304] 0 2024-04-05T03:02:48.429 INFO:tasks.workunit.client.1.smithi203.stdout:5/697: link d3/d4/dc/d67/db5/dda/lea d3/d1f/d59/d64/d12/leb 0 2024-04-05T03:02:48.431 INFO:tasks.workunit.client.1.smithi203.stdout:8/728: creat d1/d12/d2f/d55/fd3 x:0 0 0 2024-04-05T03:02:48.432 INFO:tasks.workunit.client.1.smithi203.stdout:7/933: mknod d8/da/d71/d10a/c123 0 2024-04-05T03:02:48.433 INFO:tasks.workunit.client.1.smithi203.stdout:7/934: creat d8/da/f124 x:0 0 0 2024-04-05T03:02:48.435 INFO:tasks.workunit.client.0.smithi153.stdout:9/251: fdatasync d1/d9/d1b/f30 0 2024-04-05T03:02:48.435 INFO:tasks.workunit.client.0.smithi153.stdout:9/252: chown d1/db 5668408 1 2024-04-05T03:02:48.436 INFO:tasks.workunit.client.1.smithi203.stdout:8/729: rmdir d1/d4b/d68/d75/dac/daa 0 2024-04-05T03:02:48.436 INFO:tasks.workunit.client.1.smithi203.stdout:8/730: fsync d1/d6/f45 0 2024-04-05T03:02:48.438 INFO:tasks.workunit.client.0.smithi153.stdout:9/253: creat d1/d10/f5b x:0 0 0 2024-04-05T03:02:48.439 INFO:tasks.workunit.client.1.smithi203.stdout:6/740: dread d0/da/d18/f26 [0,4194304] 0 2024-04-05T03:02:48.439 INFO:tasks.workunit.client.1.smithi203.stdout:6/741: stat d0/db4/fc8 0 2024-04-05T03:02:48.441 INFO:tasks.workunit.client.0.smithi153.stdout:9/254: mkdir d1/d10/d36/d57/d5c 0 2024-04-05T03:02:48.441 INFO:tasks.workunit.client.1.smithi203.stdout:7/935: creat d8/f125 x:0 0 0 2024-04-05T03:02:48.445 INFO:tasks.workunit.client.1.smithi203.stdout:7/936: rmdir d8/d32/d16/d1a/d3d/d59/de7 39 2024-04-05T03:02:48.447 INFO:tasks.workunit.client.0.smithi153.stdout:9/255: getdents d1/d10/d36 0 2024-04-05T03:02:48.447 INFO:tasks.workunit.client.1.smithi203.stdout:4/544: truncate da/f11 2751874 0 2024-04-05T03:02:48.448 INFO:tasks.workunit.client.0.smithi153.stdout:9/256: readlink d1/d10/d36/l54 0 2024-04-05T03:02:48.448 INFO:tasks.workunit.client.1.smithi203.stdout:1/709: write d7/d24/f34 [2198384,113556] 0 2024-04-05T03:02:48.448 INFO:tasks.workunit.client.1.smithi203.stdout:7/937: rmdir d8/d32/d5d/dbe/dc9/ddd 39 2024-04-05T03:02:48.449 INFO:tasks.workunit.client.0.smithi153.stdout:9/257: creat d1/d9/d1b/f5d x:0 0 0 2024-04-05T03:02:48.450 INFO:tasks.workunit.client.1.smithi203.stdout:1/710: write d7/d14/d2b/d58/f9a [2198499,32645] 0 2024-04-05T03:02:48.450 INFO:tasks.workunit.client.1.smithi203.stdout:5/698: dread d3/d1f/f93 [0,4194304] 0 2024-04-05T03:02:48.451 INFO:tasks.workunit.client.0.smithi153.stdout:9/258: truncate d1/d10/d16/f38 1317582 0 2024-04-05T03:02:48.452 INFO:tasks.workunit.client.1.smithi203.stdout:4/545: creat da/d29/d6a/d80/fa6 x:0 0 0 2024-04-05T03:02:48.452 INFO:tasks.workunit.client.1.smithi203.stdout:1/711: fsync d7/d14/d2b/f72 0 2024-04-05T03:02:48.458 INFO:tasks.workunit.client.1.smithi203.stdout:7/938: dwrite d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:48.458 INFO:tasks.workunit.client.0.smithi153.stdout:9/259: dwrite d1/d9/d1b/f5d [0,4194304] 0 2024-04-05T03:02:48.462 INFO:tasks.workunit.client.1.smithi203.stdout:5/699: dread d3/d4/fd4 [0,4194304] 0 2024-04-05T03:02:48.463 INFO:tasks.workunit.client.1.smithi203.stdout:1/712: dwrite d7/d14/d2b/f5a [4194304,4194304] 0 2024-04-05T03:02:48.463 INFO:tasks.workunit.client.0.smithi153.stdout:9/260: rmdir d1/d9/d1b 39 2024-04-05T03:02:48.465 INFO:tasks.workunit.client.1.smithi203.stdout:5/700: symlink d3/d4/d9e/lec 0 2024-04-05T03:02:48.465 INFO:tasks.workunit.client.1.smithi203.stdout:5/701: chown d3/d4/c7b 557613643 1 2024-04-05T03:02:48.466 INFO:tasks.workunit.client.1.smithi203.stdout:8/731: write d1/d12/d1a/d6d/fcc [1593523,67968] 0 2024-04-05T03:02:48.468 INFO:tasks.workunit.client.0.smithi153.stdout:9/261: creat d1/d9/f5e x:0 0 0 2024-04-05T03:02:48.469 INFO:tasks.workunit.client.1.smithi203.stdout:7/939: mkdir d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/d126 0 2024-04-05T03:02:48.470 INFO:tasks.workunit.client.1.smithi203.stdout:6/742: write d0/da/d18/d38/d86/f2d [884186,65371] 0 2024-04-05T03:02:48.472 INFO:tasks.workunit.client.1.smithi203.stdout:1/713: unlink d7/d24/d33/cca 0 2024-04-05T03:02:48.473 INFO:tasks.workunit.client.0.smithi153.stdout:9/262: dread d1/d10/f19 [4194304,4194304] 0 2024-04-05T03:02:48.474 INFO:tasks.workunit.client.1.smithi203.stdout:6/743: write d0/da/f10 [4036730,87194] 0 2024-04-05T03:02:48.476 INFO:tasks.workunit.client.1.smithi203.stdout:5/702: dwrite d3/d1f/d59/d60/dca/fcf [0,4194304] 0 2024-04-05T03:02:48.482 INFO:tasks.workunit.client.0.smithi153.stdout:9/263: dwrite d1/d9/d1b/f28 [4194304,4194304] 0 2024-04-05T03:02:48.484 INFO:tasks.workunit.client.1.smithi203.stdout:8/732: rename d1/d6/c8 to d1/d4b/d5f/cd4 0 2024-04-05T03:02:48.484 INFO:tasks.workunit.client.0.smithi153.stdout:9/264: write d1/f1f [4894687,94240] 0 2024-04-05T03:02:48.485 INFO:tasks.workunit.client.0.smithi153.stdout:9/265: chown d1/d10/d16/l59 3717 1 2024-04-05T03:02:48.486 INFO:tasks.workunit.client.1.smithi203.stdout:1/714: dwrite d7/d24/d33/d48/d4b/d51/f67 [0,4194304] 0 2024-04-05T03:02:48.486 INFO:tasks.workunit.client.1.smithi203.stdout:5/703: creat d3/d4/d94/fed x:0 0 0 2024-04-05T03:02:48.489 INFO:tasks.workunit.client.1.smithi203.stdout:8/733: rmdir d1 39 2024-04-05T03:02:48.491 INFO:tasks.workunit.client.1.smithi203.stdout:5/704: fdatasync f2 0 2024-04-05T03:02:48.492 INFO:tasks.workunit.client.0.smithi153.stdout:9/266: dread d1/d9/d1b/f5d [0,4194304] 0 2024-04-05T03:02:48.494 INFO:tasks.workunit.client.1.smithi203.stdout:4/546: dwrite da/d3c/f44 [0,4194304] 0 2024-04-05T03:02:48.498 INFO:tasks.workunit.client.0.smithi153.stdout:9/267: rename d1/d10/f51 to d1/d10/f5f 0 2024-04-05T03:02:48.499 INFO:tasks.workunit.client.0.smithi153.stdout:9/268: chown d1/d10 1571 1 2024-04-05T03:02:48.501 INFO:tasks.workunit.client.1.smithi203.stdout:8/734: write d1/d12/d2f/d55/fd3 [102737,14942] 0 2024-04-05T03:02:48.501 INFO:tasks.workunit.client.1.smithi203.stdout:8/735: write d1/d12/d2f/d55/fd3 [464115,90115] 0 2024-04-05T03:02:48.502 INFO:tasks.workunit.client.1.smithi203.stdout:4/547: mkdir da/d29/d2e/d57/da7 0 2024-04-05T03:02:48.504 INFO:tasks.workunit.client.1.smithi203.stdout:1/715: creat d7/d14/d2b/d58/d86/d91/da6/fe1 x:0 0 0 2024-04-05T03:02:48.507 INFO:tasks.workunit.client.1.smithi203.stdout:1/716: dread - d7/d14/d2b/fd9 zero size 2024-04-05T03:02:48.509 INFO:tasks.workunit.client.1.smithi203.stdout:4/548: getdents da/d3c 0 2024-04-05T03:02:48.510 INFO:tasks.workunit.client.1.smithi203.stdout:5/705: dwrite d3/d1f/d59/d60/f72 [0,4194304] 0 2024-04-05T03:02:48.514 INFO:tasks.workunit.client.0.smithi153.stdout:0/378: sync 2024-04-05T03:02:48.515 INFO:tasks.workunit.client.1.smithi203.stdout:8/736: dread d1/d12/d1a/f46 [0,4194304] 0 2024-04-05T03:02:48.517 INFO:tasks.workunit.client.1.smithi203.stdout:7/940: dread d8/d32/dc3/dc8/fe9 [0,4194304] 0 2024-04-05T03:02:48.517 INFO:tasks.workunit.client.1.smithi203.stdout:5/706: creat d3/d5f/d92/fee x:0 0 0 2024-04-05T03:02:48.518 INFO:tasks.workunit.client.0.smithi153.stdout:0/379: mknod d9/df/c6b 0 2024-04-05T03:02:48.520 INFO:tasks.workunit.client.1.smithi203.stdout:8/737: rmdir d1 39 2024-04-05T03:02:48.522 INFO:tasks.workunit.client.1.smithi203.stdout:1/717: dwrite d7/d14/d83/fb4 [0,4194304] 0 2024-04-05T03:02:48.525 INFO:tasks.workunit.client.1.smithi203.stdout:5/707: getdents d3/d1f/d59 0 2024-04-05T03:02:48.526 INFO:tasks.workunit.client.1.smithi203.stdout:4/549: dwrite da/d29/d2e/d57/f66 [0,4194304] 0 2024-04-05T03:02:48.527 INFO:tasks.workunit.client.1.smithi203.stdout:1/718: creat d7/d24/d33/d48/d4b/d51/fe2 x:0 0 0 2024-04-05T03:02:48.528 INFO:tasks.workunit.client.0.smithi153.stdout:9/269: fsync d1/f4 0 2024-04-05T03:02:48.529 INFO:tasks.workunit.client.1.smithi203.stdout:8/738: rename d1/d12/d2f/f61 to d1/d4b/d5f/fd5 0 2024-04-05T03:02:48.531 INFO:tasks.workunit.client.1.smithi203.stdout:8/739: write d1/d12/d2f/d1f/f50 [8051513,62208] 0 2024-04-05T03:02:48.532 INFO:tasks.workunit.client.1.smithi203.stdout:4/550: mknod da/d29/d2e/d53/d78/ca8 0 2024-04-05T03:02:48.534 INFO:tasks.workunit.client.0.smithi153.stdout:9/270: rename d1/d10/d16/f27 to d1/d10/d36/f60 0 2024-04-05T03:02:48.534 INFO:tasks.workunit.client.1.smithi203.stdout:7/941: rename d8/d32/d16/d1a/d3d/d59/de7/d52/l104 to d8/d32/d16/d1a/d3d/l127 0 2024-04-05T03:02:48.536 INFO:tasks.workunit.client.0.smithi153.stdout:9/271: write d1/d10/f39 [8632150,87788] 0 2024-04-05T03:02:48.536 INFO:tasks.workunit.client.0.smithi153.stdout:9/272: chown d1/db 807816 1 2024-04-05T03:02:48.537 INFO:tasks.workunit.client.0.smithi153.stdout:0/380: write d9/f1a [1788548,40551] 0 2024-04-05T03:02:48.538 INFO:tasks.workunit.client.0.smithi153.stdout:0/381: rename d9 to d9/df/d5f/d6c 22 2024-04-05T03:02:48.538 INFO:tasks.workunit.client.0.smithi153.stdout:9/273: write d1/d9/f5e [606534,12578] 0 2024-04-05T03:02:48.541 INFO:tasks.workunit.client.1.smithi203.stdout:7/942: creat d8/d32/d5d/d6d/daf/f128 x:0 0 0 2024-04-05T03:02:48.541 INFO:tasks.workunit.client.1.smithi203.stdout:4/551: dwrite da/d29/d2e/f61 [0,4194304] 0 2024-04-05T03:02:48.541 INFO:tasks.workunit.client.1.smithi203.stdout:8/740: getdents d1/d6/d2e/d5c/dc0 0 2024-04-05T03:02:48.543 INFO:tasks.workunit.client.1.smithi203.stdout:4/552: stat da/d29/d2e/d53 0 2024-04-05T03:02:48.544 INFO:tasks.workunit.client.0.smithi153.stdout:9/274: rename d1/d10/d16/l55 to d1/d9/d1b/l61 0 2024-04-05T03:02:48.544 INFO:tasks.workunit.client.0.smithi153.stdout:0/382: getdents d9/d14/d67 0 2024-04-05T03:02:48.544 INFO:tasks.workunit.client.1.smithi203.stdout:4/553: readlink da/d29/d2e/d53/l76 0 2024-04-05T03:02:48.544 INFO:tasks.workunit.client.0.smithi153.stdout:9/275: chown d1/d9/d1b/f28 85387 1 2024-04-05T03:02:48.547 INFO:tasks.workunit.client.1.smithi203.stdout:7/943: getdents d8/d32/d16/d35/d64 0 2024-04-05T03:02:48.548 INFO:tasks.workunit.client.0.smithi153.stdout:0/383: rename d9/d65 to d9/d68/d6d 0 2024-04-05T03:02:48.550 INFO:tasks.workunit.client.0.smithi153.stdout:9/276: creat d1/d10/d36/f62 x:0 0 0 2024-04-05T03:02:48.551 INFO:tasks.workunit.client.0.smithi153.stdout:0/384: mkdir d9/d68/d6d/d6e 0 2024-04-05T03:02:48.552 INFO:tasks.workunit.client.1.smithi203.stdout:4/554: dwrite da/d3c/d4c/f87 [0,4194304] 0 2024-04-05T03:02:48.553 INFO:tasks.workunit.client.0.smithi153.stdout:9/277: unlink d1/d10/f19 0 2024-04-05T03:02:48.554 INFO:tasks.workunit.client.1.smithi203.stdout:7/944: dread d8/d32/d16/d35/fcf [0,4194304] 0 2024-04-05T03:02:48.559 INFO:tasks.workunit.client.0.smithi153.stdout:9/278: creat d1/d10/d36/d57/d5c/f63 x:0 0 0 2024-04-05T03:02:48.559 INFO:tasks.workunit.client.0.smithi153.stdout:9/279: readlink d1/d10/l2f 0 2024-04-05T03:02:48.560 INFO:tasks.workunit.client.0.smithi153.stdout:0/385: rmdir d9/d68/d6d/d6e 0 2024-04-05T03:02:48.561 INFO:tasks.workunit.client.0.smithi153.stdout:9/280: write d1/d10/f5b [674161,68980] 0 2024-04-05T03:02:48.561 INFO:tasks.workunit.client.1.smithi203.stdout:8/741: dread d1/d6/f41 [0,4194304] 0 2024-04-05T03:02:48.562 INFO:tasks.workunit.client.1.smithi203.stdout:7/945: link d8/d32/d5d/d6d/daf/f128 d8/d32/d16/f129 0 2024-04-05T03:02:48.563 INFO:tasks.workunit.client.0.smithi153.stdout:9/281: fsync d1/d9/d1b/f5d 0 2024-04-05T03:02:48.563 INFO:tasks.workunit.client.1.smithi203.stdout:4/555: dwrite da/f9a [0,4194304] 0 2024-04-05T03:02:48.564 INFO:tasks.workunit.client.0.smithi153.stdout:9/282: write d1/d10/d16/f53 [261115,110242] 0 2024-04-05T03:02:48.565 INFO:tasks.workunit.client.0.smithi153.stdout:9/283: dread - d1/d10/f5f zero size 2024-04-05T03:02:48.566 INFO:tasks.workunit.client.0.smithi153.stdout:0/386: rename d9/df/d1e/d45/d62/d64 to d9/d14/d67/d6f 0 2024-04-05T03:02:48.568 INFO:tasks.workunit.client.1.smithi203.stdout:8/742: link d1/d12/f31 d1/d6/d2e/d5c/fd6 0 2024-04-05T03:02:48.570 INFO:tasks.workunit.client.1.smithi203.stdout:8/743: symlink d1/d12/d2f/ld7 0 2024-04-05T03:02:48.572 INFO:tasks.workunit.client.1.smithi203.stdout:8/744: stat d1/d12/d1a/f1d 0 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:1/314: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:8/316: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:6/376: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:4/464: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:5/363: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:7/379: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:2/385: sync 2024-04-05T03:02:48.574 INFO:tasks.workunit.client.0.smithi153.stdout:3/349: sync 2024-04-05T03:02:48.576 INFO:tasks.workunit.client.1.smithi203.stdout:8/745: unlink d1/d12/d17/d47/d53/d69/c89 0 2024-04-05T03:02:48.576 INFO:tasks.workunit.client.0.smithi153.stdout:5/364: readlink d0/d25/l79 0 2024-04-05T03:02:48.578 INFO:tasks.workunit.client.0.smithi153.stdout:9/284: dread d1/d9/f20 [0,4194304] 0 2024-04-05T03:02:48.578 INFO:tasks.workunit.client.0.smithi153.stdout:6/377: chown d4/dc/df/c11 0 1 2024-04-05T03:02:48.579 INFO:tasks.workunit.client.0.smithi153.stdout:3/350: creat d1d/d26/d57/f7f x:0 0 0 2024-04-05T03:02:48.579 INFO:tasks.workunit.client.0.smithi153.stdout:1/315: creat d5/d8/d20/d4b/d18/d16/f5b x:0 0 0 2024-04-05T03:02:48.580 INFO:tasks.workunit.client.1.smithi203.stdout:8/746: symlink d1/d6/d2e/d5c/d99/dbc/ld8 0 2024-04-05T03:02:48.580 INFO:tasks.workunit.client.0.smithi153.stdout:2/386: symlink d9/d68/d4b/d7a/d59/l7d 0 2024-04-05T03:02:48.581 INFO:tasks.workunit.client.0.smithi153.stdout:9/285: stat d1/d10/f5f 0 2024-04-05T03:02:48.582 INFO:tasks.workunit.client.0.smithi153.stdout:7/380: rename d2/d51/d6a to d2/da/d47/d5d/d7c 0 2024-04-05T03:02:48.583 INFO:tasks.workunit.client.1.smithi203.stdout:8/747: symlink d1/d12/d2f/d55/ld9 0 2024-04-05T03:02:48.586 INFO:tasks.workunit.client.1.smithi203.stdout:8/748: creat d1/d4b/d68/d75/dac/da4/fda x:0 0 0 2024-04-05T03:02:48.587 INFO:tasks.workunit.client.1.smithi203.stdout:8/749: fsync d1/f58 0 2024-04-05T03:02:48.587 INFO:tasks.workunit.client.0.smithi153.stdout:5/365: creat d0/d25/d4d/d62/d37/d38/f87 x:0 0 0 2024-04-05T03:02:48.588 INFO:tasks.workunit.client.1.smithi203.stdout:8/750: chown d1/d6/l64 14289 1 2024-04-05T03:02:48.589 INFO:tasks.workunit.client.0.smithi153.stdout:6/378: stat d4/dc/df/c6f 0 2024-04-05T03:02:48.593 INFO:tasks.workunit.client.0.smithi153.stdout:9/286: dwrite d1/d10/d36/f48 [0,4194304] 0 2024-04-05T03:02:48.594 INFO:tasks.workunit.client.1.smithi203.stdout:8/751: dwrite d1/d4b/d68/d75/d77/f87 [0,4194304] 0 2024-04-05T03:02:48.594 INFO:tasks.workunit.client.1.smithi203.stdout:7/946: dread d8/da/fd [0,4194304] 0 2024-04-05T03:02:48.599 INFO:tasks.workunit.client.0.smithi153.stdout:1/316: rename d5/d8/d20/d4b/d18/f37 to d5/d8/d20/d3f/f5c 0 2024-04-05T03:02:48.599 INFO:tasks.workunit.client.0.smithi153.stdout:5/366: dwrite d0/f16 [4194304,4194304] 0 2024-04-05T03:02:48.601 INFO:tasks.workunit.client.0.smithi153.stdout:0/387: dread d9/db/f4b [0,4194304] 0 2024-04-05T03:02:48.601 INFO:tasks.workunit.client.0.smithi153.stdout:8/317: dread d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:48.604 INFO:tasks.workunit.client.1.smithi203.stdout:8/752: rename d1/d12/d17/fcf to d1/fdb 0 2024-04-05T03:02:48.605 INFO:tasks.workunit.client.0.smithi153.stdout:5/367: write d0/d9/f18 [78598,10266] 0 2024-04-05T03:02:48.606 INFO:tasks.workunit.client.1.smithi203.stdout:8/753: write d1/d4b/d68/d75/d77/f87 [755710,9811] 0 2024-04-05T03:02:48.609 INFO:tasks.workunit.client.1.smithi203.stdout:8/754: mknod d1/d12/d17/db8/cdc 0 2024-04-05T03:02:48.610 INFO:tasks.workunit.client.0.smithi153.stdout:4/465: getdents da/db/d20/d35/d41 0 2024-04-05T03:02:48.611 INFO:tasks.workunit.client.0.smithi153.stdout:6/379: read d4/dc/df/d16/d17/f25 [266309,59635] 0 2024-04-05T03:02:48.612 INFO:tasks.workunit.client.0.smithi153.stdout:3/351: link l16 d1d/d26/l80 0 2024-04-05T03:02:48.612 INFO:tasks.workunit.client.0.smithi153.stdout:4/466: dread - da/f4d zero size 2024-04-05T03:02:48.613 INFO:tasks.workunit.client.1.smithi203.stdout:8/755: write d1/d12/d17/d10/f49 [1055374,82408] 0 2024-04-05T03:02:48.613 INFO:tasks.workunit.client.1.smithi203.stdout:8/756: stat d1/d12/d1a/f1d 0 2024-04-05T03:02:48.614 INFO:tasks.workunit.client.1.smithi203.stdout:8/757: readlink d1/d12/d1a/la5 0 2024-04-05T03:02:48.614 INFO:tasks.workunit.client.0.smithi153.stdout:4/467: dread - da/de/d56/d6a/f87 zero size 2024-04-05T03:02:48.615 INFO:tasks.workunit.client.0.smithi153.stdout:8/318: creat d7/d13/d19/d1b/d21/f69 x:0 0 0 2024-04-05T03:02:48.617 INFO:tasks.workunit.client.1.smithi203.stdout:8/758: creat d1/d12/d1a/fdd x:0 0 0 2024-04-05T03:02:48.618 INFO:tasks.workunit.client.0.smithi153.stdout:1/317: dwrite d5/d8/d20/d4b/d1e/f4f [0,4194304] 0 2024-04-05T03:02:48.618 INFO:tasks.workunit.client.0.smithi153.stdout:8/319: write d7/d13/d19/d31/f41 [948602,124251] 0 2024-04-05T03:02:48.618 INFO:tasks.workunit.client.0.smithi153.stdout:9/287: creat d1/d10/f64 x:0 0 0 2024-04-05T03:02:48.619 INFO:tasks.workunit.client.0.smithi153.stdout:3/352: unlink d1d/f29 0 2024-04-05T03:02:48.620 INFO:tasks.workunit.client.1.smithi203.stdout:8/759: symlink d1/d4b/d5f/d74/lde 0 2024-04-05T03:02:48.621 INFO:tasks.workunit.client.0.smithi153.stdout:3/353: truncate d1d/d27/f4d 5071118 0 2024-04-05T03:02:48.621 INFO:tasks.workunit.client.0.smithi153.stdout:0/388: symlink d9/df/d1d/d61/l70 0 2024-04-05T03:02:48.622 INFO:tasks.workunit.client.0.smithi153.stdout:0/389: readlink d9/df/l2e 0 2024-04-05T03:02:48.623 INFO:tasks.workunit.client.0.smithi153.stdout:3/354: write d1d/d26/d57/f76 [455374,30424] 0 2024-04-05T03:02:48.624 INFO:tasks.workunit.client.0.smithi153.stdout:4/468: creat da/de/d56/d93/f9b x:0 0 0 2024-04-05T03:02:48.626 INFO:tasks.workunit.client.1.smithi203.stdout:8/760: link d1/d12/d2f/d1f/f23 d1/d6/d8f/fdf 0 2024-04-05T03:02:48.629 INFO:tasks.workunit.client.0.smithi153.stdout:1/318: dwrite d5/d8/d20/d4b/d18/d16/d45/f4a [0,4194304] 0 2024-04-05T03:02:48.629 INFO:tasks.workunit.client.1.smithi203.stdout:8/761: rmdir d1/d12 39 2024-04-05T03:02:48.630 INFO:tasks.workunit.client.0.smithi153.stdout:6/380: rename d4/dc/f2e to d4/dc/df/d16/d33/f83 0 2024-04-05T03:02:48.631 INFO:tasks.workunit.client.0.smithi153.stdout:2/387: write d9/da/d19/f46 [682284,24961] 0 2024-04-05T03:02:48.634 INFO:tasks.workunit.client.1.smithi203.stdout:7/947: dread d8/d32/d16/d1a/d3d/d59/de7/d52/f2e [0,4194304] 0 2024-04-05T03:02:48.634 INFO:tasks.workunit.client.0.smithi153.stdout:3/355: dwrite d1d/f28 [0,4194304] 0 2024-04-05T03:02:48.636 INFO:tasks.workunit.client.1.smithi203.stdout:8/762: unlink d1/d12/d2f/d5b/fa7 0 2024-04-05T03:02:48.640 INFO:tasks.workunit.client.0.smithi153.stdout:3/356: truncate d1d/d27/d2b/f35 509557 0 2024-04-05T03:02:48.640 INFO:tasks.workunit.client.1.smithi203.stdout:8/763: mkdir d1/d12/d17/d47/d53/de0 0 2024-04-05T03:02:48.641 INFO:tasks.workunit.client.0.smithi153.stdout:6/381: dread d4/d38/d4c/f6a [0,4194304] 0 2024-04-05T03:02:48.642 INFO:tasks.workunit.client.0.smithi153.stdout:3/357: write d1d/f3e [1531449,43612] 0 2024-04-05T03:02:48.644 INFO:tasks.workunit.client.0.smithi153.stdout:2/388: rename d9/d68/d4b to d9/da/d19/d2e/d7e 0 2024-04-05T03:02:48.647 INFO:tasks.workunit.client.0.smithi153.stdout:0/390: creat d9/db/d42/f71 x:0 0 0 2024-04-05T03:02:48.647 INFO:tasks.workunit.client.0.smithi153.stdout:5/368: dwrite d0/f44 [0,4194304] 0 2024-04-05T03:02:48.647 INFO:tasks.workunit.client.1.smithi203.stdout:8/764: write d1/d12/d2f/d5b/fb3 [423469,3383] 0 2024-04-05T03:02:48.650 INFO:tasks.workunit.client.0.smithi153.stdout:1/319: creat d5/dd/f5d x:0 0 0 2024-04-05T03:02:48.652 INFO:tasks.workunit.client.1.smithi203.stdout:7/948: getdents d8/da/d72 0 2024-04-05T03:02:48.655 INFO:tasks.workunit.client.1.smithi203.stdout:8/765: creat d1/d12/da8/fe1 x:0 0 0 2024-04-05T03:02:48.656 INFO:tasks.workunit.client.0.smithi153.stdout:2/389: rename d9/da/d19/d2e/f3a to d9/d68/f7f 0 2024-04-05T03:02:48.660 INFO:tasks.workunit.client.0.smithi153.stdout:4/469: truncate da/de/d56/d6a/d2d/f57 3571032 0 2024-04-05T03:02:48.662 INFO:tasks.workunit.client.0.smithi153.stdout:9/288: dwrite d1/d9/f20 [0,4194304] 0 2024-04-05T03:02:48.663 INFO:tasks.workunit.client.0.smithi153.stdout:0/391: dwrite d9/db/f13 [0,4194304] 0 2024-04-05T03:02:48.663 INFO:tasks.workunit.client.0.smithi153.stdout:2/390: truncate d9/da/d36/f63 982510 0 2024-04-05T03:02:48.664 INFO:tasks.workunit.client.1.smithi203.stdout:8/766: dwrite d1/d12/d1a/f42 [0,4194304] 0 2024-04-05T03:02:48.665 INFO:tasks.workunit.client.0.smithi153.stdout:1/320: mknod d5/d8/d20/d4b/d18/d16/d48/c5e 0 2024-04-05T03:02:48.667 INFO:tasks.workunit.client.0.smithi153.stdout:6/382: getdents d4/d38/d4c 0 2024-04-05T03:02:48.668 INFO:tasks.workunit.client.0.smithi153.stdout:4/470: creat da/d2a/f9c x:0 0 0 2024-04-05T03:02:48.670 INFO:tasks.workunit.client.0.smithi153.stdout:5/369: rename d0/d25/d4d/d62/d37/d39/c81 to d0/d2c/d3f/c88 0 2024-04-05T03:02:48.673 INFO:tasks.workunit.client.1.smithi203.stdout:8/767: dwrite d1/d4b/d68/d75/dac/da4/fc9 [0,4194304] 0 2024-04-05T03:02:48.675 INFO:tasks.workunit.client.0.smithi153.stdout:9/289: dwrite d1/d10/d36/d57/d5c/f63 [0,4194304] 0 2024-04-05T03:02:48.678 INFO:tasks.workunit.client.1.smithi203.stdout:7/949: write d8/d32/d16/fa6 [3280223,37867] 0 2024-04-05T03:02:48.679 INFO:tasks.workunit.client.1.smithi203.stdout:7/950: write d8/d32/d5d/d6d/fbf [3357898,48418] 0 2024-04-05T03:02:48.680 INFO:tasks.workunit.client.1.smithi203.stdout:8/768: symlink d1/d6/d2e/d5c/d99/le2 0 2024-04-05T03:02:48.680 INFO:tasks.workunit.client.0.smithi153.stdout:8/320: dread d7/fe [0,4194304] 0 2024-04-05T03:02:48.681 INFO:tasks.workunit.client.1.smithi203.stdout:7/951: dread - d8/d32/dc3/d121/d82/f122 zero size 2024-04-05T03:02:48.681 INFO:tasks.workunit.client.0.smithi153.stdout:0/392: chown d9/d14/f19 83913277 1 2024-04-05T03:02:48.683 INFO:tasks.workunit.client.0.smithi153.stdout:1/321: creat d5/d8/d20/f5f x:0 0 0 2024-04-05T03:02:48.685 INFO:tasks.workunit.client.0.smithi153.stdout:4/471: dwrite da/de/d56/d6a/f44 [0,4194304] 0 2024-04-05T03:02:48.685 INFO:tasks.workunit.client.1.smithi203.stdout:7/952: mknod d8/d32/d16/d1a/d3d/d59/d107/c12a 0 2024-04-05T03:02:48.686 INFO:tasks.workunit.client.0.smithi153.stdout:8/321: truncate d7/d2a/f65 1012261 0 2024-04-05T03:02:48.689 INFO:tasks.workunit.client.1.smithi203.stdout:7/953: fsync d8/d32/d16/d1a/d3d/d59/de3/f111 0 2024-04-05T03:02:48.690 INFO:tasks.workunit.client.0.smithi153.stdout:2/391: rename d9/da/d37/c61 to d9/da/d19/d2e/d7e/d7a/d6d/c80 0 2024-04-05T03:02:48.693 INFO:tasks.workunit.client.0.smithi153.stdout:9/290: chown d1/d10/d16/l1e 23074 1 2024-04-05T03:02:48.698 INFO:tasks.workunit.client.0.smithi153.stdout:8/322: creat d7/d13/d19/d31/f6a x:0 0 0 2024-04-05T03:02:48.699 INFO:tasks.workunit.client.0.smithi153.stdout:1/322: dwrite d5/d8/d20/d4b/d18/d16/d33/f3c [0,4194304] 0 2024-04-05T03:02:48.699 INFO:tasks.workunit.client.0.smithi153.stdout:6/383: creat d4/dc/df/d16/d17/f84 x:0 0 0 2024-04-05T03:02:48.700 INFO:tasks.workunit.client.0.smithi153.stdout:9/291: mknod d1/d10/d16/d22/c65 0 2024-04-05T03:02:48.702 INFO:tasks.workunit.client.0.smithi153.stdout:4/472: mknod da/db/d8b/c9d 0 2024-04-05T03:02:48.705 INFO:tasks.workunit.client.0.smithi153.stdout:3/358: rename d1d/d26/f69 to d1d/d26/d57/d6b/f81 0 2024-04-05T03:02:48.705 INFO:tasks.workunit.client.0.smithi153.stdout:4/473: read - da/db/f46 zero size 2024-04-05T03:02:48.706 INFO:tasks.workunit.client.0.smithi153.stdout:2/392: creat d9/da/d36/f81 x:0 0 0 2024-04-05T03:02:48.707 INFO:tasks.workunit.client.0.smithi153.stdout:0/393: getdents d9/df/d1e/d45 0 2024-04-05T03:02:48.709 INFO:tasks.workunit.client.0.smithi153.stdout:8/323: creat d7/d2a/d50/f6b x:0 0 0 2024-04-05T03:02:48.711 INFO:tasks.workunit.client.0.smithi153.stdout:1/323: creat d5/d8/d20/d2f/f60 x:0 0 0 2024-04-05T03:02:48.711 INFO:tasks.workunit.client.0.smithi153.stdout:0/394: chown d9/df/d1e/c41 243493278 1 2024-04-05T03:02:48.713 INFO:tasks.workunit.client.0.smithi153.stdout:3/359: creat d1d/d26/f82 x:0 0 0 2024-04-05T03:02:48.713 INFO:tasks.workunit.client.0.smithi153.stdout:0/395: chown d9/d14/d67 103643545 1 2024-04-05T03:02:48.716 INFO:tasks.workunit.client.0.smithi153.stdout:0/396: fsync d9/df/f17 0 2024-04-05T03:02:48.717 INFO:tasks.workunit.client.0.smithi153.stdout:6/384: dwrite d4/dc/df/d16/d5f/f74 [0,4194304] 0 2024-04-05T03:02:48.717 INFO:tasks.workunit.client.0.smithi153.stdout:9/292: dwrite d1/d10/d36/f62 [0,4194304] 0 2024-04-05T03:02:48.718 INFO:tasks.workunit.client.0.smithi153.stdout:0/397: stat d9/df/d1e/d45/c51 0 2024-04-05T03:02:48.719 INFO:tasks.workunit.client.1.smithi203.stdout:3/590: sync 2024-04-05T03:02:48.719 INFO:tasks.workunit.client.0.smithi153.stdout:2/393: read d9/da/d19/f46 [57324,82187] 0 2024-04-05T03:02:48.720 INFO:tasks.workunit.client.0.smithi153.stdout:9/293: fdatasync d1/f2a 0 2024-04-05T03:02:48.720 INFO:tasks.workunit.client.0.smithi153.stdout:4/474: creat da/db/d20/d45/f9e x:0 0 0 2024-04-05T03:02:48.721 INFO:tasks.workunit.client.0.smithi153.stdout:8/324: mkdir d7/d13/d19/d1b/d6c 0 2024-04-05T03:02:48.721 INFO:tasks.workunit.client.0.smithi153.stdout:1/324: mkdir d5/d8/d20/d4b/d18/d61 0 2024-04-05T03:02:48.722 INFO:tasks.workunit.client.0.smithi153.stdout:4/475: chown da/db/d20/d3c/d81 1 1 2024-04-05T03:02:48.723 INFO:tasks.workunit.client.0.smithi153.stdout:9/294: chown d1/d9/fc 16165522 1 2024-04-05T03:02:48.724 INFO:tasks.workunit.client.0.smithi153.stdout:3/360: rename d1d/f3e to d1d/d32/f83 0 2024-04-05T03:02:48.727 INFO:tasks.workunit.client.1.smithi203.stdout:3/591: link d0/dc/d34/f61 d0/dc/d1a/d1d/d53/d5a/fab 0 2024-04-05T03:02:48.728 INFO:tasks.workunit.client.1.smithi203.stdout:3/592: write d0/dc/d38/f9e [989387,91134] 0 2024-04-05T03:02:48.731 INFO:tasks.workunit.client.0.smithi153.stdout:3/361: dread d1d/d27/f6f [0,4194304] 0 2024-04-05T03:02:48.734 INFO:tasks.workunit.client.1.smithi203.stdout:8/769: write d1/d12/d1a/f1e [3154081,3874] 0 2024-04-05T03:02:48.734 INFO:tasks.workunit.client.0.smithi153.stdout:6/385: dwrite d4/d38/f75 [0,4194304] 0 2024-04-05T03:02:48.735 INFO:tasks.workunit.client.1.smithi203.stdout:7/954: truncate d8/d32/d5d/d6d/f96 5331079 0 2024-04-05T03:02:48.736 INFO:tasks.workunit.client.1.smithi203.stdout:8/770: write d1/d4b/d68/d75/dac/da4/fda [843711,21331] 0 2024-04-05T03:02:48.739 INFO:tasks.workunit.client.1.smithi203.stdout:7/955: mknod d8/d32/d16/d1a/d5a/db5/c12b 0 2024-04-05T03:02:48.741 INFO:tasks.workunit.client.0.smithi153.stdout:5/370: dread d0/d25/d4d/d62/f1a [4194304,4194304] 0 2024-04-05T03:02:48.741 INFO:tasks.workunit.client.0.smithi153.stdout:6/386: chown d4/d38/f75 10830 1 2024-04-05T03:02:48.742 INFO:tasks.workunit.client.1.smithi203.stdout:8/771: dwrite d1/d4b/d5f/d74/fb5 [0,4194304] 0 2024-04-05T03:02:48.742 INFO:tasks.workunit.client.1.smithi203.stdout:7/956: symlink d8/d32/dc3/dd8/l12c 0 2024-04-05T03:02:48.742 INFO:tasks.workunit.client.1.smithi203.stdout:8/772: chown d1/d4b/d68/d75 0 1 2024-04-05T03:02:48.746 INFO:tasks.workunit.client.0.smithi153.stdout:2/394: dwrite d9/d68/f7f [0,4194304] 0 2024-04-05T03:02:48.746 INFO:tasks.workunit.client.0.smithi153.stdout:0/398: dwrite d9/df/d1d/f3b [0,4194304] 0 2024-04-05T03:02:48.746 INFO:tasks.workunit.client.0.smithi153.stdout:0/399: readlink d9/db/l33 0 2024-04-05T03:02:48.747 INFO:tasks.workunit.client.0.smithi153.stdout:6/387: dread d4/dc/df/d16/d5f/f74 [0,4194304] 0 2024-04-05T03:02:48.747 INFO:tasks.workunit.client.1.smithi203.stdout:7/957: symlink d8/d32/d5d/d6d/dca/d9b/l12d 0 2024-04-05T03:02:48.751 INFO:tasks.workunit.client.0.smithi153.stdout:2/395: read d9/da/d19/d2e/d48/f1a [833771,111041] 0 2024-04-05T03:02:48.753 INFO:tasks.workunit.client.1.smithi203.stdout:7/958: rename d8/d32/d16/d1a/d3d/d59/de7/d73/cd3 to d8/da/d72/c12e 0 2024-04-05T03:02:48.757 INFO:tasks.workunit.client.1.smithi203.stdout:7/959: rename d8/d32/d16/d1a/d30/f89 to d8/d32/d16/d35/d64/f12f 0 2024-04-05T03:02:48.758 INFO:tasks.workunit.client.1.smithi203.stdout:7/960: chown d8/d32/d5d/d6d/dca/f40 12519612 1 2024-04-05T03:02:48.761 INFO:tasks.workunit.client.1.smithi203.stdout:3/593: write d0/dc/d1a/d1d/d53/d5a/fab [1221523,22813] 0 2024-04-05T03:02:48.763 INFO:tasks.workunit.client.1.smithi203.stdout:3/594: stat d0/dc/d1a/d1d/d53/f82 0 2024-04-05T03:02:48.765 INFO:tasks.workunit.client.1.smithi203.stdout:7/961: dwrite d8/d32/d16/d1a/d3d/d59/de3/f10b [0,4194304] 0 2024-04-05T03:02:48.767 INFO:tasks.workunit.client.1.smithi203.stdout:7/962: write d8/d32/d16/d1a/d3d/d59/de7/d73/da8/fc0 [1685338,55584] 0 2024-04-05T03:02:48.769 INFO:tasks.workunit.client.1.smithi203.stdout:7/963: mknod d8/da/c130 0 2024-04-05T03:02:48.795 INFO:tasks.workunit.client.1.smithi203.stdout:7/964: dread d8/d32/d16/fa6 [0,4194304] 0 2024-04-05T03:02:48.795 INFO:tasks.workunit.client.1.smithi203.stdout:7/965: chown d8/d32/d16/d1a/d3d/d59/de3/f10b 543105 1 2024-04-05T03:02:48.796 INFO:tasks.workunit.client.1.smithi203.stdout:7/966: dread - d8/d32/dc3/d121/d82/f122 zero size 2024-04-05T03:02:48.804 INFO:tasks.workunit.client.1.smithi203.stdout:7/967: rmdir d8 39 2024-04-05T03:02:48.805 INFO:tasks.workunit.client.1.smithi203.stdout:3/595: dread d0/dc/d1a/d76/fa6 [0,4194304] 0 2024-04-05T03:02:48.806 INFO:tasks.workunit.client.1.smithi203.stdout:7/968: write d8/f33 [5244758,42972] 0 2024-04-05T03:02:48.808 INFO:tasks.workunit.client.1.smithi203.stdout:7/969: stat d8/d32/d16/d1a/d3d/d59/de7/d52/f5b 0 2024-04-05T03:02:48.811 INFO:tasks.workunit.client.1.smithi203.stdout:3/596: dwrite d0/dc/d1a/d8d/d96/f5e [0,4194304] 0 2024-04-05T03:02:48.813 INFO:tasks.workunit.client.1.smithi203.stdout:3/597: write d0/dc/d1a/d8d/d96/f51 [2003220,77842] 0 2024-04-05T03:02:48.813 INFO:tasks.workunit.client.1.smithi203.stdout:7/970: fsync d8/d32/d16/d1a/d3d/d59/de7/d52/f2e 0 2024-04-05T03:02:48.823 INFO:tasks.workunit.client.1.smithi203.stdout:3/598: write d0/dc/d1a/d1d/f2f [383282,90281] 0 2024-04-05T03:02:48.825 INFO:tasks.workunit.client.1.smithi203.stdout:3/599: fdatasync d0/dc/d1a/d1f/f32 0 2024-04-05T03:02:48.826 INFO:tasks.workunit.client.0.smithi153.stdout:8/325: readlink d7/d13/d19/l66 0 2024-04-05T03:02:48.828 INFO:tasks.workunit.client.1.smithi203.stdout:7/971: dwrite d8/d32/d5d/d6d/f8d [0,4194304] 0 2024-04-05T03:02:48.828 INFO:tasks.workunit.client.0.smithi153.stdout:8/326: truncate d7/d13/d19/d1b/d33/f63 4394434 0 2024-04-05T03:02:48.828 INFO:tasks.workunit.client.1.smithi203.stdout:7/972: readlink d8/d32/dc3/dd8/l12c 0 2024-04-05T03:02:48.830 INFO:tasks.workunit.client.0.smithi153.stdout:8/327: dread d7/fe [4194304,4194304] 0 2024-04-05T03:02:48.830 INFO:tasks.workunit.client.0.smithi153.stdout:4/476: rmdir da/db/d20/d3c/d81 39 2024-04-05T03:02:48.830 INFO:tasks.workunit.client.0.smithi153.stdout:8/328: readlink d7/d2a/l40 0 2024-04-05T03:02:48.831 INFO:tasks.workunit.client.1.smithi203.stdout:7/973: unlink d8/f3e 0 2024-04-05T03:02:48.832 INFO:tasks.workunit.client.0.smithi153.stdout:9/295: symlink d1/d10/d36/l66 0 2024-04-05T03:02:48.832 INFO:tasks.workunit.client.1.smithi203.stdout:3/600: truncate d0/f5 265497 0 2024-04-05T03:02:48.833 INFO:tasks.workunit.client.1.smithi203.stdout:7/974: stat d8/d32/d16/d1a/d3d/d59/lb2 0 2024-04-05T03:02:48.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/325: mkdir d5/d8/d20/d4b/d18/d16/d2c/d62 0 2024-04-05T03:02:48.834 INFO:tasks.workunit.client.1.smithi203.stdout:7/975: truncate d8/da/f1d 5233555 0 2024-04-05T03:02:48.837 INFO:tasks.workunit.client.0.smithi153.stdout:3/362: fdatasync d1d/f21 0 2024-04-05T03:02:48.841 INFO:tasks.workunit.client.1.smithi203.stdout:3/601: dwrite d0/dc/d38/d5d/f8b [0,4194304] 0 2024-04-05T03:02:48.842 INFO:tasks.workunit.client.1.smithi203.stdout:7/976: dwrite d8/fb7 [0,4194304] 0 2024-04-05T03:02:48.848 INFO:tasks.workunit.client.0.smithi153.stdout:8/329: unlink d7/d13/f15 0 2024-04-05T03:02:48.849 INFO:tasks.workunit.client.1.smithi203.stdout:3/602: dread d0/dc/d1a/d1d/d53/f98 [0,4194304] 0 2024-04-05T03:02:48.851 INFO:tasks.workunit.client.0.smithi153.stdout:4/477: read - da/db/d20/d35/d38/f5c zero size 2024-04-05T03:02:48.851 INFO:tasks.workunit.client.0.smithi153.stdout:9/296: dread d1/d9/d1b/f26 [0,4194304] 0 2024-04-05T03:02:48.854 INFO:tasks.workunit.client.1.smithi203.stdout:3/603: mkdir d0/dc/d1a/d1f/dac 0 2024-04-05T03:02:48.855 INFO:tasks.workunit.client.1.smithi203.stdout:7/977: dwrite d8/d32/d16/d1a/d21/fbd [0,4194304] 0 2024-04-05T03:02:48.860 INFO:tasks.workunit.client.1.smithi203.stdout:3/604: rmdir d0/dc/d38/d5d/d66/d80 39 2024-04-05T03:02:48.861 INFO:tasks.workunit.client.1.smithi203.stdout:3/605: symlink d0/dc/d1a/d1f/lad 0 2024-04-05T03:02:48.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/606: fsync d0/d41/faa 0 2024-04-05T03:02:48.862 INFO:tasks.workunit.client.0.smithi153.stdout:5/371: rename d0/d2c/c50 to d0/d25/c89 0 2024-04-05T03:02:48.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/607: readlink d0/l22 0 2024-04-05T03:02:48.869 INFO:tasks.workunit.client.1.smithi203.stdout:3/608: dwrite d0/dc/d1a/d76/f9f [0,4194304] 0 2024-04-05T03:02:48.871 INFO:tasks.workunit.client.0.smithi153.stdout:4/478: creat da/de/d56/f9f x:0 0 0 2024-04-05T03:02:48.872 INFO:tasks.workunit.client.0.smithi153.stdout:5/372: dwrite d0/d25/f2b [0,4194304] 0 2024-04-05T03:02:48.874 INFO:tasks.workunit.client.1.smithi203.stdout:3/609: creat d0/dc/d1a/d1f/da2/fae x:0 0 0 2024-04-05T03:02:48.875 INFO:tasks.workunit.client.0.smithi153.stdout:2/396: rename d9/da/d19/d2e/d48/c67 to d9/da/d19/d2e/d7e/d53/c82 0 2024-04-05T03:02:48.876 INFO:tasks.workunit.client.0.smithi153.stdout:8/330: mkdir d7/d13/d19/d1b/d6d 0 2024-04-05T03:02:48.878 INFO:tasks.workunit.client.0.smithi153.stdout:4/479: creat da/de/d56/fa0 x:0 0 0 2024-04-05T03:02:48.879 INFO:tasks.workunit.client.1.smithi203.stdout:3/610: dread d0/dc/d38/d5d/f62 [0,4194304] 0 2024-04-05T03:02:48.879 INFO:tasks.workunit.client.0.smithi153.stdout:0/400: dwrite d9/df/d1d/d61/f2b [0,4194304] 0 2024-04-05T03:02:48.880 INFO:tasks.workunit.client.0.smithi153.stdout:5/373: mknod d0/d25/c8a 0 2024-04-05T03:02:48.881 INFO:tasks.workunit.client.0.smithi153.stdout:5/374: chown d0/d25/f70 0 1 2024-04-05T03:02:48.883 INFO:tasks.workunit.client.0.smithi153.stdout:9/297: creat d1/d10/f67 x:0 0 0 2024-04-05T03:02:48.883 INFO:tasks.workunit.client.1.smithi203.stdout:3/611: mkdir d0/dc/d1a/d1f/da1/daf 0 2024-04-05T03:02:48.884 INFO:tasks.workunit.client.0.smithi153.stdout:8/331: unlink d7/l1d 0 2024-04-05T03:02:48.884 INFO:tasks.workunit.client.0.smithi153.stdout:2/397: unlink d9/d77/f5a 0 2024-04-05T03:02:48.884 INFO:tasks.workunit.client.0.smithi153.stdout:0/401: mkdir d9/df/d1d/d72 0 2024-04-05T03:02:48.884 INFO:tasks.workunit.client.0.smithi153.stdout:3/363: link f3 d1d/d75/f84 0 2024-04-05T03:02:48.887 INFO:tasks.workunit.client.0.smithi153.stdout:9/298: chown d1/d10/d16/f53 8217130 1 2024-04-05T03:02:48.888 INFO:tasks.workunit.client.0.smithi153.stdout:0/402: write d9/f4e [218242,110353] 0 2024-04-05T03:02:48.895 INFO:tasks.workunit.client.1.smithi203.stdout:7/978: dread d8/d32/f44 [4194304,4194304] 0 2024-04-05T03:02:48.896 INFO:tasks.workunit.client.0.smithi153.stdout:9/299: mkdir d1/d10/d16/d22/d2c/d68 0 2024-04-05T03:02:48.902 INFO:tasks.workunit.client.0.smithi153.stdout:5/375: dwrite d0/d9/d22/f28 [0,4194304] 0 2024-04-05T03:02:48.903 INFO:tasks.workunit.client.0.smithi153.stdout:8/332: dwrite d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:48.903 INFO:tasks.workunit.client.0.smithi153.stdout:9/300: fsync d1/d10/d16/f23 0 2024-04-05T03:02:48.903 INFO:tasks.workunit.client.1.smithi203.stdout:7/979: link d8/d32/d16/d1a/d3d/d59/c118 d8/d32/d16/d1a/d21/c131 0 2024-04-05T03:02:48.909 INFO:tasks.workunit.client.0.smithi153.stdout:2/398: dwrite d9/da/f64 [0,4194304] 0 2024-04-05T03:02:48.917 INFO:tasks.workunit.client.1.smithi203.stdout:3/612: truncate d0/dc/d1a/d76/f94 2785299 0 2024-04-05T03:02:48.918 INFO:tasks.workunit.client.1.smithi203.stdout:7/980: link d8/da/d72/cc6 d8/d32/d16/d1a/d3d/d59/de7/d5e/dcd/d126/c132 0 2024-04-05T03:02:48.921 INFO:tasks.workunit.client.1.smithi203.stdout:7/981: creat d8/d32/dc3/d121/f133 x:0 0 0 2024-04-05T03:02:48.922 INFO:tasks.workunit.client.1.smithi203.stdout:7/982: dread - d8/f125 zero size 2024-04-05T03:02:48.922 INFO:tasks.workunit.client.0.smithi153.stdout:9/301: link d1/d1d/c33 d1/c69 0 2024-04-05T03:02:48.924 INFO:tasks.workunit.client.1.smithi203.stdout:7/983: mknod d8/d32/d16/d92/c134 0 2024-04-05T03:02:48.924 INFO:tasks.workunit.client.0.smithi153.stdout:5/376: rename d0/c29 to d0/d25/d4d/d62/d37/c8b 0 2024-04-05T03:02:48.925 INFO:tasks.workunit.client.0.smithi153.stdout:2/399: getdents d9/da/d19/d2e/d7e/d7a/d59 0 2024-04-05T03:02:48.925 INFO:tasks.workunit.client.0.smithi153.stdout:5/377: read d0/d9/d22/f41 [385456,17961] 0 2024-04-05T03:02:48.927 INFO:tasks.workunit.client.1.smithi203.stdout:7/984: symlink d8/d32/d16/d1a/d3d/d59/de7/d73/da8/l135 0 2024-04-05T03:02:48.928 INFO:tasks.workunit.client.0.smithi153.stdout:9/302: rename d1/d10/d16/f53 to d1/d9/d1b/f6a 0 2024-04-05T03:02:48.928 INFO:tasks.workunit.client.1.smithi203.stdout:7/985: fdatasync d8/d32/d5d/d6d/f8f 0 2024-04-05T03:02:48.928 INFO:tasks.workunit.client.1.smithi203.stdout:3/613: dread d0/d21/d81/f5b [0,4194304] 0 2024-04-05T03:02:48.930 INFO:tasks.workunit.client.1.smithi203.stdout:3/614: chown d0/dc/d1a/d8d/d96/l50 909 1 2024-04-05T03:02:48.933 INFO:tasks.workunit.client.0.smithi153.stdout:5/378: unlink d0/d25/d4d/f5a 0 2024-04-05T03:02:48.934 INFO:tasks.workunit.client.0.smithi153.stdout:9/303: symlink d1/d1d/l6b 0 2024-04-05T03:02:48.934 INFO:tasks.workunit.client.0.smithi153.stdout:5/379: stat d0/d25/d4d/d62/l7c 0 2024-04-05T03:02:48.934 INFO:tasks.workunit.client.0.smithi153.stdout:3/364: write d1d/f36 [962302,88846] 0 2024-04-05T03:02:48.940 INFO:tasks.workunit.client.0.smithi153.stdout:4/480: dread f4 [0,4194304] 0 2024-04-05T03:02:48.941 INFO:tasks.workunit.client.0.smithi153.stdout:2/400: getdents d9/da 0 2024-04-05T03:02:48.942 INFO:tasks.workunit.client.0.smithi153.stdout:2/401: fsync d9/da/d19/f1f 0 2024-04-05T03:02:48.942 INFO:tasks.workunit.client.0.smithi153.stdout:9/304: mkdir d1/d6c 0 2024-04-05T03:02:48.943 INFO:tasks.workunit.client.1.smithi203.stdout:7/986: truncate d8/d32/d5d/d6d/f101 3590082 0 2024-04-05T03:02:48.943 INFO:tasks.workunit.client.0.smithi153.stdout:4/481: write da/de/f19 [2452739,35359] 0 2024-04-05T03:02:48.944 INFO:tasks.workunit.client.0.smithi153.stdout:5/380: mknod d0/d2c/d5e/c8c 0 2024-04-05T03:02:48.944 INFO:tasks.workunit.client.0.smithi153.stdout:3/365: chown d1d/d26/d3c/c4b 53151 1 2024-04-05T03:02:48.947 INFO:tasks.workunit.client.0.smithi153.stdout:0/403: dread d9/df/f18 [0,4194304] 0 2024-04-05T03:02:48.948 INFO:tasks.workunit.client.0.smithi153.stdout:4/482: write da/de/d56/d6a/d2d/f4a [2525197,65226] 0 2024-04-05T03:02:48.949 INFO:tasks.workunit.client.0.smithi153.stdout:0/404: write d9/db/d42/f71 [857278,96655] 0 2024-04-05T03:02:48.950 INFO:tasks.workunit.client.0.smithi153.stdout:4/483: chown da/db/d20 27686 1 2024-04-05T03:02:48.952 INFO:tasks.workunit.client.1.smithi203.stdout:7/987: dwrite d8/d32/d5d/f69 [0,4194304] 0 2024-04-05T03:02:48.952 INFO:tasks.workunit.client.0.smithi153.stdout:8/333: dread d7/d2a/f2c [0,4194304] 0 2024-04-05T03:02:48.953 INFO:tasks.workunit.client.0.smithi153.stdout:7/381: sync 2024-04-05T03:02:48.954 INFO:tasks.workunit.client.1.smithi203.stdout:7/988: stat d8/d32/dc3/d121/lf6 0 2024-04-05T03:02:48.956 INFO:tasks.workunit.client.1.smithi203.stdout:7/989: mknod d8/d32/d5d/d6d/dca/c136 0 2024-04-05T03:02:48.958 INFO:tasks.workunit.client.1.smithi203.stdout:7/990: chown d8/d32/d5d/dbe/dc9/ddd 96889 1 2024-04-05T03:02:48.959 INFO:tasks.workunit.client.1.smithi203.stdout:7/991: dread - d8/f125 zero size 2024-04-05T03:02:48.962 INFO:tasks.workunit.client.1.smithi203.stdout:7/992: creat d8/d32/d16/d92/f137 x:0 0 0 2024-04-05T03:02:48.973 INFO:tasks.workunit.client.1.smithi203.stdout:7/993: write d8/fe [4181537,61419] 0 2024-04-05T03:02:48.975 INFO:tasks.workunit.client.0.smithi153.stdout:1/326: sync 2024-04-05T03:02:48.975 INFO:tasks.workunit.client.0.smithi153.stdout:6/388: sync 2024-04-05T03:02:48.976 INFO:tasks.workunit.client.0.smithi153.stdout:1/327: readlink d5/d8/d20/d2f/l35 0 2024-04-05T03:02:48.976 INFO:tasks.workunit.client.0.smithi153.stdout:6/389: fdatasync d4/dc/df/d16/d36/d52/f55 0 2024-04-05T03:02:48.978 INFO:tasks.workunit.client.0.smithi153.stdout:9/305: write d1/d9/d1b/f6a [1265618,15401] 0 2024-04-05T03:02:48.979 INFO:tasks.workunit.client.0.smithi153.stdout:2/402: creat d9/da/d19/d2e/d7e/d53/f83 x:0 0 0 2024-04-05T03:02:48.980 INFO:tasks.workunit.client.1.smithi203.stdout:7/994: link d8/d32/d16/d1a/d3d/l127 d8/d32/d16/d1a/l138 0 2024-04-05T03:02:48.982 INFO:tasks.workunit.client.0.smithi153.stdout:0/405: fdatasync d9/d14/f19 0 2024-04-05T03:02:48.983 INFO:tasks.workunit.client.1.smithi203.stdout:7/995: mknod d8/da/d71/c139 0 2024-04-05T03:02:48.985 INFO:tasks.workunit.client.1.smithi203.stdout:7/996: symlink d8/d32/d5d/l13a 0 2024-04-05T03:02:48.988 INFO:tasks.workunit.client.0.smithi153.stdout:5/381: link d0/f75 d0/d25/d4d/f8d 0 2024-04-05T03:02:48.989 INFO:tasks.workunit.client.0.smithi153.stdout:5/382: stat d0/d9/d22/l2e 0 2024-04-05T03:02:48.992 INFO:tasks.workunit.client.0.smithi153.stdout:6/390: creat d4/dc/df/d16/d17/d54/f85 x:0 0 0 2024-04-05T03:02:48.993 INFO:tasks.workunit.client.0.smithi153.stdout:6/391: chown d4/dc/df/c11 112306831 1 2024-04-05T03:02:48.994 INFO:tasks.workunit.client.0.smithi153.stdout:6/392: read - d4/dc/df/d16/d24/d5e/f60 zero size 2024-04-05T03:02:48.998 INFO:tasks.workunit.client.0.smithi153.stdout:7/382: fdatasync d2/da/d11/d39/f63 0 2024-04-05T03:02:49.005 INFO:tasks.workunit.client.0.smithi153.stdout:2/403: link d9/da/d19/d2e/d7e/d53/f83 d9/da/d19/d2e/d7e/d7a/f84 0 2024-04-05T03:02:49.006 INFO:tasks.workunit.client.0.smithi153.stdout:6/393: rename d4/dc/c43 to d4/d38/d4c/c86 0 2024-04-05T03:02:49.013 INFO:tasks.workunit.client.0.smithi153.stdout:3/366: getdents d1d/d75 0 2024-04-05T03:02:49.014 INFO:tasks.workunit.client.0.smithi153.stdout:3/367: chown d1d/d26/d3c/f46 1839749 1 2024-04-05T03:02:49.016 INFO:tasks.workunit.client.0.smithi153.stdout:2/404: mknod d9/da/d19/d2e/d4f/c85 0 2024-04-05T03:02:49.017 INFO:tasks.workunit.client.0.smithi153.stdout:6/394: creat d4/dc/df/d16/d36/d4d/f87 x:0 0 0 2024-04-05T03:02:49.018 INFO:tasks.workunit.client.0.smithi153.stdout:0/406: link d9/df/l2e d9/db/d42/d43/l73 0 2024-04-05T03:02:49.025 INFO:tasks.workunit.client.0.smithi153.stdout:3/368: symlink d1d/d27/l85 0 2024-04-05T03:02:49.026 INFO:tasks.workunit.client.0.smithi153.stdout:2/405: link d9/da/d36/f4d d9/da/d19/d2e/d4f/f86 0 2024-04-05T03:02:49.029 INFO:tasks.workunit.client.0.smithi153.stdout:3/369: creat d1d/d27/d2b/f86 x:0 0 0 2024-04-05T03:02:49.032 INFO:tasks.workunit.client.0.smithi153.stdout:3/370: stat d1d/c1f 0 2024-04-05T03:02:49.032 INFO:tasks.workunit.client.1.smithi203.stdout:9/695: sync 2024-04-05T03:02:49.033 INFO:tasks.workunit.client.0.smithi153.stdout:6/395: getdents d4/dc/df/d16/d5f/d73 0 2024-04-05T03:02:49.034 INFO:tasks.workunit.client.0.smithi153.stdout:6/396: dread - d4/dc/df/d16/d5f/f7c zero size 2024-04-05T03:02:49.035 INFO:tasks.workunit.client.0.smithi153.stdout:3/371: rename d1d/d32/c7d to d1d/d26/d57/c87 0 2024-04-05T03:02:49.035 INFO:tasks.workunit.client.0.smithi153.stdout:6/397: write d4/dc/df/d16/d24/d5e/f62 [929009,97474] 0 2024-04-05T03:02:49.036 INFO:tasks.workunit.client.1.smithi203.stdout:9/696: getdents d13/d1b/d33/d42/d67 0 2024-04-05T03:02:49.038 INFO:tasks.workunit.client.0.smithi153.stdout:6/398: truncate d4/dc/df/d16/d17/f2a 4757163 0 2024-04-05T03:02:49.038 INFO:tasks.workunit.client.0.smithi153.stdout:6/399: stat d4/dc/df/f12 0 2024-04-05T03:02:49.039 INFO:tasks.workunit.client.1.smithi203.stdout:9/697: symlink d13/d1b/d6b/d9e/d59/d5f/le5 0 2024-04-05T03:02:49.042 INFO:tasks.workunit.client.0.smithi153.stdout:5/383: getdents d0/d25/d4d 0 2024-04-05T03:02:49.044 INFO:tasks.workunit.client.0.smithi153.stdout:5/384: mknod d0/d25/d4d/d62/d37/d38/c8e 0 2024-04-05T03:02:49.045 INFO:tasks.workunit.client.0.smithi153.stdout:5/385: creat d0/d2c/f8f x:0 0 0 2024-04-05T03:02:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:48 smithi203 ceph-mon[39911]: pgmap v9: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 77 MiB/s rd, 266 MiB/s wr, 488 op/s 2024-04-05T03:02:49.047 INFO:tasks.workunit.client.0.smithi153.stdout:8/334: write f4 [1763550,78192] 0 2024-04-05T03:02:49.049 INFO:tasks.workunit.client.0.smithi153.stdout:5/386: creat d0/d4e/f90 x:0 0 0 2024-04-05T03:02:49.049 INFO:tasks.workunit.client.0.smithi153.stdout:7/383: write d2/da/d11/d39/f63 [1491251,89603] 0 2024-04-05T03:02:49.049 INFO:tasks.workunit.client.0.smithi153.stdout:8/335: creat d7/d2a/f6e x:0 0 0 2024-04-05T03:02:49.056 INFO:tasks.workunit.client.0.smithi153.stdout:8/336: dwrite f0 [8388608,4194304] 0 2024-04-05T03:02:49.056 INFO:tasks.workunit.client.0.smithi153.stdout:7/384: dread d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:49.058 INFO:tasks.workunit.client.1.smithi203.stdout:9/698: dread d13/d5c/dd1/f90 [0,4194304] 0 2024-04-05T03:02:49.061 INFO:tasks.workunit.client.1.smithi203.stdout:9/699: unlink d13/d5c/dd1/l7c 0 2024-04-05T03:02:49.063 INFO:tasks.workunit.client.0.smithi153.stdout:7/385: creat d2/da/f7d x:0 0 0 2024-04-05T03:02:49.064 INFO:tasks.workunit.client.1.smithi203.stdout:9/700: creat d13/fe6 x:0 0 0 2024-04-05T03:02:49.068 INFO:tasks.workunit.client.0.smithi153.stdout:8/337: getdents d7/d13/d19/d1b/d4f/d5f/d54 0 2024-04-05T03:02:49.070 INFO:tasks.workunit.client.0.smithi153.stdout:3/372: rmdir d1d/d27 39 2024-04-05T03:02:49.070 INFO:tasks.workunit.client.1.smithi203.stdout:9/701: dwrite d13/f2a [0,4194304] 0 2024-04-05T03:02:49.070 INFO:tasks.workunit.client.0.smithi153.stdout:3/373: write d1d/f36 [1690978,22621] 0 2024-04-05T03:02:49.071 INFO:tasks.workunit.client.1.smithi203.stdout:9/702: readlink d13/d5c/dd1/d51/da1/la2 0 2024-04-05T03:02:49.072 INFO:tasks.workunit.client.0.smithi153.stdout:8/338: symlink d7/d46/l6f 0 2024-04-05T03:02:49.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:48 smithi153 ceph-mon[29381]: pgmap v9: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 77 MiB/s rd, 266 MiB/s wr, 488 op/s 2024-04-05T03:02:49.073 INFO:tasks.workunit.client.0.smithi153.stdout:7/386: getdents d2/da/d42 0 2024-04-05T03:02:49.073 INFO:tasks.workunit.client.1.smithi203.stdout:9/703: stat d13/d5c/d9d/cbb 0 2024-04-05T03:02:49.074 INFO:tasks.workunit.client.1.smithi203.stdout:9/704: fsync d13/d5c/dd1/d31/f99 0 2024-04-05T03:02:49.076 INFO:tasks.workunit.client.0.smithi153.stdout:3/374: chown c8 4052792 1 2024-04-05T03:02:49.077 INFO:tasks.workunit.client.1.smithi203.stdout:9/705: unlink d13/l1e 0 2024-04-05T03:02:49.078 INFO:tasks.workunit.client.1.smithi203.stdout:9/706: write d13/d1b/d6b/fbf [488785,34360] 0 2024-04-05T03:02:49.078 INFO:tasks.workunit.client.0.smithi153.stdout:0/407: creat d9/db/d42/f74 x:0 0 0 2024-04-05T03:02:49.079 INFO:tasks.workunit.client.0.smithi153.stdout:7/387: getdents d2/d51/d77 0 2024-04-05T03:02:49.080 INFO:tasks.workunit.client.1.smithi203.stdout:9/707: write d13/d5c/dd1/d31/f3e [1841951,91856] 0 2024-04-05T03:02:49.082 INFO:tasks.workunit.client.0.smithi153.stdout:5/387: mknod d0/d25/d4d/d62/c91 0 2024-04-05T03:02:49.084 INFO:tasks.workunit.client.0.smithi153.stdout:8/339: dwrite d7/f8 [0,4194304] 0 2024-04-05T03:02:49.084 INFO:tasks.workunit.client.1.smithi203.stdout:9/708: link d13/d1b/d6b/d9e/d77/fd3 d13/dc8/dd2/fe7 0 2024-04-05T03:02:49.085 INFO:tasks.workunit.client.0.smithi153.stdout:9/306: creat d1/d10/f6d x:0 0 0 2024-04-05T03:02:49.085 INFO:tasks.workunit.client.0.smithi153.stdout:5/388: dread - d0/d25/d4d/d62/d5f/d86/f77 zero size 2024-04-05T03:02:49.087 INFO:tasks.workunit.client.0.smithi153.stdout:6/400: write d4/d38/d4c/f6a [529883,106925] 0 2024-04-05T03:02:49.087 INFO:tasks.workunit.client.0.smithi153.stdout:0/408: symlink d9/df/d1d/d72/l75 0 2024-04-05T03:02:49.087 INFO:tasks.workunit.client.0.smithi153.stdout:8/340: chown d7/d2a/f2c 1 1 2024-04-05T03:02:49.088 INFO:tasks.workunit.client.1.smithi203.stdout:9/709: mknod d13/d1b/d6b/d9e/d77/ce8 0 2024-04-05T03:02:49.088 INFO:tasks.workunit.client.0.smithi153.stdout:7/388: mknod d2/da/c7e 0 2024-04-05T03:02:49.090 INFO:tasks.workunit.client.1.smithi203.stdout:9/710: mkdir d13/de9 0 2024-04-05T03:02:49.092 INFO:tasks.workunit.client.0.smithi153.stdout:4/484: mkdir da/de/da1 0 2024-04-05T03:02:49.096 INFO:tasks.workunit.client.1.smithi203.stdout:9/711: unlink d13/d1b/d33/d63/l8b 0 2024-04-05T03:02:49.097 INFO:tasks.workunit.client.0.smithi153.stdout:5/389: unlink d0/d25/d4d/d62/d37/c8b 0 2024-04-05T03:02:49.099 INFO:tasks.workunit.client.0.smithi153.stdout:0/409: fdatasync d9/df/d1d/d61/f1b 0 2024-04-05T03:02:49.099 INFO:tasks.workunit.client.0.smithi153.stdout:6/401: unlink d4/dc/df/d16/d36/c82 0 2024-04-05T03:02:49.099 INFO:tasks.workunit.client.1.smithi203.stdout:9/712: unlink d13/d1b/d6b/d9e/d59/f85 0 2024-04-05T03:02:49.101 INFO:tasks.workunit.client.0.smithi153.stdout:7/389: dwrite d2/da/d3a/d41/d7a/d55/d6f/f78 [0,4194304] 0 2024-04-05T03:02:49.103 INFO:tasks.workunit.client.1.smithi203.stdout:9/713: rename d13/c16 to d13/d1b/d33/d42/cea 0 2024-04-05T03:02:49.106 INFO:tasks.workunit.client.1.smithi203.stdout:9/714: read d13/d1b/fad [3307611,123018] 0 2024-04-05T03:02:49.107 INFO:tasks.workunit.client.0.smithi153.stdout:5/390: dwrite d0/f16 [0,4194304] 0 2024-04-05T03:02:49.107 INFO:tasks.workunit.client.0.smithi153.stdout:0/410: creat d9/db/d42/d43/f76 x:0 0 0 2024-04-05T03:02:49.107 INFO:tasks.workunit.client.0.smithi153.stdout:3/375: write d1d/d27/f64 [1773436,123003] 0 2024-04-05T03:02:49.110 INFO:tasks.workunit.client.0.smithi153.stdout:9/307: dread d1/f4 [0,4194304] 0 2024-04-05T03:02:49.110 INFO:tasks.workunit.client.0.smithi153.stdout:1/328: link d5/d8/l2e d5/d8/d20/l63 0 2024-04-05T03:02:49.112 INFO:tasks.workunit.client.0.smithi153.stdout:7/390: symlink d2/da/d42/l7f 0 2024-04-05T03:02:49.114 INFO:tasks.workunit.client.0.smithi153.stdout:7/391: write d2/f5 [3645053,20928] 0 2024-04-05T03:02:49.114 INFO:tasks.workunit.client.0.smithi153.stdout:8/341: getdents d7/d13/d19/d1b/d4f/d5f 0 2024-04-05T03:02:49.119 INFO:tasks.workunit.client.0.smithi153.stdout:9/308: dwrite d1/d10/d16/f38 [0,4194304] 0 2024-04-05T03:02:49.119 INFO:tasks.workunit.client.0.smithi153.stdout:5/391: chown d0/d25/c89 117219 1 2024-04-05T03:02:49.122 INFO:tasks.workunit.client.0.smithi153.stdout:4/485: truncate da/db/d20/d45/f4e 3460716 0 2024-04-05T03:02:49.124 INFO:tasks.workunit.client.0.smithi153.stdout:0/411: mknod d9/db/c77 0 2024-04-05T03:02:49.125 INFO:tasks.workunit.client.1.smithi203.stdout:9/715: write d13/d5c/dd1/d51/f91 [1678838,128650] 0 2024-04-05T03:02:49.125 INFO:tasks.workunit.client.0.smithi153.stdout:4/486: write da/db/d20/d35/d38/d3e/f71 [3241587,128959] 0 2024-04-05T03:02:49.129 INFO:tasks.workunit.client.1.smithi203.stdout:9/716: mkdir d13/d5c/dd1/d31/deb 0 2024-04-05T03:02:49.130 INFO:tasks.workunit.client.0.smithi153.stdout:5/392: symlink d0/d9/d22/l92 0 2024-04-05T03:02:49.130 INFO:tasks.workunit.client.0.smithi153.stdout:7/392: dread d2/da/d3a/d41/d31/f45 [0,4194304] 0 2024-04-05T03:02:49.130 INFO:tasks.workunit.client.0.smithi153.stdout:3/376: mknod d1d/d26/d3c/c88 0 2024-04-05T03:02:49.132 INFO:tasks.workunit.client.1.smithi203.stdout:9/717: symlink d13/d5c/lec 0 2024-04-05T03:02:49.132 INFO:tasks.workunit.client.0.smithi153.stdout:5/393: write d0/d9/f21 [1175962,90398] 0 2024-04-05T03:02:49.132 INFO:tasks.workunit.client.0.smithi153.stdout:5/394: chown d0/d25/d68 34060892 1 2024-04-05T03:02:49.138 INFO:tasks.workunit.client.0.smithi153.stdout:8/342: unlink d7/d13/d19/d1b/l26 0 2024-04-05T03:02:49.138 INFO:tasks.workunit.client.0.smithi153.stdout:0/412: mknod d9/df/d1d/c78 0 2024-04-05T03:02:49.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/487: rmdir da/db/d20/d35/d38/d76 39 2024-04-05T03:02:49.140 INFO:tasks.workunit.client.0.smithi153.stdout:6/402: dwrite d4/dc/df/d16/d17/f3b [0,4194304] 0 2024-04-05T03:02:49.141 INFO:tasks.workunit.client.0.smithi153.stdout:9/309: creat d1/d6c/f6e x:0 0 0 2024-04-05T03:02:49.142 INFO:tasks.workunit.client.0.smithi153.stdout:4/488: chown da/db/d20/f6e 455190 1 2024-04-05T03:02:49.142 INFO:tasks.workunit.client.0.smithi153.stdout:7/393: mknod d2/da/d3a/d41/d7a/d55/c80 0 2024-04-05T03:02:49.144 INFO:tasks.workunit.client.0.smithi153.stdout:9/310: write d1/d9/d1b/f1c [1624882,100002] 0 2024-04-05T03:02:49.146 INFO:tasks.workunit.client.0.smithi153.stdout:8/343: symlink d7/d13/d19/d31/l70 0 2024-04-05T03:02:49.146 INFO:tasks.workunit.client.0.smithi153.stdout:3/377: truncate f3 1771015 0 2024-04-05T03:02:49.149 INFO:tasks.workunit.client.0.smithi153.stdout:3/378: write d1d/d26/d3c/f73 [448849,37951] 0 2024-04-05T03:02:49.154 INFO:tasks.workunit.client.1.smithi203.stdout:9/718: dwrite d13/d1b/d33/d64/f9c [0,4194304] 0 2024-04-05T03:02:49.154 INFO:tasks.workunit.client.0.smithi153.stdout:1/329: truncate d5/d8/d20/d4b/d18/d16/f36 1104208 0 2024-04-05T03:02:49.155 INFO:tasks.workunit.client.0.smithi153.stdout:9/311: creat d1/d10/d36/d57/f6f x:0 0 0 2024-04-05T03:02:49.155 INFO:tasks.workunit.client.0.smithi153.stdout:1/330: stat d5/d8/d20/d4b/d18/d16/d33/f3c 0 2024-04-05T03:02:49.155 INFO:tasks.workunit.client.0.smithi153.stdout:1/331: write d5/d8/d20/d4b/d18/d16/f2d [4507326,107132] 0 2024-04-05T03:02:49.156 INFO:tasks.workunit.client.1.smithi203.stdout:9/719: creat d13/d1b/d6b/d9e/d77/fed x:0 0 0 2024-04-05T03:02:49.156 INFO:tasks.workunit.client.0.smithi153.stdout:6/403: symlink d4/dc/df/d16/d33/l88 0 2024-04-05T03:02:49.156 INFO:tasks.workunit.client.0.smithi153.stdout:1/332: stat d5/d8/d20/f25 0 2024-04-05T03:02:49.158 INFO:tasks.workunit.client.1.smithi203.stdout:9/720: dread d13/d1b/fbd [0,4194304] 0 2024-04-05T03:02:49.158 INFO:tasks.workunit.client.0.smithi153.stdout:6/404: chown d4/dc/df/d16/d33/l59 991896 1 2024-04-05T03:02:49.161 INFO:tasks.workunit.client.0.smithi153.stdout:8/344: mknod d7/d13/d19/d1b/d4f/d5f/d54/c71 0 2024-04-05T03:02:49.164 INFO:tasks.workunit.client.0.smithi153.stdout:9/312: symlink d1/d10/d36/d57/d5c/l70 0 2024-04-05T03:02:49.164 INFO:tasks.workunit.client.1.smithi203.stdout:9/721: rmdir d13/d5c/dd1/d51/da1/db1 0 2024-04-05T03:02:49.165 INFO:tasks.workunit.client.1.smithi203.stdout:9/722: chown d13/d5c/dd1/d51/da1/la2 3261033 1 2024-04-05T03:02:49.166 INFO:tasks.workunit.client.0.smithi153.stdout:9/313: dread - d1/d10/f5f zero size 2024-04-05T03:02:49.166 INFO:tasks.workunit.client.1.smithi203.stdout:9/723: write f12 [734020,127037] 0 2024-04-05T03:02:49.169 INFO:tasks.workunit.client.1.smithi203.stdout:9/724: creat d13/d5c/dd1/d51/da1/dba/fee x:0 0 0 2024-04-05T03:02:49.169 INFO:tasks.workunit.client.1.smithi203.stdout:9/725: dread - d13/fe6 zero size 2024-04-05T03:02:49.171 INFO:tasks.workunit.client.1.smithi203.stdout:9/726: chown d13/d1b/d33/l43 84 1 2024-04-05T03:02:49.171 INFO:tasks.workunit.client.0.smithi153.stdout:1/333: dwrite d5/dd/f13 [4194304,4194304] 0 2024-04-05T03:02:49.172 INFO:tasks.workunit.client.0.smithi153.stdout:9/314: truncate d1/d10/f6d 574117 0 2024-04-05T03:02:49.172 INFO:tasks.workunit.client.0.smithi153.stdout:0/413: truncate d9/d14/f3f 252249 0 2024-04-05T03:02:49.172 INFO:tasks.workunit.client.1.smithi203.stdout:9/727: dread - d13/d1b/d6b/d7a/fbe zero size 2024-04-05T03:02:49.172 INFO:tasks.workunit.client.0.smithi153.stdout:5/395: dread d0/d9/f20 [0,4194304] 0 2024-04-05T03:02:49.173 INFO:tasks.workunit.client.1.smithi203.stdout:0/708: sync 2024-04-05T03:02:49.174 INFO:tasks.workunit.client.0.smithi153.stdout:5/396: fdatasync d0/d25/f2f 0 2024-04-05T03:02:49.175 INFO:tasks.workunit.client.1.smithi203.stdout:9/728: chown d13/d1b/c50 21873 1 2024-04-05T03:02:49.175 INFO:tasks.workunit.client.1.smithi203.stdout:9/729: chown d13/d1b 0 1 2024-04-05T03:02:49.175 INFO:tasks.workunit.client.0.smithi153.stdout:3/379: getdents d1d/d26/d57 0 2024-04-05T03:02:49.176 INFO:tasks.workunit.client.0.smithi153.stdout:6/405: creat d4/f89 x:0 0 0 2024-04-05T03:02:49.176 INFO:tasks.workunit.client.1.smithi203.stdout:9/730: dread - d13/d1b/d6b/d9e/d59/d9b/fd9 zero size 2024-04-05T03:02:49.177 INFO:tasks.workunit.client.0.smithi153.stdout:1/334: write d5/d8/d20/d2f/f60 [637842,75859] 0 2024-04-05T03:02:49.178 INFO:tasks.workunit.client.0.smithi153.stdout:0/414: mkdir d9/df/d1e/d45/d62/d79 0 2024-04-05T03:02:49.180 INFO:tasks.workunit.client.1.smithi203.stdout:9/731: truncate d13/d1b/d6b/d9e/d59/dbc/fc4 231954 0 2024-04-05T03:02:49.181 INFO:tasks.workunit.client.0.smithi153.stdout:9/315: mkdir d1/d10/d36/d71 0 2024-04-05T03:02:49.182 INFO:tasks.workunit.client.0.smithi153.stdout:6/406: write d4/dc/df/d16/d24/f29 [3319176,5120] 0 2024-04-05T03:02:49.184 INFO:tasks.workunit.client.0.smithi153.stdout:4/489: dread da/f1d [0,4194304] 0 2024-04-05T03:02:49.187 INFO:tasks.workunit.client.0.smithi153.stdout:7/394: dwrite d2/da/d3a/f71 [0,4194304] 0 2024-04-05T03:02:49.188 INFO:tasks.workunit.client.0.smithi153.stdout:1/335: creat d5/d8/d20/d3f/f64 x:0 0 0 2024-04-05T03:02:49.189 INFO:tasks.workunit.client.0.smithi153.stdout:7/395: chown d2/da/d3a/d41/d7a/d54/c6b 106 1 2024-04-05T03:02:49.190 INFO:tasks.workunit.client.0.smithi153.stdout:3/380: unlink d1d/d26/d3c/d3f/c5d 0 2024-04-05T03:02:49.191 INFO:tasks.workunit.client.0.smithi153.stdout:3/381: chown d1d/d27/d2b/l66 468378 1 2024-04-05T03:02:49.192 INFO:tasks.workunit.client.1.smithi203.stdout:9/732: write d13/d1b/d33/f65 [207764,58200] 0 2024-04-05T03:02:49.192 INFO:tasks.workunit.client.0.smithi153.stdout:4/490: symlink da/db/la2 0 2024-04-05T03:02:49.193 INFO:tasks.workunit.client.0.smithi153.stdout:4/491: write da/db/d20/d45/f82 [407708,38340] 0 2024-04-05T03:02:49.194 INFO:tasks.workunit.client.0.smithi153.stdout:5/397: truncate d0/d25/f2f 3925089 0 2024-04-05T03:02:49.195 INFO:tasks.workunit.client.1.smithi203.stdout:9/733: write d13/d1b/d6b/d9e/d77/fed [514886,8960] 0 2024-04-05T03:02:49.197 INFO:tasks.workunit.client.1.smithi203.stdout:0/709: dwrite dd/d17/d1d/d22/d45/d51/d69/f7e [0,4194304] 0 2024-04-05T03:02:49.198 INFO:tasks.workunit.client.0.smithi153.stdout:6/407: creat d4/dc/df/d16/d24/d4e/f8a x:0 0 0 2024-04-05T03:02:49.199 INFO:tasks.workunit.client.0.smithi153.stdout:1/336: chown d5/d8/d20/d4b/d18/d16/d45/c5a 3828331 1 2024-04-05T03:02:49.203 INFO:tasks.workunit.client.0.smithi153.stdout:9/316: symlink d1/d10/d16/d22/d2c/d68/l72 0 2024-04-05T03:02:49.205 INFO:tasks.workunit.client.0.smithi153.stdout:7/396: creat d2/da/d47/d5d/d7c/f81 x:0 0 0 2024-04-05T03:02:49.205 INFO:tasks.workunit.client.1.smithi203.stdout:9/734: dwrite d13/d5c/dd1/d31/f60 [0,4194304] 0 2024-04-05T03:02:49.206 INFO:tasks.workunit.client.0.smithi153.stdout:3/382: creat d1d/d32/f89 x:0 0 0 2024-04-05T03:02:49.207 INFO:tasks.workunit.client.1.smithi203.stdout:9/735: chown d13/d1b/d6b/d9e/d59/fb2 442429842 1 2024-04-05T03:02:49.207 INFO:tasks.workunit.client.1.smithi203.stdout:0/710: dwrite dd/d17/d1d/d26/d34/f47 [0,4194304] 0 2024-04-05T03:02:49.209 INFO:tasks.workunit.client.1.smithi203.stdout:9/736: fdatasync d13/d1b/d6b/d9e/d59/dbc/fdd 0 2024-04-05T03:02:49.209 INFO:tasks.workunit.client.1.smithi203.stdout:0/711: chown dd/d17/d1d/d22/d53/d7f/f83 2708390 1 2024-04-05T03:02:49.210 INFO:tasks.workunit.client.0.smithi153.stdout:4/492: creat da/de/d56/d6a/d2d/fa3 x:0 0 0 2024-04-05T03:02:49.211 INFO:tasks.workunit.client.0.smithi153.stdout:5/398: creat d0/d25/d4d/d62/f93 x:0 0 0 2024-04-05T03:02:49.213 INFO:tasks.workunit.client.0.smithi153.stdout:6/408: dread d4/dc/df/d16/d33/f83 [0,4194304] 0 2024-04-05T03:02:49.213 INFO:tasks.workunit.client.0.smithi153.stdout:1/337: rmdir d5/d8/d20/d2f 39 2024-04-05T03:02:49.214 INFO:tasks.workunit.client.0.smithi153.stdout:6/409: stat d4/d38/d4c/c81 0 2024-04-05T03:02:49.214 INFO:tasks.workunit.client.0.smithi153.stdout:1/338: chown d5/d8/d20/c57 261890 1 2024-04-05T03:02:49.216 INFO:tasks.workunit.client.1.smithi203.stdout:0/712: getdents dd/d17/d1d 0 2024-04-05T03:02:49.216 INFO:tasks.workunit.client.0.smithi153.stdout:9/317: creat d1/d9/d1b/f73 x:0 0 0 2024-04-05T03:02:49.219 INFO:tasks.workunit.client.0.smithi153.stdout:0/415: getdents d9/df/d1e 0 2024-04-05T03:02:49.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/493: creat da/db/d8b/fa4 x:0 0 0 2024-04-05T03:02:49.223 INFO:tasks.workunit.client.1.smithi203.stdout:0/713: dwrite dd/d17/d1d/d22/f65 [4194304,4194304] 0 2024-04-05T03:02:49.225 INFO:tasks.workunit.client.0.smithi153.stdout:6/410: dwrite d4/dc/df/d16/d5f/d73/f7e [0,4194304] 0 2024-04-05T03:02:49.225 INFO:tasks.workunit.client.0.smithi153.stdout:9/318: rename d1/d10/d36/d57/d5c/f63 to d1/d10/d16/d22/f74 0 2024-04-05T03:02:49.228 INFO:tasks.workunit.client.0.smithi153.stdout:9/319: write d1/d9/d1b/f5d [2033415,55951] 0 2024-04-05T03:02:49.228 INFO:tasks.workunit.client.0.smithi153.stdout:3/383: rmdir d1d/d26/d3c 39 2024-04-05T03:02:49.229 INFO:tasks.workunit.client.1.smithi203.stdout:0/714: getdents dd/d17/d1d/d22/d53/db0 0 2024-04-05T03:02:49.229 INFO:tasks.workunit.client.0.smithi153.stdout:3/384: dread - d1d/d26/d57/f59 zero size 2024-04-05T03:02:49.230 INFO:tasks.workunit.client.1.smithi203.stdout:0/715: chown dd/d17/d1d/d22/f9c 18727 1 2024-04-05T03:02:49.230 INFO:tasks.workunit.client.0.smithi153.stdout:5/399: mkdir d0/d94 0 2024-04-05T03:02:49.231 INFO:tasks.workunit.client.1.smithi203.stdout:0/716: write dd/d17/f40 [1581102,57707] 0 2024-04-05T03:02:49.235 INFO:tasks.workunit.client.1.smithi203.stdout:0/717: write dd/d17/d1d/d22/dc1/fd3 [2098968,32761] 0 2024-04-05T03:02:49.236 INFO:tasks.workunit.client.0.smithi153.stdout:3/385: write d1d/f70 [288001,36963] 0 2024-04-05T03:02:49.238 INFO:tasks.workunit.client.0.smithi153.stdout:4/494: write da/db/d20/d3c/d81/f8a [972032,87164] 0 2024-04-05T03:02:49.239 INFO:tasks.workunit.client.0.smithi153.stdout:6/411: creat d4/d38/f8b x:0 0 0 2024-04-05T03:02:49.241 INFO:tasks.workunit.client.0.smithi153.stdout:0/416: dread d9/db/fd [0,4194304] 0 2024-04-05T03:02:49.242 INFO:tasks.workunit.client.1.smithi203.stdout:0/718: link dd/d17/d1d/d26/d34/d56/cc3 dd/d17/d1d/d22/d53/d61/cd6 0 2024-04-05T03:02:49.244 INFO:tasks.workunit.client.1.smithi203.stdout:0/719: write dd/d17/d1d/d26/d34/d56/fa7 [28337,101148] 0 2024-04-05T03:02:49.244 INFO:tasks.workunit.client.0.smithi153.stdout:9/320: dwrite d1/d9/d1b/f30 [0,4194304] 0 2024-04-05T03:02:49.246 INFO:tasks.workunit.client.0.smithi153.stdout:6/412: creat d4/dc/df/d16/d5f/f8c x:0 0 0 2024-04-05T03:02:49.247 INFO:tasks.workunit.client.0.smithi153.stdout:4/495: creat da/de/d56/d6a/d2d/fa5 x:0 0 0 2024-04-05T03:02:49.248 INFO:tasks.workunit.client.0.smithi153.stdout:3/386: read d1d/d32/f83 [2645168,119413] 0 2024-04-05T03:02:49.248 INFO:tasks.workunit.client.0.smithi153.stdout:0/417: creat d9/d14/f7a x:0 0 0 2024-04-05T03:02:49.251 INFO:tasks.workunit.client.0.smithi153.stdout:6/413: fdatasync d4/dc/df/f13 0 2024-04-05T03:02:49.251 INFO:tasks.workunit.client.0.smithi153.stdout:9/321: creat d1/d9/d1b/f75 x:0 0 0 2024-04-05T03:02:49.252 INFO:tasks.workunit.client.0.smithi153.stdout:3/387: read - d1d/d27/d2b/f86 zero size 2024-04-05T03:02:49.252 INFO:tasks.workunit.client.0.smithi153.stdout:7/397: dwrite d2/da/d42/f4b [0,4194304] 0 2024-04-05T03:02:49.253 INFO:tasks.workunit.client.0.smithi153.stdout:3/388: chown d1d/d27 3 1 2024-04-05T03:02:49.257 INFO:tasks.workunit.client.0.smithi153.stdout:7/398: write d2/da/d3a/d41/d7a/f3c [832478,18261] 0 2024-04-05T03:02:49.257 INFO:tasks.workunit.client.0.smithi153.stdout:5/400: link d0/d25/d4d/d62/d37/d38/c6d d0/d25/d4d/d62/d37/d38/c95 0 2024-04-05T03:02:49.260 INFO:tasks.workunit.client.0.smithi153.stdout:0/418: getdents d9/df/d1e/d45/d62/d79 0 2024-04-05T03:02:49.262 INFO:tasks.workunit.client.0.smithi153.stdout:4/496: dwrite da/de/d56/d6a/f44 [0,4194304] 0 2024-04-05T03:02:49.262 INFO:tasks.workunit.client.0.smithi153.stdout:0/419: chown d9/df/d1e/d45/f6a 52400246 1 2024-04-05T03:02:49.263 INFO:tasks.workunit.client.0.smithi153.stdout:5/401: write d0/d9/f1e [1517085,23109] 0 2024-04-05T03:02:49.264 INFO:tasks.workunit.client.0.smithi153.stdout:6/414: rename d4/dc/df/d16/d17/d54/d45 to d4/dc/df/d16/d24/d8d 0 2024-04-05T03:02:49.269 INFO:tasks.workunit.client.0.smithi153.stdout:7/399: creat d2/da/d3a/d41/d7a/f82 x:0 0 0 2024-04-05T03:02:49.269 INFO:tasks.workunit.client.0.smithi153.stdout:0/420: read d9/df/f39 [2787865,75179] 0 2024-04-05T03:02:49.270 INFO:tasks.workunit.client.0.smithi153.stdout:5/402: creat d0/d25/f96 x:0 0 0 2024-04-05T03:02:49.270 INFO:tasks.workunit.client.0.smithi153.stdout:1/339: dread d5/d8/d20/d3f/f5c [0,4194304] 0 2024-04-05T03:02:49.274 INFO:tasks.workunit.client.0.smithi153.stdout:7/400: write d2/f1d [4221856,9375] 0 2024-04-05T03:02:49.276 INFO:tasks.workunit.client.0.smithi153.stdout:1/340: getdents d5/dd 0 2024-04-05T03:02:49.276 INFO:tasks.workunit.client.0.smithi153.stdout:6/415: dwrite d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:49.277 INFO:tasks.workunit.client.0.smithi153.stdout:7/401: write d2/da/d3a/d41/d31/f5f [2446644,14875] 0 2024-04-05T03:02:49.278 INFO:tasks.workunit.client.0.smithi153.stdout:7/402: write d2/da/f7d [956745,47969] 0 2024-04-05T03:02:49.282 INFO:tasks.workunit.client.0.smithi153.stdout:6/416: write d4/dc/df/d16/d24/d4e/f8a [732041,109374] 0 2024-04-05T03:02:49.285 INFO:tasks.workunit.client.0.smithi153.stdout:1/341: creat d5/d8/f65 x:0 0 0 2024-04-05T03:02:49.286 INFO:tasks.workunit.client.0.smithi153.stdout:7/403: write d2/da/d47/f60 [5041519,104089] 0 2024-04-05T03:02:49.288 INFO:tasks.workunit.client.0.smithi153.stdout:1/342: creat d5/dd/d1b/f66 x:0 0 0 2024-04-05T03:02:49.289 INFO:tasks.workunit.client.0.smithi153.stdout:7/404: mknod d2/da/d47/c83 0 2024-04-05T03:02:49.291 INFO:tasks.workunit.client.0.smithi153.stdout:1/343: rmdir d5/d8/d20/d4b/d51 39 2024-04-05T03:02:49.296 INFO:tasks.workunit.client.0.smithi153.stdout:6/417: write d4/f6 [2125963,70087] 0 2024-04-05T03:02:49.299 INFO:tasks.workunit.client.0.smithi153.stdout:4/497: dwrite da/de/d56/d6a/f7f [0,4194304] 0 2024-04-05T03:02:49.300 INFO:tasks.workunit.client.0.smithi153.stdout:1/344: dwrite d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:49.300 INFO:tasks.workunit.client.0.smithi153.stdout:3/389: dread d1d/d26/d57/d6b/f67 [0,4194304] 0 2024-04-05T03:02:49.301 INFO:tasks.workunit.client.0.smithi153.stdout:4/498: readlink da/l4c 0 2024-04-05T03:02:49.302 INFO:tasks.workunit.client.0.smithi153.stdout:5/403: dread d0/d25/d4d/d62/f10 [0,4194304] 0 2024-04-05T03:02:49.306 INFO:tasks.workunit.client.0.smithi153.stdout:6/418: unlink d4/dc/df/d16/d5f/c65 0 2024-04-05T03:02:49.309 INFO:tasks.workunit.client.0.smithi153.stdout:5/404: write d0/d25/d4d/d62/f93 [310350,127983] 0 2024-04-05T03:02:49.311 INFO:tasks.workunit.client.0.smithi153.stdout:0/421: dread f3 [0,4194304] 0 2024-04-05T03:02:49.314 INFO:tasks.workunit.client.0.smithi153.stdout:1/345: symlink d5/d8/d20/d4b/d18/d16/d48/l67 0 2024-04-05T03:02:49.315 INFO:tasks.workunit.client.0.smithi153.stdout:5/405: chown d0/d25/d4d/d62/d5f/d86/f77 121 1 2024-04-05T03:02:49.316 INFO:tasks.workunit.client.0.smithi153.stdout:4/499: dwrite da/de/d56/d78/f7e [0,4194304] 0 2024-04-05T03:02:49.317 INFO:tasks.workunit.client.0.smithi153.stdout:6/419: link d4/dc/df/d16/d36/d52/c5d d4/d38/d4c/c8e 0 2024-04-05T03:02:49.319 INFO:tasks.workunit.client.0.smithi153.stdout:3/390: rename d1d/c31 to d1d/d26/d3c/c8a 0 2024-04-05T03:02:49.320 INFO:tasks.workunit.client.0.smithi153.stdout:1/346: mkdir d5/d8/d20/d4b/d18/d16/d45/d68 0 2024-04-05T03:02:49.321 INFO:tasks.workunit.client.0.smithi153.stdout:5/406: write d0/d25/d4d/d62/d37/f4f [3156063,8767] 0 2024-04-05T03:02:49.322 INFO:tasks.workunit.client.0.smithi153.stdout:0/422: rename d9/d14/f53 to d9/df/d1d/d72/f7b 0 2024-04-05T03:02:49.324 INFO:tasks.workunit.client.0.smithi153.stdout:7/405: dread d2/f1d [0,4194304] 0 2024-04-05T03:02:49.327 INFO:tasks.workunit.client.0.smithi153.stdout:6/420: mknod d4/dc/df/d16/d24/d8d/c8f 0 2024-04-05T03:02:49.328 INFO:tasks.workunit.client.0.smithi153.stdout:5/407: chown d0/d25/d4d/d62/l54 709690605 1 2024-04-05T03:02:49.329 INFO:tasks.workunit.client.0.smithi153.stdout:0/423: fdatasync d9/df/d1d/d72/f7b 0 2024-04-05T03:02:49.329 INFO:tasks.workunit.client.0.smithi153.stdout:4/500: dread f4 [0,4194304] 0 2024-04-05T03:02:49.331 INFO:tasks.workunit.client.0.smithi153.stdout:7/406: read d2/da/d3a/d41/d7a/f4e [6302203,49213] 0 2024-04-05T03:02:49.335 INFO:tasks.workunit.client.0.smithi153.stdout:4/501: creat da/db/d20/d45/fa6 x:0 0 0 2024-04-05T03:02:49.336 INFO:tasks.workunit.client.0.smithi153.stdout:7/407: chown d2/da/d3a/d41/d7a/d55/d6f/f78 112735834 1 2024-04-05T03:02:49.337 INFO:tasks.workunit.client.0.smithi153.stdout:1/347: dwrite d5/d8/d20/f5f [0,4194304] 0 2024-04-05T03:02:49.337 INFO:tasks.workunit.client.0.smithi153.stdout:6/421: read d4/fb [3819596,98378] 0 2024-04-05T03:02:49.340 INFO:tasks.workunit.client.0.smithi153.stdout:1/348: chown d5/d8/c30 9 1 2024-04-05T03:02:49.341 INFO:tasks.workunit.client.0.smithi153.stdout:7/408: dread - d2/da/d42/f44 zero size 2024-04-05T03:02:49.343 INFO:tasks.workunit.client.0.smithi153.stdout:3/391: dread fd [0,4194304] 0 2024-04-05T03:02:49.344 INFO:tasks.workunit.client.0.smithi153.stdout:0/424: dwrite d9/db/f13 [0,4194304] 0 2024-04-05T03:02:49.349 INFO:tasks.workunit.client.0.smithi153.stdout:4/502: mkdir da/db/d20/d35/d38/d76/d90/da7 0 2024-04-05T03:02:49.353 INFO:tasks.workunit.client.0.smithi153.stdout:7/409: symlink d2/da/d3a/d41/d7a/d55/d6f/l84 0 2024-04-05T03:02:49.354 INFO:tasks.workunit.client.0.smithi153.stdout:4/503: rmdir da/de/d56/d6a 39 2024-04-05T03:02:49.355 INFO:tasks.workunit.client.0.smithi153.stdout:6/422: dwrite d4/dc/df/d16/d36/d4d/f87 [0,4194304] 0 2024-04-05T03:02:49.355 INFO:tasks.workunit.client.0.smithi153.stdout:1/349: dread d5/dd/f13 [0,4194304] 0 2024-04-05T03:02:49.360 INFO:tasks.workunit.client.0.smithi153.stdout:7/410: mkdir d2/da/d3a/d41/d7a/d54/d85 0 2024-04-05T03:02:49.360 INFO:tasks.workunit.client.0.smithi153.stdout:3/392: dwrite d1d/d75/f77 [0,4194304] 0 2024-04-05T03:02:49.361 INFO:tasks.workunit.client.0.smithi153.stdout:1/350: readlink d5/d8/d20/d2f/l35 0 2024-04-05T03:02:49.361 INFO:tasks.workunit.client.0.smithi153.stdout:5/408: dread d0/d25/d4d/f66 [0,4194304] 0 2024-04-05T03:02:49.363 INFO:tasks.workunit.client.0.smithi153.stdout:4/504: dread da/de/d56/f61 [0,4194304] 0 2024-04-05T03:02:49.363 INFO:tasks.workunit.client.0.smithi153.stdout:7/411: write d2/da/d3a/d41/f1e [4422506,76045] 0 2024-04-05T03:02:49.365 INFO:tasks.workunit.client.0.smithi153.stdout:6/423: mkdir d4/dc/df/d16/d44/d90 0 2024-04-05T03:02:49.367 INFO:tasks.workunit.client.0.smithi153.stdout:5/409: symlink d0/d9/d22/l97 0 2024-04-05T03:02:49.369 INFO:tasks.workunit.client.0.smithi153.stdout:7/412: symlink d2/da/d11/d39/l86 0 2024-04-05T03:02:49.369 INFO:tasks.workunit.client.0.smithi153.stdout:5/410: write d0/f3c [425676,91790] 0 2024-04-05T03:02:49.369 INFO:tasks.workunit.client.0.smithi153.stdout:7/413: chown d2/f6 62510983 1 2024-04-05T03:02:49.370 INFO:tasks.workunit.client.0.smithi153.stdout:5/411: write d0/d25/d4d/d62/f93 [71292,3203] 0 2024-04-05T03:02:49.371 INFO:tasks.workunit.client.0.smithi153.stdout:3/393: unlink d1d/d26/d57/d6b/l68 0 2024-04-05T03:02:49.373 INFO:tasks.workunit.client.0.smithi153.stdout:1/351: rename d5/dd/l39 to d5/d8/d20/d4b/d18/d16/d2c/l69 0 2024-04-05T03:02:49.374 INFO:tasks.workunit.client.0.smithi153.stdout:4/505: mkdir da/db/d20/d35/d41/da8 0 2024-04-05T03:02:49.376 INFO:tasks.workunit.client.0.smithi153.stdout:7/414: creat d2/da/d47/d5d/d7c/f87 x:0 0 0 2024-04-05T03:02:49.379 INFO:tasks.workunit.client.0.smithi153.stdout:6/424: rename d4/dc/df/f12 to d4/dc/df/d16/d36/d4d/f91 0 2024-04-05T03:02:49.380 INFO:tasks.workunit.client.0.smithi153.stdout:1/352: dwrite d5/d8/d20/d4b/d18/d16/f5b [0,4194304] 0 2024-04-05T03:02:49.380 INFO:tasks.workunit.client.0.smithi153.stdout:5/412: getdents d0/d94 0 2024-04-05T03:02:49.383 INFO:tasks.workunit.client.0.smithi153.stdout:4/506: mknod da/db/d20/d45/ca9 0 2024-04-05T03:02:49.383 INFO:tasks.workunit.client.0.smithi153.stdout:6/425: write d4/f20 [5233743,112161] 0 2024-04-05T03:02:49.384 INFO:tasks.workunit.client.0.smithi153.stdout:7/415: mkdir d2/da/d11/d88 0 2024-04-05T03:02:49.386 INFO:tasks.workunit.client.0.smithi153.stdout:0/425: dread d9/df/d1e/f54 [0,4194304] 0 2024-04-05T03:02:49.386 INFO:tasks.workunit.client.0.smithi153.stdout:1/353: symlink d5/d8/d20/d3f/l6a 0 2024-04-05T03:02:49.391 INFO:tasks.workunit.client.0.smithi153.stdout:4/507: rmdir da/db/d20/d35/d41/d67 39 2024-04-05T03:02:49.392 INFO:tasks.workunit.client.0.smithi153.stdout:6/426: fsync d4/fb 0 2024-04-05T03:02:49.393 INFO:tasks.workunit.client.0.smithi153.stdout:7/416: rmdir d2/da/d3a/d41/d7a/d55/d6f 39 2024-04-05T03:02:49.394 INFO:tasks.workunit.client.0.smithi153.stdout:7/417: chown d2/da/d3a/d41/d7a/d54/c6b 121 1 2024-04-05T03:02:49.396 INFO:tasks.workunit.client.0.smithi153.stdout:3/394: link c19 d1d/d32/c8b 0 2024-04-05T03:02:49.399 INFO:tasks.workunit.client.0.smithi153.stdout:5/413: rename d0/d25/d4d/d62/d37/d39/f3b to d0/d25/d4d/d62/d37/d38/f98 0 2024-04-05T03:02:49.401 INFO:tasks.workunit.client.0.smithi153.stdout:6/427: stat d4/dc/df/d16/d17/f3a 0 2024-04-05T03:02:49.402 INFO:tasks.workunit.client.0.smithi153.stdout:7/418: symlink d2/da/d47/l89 0 2024-04-05T03:02:49.403 INFO:tasks.workunit.client.0.smithi153.stdout:7/419: dread - d2/da/d3a/d41/d31/f37 zero size 2024-04-05T03:02:49.403 INFO:tasks.workunit.client.0.smithi153.stdout:0/426: link d9/db/f49 d9/d14/d67/d6f/f7c 0 2024-04-05T03:02:49.404 INFO:tasks.workunit.client.0.smithi153.stdout:3/395: unlink d1d/d26/d3c/f44 0 2024-04-05T03:02:49.405 INFO:tasks.workunit.client.0.smithi153.stdout:3/396: stat d1d/d32/f45 0 2024-04-05T03:02:49.405 INFO:tasks.workunit.client.1.smithi203.stdout:2/853: sync 2024-04-05T03:02:49.407 INFO:tasks.workunit.client.0.smithi153.stdout:3/397: read d1d/d27/d2b/f2d [3375538,67599] 0 2024-04-05T03:02:49.408 INFO:tasks.workunit.client.0.smithi153.stdout:5/414: creat d0/d25/d68/f99 x:0 0 0 2024-04-05T03:02:49.409 INFO:tasks.workunit.client.0.smithi153.stdout:5/415: truncate d0/d25/d68/f99 310752 0 2024-04-05T03:02:49.409 INFO:tasks.workunit.client.1.smithi203.stdout:2/854: mkdir d5/d10/d35/d36/db6/d126 0 2024-04-05T03:02:49.410 INFO:tasks.workunit.client.0.smithi153.stdout:5/416: readlink d0/d25/d4d/d62/l54 0 2024-04-05T03:02:49.410 INFO:tasks.workunit.client.0.smithi153.stdout:1/354: link d5/d8/d20/d4b/d18/d16/f36 d5/d8/d20/d4b/d18/d16/d33/f6b 0 2024-04-05T03:02:49.412 INFO:tasks.workunit.client.1.smithi203.stdout:2/855: creat d5/d10/db8/f127 x:0 0 0 2024-04-05T03:02:49.413 INFO:tasks.workunit.client.0.smithi153.stdout:6/428: dread d4/dc/f3c [0,4194304] 0 2024-04-05T03:02:49.415 INFO:tasks.workunit.client.1.smithi203.stdout:2/856: creat d5/d10/db8/dab/de1/dbd/f128 x:0 0 0 2024-04-05T03:02:49.417 INFO:tasks.workunit.client.1.smithi203.stdout:2/857: mknod d5/c129 0 2024-04-05T03:02:49.419 INFO:tasks.workunit.client.0.smithi153.stdout:6/429: dwrite d4/dc/df/d16/d36/d4d/f66 [0,4194304] 0 2024-04-05T03:02:49.420 INFO:tasks.workunit.client.0.smithi153.stdout:0/427: link d9/df/d1e/f54 d9/df/d1e/d45/d62/f7d 0 2024-04-05T03:02:49.420 INFO:tasks.workunit.client.0.smithi153.stdout:4/508: link da/de/d56/d6a/c75 da/db/d20/d35/d41/d67/caa 0 2024-04-05T03:02:49.421 INFO:tasks.workunit.client.1.smithi203.stdout:2/858: truncate d5/d10/d35/d36/db6/fa7 2446130 0 2024-04-05T03:02:49.422 INFO:tasks.workunit.client.0.smithi153.stdout:0/428: chown d9/db/f44 6805854 1 2024-04-05T03:02:49.424 INFO:tasks.workunit.client.1.smithi203.stdout:2/859: fsync d5/d10/d35/d36/db6/d87/f89 0 2024-04-05T03:02:49.425 INFO:tasks.workunit.client.0.smithi153.stdout:1/355: mknod d5/d8/d20/d4b/d51/c6c 0 2024-04-05T03:02:49.427 INFO:tasks.workunit.client.1.smithi203.stdout:2/860: truncate d5/d66/d7d/f94 4024247 0 2024-04-05T03:02:49.429 INFO:tasks.workunit.client.0.smithi153.stdout:5/417: unlink d0/d2c/d3f/f59 0 2024-04-05T03:02:49.429 INFO:tasks.workunit.client.0.smithi153.stdout:6/430: dwrite d4/dc/df/d16/d36/d4d/d64/f6c [0,4194304] 0 2024-04-05T03:02:49.431 INFO:tasks.workunit.client.0.smithi153.stdout:3/398: getdents d1d/d26 0 2024-04-05T03:02:49.433 INFO:tasks.workunit.client.0.smithi153.stdout:0/429: mkdir d9/db/d42/d7e 0 2024-04-05T03:02:49.434 INFO:tasks.workunit.client.0.smithi153.stdout:6/431: mknod d4/dc/df/d16/d24/c92 0 2024-04-05T03:02:49.437 INFO:tasks.workunit.client.0.smithi153.stdout:0/430: stat d9/db/d42/d43/l73 0 2024-04-05T03:02:49.441 INFO:tasks.workunit.client.0.smithi153.stdout:5/418: dread d0/d25/d4d/d62/d37/d38/f98 [0,4194304] 0 2024-04-05T03:02:49.445 INFO:tasks.workunit.client.0.smithi153.stdout:3/399: dwrite d1d/d26/f82 [0,4194304] 0 2024-04-05T03:02:49.446 INFO:tasks.workunit.client.0.smithi153.stdout:5/419: dread - d0/d25/d4d/d62/d5f/f7e zero size 2024-04-05T03:02:49.449 INFO:tasks.workunit.client.0.smithi153.stdout:1/356: dwrite d5/dd/d1b/f1d [0,4194304] 0 2024-04-05T03:02:49.453 INFO:tasks.workunit.client.0.smithi153.stdout:6/432: write d4/dc/df/f39 [3442177,52571] 0 2024-04-05T03:02:49.455 INFO:tasks.workunit.client.0.smithi153.stdout:3/400: read d1d/d26/d57/d6b/f67 [824660,37245] 0 2024-04-05T03:02:49.455 INFO:tasks.workunit.client.0.smithi153.stdout:4/509: dread da/f26 [0,4194304] 0 2024-04-05T03:02:49.456 INFO:tasks.workunit.client.0.smithi153.stdout:1/357: mknod d5/d8/d20/d3f/c6d 0 2024-04-05T03:02:49.459 INFO:tasks.workunit.client.0.smithi153.stdout:5/420: write d0/d25/d4d/d62/d5f/d86/f85 [589911,85779] 0 2024-04-05T03:02:49.459 INFO:tasks.workunit.client.0.smithi153.stdout:0/431: dwrite d9/f4e [0,4194304] 0 2024-04-05T03:02:49.463 INFO:tasks.workunit.client.0.smithi153.stdout:0/432: chown d9/db/d42/f71 1 1 2024-04-05T03:02:49.464 INFO:tasks.workunit.client.1.smithi203.stdout:2/861: dread d5/d10/d35/d2d/d7c/d72/d8f/fb9 [0,4194304] 0 2024-04-05T03:02:49.466 INFO:tasks.workunit.client.0.smithi153.stdout:4/510: creat da/db/d20/d3c/fab x:0 0 0 2024-04-05T03:02:49.467 INFO:tasks.workunit.client.0.smithi153.stdout:1/358: creat d5/d8/f6e x:0 0 0 2024-04-05T03:02:49.467 INFO:tasks.workunit.client.0.smithi153.stdout:0/433: chown d9/db/f44 3 1 2024-04-05T03:02:49.468 INFO:tasks.workunit.client.0.smithi153.stdout:6/433: rename d4/dc/df/d16/d33/f47 to d4/d38/f93 0 2024-04-05T03:02:49.469 INFO:tasks.workunit.client.1.smithi203.stdout:2/862: mknod d5/d10/d35/c12a 0 2024-04-05T03:02:49.469 INFO:tasks.workunit.client.1.smithi203.stdout:2/863: fsync d5/d66/d7d/f10e 0 2024-04-05T03:02:49.470 INFO:tasks.workunit.client.0.smithi153.stdout:4/511: write da/db/f46 [892126,7383] 0 2024-04-05T03:02:49.472 INFO:tasks.workunit.client.0.smithi153.stdout:3/401: dwrite d1d/f72 [0,4194304] 0 2024-04-05T03:02:49.472 INFO:tasks.workunit.client.1.smithi203.stdout:2/864: creat d5/d10/d44/f12b x:0 0 0 2024-04-05T03:02:49.477 INFO:tasks.workunit.client.1.smithi203.stdout:2/865: rmdir d5/d10/db8/dbe/d115 0 2024-04-05T03:02:49.477 INFO:tasks.workunit.client.0.smithi153.stdout:5/421: dwrite d0/d25/d4d/d62/d37/d38/f87 [0,4194304] 0 2024-04-05T03:02:49.479 INFO:tasks.workunit.client.0.smithi153.stdout:4/512: mknod da/de/d56/d6a/cac 0 2024-04-05T03:02:49.480 INFO:tasks.workunit.client.0.smithi153.stdout:6/434: truncate d4/dc/df/d16/d24/d4e/f5a 5921626 0 2024-04-05T03:02:49.481 INFO:tasks.workunit.client.0.smithi153.stdout:1/359: truncate d5/d8/d20/d4b/d18/d16/f22 4521452 0 2024-04-05T03:02:49.483 INFO:tasks.workunit.client.1.smithi203.stdout:2/866: truncate d5/d10/d35/d2d/d7c/d72/d8f/fa5 778393 0 2024-04-05T03:02:49.483 INFO:tasks.workunit.client.0.smithi153.stdout:5/422: stat d0/d25/d4d/d62/d5f/d86/f74 0 2024-04-05T03:02:49.485 INFO:tasks.workunit.client.0.smithi153.stdout:1/360: dread - d5/d8/d20/d2f/d40/f4c zero size 2024-04-05T03:02:49.485 INFO:tasks.workunit.client.0.smithi153.stdout:0/434: dwrite d9/df/d1e/f50 [0,4194304] 0 2024-04-05T03:02:49.486 INFO:tasks.workunit.client.0.smithi153.stdout:5/423: chown d0/d25/d4d/d62/l55 2 1 2024-04-05T03:02:49.487 INFO:tasks.workunit.client.0.smithi153.stdout:5/424: write d0/d9/d22/f33 [1242924,121432] 0 2024-04-05T03:02:49.487 INFO:tasks.workunit.client.0.smithi153.stdout:3/402: read d1d/d27/d2b/f2f [344569,11913] 0 2024-04-05T03:02:49.489 INFO:tasks.workunit.client.0.smithi153.stdout:5/425: fdatasync d0/d25/d4d/d62/d5f/f7e 0 2024-04-05T03:02:49.493 INFO:tasks.workunit.client.1.smithi203.stdout:2/867: dwrite d5/d10/d35/d2d/d7c/d72/d8f/fe4 [0,4194304] 0 2024-04-05T03:02:49.493 INFO:tasks.workunit.client.0.smithi153.stdout:6/435: mknod d4/dc/df/c94 0 2024-04-05T03:02:49.496 INFO:tasks.workunit.client.1.smithi203.stdout:2/868: readlink d5/l18 0 2024-04-05T03:02:49.497 INFO:tasks.workunit.client.0.smithi153.stdout:8/345: sync 2024-04-05T03:02:49.498 INFO:tasks.workunit.client.0.smithi153.stdout:2/406: sync 2024-04-05T03:02:49.498 INFO:tasks.workunit.client.0.smithi153.stdout:9/322: sync 2024-04-05T03:02:49.498 INFO:tasks.workunit.client.0.smithi153.stdout:1/361: mknod d5/d8/d20/d4b/d18/d16/d2c/c6f 0 2024-04-05T03:02:49.499 INFO:tasks.workunit.client.1.smithi203.stdout:2/869: creat d5/d10/d35/d2d/f12c x:0 0 0 2024-04-05T03:02:49.500 INFO:tasks.workunit.client.0.smithi153.stdout:5/426: creat d0/d25/d4d/d62/d5f/d86/f9a x:0 0 0 2024-04-05T03:02:49.500 INFO:tasks.workunit.client.0.smithi153.stdout:8/346: write d7/d2a/f65 [1759347,62819] 0 2024-04-05T03:02:49.501 INFO:tasks.workunit.client.0.smithi153.stdout:4/513: creat da/db/d20/fad x:0 0 0 2024-04-05T03:02:49.501 INFO:tasks.workunit.client.0.smithi153.stdout:2/407: dread - d9/da/d19/d2e/d7e/d7a/f84 zero size 2024-04-05T03:02:49.503 INFO:tasks.workunit.client.0.smithi153.stdout:1/362: stat d5/d8/d20/d4b/f11 0 2024-04-05T03:02:49.505 INFO:tasks.workunit.client.0.smithi153.stdout:3/403: mknod d1d/d26/d3c/d3f/d7c/c8c 0 2024-04-05T03:02:49.505 INFO:tasks.workunit.client.0.smithi153.stdout:9/323: creat d1/db/f76 x:0 0 0 2024-04-05T03:02:49.506 INFO:tasks.workunit.client.1.smithi203.stdout:2/870: getdents d5/d63 0 2024-04-05T03:02:49.507 INFO:tasks.workunit.client.0.smithi153.stdout:3/404: readlink d1d/d32/l5c 0 2024-04-05T03:02:49.507 INFO:tasks.workunit.client.0.smithi153.stdout:6/436: symlink d4/l95 0 2024-04-05T03:02:49.507 INFO:tasks.workunit.client.0.smithi153.stdout:5/427: fsync d0/d25/f36 0 2024-04-05T03:02:49.508 INFO:tasks.workunit.client.1.smithi203.stdout:2/871: symlink d5/d10/d2f/l12d 0 2024-04-05T03:02:49.509 INFO:tasks.workunit.client.0.smithi153.stdout:0/435: creat d9/db/d42/f7f x:0 0 0 2024-04-05T03:02:49.509 INFO:tasks.workunit.client.0.smithi153.stdout:3/405: chown d1d/d32/l5c 6078864 1 2024-04-05T03:02:49.509 INFO:tasks.workunit.client.0.smithi153.stdout:9/324: dread - d1/d1d/f5a zero size 2024-04-05T03:02:49.509 INFO:tasks.workunit.client.0.smithi153.stdout:2/408: mknod d9/da/d19/d2e/d48/c87 0 2024-04-05T03:02:49.509 INFO:tasks.workunit.client.0.smithi153.stdout:4/514: creat da/db/d20/d35/d41/d67/fae x:0 0 0 2024-04-05T03:02:49.513 INFO:tasks.workunit.client.1.smithi203.stdout:2/872: rename d5/d10/d35/d36/db6/d126 to d5/d10/d35/d2d/d7c/d29/d67/d118/d12e 0 2024-04-05T03:02:49.515 INFO:tasks.workunit.client.0.smithi153.stdout:1/363: creat d5/d8/d20/d4b/d18/d16/d33/f70 x:0 0 0 2024-04-05T03:02:49.516 INFO:tasks.workunit.client.0.smithi153.stdout:9/325: symlink d1/d3a/l77 0 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:6/744: sync 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:3/615: sync 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:1/719: sync 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:5/708: sync 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:7/997: sync 2024-04-05T03:02:49.518 INFO:tasks.workunit.client.1.smithi203.stdout:4/556: sync 2024-04-05T03:02:49.519 INFO:tasks.workunit.client.1.smithi203.stdout:8/773: sync 2024-04-05T03:02:49.519 INFO:tasks.workunit.client.0.smithi153.stdout:2/409: creat d9/d77/f88 x:0 0 0 2024-04-05T03:02:49.519 INFO:tasks.workunit.client.0.smithi153.stdout:3/406: symlink d1d/d75/l8d 0 2024-04-05T03:02:49.520 INFO:tasks.workunit.client.0.smithi153.stdout:4/515: mknod da/de/d56/d78/d8f/d68/caf 0 2024-04-05T03:02:49.521 INFO:tasks.workunit.client.1.smithi203.stdout:4/557: fdatasync da/d3c/d4c/f73 0 2024-04-05T03:02:49.521 INFO:tasks.workunit.client.0.smithi153.stdout:1/364: fsync d5/d8/d20/d4b/fe 0 2024-04-05T03:02:49.522 INFO:tasks.workunit.client.1.smithi203.stdout:1/720: mknod d7/d90/ce3 0 2024-04-05T03:02:49.522 INFO:tasks.workunit.client.1.smithi203.stdout:6/745: write d0/da/d54/d4c/fca [5059,25401] 0 2024-04-05T03:02:49.522 INFO:tasks.workunit.client.0.smithi153.stdout:9/326: write d1/d9/d1b/f73 [433898,37092] 0 2024-04-05T03:02:49.526 INFO:tasks.workunit.client.0.smithi153.stdout:5/428: dwrite d0/d25/f70 [0,4194304] 0 2024-04-05T03:02:49.526 INFO:tasks.workunit.client.0.smithi153.stdout:4/516: truncate da/de/d56/d6a/f87 214485 0 2024-04-05T03:02:49.527 INFO:tasks.workunit.client.0.smithi153.stdout:3/407: creat d1d/d26/d3c/d3f/d7c/f8e x:0 0 0 2024-04-05T03:02:49.529 INFO:tasks.workunit.client.1.smithi203.stdout:8/774: dwrite d1/d6/fab [0,4194304] 0 2024-04-05T03:02:49.529 INFO:tasks.workunit.client.1.smithi203.stdout:7/998: dread d8/d32/d16/d1a/d21/f23 [0,4194304] 0 2024-04-05T03:02:49.530 INFO:tasks.workunit.client.1.smithi203.stdout:8/775: readlink d1/d12/d2f/l1b 0 2024-04-05T03:02:49.530 INFO:tasks.workunit.client.0.smithi153.stdout:3/408: chown d1d/d27/d2b/f2f 528088 1 2024-04-05T03:02:49.531 INFO:tasks.workunit.client.0.smithi153.stdout:9/327: rename d1/d10/f6d to d1/d10/d36/d57/f78 0 2024-04-05T03:02:49.531 INFO:tasks.workunit.client.1.smithi203.stdout:5/709: creat d3/d4/dc/d67/db5/dda/fef x:0 0 0 2024-04-05T03:02:49.534 INFO:tasks.workunit.client.1.smithi203.stdout:1/721: mknod d7/d14/d9d/ce4 0 2024-04-05T03:02:49.538 INFO:tasks.workunit.client.0.smithi153.stdout:1/365: unlink d5/d8/l2e 0 2024-04-05T03:02:49.539 INFO:tasks.workunit.client.1.smithi203.stdout:7/999: rmdir d8/d32/dc3/dc8 39 2024-04-05T03:02:49.539 INFO:tasks.workunit.client.0.smithi153.stdout:5/429: creat d0/d25/d68/f9b x:0 0 0 2024-04-05T03:02:49.540 INFO:tasks.workunit.client.1.smithi203.stdout:2/873: write d5/d10/d35/d36/f6c [8442252,79736] 0 2024-04-05T03:02:49.540 INFO:tasks.workunit.client.1.smithi203.stdout:4/558: mknod da/d29/ca9 0 2024-04-05T03:02:49.541 INFO:tasks.workunit.client.0.smithi153.stdout:7/420: sync 2024-04-05T03:02:49.542 INFO:tasks.workunit.client.0.smithi153.stdout:3/409: dwrite d1d/d75/f77 [0,4194304] 0 2024-04-05T03:02:49.542 INFO:tasks.workunit.client.0.smithi153.stdout:6/437: dread d4/f51 [0,4194304] 0 2024-04-05T03:02:49.543 INFO:tasks.workunit.client.1.smithi203.stdout:5/710: creat d3/d4/dc/d67/dac/ff0 x:0 0 0 2024-04-05T03:02:49.543 INFO:tasks.workunit.client.1.smithi203.stdout:2/874: fsync d5/d10/db8/dab/de1/f9a 0 2024-04-05T03:02:49.543 INFO:tasks.workunit.client.1.smithi203.stdout:4/559: fsync da/d29/d6a/d80/fa0 0 2024-04-05T03:02:49.543 INFO:tasks.workunit.client.1.smithi203.stdout:5/711: chown d3/d4/d94/cd0 1209619648 1 2024-04-05T03:02:49.544 INFO:tasks.workunit.client.0.smithi153.stdout:8/347: truncate d7/d2a/f65 1485270 0 2024-04-05T03:02:49.544 INFO:tasks.workunit.client.0.smithi153.stdout:8/348: chown d7 428871 1 2024-04-05T03:02:49.544 INFO:tasks.workunit.client.0.smithi153.stdout:7/421: chown d2/da/d47/l89 119 1 2024-04-05T03:02:49.545 INFO:tasks.workunit.client.1.smithi203.stdout:8/776: rename d1/d12/d1a/l60 to d1/d4b/d5f/le3 0 2024-04-05T03:02:49.545 INFO:tasks.workunit.client.1.smithi203.stdout:1/722: dread d7/f99 [0,4194304] 0 2024-04-05T03:02:49.545 INFO:tasks.workunit.client.0.smithi153.stdout:9/328: symlink d1/d10/d16/d22/l79 0 2024-04-05T03:02:49.546 INFO:tasks.workunit.client.1.smithi203.stdout:6/746: mknod d0/da/d54/ce0 0 2024-04-05T03:02:49.548 INFO:tasks.workunit.client.1.smithi203.stdout:3/616: dread d0/d41/f4a [0,4194304] 0 2024-04-05T03:02:49.548 INFO:tasks.workunit.client.1.smithi203.stdout:1/723: write d7/d24/d33/d48/d4b/d6b/db9/fd2 [2872274,584] 0 2024-04-05T03:02:49.549 INFO:tasks.workunit.client.1.smithi203.stdout:4/560: rmdir da/d29/d2e 39 2024-04-05T03:02:49.549 INFO:tasks.workunit.client.1.smithi203.stdout:6/747: read - d0/da/d54/d4c/da1/db7/d71/fad zero size 2024-04-05T03:02:49.550 INFO:tasks.workunit.client.0.smithi153.stdout:6/438: dread - d4/d38/f8b zero size 2024-04-05T03:02:49.552 INFO:tasks.workunit.client.1.smithi203.stdout:5/712: chown d3/d1f/d59/d64/ca9 2 1 2024-04-05T03:02:49.552 INFO:tasks.workunit.client.0.smithi153.stdout:4/517: dwrite f9 [0,4194304] 0 2024-04-05T03:02:49.552 INFO:tasks.workunit.client.0.smithi153.stdout:6/439: chown d4/dc/df/d16/d24 34180 1 2024-04-05T03:02:49.554 INFO:tasks.workunit.client.0.smithi153.stdout:9/329: write d1/d10/d36/f48 [1003927,51539] 0 2024-04-05T03:02:49.555 INFO:tasks.workunit.client.1.smithi203.stdout:1/724: truncate d7/d24/d33/fad 603588 0 2024-04-05T03:02:49.555 INFO:tasks.workunit.client.0.smithi153.stdout:5/430: truncate d0/d2c/f53 32498 0 2024-04-05T03:02:49.555 INFO:tasks.workunit.client.0.smithi153.stdout:2/410: getdents d9/d77 0 2024-04-05T03:02:49.556 INFO:tasks.workunit.client.0.smithi153.stdout:8/349: mkdir d7/d13/d19/d1b/d4f/d72 0 2024-04-05T03:02:49.557 INFO:tasks.workunit.client.1.smithi203.stdout:6/748: chown d0/da/d54/d4c/da1/db7/d71/l72 15349767 1 2024-04-05T03:02:49.557 INFO:tasks.workunit.client.1.smithi203.stdout:3/617: rename d0/dc/d38/l79 to d0/dc/d1a/d1f/da1/lb0 0 2024-04-05T03:02:49.558 INFO:tasks.workunit.client.0.smithi153.stdout:2/411: stat d9/da/d37/f7b 0 2024-04-05T03:02:49.562 INFO:tasks.workunit.client.1.smithi203.stdout:5/713: creat d3/d4/dc/d67/dac/ff1 x:0 0 0 2024-04-05T03:02:49.562 INFO:tasks.workunit.client.1.smithi203.stdout:8/777: dwrite d1/d12/d2f/d1f/f50 [0,4194304] 0 2024-04-05T03:02:49.562 INFO:tasks.workunit.client.0.smithi153.stdout:0/436: fdatasync d9/df/d1d/d61/f3d 0 2024-04-05T03:02:49.562 INFO:tasks.workunit.client.0.smithi153.stdout:2/412: chown d9/da/d19 798 1 2024-04-05T03:02:49.563 INFO:tasks.workunit.client.0.smithi153.stdout:2/413: write d9/da/f49 [1988010,48162] 0 2024-04-05T03:02:49.564 INFO:tasks.workunit.client.0.smithi153.stdout:4/518: truncate da/db/f1a 1389177 0 2024-04-05T03:02:49.565 INFO:tasks.workunit.client.1.smithi203.stdout:6/749: symlink d0/da/d54/d4c/da1/db7/d71/le1 0 2024-04-05T03:02:49.565 INFO:tasks.workunit.client.1.smithi203.stdout:1/725: dread d7/f77 [0,4194304] 0 2024-04-05T03:02:49.567 INFO:tasks.workunit.client.0.smithi153.stdout:6/440: dwrite d4/dc/df/f13 [0,4194304] 0 2024-04-05T03:02:49.568 INFO:tasks.workunit.client.1.smithi203.stdout:5/714: mkdir d3/d4/dc/df2 0 2024-04-05T03:02:49.569 INFO:tasks.workunit.client.0.smithi153.stdout:9/330: write d1/d10/d36/d57/f78 [228396,118539] 0 2024-04-05T03:02:49.569 INFO:tasks.workunit.client.1.smithi203.stdout:1/726: fsync d7/d14/f1e 0 2024-04-05T03:02:49.570 INFO:tasks.workunit.client.1.smithi203.stdout:0/720: sync 2024-04-05T03:02:49.571 INFO:tasks.workunit.client.0.smithi153.stdout:7/422: dread d2/da/d11/d39/f63 [0,4194304] 0 2024-04-05T03:02:49.572 INFO:tasks.workunit.client.1.smithi203.stdout:9/737: sync 2024-04-05T03:02:49.573 INFO:tasks.workunit.client.0.smithi153.stdout:7/423: write d2/da/d3a/d41/d7a/f82 [615839,104490] 0 2024-04-05T03:02:49.576 INFO:tasks.workunit.client.0.smithi153.stdout:1/366: write d5/d8/d20/d4b/d18/d16/f36 [536918,11871] 0 2024-04-05T03:02:49.576 INFO:tasks.workunit.client.1.smithi203.stdout:4/561: rename da/l20 to da/d29/d2e/d57/da7/laa 0 2024-04-05T03:02:49.578 INFO:tasks.workunit.client.1.smithi203.stdout:5/715: dwrite d3/d4/dc/d67/dac/fb6 [4194304,4194304] 0 2024-04-05T03:02:49.579 INFO:tasks.workunit.client.0.smithi153.stdout:5/431: rename d0/d25/d4d/d62/d37/d39 to d0/d25/d4d/d9c 0 2024-04-05T03:02:49.580 INFO:tasks.workunit.client.1.smithi203.stdout:8/778: mknod d1/d90/ce4 0 2024-04-05T03:02:49.580 INFO:tasks.workunit.client.1.smithi203.stdout:3/618: creat d0/dc/d38/d92/fb1 x:0 0 0 2024-04-05T03:02:49.583 INFO:tasks.workunit.client.0.smithi153.stdout:0/437: creat d9/df/d1e/f80 x:0 0 0 2024-04-05T03:02:49.583 INFO:tasks.workunit.client.1.smithi203.stdout:5/716: dread d3/d4/fd4 [0,4194304] 0 2024-04-05T03:02:49.586 INFO:tasks.workunit.client.1.smithi203.stdout:3/619: symlink d0/dc/d38/lb2 0 2024-04-05T03:02:49.590 INFO:tasks.workunit.client.1.smithi203.stdout:2/875: dwrite d5/d10/d44/fd7 [0,4194304] 0 2024-04-05T03:02:49.590 INFO:tasks.workunit.client.1.smithi203.stdout:5/717: fsync d3/d1f/f93 0 2024-04-05T03:02:49.591 INFO:tasks.workunit.client.0.smithi153.stdout:2/414: symlink d9/da/d19/d2e/d7e/d7a/l89 0 2024-04-05T03:02:49.592 INFO:tasks.workunit.client.0.smithi153.stdout:2/415: fsync d9/da/f49 0 2024-04-05T03:02:49.592 INFO:tasks.workunit.client.1.smithi203.stdout:5/718: chown d3/d4/dc/l5c 1301667 1 2024-04-05T03:02:49.593 INFO:tasks.workunit.client.1.smithi203.stdout:3/620: mknod d0/dc/d1a/d1d/cb3 0 2024-04-05T03:02:49.594 INFO:tasks.workunit.client.0.smithi153.stdout:2/416: truncate d9/da/d37/f7b 717352 0 2024-04-05T03:02:49.594 INFO:tasks.workunit.client.1.smithi203.stdout:8/779: creat d1/fe5 x:0 0 0 2024-04-05T03:02:49.595 INFO:tasks.workunit.client.0.smithi153.stdout:2/417: stat d9/da/d19/d2e/d4f/c5c 0 2024-04-05T03:02:49.595 INFO:tasks.workunit.client.0.smithi153.stdout:6/441: rmdir d4/d38 39 2024-04-05T03:02:49.596 INFO:tasks.workunit.client.1.smithi203.stdout:3/621: write d0/d41/f9a [977822,75777] 0 2024-04-05T03:02:49.596 INFO:tasks.workunit.client.1.smithi203.stdout:5/719: mknod d3/d5f/d92/cf3 0 2024-04-05T03:02:49.598 INFO:tasks.workunit.client.0.smithi153.stdout:3/410: dread d1d/d26/f4c [0,4194304] 0 2024-04-05T03:02:49.598 INFO:tasks.workunit.client.1.smithi203.stdout:2/876: unlink d5/d10/d35/d2d/d7c/f25 0 2024-04-05T03:02:49.600 INFO:tasks.workunit.client.0.smithi153.stdout:5/432: creat d0/d25/d4d/d62/f9d x:0 0 0 2024-04-05T03:02:49.600 INFO:tasks.workunit.client.1.smithi203.stdout:5/720: chown d3/d5f/d92/fc9 38427917 1 2024-04-05T03:02:49.601 INFO:tasks.workunit.client.0.smithi153.stdout:8/350: write f6 [1274136,110254] 0 2024-04-05T03:02:49.603 INFO:tasks.workunit.client.1.smithi203.stdout:8/780: dread d1/d4b/d68/d75/dac/da4/fc9 [0,4194304] 0 2024-04-05T03:02:49.604 INFO:tasks.workunit.client.1.smithi203.stdout:3/622: symlink d0/d21/lb4 0 2024-04-05T03:02:49.605 INFO:tasks.workunit.client.1.smithi203.stdout:8/781: truncate d1/f63 4916782 0 2024-04-05T03:02:49.606 INFO:tasks.workunit.client.0.smithi153.stdout:2/418: mkdir d9/da/d37/d8a 0 2024-04-05T03:02:49.607 INFO:tasks.workunit.client.1.smithi203.stdout:5/721: creat d3/d4/dc/ff4 x:0 0 0 2024-04-05T03:02:49.608 INFO:tasks.workunit.client.1.smithi203.stdout:1/727: dread d7/d24/d33/d48/d4b/d6b/db9/fce [0,4194304] 0 2024-04-05T03:02:49.610 INFO:tasks.workunit.client.1.smithi203.stdout:8/782: creat d1/d4b/d68/d75/d77/fe6 x:0 0 0 2024-04-05T03:02:49.611 INFO:tasks.workunit.client.1.smithi203.stdout:8/783: dread - d1/d4b/d5f/daf/fba zero size 2024-04-05T03:02:49.611 INFO:tasks.workunit.client.0.smithi153.stdout:6/442: creat d4/dc/df/d16/d44/f96 x:0 0 0 2024-04-05T03:02:49.612 INFO:tasks.workunit.client.0.smithi153.stdout:6/443: chown d4/dc/df/d16 37911 1 2024-04-05T03:02:49.612 INFO:tasks.workunit.client.1.smithi203.stdout:3/623: rename d0/dc/l1c to d0/d21/lb5 0 2024-04-05T03:02:49.613 INFO:tasks.workunit.client.1.smithi203.stdout:8/784: write d1/d4b/d5f/daf/fba [428749,95012] 0 2024-04-05T03:02:49.613 INFO:tasks.workunit.client.0.smithi153.stdout:3/411: creat d1d/d27/f8f x:0 0 0 2024-04-05T03:02:49.614 INFO:tasks.workunit.client.1.smithi203.stdout:6/750: write d0/da/d18/d38/d2a/d44/d78/f89 [608284,6959] 0 2024-04-05T03:02:49.615 INFO:tasks.workunit.client.0.smithi153.stdout:5/433: creat d0/d4e/f9e x:0 0 0 2024-04-05T03:02:49.617 INFO:tasks.workunit.client.1.smithi203.stdout:1/728: mknod d7/d24/d33/d48/d4b/d6b/db9/ce5 0 2024-04-05T03:02:49.619 INFO:tasks.workunit.client.0.smithi153.stdout:0/438: mknod d9/c81 0 2024-04-05T03:02:49.620 INFO:tasks.workunit.client.0.smithi153.stdout:4/519: creat da/fb0 x:0 0 0 2024-04-05T03:02:49.621 INFO:tasks.workunit.client.1.smithi203.stdout:4/562: write da/f11 [3523320,122956] 0 2024-04-05T03:02:49.622 INFO:tasks.workunit.client.1.smithi203.stdout:3/624: chown d0/dc/d1a/d1d/f2f 407915162 1 2024-04-05T03:02:49.622 INFO:tasks.workunit.client.0.smithi153.stdout:2/419: symlink d9/da/d19/d2e/d60/l8b 0 2024-04-05T03:02:49.624 INFO:tasks.workunit.client.0.smithi153.stdout:7/424: creat d2/da/d3a/d41/f8a x:0 0 0 2024-04-05T03:02:49.624 INFO:tasks.workunit.client.1.smithi203.stdout:9/738: dread d13/d1b/d33/d64/f6d [0,4194304] 0 2024-04-05T03:02:49.625 INFO:tasks.workunit.client.1.smithi203.stdout:8/785: dwrite d1/d6/d8f/fb0 [0,4194304] 0 2024-04-05T03:02:49.625 INFO:tasks.workunit.client.1.smithi203.stdout:6/751: mknod d0/da/d54/d4c/d7f/ce2 0 2024-04-05T03:02:49.625 INFO:tasks.workunit.client.0.smithi153.stdout:6/444: dwrite d4/dc/df/d16/d17/f84 [0,4194304] 0 2024-04-05T03:02:49.626 INFO:tasks.workunit.client.1.smithi203.stdout:3/625: rmdir d0/dc/d1a 39 2024-04-05T03:02:49.628 INFO:tasks.workunit.client.0.smithi153.stdout:2/420: write d9/da/d19/f1f [10839894,43985] 0 2024-04-05T03:02:49.629 INFO:tasks.workunit.client.0.smithi153.stdout:4/520: write da/d2a/f33 [3807977,100980] 0 2024-04-05T03:02:49.629 INFO:tasks.workunit.client.0.smithi153.stdout:3/412: dwrite d1d/f36 [0,4194304] 0 2024-04-05T03:02:49.632 INFO:tasks.workunit.client.0.smithi153.stdout:6/445: write d4/dc/df/d16/d5f/f8c [227708,12469] 0 2024-04-05T03:02:49.633 INFO:tasks.workunit.client.1.smithi203.stdout:8/786: write d1/d12/d2f/d5b/fb3 [967283,64614] 0 2024-04-05T03:02:49.633 INFO:tasks.workunit.client.1.smithi203.stdout:4/563: truncate da/d29/d2e/d53/f97 550120 0 2024-04-05T03:02:49.634 INFO:tasks.workunit.client.1.smithi203.stdout:6/752: dread d0/da/d54/f3f [0,4194304] 0 2024-04-05T03:02:49.634 INFO:tasks.workunit.client.0.smithi153.stdout:1/367: mknod d5/d8/d20/d4b/d1e/c71 0 2024-04-05T03:02:49.634 INFO:tasks.workunit.client.0.smithi153.stdout:7/425: write d2/da/d42/f44 [347738,58064] 0 2024-04-05T03:02:49.636 INFO:tasks.workunit.client.0.smithi153.stdout:4/521: dread - da/de/d56/fa0 zero size 2024-04-05T03:02:49.636 INFO:tasks.workunit.client.1.smithi203.stdout:3/626: unlink d0/dc/le 0 2024-04-05T03:02:49.637 INFO:tasks.workunit.client.0.smithi153.stdout:0/439: rename d9/db/f2d to d9/db/d42/d43/f82 0 2024-04-05T03:02:49.638 INFO:tasks.workunit.client.0.smithi153.stdout:5/434: truncate d0/d25/f2f 1843361 0 2024-04-05T03:02:49.639 INFO:tasks.workunit.client.0.smithi153.stdout:4/522: chown da/fb0 248053238 1 2024-04-05T03:02:49.640 INFO:tasks.workunit.client.1.smithi203.stdout:9/739: symlink d13/d1b/d6b/d9e/d59/d9b/lef 0 2024-04-05T03:02:49.641 INFO:tasks.workunit.client.1.smithi203.stdout:8/787: rename d1/d4b/d68/d75 to d1/d4b/d68/d75/d77/de7 22 2024-04-05T03:02:49.642 INFO:tasks.workunit.client.0.smithi153.stdout:9/331: dread d1/d10/f39 [4194304,4194304] 0 2024-04-05T03:02:49.643 INFO:tasks.workunit.client.0.smithi153.stdout:2/421: symlink d9/d62/l8c 0 2024-04-05T03:02:49.643 INFO:tasks.workunit.client.1.smithi203.stdout:2/877: dwrite d5/d10/d1a/f4b [0,4194304] 0 2024-04-05T03:02:49.644 INFO:tasks.workunit.client.0.smithi153.stdout:3/413: mkdir d1d/d26/d3c/d3f/d90 0 2024-04-05T03:02:49.644 INFO:tasks.workunit.client.0.smithi153.stdout:1/368: chown d5/dd/c3b 106394887 1 2024-04-05T03:02:49.645 INFO:tasks.workunit.client.1.smithi203.stdout:9/740: fdatasync d13/d1b/d33/d42/fb0 0 2024-04-05T03:02:49.645 INFO:tasks.workunit.client.1.smithi203.stdout:4/564: mkdir da/d29/d2e/d57/da7/dab 0 2024-04-05T03:02:49.647 INFO:tasks.workunit.client.0.smithi153.stdout:7/426: mknod d2/da/d47/d5d/c8b 0 2024-04-05T03:02:49.647 INFO:tasks.workunit.client.0.smithi153.stdout:9/332: chown d1/d1d/l42 21 1 2024-04-05T03:02:49.649 INFO:tasks.workunit.client.0.smithi153.stdout:5/435: creat d0/d4e/f9f x:0 0 0 2024-04-05T03:02:49.652 INFO:tasks.workunit.client.0.smithi153.stdout:8/351: truncate d7/d13/f18 1991931 0 2024-04-05T03:02:49.653 INFO:tasks.workunit.client.0.smithi153.stdout:3/414: mknod d1d/d27/c91 0 2024-04-05T03:02:49.653 INFO:tasks.workunit.client.0.smithi153.stdout:0/440: dwrite d9/db/d42/f7f [0,4194304] 0 2024-04-05T03:02:49.654 INFO:tasks.workunit.client.1.smithi203.stdout:6/753: fsync d0/da/d54/d4c/da1/db7/fcb 0 2024-04-05T03:02:49.655 INFO:tasks.workunit.client.0.smithi153.stdout:6/446: creat d4/dc/f97 x:0 0 0 2024-04-05T03:02:49.655 INFO:tasks.workunit.client.1.smithi203.stdout:1/729: dread d7/f1c [0,4194304] 0 2024-04-05T03:02:49.656 INFO:tasks.workunit.client.1.smithi203.stdout:9/741: write d13/d5c/d9d/fe4 [718317,77089] 0 2024-04-05T03:02:49.658 INFO:tasks.workunit.client.0.smithi153.stdout:4/523: mkdir da/db/d20/d45/db1 0 2024-04-05T03:02:49.658 INFO:tasks.workunit.client.0.smithi153.stdout:2/422: creat d9/da/d19/d2e/d7e/d7a/d59/f8d x:0 0 0 2024-04-05T03:02:49.658 INFO:tasks.workunit.client.1.smithi203.stdout:6/754: chown d0/da/d54/d4c/fca 0 1 2024-04-05T03:02:49.659 INFO:tasks.workunit.client.1.smithi203.stdout:5/722: dwrite d3/d1f/d59/d64/d12/f46 [0,4194304] 0 2024-04-05T03:02:49.660 INFO:tasks.workunit.client.0.smithi153.stdout:9/333: creat d1/db/f7a x:0 0 0 2024-04-05T03:02:49.661 INFO:tasks.workunit.client.1.smithi203.stdout:6/755: readlink d0/da/d54/d82/la0 0 2024-04-05T03:02:49.662 INFO:tasks.workunit.client.0.smithi153.stdout:1/369: mknod d5/d8/d20/d4b/d51/c72 0 2024-04-05T03:02:49.662 INFO:tasks.workunit.client.0.smithi153.stdout:9/334: fsync d1/f2e 0 2024-04-05T03:02:49.665 INFO:tasks.workunit.client.0.smithi153.stdout:4/524: creat da/de/fb2 x:0 0 0 2024-04-05T03:02:49.666 INFO:tasks.workunit.client.0.smithi153.stdout:8/352: mkdir d7/d13/d19/d1b/d4f/d72/d73 0 2024-04-05T03:02:49.666 INFO:tasks.workunit.client.1.smithi203.stdout:3/627: dwrite d0/dc/d38/d92/fb1 [0,4194304] 0 2024-04-05T03:02:49.666 INFO:tasks.workunit.client.1.smithi203.stdout:3/628: write d0/d21/f6c [2923023,32075] 0 2024-04-05T03:02:49.667 INFO:tasks.workunit.client.0.smithi153.stdout:5/436: dwrite d0/d25/d4d/d62/d5f/d86/f9a [0,4194304] 0 2024-04-05T03:02:49.670 INFO:tasks.workunit.client.1.smithi203.stdout:3/629: read d0/dc/f54 [1599894,65120] 0 2024-04-05T03:02:49.674 INFO:tasks.workunit.client.1.smithi203.stdout:5/723: dwrite d3/d4/f8e [0,4194304] 0 2024-04-05T03:02:49.674 INFO:tasks.workunit.client.0.smithi153.stdout:7/427: rename d2/da/d3a/d41/d31/f5f to d2/da/d3a/d41/f8c 0 2024-04-05T03:02:49.675 INFO:tasks.workunit.client.0.smithi153.stdout:1/370: write d5/d8/d20/d4b/d18/d16/f2d [4913529,87014] 0 2024-04-05T03:02:49.678 INFO:tasks.workunit.client.0.smithi153.stdout:3/415: dwrite d1d/d27/d2b/f7e [0,4194304] 0 2024-04-05T03:02:49.682 INFO:tasks.workunit.client.0.smithi153.stdout:0/441: rmdir d9/df/d1e/d45/d58 0 2024-04-05T03:02:49.683 INFO:tasks.workunit.client.0.smithi153.stdout:7/428: write d2/f5 [3144957,104174] 0 2024-04-05T03:02:49.684 INFO:tasks.workunit.client.0.smithi153.stdout:9/335: creat d1/d9/d1b/f7b x:0 0 0 2024-04-05T03:02:49.685 INFO:tasks.workunit.client.1.smithi203.stdout:5/724: dwrite d3/d1f/d59/d64/d12/f46 [0,4194304] 0 2024-04-05T03:02:49.686 INFO:tasks.workunit.client.0.smithi153.stdout:3/416: dread d1d/d27/f6f [0,4194304] 0 2024-04-05T03:02:49.687 INFO:tasks.workunit.client.0.smithi153.stdout:1/371: fdatasync d5/d8/d20/d4b/d18/d16/f22 0 2024-04-05T03:02:49.689 INFO:tasks.workunit.client.1.smithi203.stdout:9/742: dread d13/d5c/d9d/fcd [0,4194304] 0 2024-04-05T03:02:49.690 INFO:tasks.workunit.client.0.smithi153.stdout:4/525: mknod da/db/d20/d35/d38/d3e/cb3 0 2024-04-05T03:02:49.690 INFO:tasks.workunit.client.0.smithi153.stdout:2/423: dwrite d9/d77/f69 [0,4194304] 0 2024-04-05T03:02:49.694 INFO:tasks.workunit.client.0.smithi153.stdout:7/429: chown d2/d51/f79 102457700 1 2024-04-05T03:02:49.702 INFO:tasks.workunit.client.0.smithi153.stdout:0/442: mknod d9/db/d42/d7e/c83 0 2024-04-05T03:02:49.703 INFO:tasks.workunit.client.0.smithi153.stdout:4/526: symlink da/de/d56/d78/d8f/lb4 0 2024-04-05T03:02:49.705 INFO:tasks.workunit.client.0.smithi153.stdout:3/417: write d1d/d26/d3c/f5b [2422825,66818] 0 2024-04-05T03:02:49.705 INFO:tasks.workunit.client.0.smithi153.stdout:7/430: mkdir d2/da/d47/d5d/d7c/d8d 0 2024-04-05T03:02:49.707 INFO:tasks.workunit.client.0.smithi153.stdout:9/336: creat d1/d10/d16/d22/d2c/d68/f7c x:0 0 0 2024-04-05T03:02:49.707 INFO:tasks.workunit.client.0.smithi153.stdout:3/418: readlink d1d/d26/d57/l6c 0 2024-04-05T03:02:49.707 INFO:tasks.workunit.client.0.smithi153.stdout:2/424: mknod d9/da/d19/d2e/d48/c8e 0 2024-04-05T03:02:49.710 INFO:tasks.workunit.client.0.smithi153.stdout:6/447: rename d4/ca to d4/dc/df/d16/d33/c98 0 2024-04-05T03:02:49.711 INFO:tasks.workunit.client.0.smithi153.stdout:4/527: chown da/db/d20/d35/d41/c64 14531 1 2024-04-05T03:02:49.712 INFO:tasks.workunit.client.0.smithi153.stdout:2/425: creat d9/da/d19/d2e/f8f x:0 0 0 2024-04-05T03:02:49.713 INFO:tasks.workunit.client.0.smithi153.stdout:3/419: mkdir d1d/d26/d3c/d3f/d90/d92 0 2024-04-05T03:02:49.715 INFO:tasks.workunit.client.0.smithi153.stdout:7/431: link d2/da/d3a/d41/d7a/f3c d2/f8e 0 2024-04-05T03:02:49.717 INFO:tasks.workunit.client.0.smithi153.stdout:6/448: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:49.719 INFO:tasks.workunit.client.0.smithi153.stdout:8/353: rename d7/d13/d19/d1b/d4f/f5d to d7/f74 0 2024-04-05T03:02:49.722 INFO:tasks.workunit.client.0.smithi153.stdout:4/528: creat da/db/d20/fb5 x:0 0 0 2024-04-05T03:02:49.725 INFO:tasks.workunit.client.0.smithi153.stdout:6/449: creat d4/dc/df/d16/d24/d4e/f99 x:0 0 0 2024-04-05T03:02:49.725 INFO:tasks.workunit.client.0.smithi153.stdout:4/529: dread da/db/d20/d45/f82 [0,4194304] 0 2024-04-05T03:02:49.727 INFO:tasks.workunit.client.0.smithi153.stdout:7/432: getdents d2/da/d47/d5d/d7c/d8d 0 2024-04-05T03:02:49.727 INFO:tasks.workunit.client.0.smithi153.stdout:4/530: write da/de/d56/d78/f7e [117712,89028] 0 2024-04-05T03:02:49.735 INFO:tasks.workunit.client.0.smithi153.stdout:7/433: unlink d2/da/d3a/d41/d31/f43 0 2024-04-05T03:02:49.739 INFO:tasks.workunit.client.0.smithi153.stdout:6/450: link d4/dc/df/c34 d4/c9a 0 2024-04-05T03:02:49.743 INFO:tasks.workunit.client.0.smithi153.stdout:7/434: rename d2/da/cd to d2/da/c8f 0 2024-04-05T03:02:49.743 INFO:tasks.workunit.client.1.smithi203.stdout:4/565: dread da/d29/f2b [0,4194304] 0 2024-04-05T03:02:49.746 INFO:tasks.workunit.client.0.smithi153.stdout:6/451: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:49.747 INFO:tasks.workunit.client.0.smithi153.stdout:8/354: dread f0 [4194304,4194304] 0 2024-04-05T03:02:49.749 INFO:tasks.workunit.client.0.smithi153.stdout:6/452: chown d4/dc/df/d16/d36/d4d/d64/f6c 15 1 2024-04-05T03:02:49.749 INFO:tasks.workunit.client.0.smithi153.stdout:4/531: rmdir da/de/d56 39 2024-04-05T03:02:49.751 INFO:tasks.workunit.client.0.smithi153.stdout:6/453: chown d4/dc/df/d16/d24/d8d/c8f 15584089 1 2024-04-05T03:02:49.752 INFO:tasks.workunit.client.0.smithi153.stdout:1/372: dwrite d5/d8/ff [0,4194304] 0 2024-04-05T03:02:49.753 INFO:tasks.workunit.client.0.smithi153.stdout:0/443: write d9/db/d42/d43/f4a [101419,121408] 0 2024-04-05T03:02:49.758 INFO:tasks.workunit.client.0.smithi153.stdout:0/444: truncate d9/d14/d67/d6f/f7c 4945169 0 2024-04-05T03:02:49.758 INFO:tasks.workunit.client.0.smithi153.stdout:3/420: rename d1d/d26 to d1d/d32/d93 0 2024-04-05T03:02:49.764 INFO:tasks.workunit.client.0.smithi153.stdout:3/421: chown d1d/d32/d93/d57/f76 3 1 2024-04-05T03:02:49.765 INFO:tasks.workunit.client.0.smithi153.stdout:7/435: rename d2/da/d47/c83 to d2/da/d47/d5d/c90 0 2024-04-05T03:02:49.767 INFO:tasks.workunit.client.0.smithi153.stdout:1/373: dwrite d5/d8/f6e [0,4194304] 0 2024-04-05T03:02:49.771 INFO:tasks.workunit.client.0.smithi153.stdout:3/422: truncate d1d/d32/d93/f4c 1513109 0 2024-04-05T03:02:49.771 INFO:tasks.workunit.client.1.smithi203.stdout:8/788: unlink d1/d4b/d68/d75/dac/da4/fda 0 2024-04-05T03:02:49.773 INFO:tasks.workunit.client.0.smithi153.stdout:2/426: dwrite d9/da/d36/f4d [0,4194304] 0 2024-04-05T03:02:49.773 INFO:tasks.workunit.client.1.smithi203.stdout:8/789: write d1/d12/d17/d47/d53/f82 [4350513,25738] 0 2024-04-05T03:02:49.777 INFO:tasks.workunit.client.1.smithi203.stdout:6/756: mknod d0/da/d54/d4c/da1/db7/d71/ce3 0 2024-04-05T03:02:49.779 INFO:tasks.workunit.client.0.smithi153.stdout:7/436: symlink d2/da/d3a/d41/d7a/d55/l91 0 2024-04-05T03:02:49.780 INFO:tasks.workunit.client.1.smithi203.stdout:6/757: truncate d0/da/d54/d4c/da1/db7/d71/fad 585136 0 2024-04-05T03:02:49.781 INFO:tasks.workunit.client.1.smithi203.stdout:9/743: mkdir d13/d1b/d6b/d9e/df0 0 2024-04-05T03:02:49.783 INFO:tasks.workunit.client.0.smithi153.stdout:6/454: rename d4/dc/df/c5b to d4/dc/df/d16/d17/c9b 0 2024-04-05T03:02:49.784 INFO:tasks.workunit.client.0.smithi153.stdout:1/374: getdents d5/d8/d20 0 2024-04-05T03:02:49.786 INFO:tasks.workunit.client.1.smithi203.stdout:1/730: symlink d7/d14/d2b/le6 0 2024-04-05T03:02:49.786 INFO:tasks.workunit.client.0.smithi153.stdout:3/423: dwrite d1d/d32/d93/d3c/f7b [0,4194304] 0 2024-04-05T03:02:49.788 INFO:tasks.workunit.client.0.smithi153.stdout:2/427: truncate d9/da/f49 3191755 0 2024-04-05T03:02:49.788 INFO:tasks.workunit.client.0.smithi153.stdout:0/445: write d9/d14/d67/d6f/f7c [2166650,105568] 0 2024-04-05T03:02:49.788 INFO:tasks.workunit.client.1.smithi203.stdout:8/790: creat d1/d6/fe8 x:0 0 0 2024-04-05T03:02:49.793 INFO:tasks.workunit.client.0.smithi153.stdout:8/355: rename d7/d13/d19/d1b/d33/f63 to d7/d13/d19/d1b/d4f/d68/f75 0 2024-04-05T03:02:49.794 INFO:tasks.workunit.client.1.smithi203.stdout:6/758: mkdir d0/da/d18/d92/de4 0 2024-04-05T03:02:49.794 INFO:tasks.workunit.client.0.smithi153.stdout:1/375: symlink d5/d8/d20/d3f/l73 0 2024-04-05T03:02:49.798 INFO:tasks.workunit.client.1.smithi203.stdout:1/731: symlink d7/d14/d2b/d58/d86/d91/da6/le7 0 2024-04-05T03:02:49.799 INFO:tasks.workunit.client.0.smithi153.stdout:6/455: mkdir d4/dc/d7a/d9c 0 2024-04-05T03:02:49.799 INFO:tasks.workunit.client.1.smithi203.stdout:8/791: creat d1/d4b/d5f/fe9 x:0 0 0 2024-04-05T03:02:49.801 INFO:tasks.workunit.client.1.smithi203.stdout:3/630: creat d0/dc/d1a/d1f/dac/fb6 x:0 0 0 2024-04-05T03:02:49.802 INFO:tasks.workunit.client.0.smithi153.stdout:3/424: mkdir d1d/d32/d93/d3c/d94 0 2024-04-05T03:02:49.802 INFO:tasks.workunit.client.0.smithi153.stdout:1/376: fsync d5/d8/d20/d3f/f5c 0 2024-04-05T03:02:49.804 INFO:tasks.workunit.client.0.smithi153.stdout:2/428: dwrite d9/da/d19/f46 [0,4194304] 0 2024-04-05T03:02:49.804 INFO:tasks.workunit.client.0.smithi153.stdout:7/437: link d2/da/d3a/d41/d7a/d55/d6f/f78 d2/da/d47/f92 0 2024-04-05T03:02:49.807 INFO:tasks.workunit.client.0.smithi153.stdout:6/456: symlink d4/dc/d7a/l9d 0 2024-04-05T03:02:49.808 INFO:tasks.workunit.client.1.smithi203.stdout:3/631: dwrite d0/d41/f9a [0,4194304] 0 2024-04-05T03:02:49.809 INFO:tasks.workunit.client.1.smithi203.stdout:2/878: getdents d5/d10/d35/d2d/d7c/d29/d67/d118 0 2024-04-05T03:02:49.811 INFO:tasks.workunit.client.1.smithi203.stdout:1/732: mkdir d7/d14/d2b/d58/de8 0 2024-04-05T03:02:49.812 INFO:tasks.workunit.client.1.smithi203.stdout:1/733: chown d7/d24/d33/d48 3094 1 2024-04-05T03:02:49.814 INFO:tasks.workunit.client.0.smithi153.stdout:8/356: dwrite d7/d13/d19/d1b/d4f/d68/f75 [0,4194304] 0 2024-04-05T03:02:49.814 INFO:tasks.workunit.client.1.smithi203.stdout:5/725: getdents d3/d1f 0 2024-04-05T03:02:49.816 INFO:tasks.workunit.client.0.smithi153.stdout:2/429: rmdir d9/da/d19/d2e/d7e/d7a/d59 39 2024-04-05T03:02:49.817 INFO:tasks.workunit.client.0.smithi153.stdout:7/438: dread d2/f1d [0,4194304] 0 2024-04-05T03:02:49.818 INFO:tasks.workunit.client.0.smithi153.stdout:1/377: dwrite f4 [8388608,4194304] 0 2024-04-05T03:02:49.818 INFO:tasks.workunit.client.1.smithi203.stdout:1/734: symlink d7/le9 0 2024-04-05T03:02:49.820 INFO:tasks.workunit.client.0.smithi153.stdout:7/439: stat d2/da/d3a/d41/d7a/f56 0 2024-04-05T03:02:49.820 INFO:tasks.workunit.client.1.smithi203.stdout:8/792: creat d1/d6/d2e/fea x:0 0 0 2024-04-05T03:02:49.822 INFO:tasks.workunit.client.0.smithi153.stdout:6/457: creat d4/dc/df/d16/d5f/d73/f9e x:0 0 0 2024-04-05T03:02:49.826 INFO:tasks.workunit.client.0.smithi153.stdout:1/378: dread - d5/dd/f5d zero size 2024-04-05T03:02:49.829 INFO:tasks.workunit.client.0.smithi153.stdout:1/379: write d5/d8/d20/d4b/d18/d16/d33/f6b [144462,85559] 0 2024-04-05T03:02:49.830 INFO:tasks.workunit.client.1.smithi203.stdout:5/726: dread d3/d1f/d49/f57 [0,4194304] 0 2024-04-05T03:02:49.830 INFO:tasks.workunit.client.1.smithi203.stdout:1/735: mkdir d7/d24/d33/d48/d4b/dea 0 2024-04-05T03:02:49.830 INFO:tasks.workunit.client.0.smithi153.stdout:7/440: rename d2/da/d42 to d2/d51/d93 0 2024-04-05T03:02:49.830 INFO:tasks.workunit.client.0.smithi153.stdout:0/446: dwrite d9/df/f59 [0,4194304] 0 2024-04-05T03:02:49.832 INFO:tasks.workunit.client.0.smithi153.stdout:7/441: dread - d2/f7b zero size 2024-04-05T03:02:49.832 INFO:tasks.workunit.client.0.smithi153.stdout:1/380: dread - d5/dd/d1b/f56 zero size 2024-04-05T03:02:49.834 INFO:tasks.workunit.client.0.smithi153.stdout:7/442: truncate d2/da/f68 294400 0 2024-04-05T03:02:49.837 INFO:tasks.workunit.client.1.smithi203.stdout:8/793: link d1/d12/da8/fe1 d1/d12/feb 0 2024-04-05T03:02:49.837 INFO:tasks.workunit.client.0.smithi153.stdout:6/458: link d4/fb d4/dc/df/d16/d36/d52/f9f 0 2024-04-05T03:02:49.839 INFO:tasks.workunit.client.1.smithi203.stdout:1/736: mkdir d7/d14/d83/deb 0 2024-04-05T03:02:49.841 INFO:tasks.workunit.client.1.smithi203.stdout:5/727: creat d3/d4/dc/df2/ff5 x:0 0 0 2024-04-05T03:02:49.842 INFO:tasks.workunit.client.0.smithi153.stdout:0/447: dwrite d9/df/d1d/f3b [0,4194304] 0 2024-04-05T03:02:49.848 INFO:tasks.workunit.client.1.smithi203.stdout:3/632: dread d0/dc/d38/f3b [0,4194304] 0 2024-04-05T03:02:49.852 INFO:tasks.workunit.client.1.smithi203.stdout:5/728: link d3/d4/dc/cb2 d3/d1f/d59/cf6 0 2024-04-05T03:02:49.853 INFO:tasks.workunit.client.1.smithi203.stdout:3/633: mknod d0/d21/d81/cb7 0 2024-04-05T03:02:49.853 INFO:tasks.workunit.client.0.smithi153.stdout:3/425: truncate d1d/d32/f49 747552 0 2024-04-05T03:02:49.855 INFO:tasks.workunit.client.0.smithi153.stdout:7/443: dwrite d2/da/d3a/d41/d7a/f4e [0,4194304] 0 2024-04-05T03:02:49.856 INFO:tasks.workunit.client.1.smithi203.stdout:3/634: rmdir d0/dc/d38/d5d 39 2024-04-05T03:02:49.857 INFO:tasks.workunit.client.0.smithi153.stdout:6/459: dwrite d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:49.860 INFO:tasks.workunit.client.1.smithi203.stdout:5/729: rename d3/d1f/d59/d64/c2f to d3/d4/dc/d67/dac/cf7 0 2024-04-05T03:02:49.862 INFO:tasks.workunit.client.0.smithi153.stdout:2/430: dread d9/da/fe [0,4194304] 0 2024-04-05T03:02:49.865 INFO:tasks.workunit.client.1.smithi203.stdout:5/730: rename d3/d4/d9e/dde/ce6 to d3/d4/dc/d67/dac/cf8 0 2024-04-05T03:02:49.865 INFO:tasks.workunit.client.0.smithi153.stdout:0/448: dwrite d9/db/f13 [0,4194304] 0 2024-04-05T03:02:49.865 INFO:tasks.workunit.client.0.smithi153.stdout:6/460: chown d4/dc/df/d16/d33/c98 3790952 1 2024-04-05T03:02:49.866 INFO:tasks.workunit.client.0.smithi153.stdout:7/444: creat d2/da/d3a/f94 x:0 0 0 2024-04-05T03:02:49.868 INFO:tasks.workunit.client.1.smithi203.stdout:5/731: stat d3/d1f/d59/d60/fad 0 2024-04-05T03:02:49.869 INFO:tasks.workunit.client.1.smithi203.stdout:4/566: truncate da/f26 332335 0 2024-04-05T03:02:49.869 INFO:tasks.workunit.client.0.smithi153.stdout:3/426: symlink d1d/d32/d93/d3c/d3f/l95 0 2024-04-05T03:02:49.870 INFO:tasks.workunit.client.1.smithi203.stdout:9/744: dwrite d13/dc8/dd2/fe7 [0,4194304] 0 2024-04-05T03:02:49.870 INFO:tasks.workunit.client.0.smithi153.stdout:0/449: mkdir d9/df/d1e/d84 0 2024-04-05T03:02:49.871 INFO:tasks.workunit.client.0.smithi153.stdout:6/461: chown d4/dc/df/d16/d36/d4d/d64/c67 5241685 1 2024-04-05T03:02:49.872 INFO:tasks.workunit.client.0.smithi153.stdout:2/431: truncate d9/da/f49 2750033 0 2024-04-05T03:02:49.874 INFO:tasks.workunit.client.0.smithi153.stdout:3/427: creat d1d/d27/f96 x:0 0 0 2024-04-05T03:02:49.874 INFO:tasks.workunit.client.0.smithi153.stdout:8/357: dread d7/d13/f14 [0,4194304] 0 2024-04-05T03:02:49.875 INFO:tasks.workunit.client.0.smithi153.stdout:7/445: mkdir d2/da/d47/d5d/d7c/d8d/d95 0 2024-04-05T03:02:49.876 INFO:tasks.workunit.client.0.smithi153.stdout:2/432: read d9/da/fb [3947256,87422] 0 2024-04-05T03:02:49.877 INFO:tasks.workunit.client.1.smithi203.stdout:6/759: dwrite d0/f9 [4194304,4194304] 0 2024-04-05T03:02:49.881 INFO:tasks.workunit.client.1.smithi203.stdout:1/737: write d7/d24/d33/d48/d4b/f5c [1122274,34714] 0 2024-04-05T03:02:49.882 INFO:tasks.workunit.client.1.smithi203.stdout:9/745: symlink d13/d1b/d6b/d9e/d59/lf1 0 2024-04-05T03:02:49.884 INFO:tasks.workunit.client.0.smithi153.stdout:3/428: creat d1d/d32/d93/d57/d6b/f97 x:0 0 0 2024-04-05T03:02:49.884 INFO:tasks.workunit.client.1.smithi203.stdout:6/760: symlink d0/da/d54/d4c/le5 0 2024-04-05T03:02:49.885 INFO:tasks.workunit.client.0.smithi153.stdout:2/433: read d9/da/d37/f7b [181076,106103] 0 2024-04-05T03:02:49.885 INFO:tasks.workunit.client.1.smithi203.stdout:9/746: write d13/d5c/dd1/d31/f60 [2149874,9833] 0 2024-04-05T03:02:49.886 INFO:tasks.workunit.client.1.smithi203.stdout:1/738: creat d7/d14/d2b/d58/d86/fec x:0 0 0 2024-04-05T03:02:49.887 INFO:tasks.workunit.client.0.smithi153.stdout:6/462: dwrite d4/dc/df/d16/d5f/f7c [0,4194304] 0 2024-04-05T03:02:49.888 INFO:tasks.workunit.client.0.smithi153.stdout:3/429: truncate d1d/f36 5006731 0 2024-04-05T03:02:49.890 INFO:tasks.workunit.client.0.smithi153.stdout:0/450: link d9/df/f17 d9/db/d42/d43/f85 0 2024-04-05T03:02:49.892 INFO:tasks.workunit.client.0.smithi153.stdout:7/446: truncate d2/f8e 1281229 0 2024-04-05T03:02:49.896 INFO:tasks.workunit.client.0.smithi153.stdout:6/463: write d4/dc/df/d16/d24/d4e/f5a [3597327,35444] 0 2024-04-05T03:02:49.897 INFO:tasks.workunit.client.0.smithi153.stdout:2/434: dread d9/da/d19/f46 [0,4194304] 0 2024-04-05T03:02:49.898 INFO:tasks.workunit.client.1.smithi203.stdout:8/794: dread d1/d12/f31 [0,4194304] 0 2024-04-05T03:02:49.904 INFO:tasks.workunit.client.0.smithi153.stdout:3/430: dwrite d1d/d27/f96 [0,4194304] 0 2024-04-05T03:02:49.904 INFO:tasks.workunit.client.0.smithi153.stdout:2/435: dread - d9/da/d19/d2e/d7e/d7a/f57 zero size 2024-04-05T03:02:49.905 INFO:tasks.workunit.client.0.smithi153.stdout:8/358: getdents d7/d13/d19/d31 0 2024-04-05T03:02:49.906 INFO:tasks.workunit.client.0.smithi153.stdout:8/359: chown d7/fe 16769730 1 2024-04-05T03:02:49.907 INFO:tasks.workunit.client.0.smithi153.stdout:0/451: getdents d9 0 2024-04-05T03:02:49.908 INFO:tasks.workunit.client.1.smithi203.stdout:8/795: getdents d1/d4b/d5f 0 2024-04-05T03:02:49.908 INFO:tasks.workunit.client.0.smithi153.stdout:1/381: dread d5/f7 [0,4194304] 0 2024-04-05T03:02:49.908 INFO:tasks.workunit.client.0.smithi153.stdout:3/431: chown d1d/d32/d93/d57/c3a 48969 1 2024-04-05T03:02:49.911 INFO:tasks.workunit.client.1.smithi203.stdout:8/796: mknod d1/d12/d1a/cec 0 2024-04-05T03:02:49.911 INFO:tasks.workunit.client.0.smithi153.stdout:6/464: unlink d4/dc/df/d16/d24/d4e/f99 0 2024-04-05T03:02:49.913 INFO:tasks.workunit.client.0.smithi153.stdout:1/382: creat d5/dd/d1b/f74 x:0 0 0 2024-04-05T03:02:49.914 INFO:tasks.workunit.client.0.smithi153.stdout:3/432: mknod d1d/d32/d93/c98 0 2024-04-05T03:02:49.914 INFO:tasks.workunit.client.1.smithi203.stdout:5/732: write d3/d4/dc/f54 [81846,117198] 0 2024-04-05T03:02:49.916 INFO:tasks.workunit.client.1.smithi203.stdout:5/733: chown d3/f7 4 1 2024-04-05T03:02:49.918 INFO:tasks.workunit.client.1.smithi203.stdout:5/734: dread - d3/d4/dc/d67/dac/ff1 zero size 2024-04-05T03:02:49.920 INFO:tasks.workunit.client.1.smithi203.stdout:4/567: dread da/f10 [0,4194304] 0 2024-04-05T03:02:49.922 INFO:tasks.workunit.client.1.smithi203.stdout:5/735: mkdir d3/d4/dc/d67/db5/dda/df9 0 2024-04-05T03:02:49.923 INFO:tasks.workunit.client.0.smithi153.stdout:6/465: creat d4/d38/d4c/fa0 x:0 0 0 2024-04-05T03:02:49.923 INFO:tasks.workunit.client.1.smithi203.stdout:5/736: fsync d3/d4/dc/d67/dac/fe7 0 2024-04-05T03:02:49.924 INFO:tasks.workunit.client.0.smithi153.stdout:6/466: dread - d4/d38/d4c/fa0 zero size 2024-04-05T03:02:49.925 INFO:tasks.workunit.client.0.smithi153.stdout:3/433: mkdir d1d/d32/d93/d3c/d3f/d90/d92/d99 0 2024-04-05T03:02:49.926 INFO:tasks.workunit.client.0.smithi153.stdout:8/360: dread d7/d13/d19/f4e [4194304,4194304] 0 2024-04-05T03:02:49.927 INFO:tasks.workunit.client.1.smithi203.stdout:5/737: read - d3/d4/fd1 zero size 2024-04-05T03:02:49.927 INFO:tasks.workunit.client.0.smithi153.stdout:8/361: chown d7/d13/d19/d1b/d4f/c60 6668 1 2024-04-05T03:02:49.927 INFO:tasks.workunit.client.0.smithi153.stdout:6/467: creat d4/dc/df/d16/d44/fa1 x:0 0 0 2024-04-05T03:02:49.930 INFO:tasks.workunit.client.0.smithi153.stdout:8/362: read d7/d13/d19/f4e [2705050,78359] 0 2024-04-05T03:02:49.930 INFO:tasks.workunit.client.1.smithi203.stdout:5/738: chown d3/d1f/d59/d64/d12/f96 6854 1 2024-04-05T03:02:49.930 INFO:tasks.workunit.client.1.smithi203.stdout:4/568: dwrite da/d3c/d4c/f87 [0,4194304] 0 2024-04-05T03:02:49.932 INFO:tasks.workunit.client.1.smithi203.stdout:8/797: truncate d1/d12/d17/d10/f15 2578716 0 2024-04-05T03:02:49.934 INFO:tasks.workunit.client.1.smithi203.stdout:5/739: stat d3/d4/dc/df2 0 2024-04-05T03:02:49.938 INFO:tasks.workunit.client.1.smithi203.stdout:6/761: dwrite d0/da/d54/d4c/f77 [4194304,4194304] 0 2024-04-05T03:02:49.941 INFO:tasks.workunit.client.1.smithi203.stdout:1/739: dread d7/d24/fc1 [4194304,4194304] 0 2024-04-05T03:02:49.943 INFO:tasks.workunit.client.1.smithi203.stdout:5/740: truncate d3/f36 2544989 0 2024-04-05T03:02:49.944 INFO:tasks.workunit.client.1.smithi203.stdout:5/741: chown d3/d4/dc/d67/dac/fb6 103433 1 2024-04-05T03:02:49.946 INFO:tasks.workunit.client.1.smithi203.stdout:5/742: write d3/d1f/d59/d64/f3a [1165476,60000] 0 2024-04-05T03:02:49.948 INFO:tasks.workunit.client.1.smithi203.stdout:8/798: link d1/f58 d1/d6/d2e/d5c/d99/dbc/fed 0 2024-04-05T03:02:49.949 INFO:tasks.workunit.client.0.smithi153.stdout:3/434: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/f9a x:0 0 0 2024-04-05T03:02:49.955 INFO:tasks.workunit.client.1.smithi203.stdout:5/743: rename d3/d4/dc/d67/c8a to d3/d4/dc/cfa 0 2024-04-05T03:02:49.957 INFO:tasks.workunit.client.0.smithi153.stdout:6/468: dwrite d4/d38/f75 [0,4194304] 0 2024-04-05T03:02:49.959 INFO:tasks.workunit.client.1.smithi203.stdout:5/744: symlink d3/d1f/d59/lfb 0 2024-04-05T03:02:49.961 INFO:tasks.workunit.client.1.smithi203.stdout:8/799: getdents d1/d4b/d68 0 2024-04-05T03:02:49.962 INFO:tasks.workunit.client.0.smithi153.stdout:6/469: mkdir d4/dc/df/d16/d5f/da2 0 2024-04-05T03:02:49.962 INFO:tasks.workunit.client.0.smithi153.stdout:1/383: dread d5/d8/d20/d4b/d18/d16/f36 [0,4194304] 0 2024-04-05T03:02:49.963 INFO:tasks.workunit.client.0.smithi153.stdout:8/363: link d7/d13/d19/l38 d7/d13/d19/d1b/d21/l76 0 2024-04-05T03:02:49.964 INFO:tasks.workunit.client.1.smithi203.stdout:1/740: write d7/d24/f6e [628752,128671] 0 2024-04-05T03:02:49.965 INFO:tasks.workunit.client.0.smithi153.stdout:3/435: dwrite d1d/d32/f83 [4194304,4194304] 0 2024-04-05T03:02:49.968 INFO:tasks.workunit.client.0.smithi153.stdout:1/384: chown d5/d8/d20/d4b/f11 181 1 2024-04-05T03:02:49.969 INFO:tasks.workunit.client.1.smithi203.stdout:1/741: symlink d7/d14/d83/led 0 2024-04-05T03:02:49.970 INFO:tasks.workunit.client.0.smithi153.stdout:8/364: creat d7/d2a/f77 x:0 0 0 2024-04-05T03:02:49.970 INFO:tasks.workunit.client.0.smithi153.stdout:7/447: dread d2/da/d3a/d41/d7a/d55/f61 [0,4194304] 0 2024-04-05T03:02:49.974 INFO:tasks.workunit.client.1.smithi203.stdout:1/742: mkdir d7/d14/d2b/d58/d86/d91/da6/dee 0 2024-04-05T03:02:49.974 INFO:tasks.workunit.client.0.smithi153.stdout:8/365: creat d7/d13/d19/d31/f78 x:0 0 0 2024-04-05T03:02:49.975 INFO:tasks.workunit.client.1.smithi203.stdout:1/743: stat d7/d24/d33/d48/d4b/d51/f7f 0 2024-04-05T03:02:49.977 INFO:tasks.workunit.client.0.smithi153.stdout:6/470: rename d4/dc/df/d16/d17/c9b to d4/dc/df/d16/d5f/da2/ca3 0 2024-04-05T03:02:49.978 INFO:tasks.workunit.client.0.smithi153.stdout:3/436: dwrite d1d/d32/d93/d3c/f73 [0,4194304] 0 2024-04-05T03:02:49.979 INFO:tasks.workunit.client.1.smithi203.stdout:1/744: mkdir d7/d24/d33/d48/d4b/dea/def 0 2024-04-05T03:02:49.979 INFO:tasks.workunit.client.0.smithi153.stdout:7/448: creat d2/da/d3a/d41/d31/f96 x:0 0 0 2024-04-05T03:02:49.979 INFO:tasks.workunit.client.0.smithi153.stdout:1/385: dread d5/d8/f6e [0,4194304] 0 2024-04-05T03:02:49.981 INFO:tasks.workunit.client.1.smithi203.stdout:1/745: dread - d7/d14/fdd zero size 2024-04-05T03:02:49.981 INFO:tasks.workunit.client.1.smithi203.stdout:4/569: dread da/d29/d2e/d57/f86 [0,4194304] 0 2024-04-05T03:02:49.983 INFO:tasks.workunit.client.0.smithi153.stdout:3/437: write d1d/d27/f8f [70348,81766] 0 2024-04-05T03:02:49.983 INFO:tasks.workunit.client.0.smithi153.stdout:7/449: mknod d2/da/d47/c97 0 2024-04-05T03:02:49.983 INFO:tasks.workunit.client.1.smithi203.stdout:4/570: chown da/d29/d2e/c33 30 1 2024-04-05T03:02:49.983 INFO:tasks.workunit.client.1.smithi203.stdout:8/800: dwrite d1/d4b/d68/f95 [0,4194304] 0 2024-04-05T03:02:49.984 INFO:tasks.workunit.client.1.smithi203.stdout:1/746: readlink d7/d24/d33/d48/d4b/d6b/la2 0 2024-04-05T03:02:49.991 INFO:tasks.workunit.client.0.smithi153.stdout:1/386: dwrite d5/dd/d1b/f74 [0,4194304] 0 2024-04-05T03:02:49.994 INFO:tasks.workunit.client.0.smithi153.stdout:1/387: truncate d5/d8/d20/d2f/d40/f4c 274436 0 2024-04-05T03:02:49.998 INFO:tasks.workunit.client.1.smithi203.stdout:0/721: sync 2024-04-05T03:02:50.001 INFO:tasks.workunit.client.0.smithi153.stdout:8/366: dread d7/d13/d19/d31/f41 [0,4194304] 0 2024-04-05T03:02:50.001 INFO:tasks.workunit.client.0.smithi153.stdout:1/388: dwrite d5/d8/d20/d4b/d18/d16/f2d [0,4194304] 0 2024-04-05T03:02:50.002 INFO:tasks.workunit.client.0.smithi153.stdout:3/438: mknod d1d/d32/d93/d57/d6b/c9b 0 2024-04-05T03:02:50.002 INFO:tasks.workunit.client.1.smithi203.stdout:8/801: read d1/d12/d2f/f2a [3697579,27896] 0 2024-04-05T03:02:50.004 INFO:tasks.workunit.client.1.smithi203.stdout:1/747: creat d7/d14/d2b/d58/ff0 x:0 0 0 2024-04-05T03:02:50.006 INFO:tasks.workunit.client.1.smithi203.stdout:5/745: dread d3/d4/d9e/fab [0,4194304] 0 2024-04-05T03:02:50.007 INFO:tasks.workunit.client.1.smithi203.stdout:5/746: stat d3/d4/dc/l38 0 2024-04-05T03:02:50.008 INFO:tasks.workunit.client.1.smithi203.stdout:1/748: truncate d7/f77 609765 0 2024-04-05T03:02:50.010 INFO:tasks.workunit.client.1.smithi203.stdout:0/722: mknod dd/d17/d1d/d22/d45/cd7 0 2024-04-05T03:02:50.011 INFO:tasks.workunit.client.0.smithi153.stdout:8/367: mknod d7/d13/d19/d1b/d4f/d5f/d54/c79 0 2024-04-05T03:02:50.014 INFO:tasks.workunit.client.1.smithi203.stdout:4/571: dwrite da/f27 [8388608,4194304] 0 2024-04-05T03:02:50.015 INFO:tasks.workunit.client.0.smithi153.stdout:1/389: creat d5/d8/f75 x:0 0 0 2024-04-05T03:02:50.016 INFO:tasks.workunit.client.0.smithi153.stdout:1/390: chown d5/d8/d20/d4b/d1e 2292 1 2024-04-05T03:02:50.016 INFO:tasks.workunit.client.1.smithi203.stdout:1/749: mknod d7/d24/d33/d48/d4b/d6b/cf1 0 2024-04-05T03:02:50.017 INFO:tasks.workunit.client.1.smithi203.stdout:1/750: chown d7/d14/d9d/dbf 24 1 2024-04-05T03:02:50.017 INFO:tasks.workunit.client.0.smithi153.stdout:8/368: creat d7/d13/d19/d1b/d6d/f7a x:0 0 0 2024-04-05T03:02:50.020 INFO:tasks.workunit.client.1.smithi203.stdout:0/723: rename dd/d17/d1d/d26/d34 to dd/d17/dd8 0 2024-04-05T03:02:50.021 INFO:tasks.workunit.client.0.smithi153.stdout:1/391: rename d5/d8/d20/d4b/d18/d16/f2d to d5/f76 0 2024-04-05T03:02:50.021 INFO:tasks.workunit.client.0.smithi153.stdout:7/450: dread d2/f5a [0,4194304] 0 2024-04-05T03:02:50.023 INFO:tasks.workunit.client.1.smithi203.stdout:0/724: fdatasync dd/d17/d1d/d26/d28/f3f 0 2024-04-05T03:02:50.024 INFO:tasks.workunit.client.0.smithi153.stdout:7/451: rmdir d2/da/d3a/d41/d7a/d54 39 2024-04-05T03:02:50.025 INFO:tasks.workunit.client.0.smithi153.stdout:7/452: readlink d2/d51/l70 0 2024-04-05T03:02:50.026 INFO:tasks.workunit.client.0.smithi153.stdout:3/439: dread d1d/d75/f84 [0,4194304] 0 2024-04-05T03:02:50.027 INFO:tasks.workunit.client.0.smithi153.stdout:3/440: fdatasync d1d/d32/d93/d57/f59 0 2024-04-05T03:02:50.027 INFO:tasks.workunit.client.0.smithi153.stdout:1/392: link d5/d8/d20/d4b/d18/d16/d48/f50 d5/d8/d20/d4b/d18/d16/d45/d68/f77 0 2024-04-05T03:02:50.028 INFO:tasks.workunit.client.0.smithi153.stdout:7/453: mknod d2/da/d11/d39/c98 0 2024-04-05T03:02:50.029 INFO:tasks.workunit.client.1.smithi203.stdout:1/751: dwrite d7/d14/d9d/dbf/fcd [0,4194304] 0 2024-04-05T03:02:50.030 INFO:tasks.workunit.client.0.smithi153.stdout:3/441: mkdir d1d/d32/d93/d3c/d3f/d9c 0 2024-04-05T03:02:50.032 INFO:tasks.workunit.client.0.smithi153.stdout:7/454: unlink d2/da/l23 0 2024-04-05T03:02:50.032 INFO:tasks.workunit.client.1.smithi203.stdout:0/725: dwrite dd/fd0 [0,4194304] 0 2024-04-05T03:02:50.034 INFO:tasks.workunit.client.1.smithi203.stdout:0/726: write dd/d17/d1d/d26/d8c/fc9 [662438,66596] 0 2024-04-05T03:02:50.034 INFO:tasks.workunit.client.0.smithi153.stdout:1/393: rename d5/d8/d20/d3f/c6d to d5/d8/d20/c78 0 2024-04-05T03:02:50.040 INFO:tasks.workunit.client.0.smithi153.stdout:7/455: mknod d2/da/d47/d5d/d7c/d8d/d95/c99 0 2024-04-05T03:02:50.040 INFO:tasks.workunit.client.1.smithi203.stdout:0/727: rmdir dd/d17/d1d/d22/d53/d7f/dbf 0 2024-04-05T03:02:50.041 INFO:tasks.workunit.client.1.smithi203.stdout:0/728: chown dd/c24 430 1 2024-04-05T03:02:50.041 INFO:tasks.workunit.client.0.smithi153.stdout:7/456: readlink d2/da/d11/d39/l86 0 2024-04-05T03:02:50.041 INFO:tasks.workunit.client.0.smithi153.stdout:3/442: link d1d/d27/l43 d1d/d32/d93/d3c/d3f/d90/d92/d99/l9d 0 2024-04-05T03:02:50.042 INFO:tasks.workunit.client.1.smithi203.stdout:8/802: write d1/d12/feb [16870,42457] 0 2024-04-05T03:02:50.043 INFO:tasks.workunit.client.0.smithi153.stdout:7/457: chown d2/f40 10091216 1 2024-04-05T03:02:50.044 INFO:tasks.workunit.client.0.smithi153.stdout:3/443: mknod d1d/d32/d93/d57/d6b/c9e 0 2024-04-05T03:02:50.045 INFO:tasks.workunit.client.1.smithi203.stdout:8/803: creat d1/d6/d2e/d5c/dc0/fee x:0 0 0 2024-04-05T03:02:50.045 INFO:tasks.workunit.client.0.smithi153.stdout:6/471: truncate d4/dc/df/d16/d5f/f74 3454214 0 2024-04-05T03:02:50.046 INFO:tasks.workunit.client.1.smithi203.stdout:0/729: truncate dd/d17/d1d/d26/d28/f2d 1555847 0 2024-04-05T03:02:50.046 INFO:tasks.workunit.client.0.smithi153.stdout:6/472: chown d4/d38/d4c/c81 79 1 2024-04-05T03:02:50.047 INFO:tasks.workunit.client.1.smithi203.stdout:5/747: write d3/f7 [3720769,104331] 0 2024-04-05T03:02:50.047 INFO:tasks.workunit.client.0.smithi153.stdout:3/444: mkdir d1d/d32/d93/d57/d6b/d9f 0 2024-04-05T03:02:50.047 INFO:tasks.workunit.client.0.smithi153.stdout:7/458: dread d2/da/d3a/d41/d31/f45 [0,4194304] 0 2024-04-05T03:02:50.048 INFO:tasks.workunit.client.0.smithi153.stdout:6/473: chown d4/dc/df/d16/d24/c92 3 1 2024-04-05T03:02:50.048 INFO:tasks.workunit.client.1.smithi203.stdout:8/804: symlink d1/d12/d1a/da0/lef 0 2024-04-05T03:02:50.049 INFO:tasks.workunit.client.1.smithi203.stdout:4/572: write da/d3c/f43 [5540515,77943] 0 2024-04-05T03:02:50.050 INFO:tasks.workunit.client.1.smithi203.stdout:8/805: chown d1/d6/f7c 2719720 1 2024-04-05T03:02:50.051 INFO:tasks.workunit.client.0.smithi153.stdout:6/474: rmdir d4/dc/df/d16/d5f/d73 39 2024-04-05T03:02:50.055 INFO:tasks.workunit.client.1.smithi203.stdout:0/730: dwrite dd/d17/d1d/d26/f98 [0,4194304] 0 2024-04-05T03:02:50.055 INFO:tasks.workunit.client.1.smithi203.stdout:8/806: mknod d1/d12/da8/cf0 0 2024-04-05T03:02:50.056 INFO:tasks.workunit.client.0.smithi153.stdout:3/445: dwrite d1d/d27/f6f [0,4194304] 0 2024-04-05T03:02:50.057 INFO:tasks.workunit.client.1.smithi203.stdout:0/731: write dd/d17/d1d/d22/d53/d7f/fc7 [745493,102594] 0 2024-04-05T03:02:50.062 INFO:tasks.workunit.client.0.smithi153.stdout:8/369: truncate d7/d13/d19/f22 256822 0 2024-04-05T03:02:50.063 INFO:tasks.workunit.client.1.smithi203.stdout:4/573: rename da/l24 to da/d29/lac 0 2024-04-05T03:02:50.064 INFO:tasks.workunit.client.0.smithi153.stdout:8/370: chown d7/d13/d19/f4e 1592089 1 2024-04-05T03:02:50.067 INFO:tasks.workunit.client.0.smithi153.stdout:7/459: dread d2/da/d3a/f46 [0,4194304] 0 2024-04-05T03:02:50.067 INFO:tasks.workunit.client.1.smithi203.stdout:0/732: symlink dd/d17/d1d/d22/d53/d7f/ld9 0 2024-04-05T03:02:50.067 INFO:tasks.workunit.client.1.smithi203.stdout:8/807: creat d1/d12/d17/d47/d53/de0/ff1 x:0 0 0 2024-04-05T03:02:50.067 INFO:tasks.workunit.client.0.smithi153.stdout:8/371: creat d7/d2a/f7b x:0 0 0 2024-04-05T03:02:50.069 INFO:tasks.workunit.client.0.smithi153.stdout:8/372: truncate d7/d2a/f77 30961 0 2024-04-05T03:02:50.074 INFO:tasks.workunit.client.1.smithi203.stdout:0/733: dread dd/d17/f48 [4194304,4194304] 0 2024-04-05T03:02:50.074 INFO:tasks.workunit.client.0.smithi153.stdout:7/460: rename d2/da/d11/f3f to d2/da/d3a/d41/d31/f9a 0 2024-04-05T03:02:50.076 INFO:tasks.workunit.client.1.smithi203.stdout:8/808: dwrite d1/d12/d17/d47/d53/de0/ff1 [0,4194304] 0 2024-04-05T03:02:50.077 INFO:tasks.workunit.client.0.smithi153.stdout:1/394: dwrite d5/d8/d20/f5f [4194304,4194304] 0 2024-04-05T03:02:50.077 INFO:tasks.workunit.client.0.smithi153.stdout:7/461: truncate d2/da/d3a/d41/d31/f96 953740 0 2024-04-05T03:02:50.078 INFO:tasks.workunit.client.1.smithi203.stdout:0/734: chown dd/d17/d1d/d22/d53/d61/cd6 6519 1 2024-04-05T03:02:50.078 INFO:tasks.workunit.client.1.smithi203.stdout:8/809: readlink d1/d12/d1a/l80 0 2024-04-05T03:02:50.082 INFO:tasks.workunit.client.0.smithi153.stdout:8/373: truncate d7/fd 4096539 0 2024-04-05T03:02:50.082 INFO:tasks.workunit.client.0.smithi153.stdout:6/475: truncate d4/d38/f75 1494548 0 2024-04-05T03:02:50.083 INFO:tasks.workunit.client.1.smithi203.stdout:0/735: rmdir dd/d17/d1d/d26/d8c 39 2024-04-05T03:02:50.083 INFO:tasks.workunit.client.1.smithi203.stdout:8/810: read d1/d6/d2e/f6b [230878,12881] 0 2024-04-05T03:02:50.083 INFO:tasks.workunit.client.0.smithi153.stdout:1/395: fdatasync d5/d8/d20/d4b/d18/d16/d33/f3e 0 2024-04-05T03:02:50.086 INFO:tasks.workunit.client.0.smithi153.stdout:7/462: unlink d2/da/d3a/d41/d31/f96 0 2024-04-05T03:02:50.089 INFO:tasks.workunit.client.0.smithi153.stdout:8/374: symlink d7/d13/d19/d1b/d4f/d5f/d54/l7c 0 2024-04-05T03:02:50.090 INFO:tasks.workunit.client.1.smithi203.stdout:5/748: dread d3/d1f/d59/d60/fb4 [0,4194304] 0 2024-04-05T03:02:50.091 INFO:tasks.workunit.client.1.smithi203.stdout:0/736: link dd/d17/d1d/d26/f98 dd/d17/d1d/d22/d53/db0/fda 0 2024-04-05T03:02:50.093 INFO:tasks.workunit.client.1.smithi203.stdout:5/749: mknod d3/d4/dd6/cfc 0 2024-04-05T03:02:50.093 INFO:tasks.workunit.client.0.smithi153.stdout:8/375: symlink d7/d13/d19/d1b/d4f/d5f/d54/d5b/l7d 0 2024-04-05T03:02:50.094 INFO:tasks.workunit.client.0.smithi153.stdout:8/376: dread - d7/d13/d19/d1b/d21/f69 zero size 2024-04-05T03:02:50.094 INFO:tasks.workunit.client.1.smithi203.stdout:8/811: dwrite d1/d6/f56 [0,4194304] 0 2024-04-05T03:02:50.095 INFO:tasks.workunit.client.0.smithi153.stdout:8/377: dread - d7/d2a/d50/f6b zero size 2024-04-05T03:02:50.095 INFO:tasks.workunit.client.0.smithi153.stdout:7/463: rename d2/da/d3a/d41/d7a/d55 to d2/d9b 0 2024-04-05T03:02:50.096 INFO:tasks.workunit.client.0.smithi153.stdout:8/378: stat d7/d13/d19/d1b/d4f/d5f 0 2024-04-05T03:02:50.096 INFO:tasks.workunit.client.1.smithi203.stdout:4/574: dwrite da/f56 [0,4194304] 0 2024-04-05T03:02:50.097 INFO:tasks.workunit.client.0.smithi153.stdout:1/396: getdents d5/d8/d20/d4b/d18/d16 0 2024-04-05T03:02:50.098 INFO:tasks.workunit.client.0.smithi153.stdout:8/379: stat d7/c24 0 2024-04-05T03:02:50.099 INFO:tasks.workunit.client.1.smithi203.stdout:0/737: chown dd/d17/d1d/d26/d8c/d6d/dc4 1085 1 2024-04-05T03:02:50.099 INFO:tasks.workunit.client.0.smithi153.stdout:1/397: chown d5/d8/d20/d4b/d18/d16/d2c/d62 3 1 2024-04-05T03:02:50.099 INFO:tasks.workunit.client.0.smithi153.stdout:8/380: chown d7/d13/d19/c5e 841 1 2024-04-05T03:02:50.101 INFO:tasks.workunit.client.0.smithi153.stdout:7/464: write d2/da/d47/d5d/d7c/f5b [2451760,22902] 0 2024-04-05T03:02:50.103 INFO:tasks.workunit.client.1.smithi203.stdout:0/738: dread dd/fd0 [0,4194304] 0 2024-04-05T03:02:50.103 INFO:tasks.workunit.client.1.smithi203.stdout:4/575: symlink da/d3c/lad 0 2024-04-05T03:02:50.104 INFO:tasks.workunit.client.0.smithi153.stdout:8/381: write d7/d13/d19/d31/f41 [321896,53419] 0 2024-04-05T03:02:50.104 INFO:tasks.workunit.client.1.smithi203.stdout:5/750: dwrite d3/d4/dc/d67/f70 [0,4194304] 0 2024-04-05T03:02:50.106 INFO:tasks.workunit.client.1.smithi203.stdout:5/751: dread - d3/fc7 zero size 2024-04-05T03:02:50.108 INFO:tasks.workunit.client.0.smithi153.stdout:1/398: dwrite d5/dd/d1b/f66 [0,4194304] 0 2024-04-05T03:02:50.109 INFO:tasks.workunit.client.1.smithi203.stdout:0/739: rename dd/d17/l1a to dd/db1/ldb 0 2024-04-05T03:02:50.110 INFO:tasks.workunit.client.1.smithi203.stdout:4/576: symlink da/d71/lae 0 2024-04-05T03:02:50.111 INFO:tasks.workunit.client.0.smithi153.stdout:1/399: dread d5/d8/d20/d4b/d18/d16/d33/f6b [0,4194304] 0 2024-04-05T03:02:50.113 INFO:tasks.workunit.client.0.smithi153.stdout:7/465: rename d2/da/d3a/d41/d7a/c17 to d2/d51/d93/c9c 0 2024-04-05T03:02:50.113 INFO:tasks.workunit.client.0.smithi153.stdout:7/466: stat d2/f1d 0 2024-04-05T03:02:50.114 INFO:tasks.workunit.client.0.smithi153.stdout:7/467: dread - d2/f7b zero size 2024-04-05T03:02:50.114 INFO:tasks.workunit.client.0.smithi153.stdout:8/382: mknod d7/d13/d19/d1b/c7e 0 2024-04-05T03:02:50.117 INFO:tasks.workunit.client.0.smithi153.stdout:1/400: rmdir d5/dd/d1b 39 2024-04-05T03:02:50.118 INFO:tasks.workunit.client.0.smithi153.stdout:8/383: creat d7/d13/d19/d1b/d6d/f7f x:0 0 0 2024-04-05T03:02:50.158 INFO:tasks.workunit.client.0.smithi153.stdout:7/468: truncate d2/da/d3a/d41/d7a/f56 3187988 0 2024-04-05T03:02:50.158 INFO:tasks.workunit.client.0.smithi153.stdout:8/384: dread - d7/d2a/f6e zero size 2024-04-05T03:02:50.164 INFO:tasks.workunit.client.0.smithi153.stdout:8/385: mknod d7/d13/d19/d1b/d6d/c80 0 2024-04-05T03:02:50.166 INFO:tasks.workunit.client.0.smithi153.stdout:7/469: mkdir d2/da/d11/d9d 0 2024-04-05T03:02:50.168 INFO:tasks.workunit.client.0.smithi153.stdout:1/401: creat d5/d8/d20/d4b/f79 x:0 0 0 2024-04-05T03:02:50.168 INFO:tasks.workunit.client.0.smithi153.stdout:8/386: mknod d7/d13/d19/d1b/d6d/c81 0 2024-04-05T03:02:50.169 INFO:tasks.workunit.client.0.smithi153.stdout:8/387: fdatasync f5 0 2024-04-05T03:02:50.170 INFO:tasks.workunit.client.0.smithi153.stdout:8/388: write f3 [1935761,32226] 0 2024-04-05T03:02:50.177 INFO:tasks.workunit.client.1.smithi203.stdout:8/812: dwrite d1/d12/d1a/f78 [0,4194304] 0 2024-04-05T03:02:50.183 INFO:tasks.workunit.client.0.smithi153.stdout:1/402: creat d5/d8/d20/d4b/d18/d16/f7a x:0 0 0 2024-04-05T03:02:50.184 INFO:tasks.workunit.client.0.smithi153.stdout:1/403: stat d5/d8/d20/d4b/d18/d16/d45/d68/f77 0 2024-04-05T03:02:50.186 INFO:tasks.workunit.client.1.smithi203.stdout:5/752: dwrite d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:50.187 INFO:tasks.workunit.client.1.smithi203.stdout:0/740: dwrite dd/d17/d1d/d26/f4f [0,4194304] 0 2024-04-05T03:02:50.187 INFO:tasks.workunit.client.0.smithi153.stdout:8/389: rmdir d7/d2a 39 2024-04-05T03:02:50.188 INFO:tasks.workunit.client.1.smithi203.stdout:8/813: chown d1/d4b/d68/la6 70093805 1 2024-04-05T03:02:50.189 INFO:tasks.workunit.client.1.smithi203.stdout:0/741: fdatasync dd/d17/d1d/d22/d53/d55/fa1 0 2024-04-05T03:02:50.189 INFO:tasks.workunit.client.1.smithi203.stdout:8/814: dread - d1/d6/d2e/fea zero size 2024-04-05T03:02:50.191 INFO:tasks.workunit.client.1.smithi203.stdout:8/815: write d1/d12/d17/d47/d53/f82 [2820238,83387] 0 2024-04-05T03:02:50.193 INFO:tasks.workunit.client.1.smithi203.stdout:5/753: fsync d3/f82 0 2024-04-05T03:02:50.194 INFO:tasks.workunit.client.1.smithi203.stdout:8/816: readlink d1/d12/d2f/d55/l71 0 2024-04-05T03:02:50.195 INFO:tasks.workunit.client.1.smithi203.stdout:0/742: write dd/d17/d1d/d26/fc0 [4730144,124989] 0 2024-04-05T03:02:50.197 INFO:tasks.workunit.client.1.smithi203.stdout:0/743: write dd/d17/d1d/d22/d53/d61/f9d [726551,92919] 0 2024-04-05T03:02:50.197 INFO:tasks.workunit.client.1.smithi203.stdout:0/744: chown ca 861 1 2024-04-05T03:02:50.199 INFO:tasks.workunit.client.0.smithi153.stdout:1/404: creat d5/d8/d20/d3f/f7b x:0 0 0 2024-04-05T03:02:50.201 INFO:tasks.workunit.client.0.smithi153.stdout:7/470: getdents d2/da/d3a/d41/d7a 0 2024-04-05T03:02:50.203 INFO:tasks.workunit.client.0.smithi153.stdout:1/405: chown d5/d8/d20/d4b/d51/c6c 398328 1 2024-04-05T03:02:50.204 INFO:tasks.workunit.client.1.smithi203.stdout:8/817: rename d1/d6/c5a to d1/d6/d2e/d5c/d99/dbc/cf2 0 2024-04-05T03:02:50.205 INFO:tasks.workunit.client.1.smithi203.stdout:8/818: chown d1/d12/cb7 47 1 2024-04-05T03:02:50.206 INFO:tasks.workunit.client.1.smithi203.stdout:8/819: chown d1/d6/d2e/d5c/lb9 423267896 1 2024-04-05T03:02:50.206 INFO:tasks.workunit.client.0.smithi153.stdout:7/471: rename d2/f7b to d2/da/d47/f9e 0 2024-04-05T03:02:50.207 INFO:tasks.workunit.client.0.smithi153.stdout:8/390: dwrite d7/d13/d19/d1b/f35 [0,4194304] 0 2024-04-05T03:02:50.207 INFO:tasks.workunit.client.1.smithi203.stdout:0/745: symlink dd/d17/d1d/d22/d80/ldc 0 2024-04-05T03:02:50.209 INFO:tasks.workunit.client.1.smithi203.stdout:0/746: stat dd/d17/f79 0 2024-04-05T03:02:50.209 INFO:tasks.workunit.client.1.smithi203.stdout:8/820: read d1/d12/d17/d10/f15 [611753,51189] 0 2024-04-05T03:02:50.210 INFO:tasks.workunit.client.0.smithi153.stdout:7/472: write d2/da/d3a/d41/d31/f9a [446711,31805] 0 2024-04-05T03:02:50.211 INFO:tasks.workunit.client.1.smithi203.stdout:0/747: write dd/d17/dd8/f47 [6793272,81471] 0 2024-04-05T03:02:50.211 INFO:tasks.workunit.client.0.smithi153.stdout:8/391: symlink d7/d13/d19/d1b/d4f/d72/d73/l82 0 2024-04-05T03:02:50.211 INFO:tasks.workunit.client.1.smithi203.stdout:0/748: chown dd/d17/d1d/d22/d53/f63 1739581 1 2024-04-05T03:02:50.214 INFO:tasks.workunit.client.0.smithi153.stdout:7/473: dread - d2/da/d3a/d41/d31/f4d zero size 2024-04-05T03:02:50.214 INFO:tasks.workunit.client.0.smithi153.stdout:8/392: dread - d7/d2a/f6e zero size 2024-04-05T03:02:50.216 INFO:tasks.workunit.client.0.smithi153.stdout:8/393: fdatasync d7/d2a/d50/f6b 0 2024-04-05T03:02:50.217 INFO:tasks.workunit.client.1.smithi203.stdout:8/821: link d1/d90/ce4 d1/d6/d8f/cf3 0 2024-04-05T03:02:50.217 INFO:tasks.workunit.client.1.smithi203.stdout:5/754: fsync d3/d1f/d59/d64/fdd 0 2024-04-05T03:02:50.218 INFO:tasks.workunit.client.1.smithi203.stdout:8/822: chown d1/d4b/d68/f95 0 1 2024-04-05T03:02:50.219 INFO:tasks.workunit.client.0.smithi153.stdout:1/406: dwrite d5/d8/d20/d4b/d1e/f4f [0,4194304] 0 2024-04-05T03:02:50.219 INFO:tasks.workunit.client.0.smithi153.stdout:7/474: mknod d2/da/d11/d39/c9f 0 2024-04-05T03:02:50.225 INFO:tasks.workunit.client.1.smithi203.stdout:5/755: dread d3/d4/f87 [0,4194304] 0 2024-04-05T03:02:50.225 INFO:tasks.workunit.client.0.smithi153.stdout:7/475: dread d2/da/d3a/d41/d31/f9a [0,4194304] 0 2024-04-05T03:02:50.227 INFO:tasks.workunit.client.0.smithi153.stdout:8/394: dwrite d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:02:50.227 INFO:tasks.workunit.client.0.smithi153.stdout:1/407: mknod d5/d8/d20/d2f/c7c 0 2024-04-05T03:02:50.228 INFO:tasks.workunit.client.0.smithi153.stdout:7/476: chown d2/d51/d59/c58 1651 1 2024-04-05T03:02:50.235 INFO:tasks.workunit.client.1.smithi203.stdout:8/823: truncate d1/d12/d2f/d5b/fb4 4070666 0 2024-04-05T03:02:50.236 INFO:tasks.workunit.client.0.smithi153.stdout:1/408: rename d5/dd/d1b/f2b to d5/d8/d20/d4b/d18/d16/d2c/f7d 0 2024-04-05T03:02:50.238 INFO:tasks.workunit.client.1.smithi203.stdout:8/824: symlink d1/d12/d17/d47/d53/dd0/lf4 0 2024-04-05T03:02:50.238 INFO:tasks.workunit.client.1.smithi203.stdout:8/825: chown d1/d12/d17/d47/d53/f82 184281 1 2024-04-05T03:02:50.239 INFO:tasks.workunit.client.0.smithi153.stdout:8/395: link d7/d13/d19/c67 d7/d46/c83 0 2024-04-05T03:02:50.239 INFO:tasks.workunit.client.1.smithi203.stdout:8/826: read - d1/d12/d1a/fdd zero size 2024-04-05T03:02:50.240 INFO:tasks.workunit.client.0.smithi153.stdout:8/396: chown d7/d13/d19/d31/f6a 511949352 1 2024-04-05T03:02:50.240 INFO:tasks.workunit.client.0.smithi153.stdout:1/409: mknod d5/d8/d20/d4b/d18/d61/c7e 0 2024-04-05T03:02:50.245 INFO:tasks.workunit.client.1.smithi203.stdout:0/749: dread dd/d17/d1d/d22/fac [0,4194304] 0 2024-04-05T03:02:50.246 INFO:tasks.workunit.client.0.smithi153.stdout:8/397: rename f4 to d7/d13/d19/d1b/d4f/d68/f84 0 2024-04-05T03:02:50.248 INFO:tasks.workunit.client.1.smithi203.stdout:0/750: rmdir dd/d17/d1d/d22/d53/d55 39 2024-04-05T03:02:50.249 INFO:tasks.workunit.client.0.smithi153.stdout:1/410: dwrite d5/d8/d20/d2f/d40/f4c [0,4194304] 0 2024-04-05T03:02:50.249 INFO:tasks.workunit.client.1.smithi203.stdout:0/751: write dd/d17/d1d/d22/dc1/fd3 [1617761,109320] 0 2024-04-05T03:02:50.252 INFO:tasks.workunit.client.1.smithi203.stdout:8/827: write d1/d12/d17/d10/f6e [881503,15561] 0 2024-04-05T03:02:50.254 INFO:tasks.workunit.client.1.smithi203.stdout:0/752: dread dd/d17/d1d/d22/fac [0,4194304] 0 2024-04-05T03:02:50.255 INFO:tasks.workunit.client.0.smithi153.stdout:7/477: dread d2/da/d3a/d41/d7a/f56 [0,4194304] 0 2024-04-05T03:02:50.257 INFO:tasks.workunit.client.0.smithi153.stdout:8/398: write d7/d2a/f2c [3699413,5424] 0 2024-04-05T03:02:50.258 INFO:tasks.workunit.client.1.smithi203.stdout:0/753: mknod dd/d17/d1d/d26/d8c/d6d/dc4/cdd 0 2024-04-05T03:02:50.259 INFO:tasks.workunit.client.0.smithi153.stdout:8/399: truncate d7/d13/d19/d1b/d6d/f7f 775997 0 2024-04-05T03:02:50.266 INFO:tasks.workunit.client.0.smithi153.stdout:8/400: dwrite d7/f16 [0,4194304] 0 2024-04-05T03:02:50.271 INFO:tasks.workunit.client.0.smithi153.stdout:1/411: dread d5/d8/d20/d4b/d18/d16/d33/f3c [0,4194304] 0 2024-04-05T03:02:50.271 INFO:tasks.workunit.client.0.smithi153.stdout:8/401: rename d7/d13/d19/d31 to d7/d85 0 2024-04-05T03:02:50.273 INFO:tasks.workunit.client.0.smithi153.stdout:8/402: fdatasync d7/d13/d19/f4e 0 2024-04-05T03:02:50.274 INFO:tasks.workunit.client.0.smithi153.stdout:8/403: write d7/d13/d19/d1b/d33/f59 [275367,9494] 0 2024-04-05T03:02:50.275 INFO:tasks.workunit.client.0.smithi153.stdout:1/412: creat d5/d8/d20/d4b/d51/f7f x:0 0 0 2024-04-05T03:02:50.276 INFO:tasks.workunit.client.1.smithi203.stdout:8/828: dread d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:50.277 INFO:tasks.workunit.client.0.smithi153.stdout:1/413: truncate d5/dd/d1b/f56 551661 0 2024-04-05T03:02:50.277 INFO:tasks.workunit.client.1.smithi203.stdout:0/754: dwrite dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:50.278 INFO:tasks.workunit.client.1.smithi203.stdout:8/829: write d1/d6/f45 [312779,92937] 0 2024-04-05T03:02:50.281 INFO:tasks.workunit.client.1.smithi203.stdout:0/755: unlink dd/d17/d1d/d22/d80/ldc 0 2024-04-05T03:02:50.288 INFO:tasks.workunit.client.1.smithi203.stdout:8/830: dwrite d1/fdb [0,4194304] 0 2024-04-05T03:02:50.289 INFO:tasks.workunit.client.1.smithi203.stdout:8/831: write d1/d4b/fb2 [279613,8587] 0 2024-04-05T03:02:50.294 INFO:tasks.workunit.client.1.smithi203.stdout:8/832: mkdir d1/d6/d2e/d5c/df5 0 2024-04-05T03:02:50.295 INFO:tasks.workunit.client.0.smithi153.stdout:5/437: sync 2024-04-05T03:02:50.295 INFO:tasks.workunit.client.0.smithi153.stdout:2/436: sync 2024-04-05T03:02:50.295 INFO:tasks.workunit.client.0.smithi153.stdout:4/532: sync 2024-04-05T03:02:50.295 INFO:tasks.workunit.client.0.smithi153.stdout:9/337: sync 2024-04-05T03:02:50.297 INFO:tasks.workunit.client.0.smithi153.stdout:5/438: write d0/d25/d4d/d62/d37/f4f [4581546,105104] 0 2024-04-05T03:02:50.297 INFO:tasks.workunit.client.0.smithi153.stdout:9/338: truncate d1/d9/f5e 863971 0 2024-04-05T03:02:50.298 INFO:tasks.workunit.client.0.smithi153.stdout:2/437: creat d9/d77/f90 x:0 0 0 2024-04-05T03:02:50.299 INFO:tasks.workunit.client.1.smithi203.stdout:8/833: rename d1/d4b/d5f/c86 to d1/d90/dcd/cf6 0 2024-04-05T03:02:50.300 INFO:tasks.workunit.client.0.smithi153.stdout:4/533: read - da/de/d56/d93/f9b zero size 2024-04-05T03:02:50.300 INFO:tasks.workunit.client.0.smithi153.stdout:2/438: chown d9/da/d19/d2e/d7e 0 1 2024-04-05T03:02:50.301 INFO:tasks.workunit.client.0.smithi153.stdout:4/534: fdatasync da/db/d20/d35/d38/f59 0 2024-04-05T03:02:50.302 INFO:tasks.workunit.client.0.smithi153.stdout:5/439: creat d0/d25/d68/fa0 x:0 0 0 2024-04-05T03:02:50.303 INFO:tasks.workunit.client.0.smithi153.stdout:8/404: write d7/f28 [4654440,123732] 0 2024-04-05T03:02:50.303 INFO:tasks.workunit.client.0.smithi153.stdout:9/339: creat d1/d9/d1b/f7d x:0 0 0 2024-04-05T03:02:50.304 INFO:tasks.workunit.client.0.smithi153.stdout:5/440: read d0/d25/d4d/f66 [2463261,4650] 0 2024-04-05T03:02:50.305 INFO:tasks.workunit.client.1.smithi203.stdout:8/834: rename d1/d12/ca3 to d1/d12/d17/cf7 0 2024-04-05T03:02:50.306 INFO:tasks.workunit.client.1.smithi203.stdout:8/835: chown d1/d6/d2e/d5c/c9b 19385 1 2024-04-05T03:02:50.307 INFO:tasks.workunit.client.1.smithi203.stdout:0/756: dread dd/d17/d1d/d22/d80/f4b [0,4194304] 0 2024-04-05T03:02:50.308 INFO:tasks.workunit.client.0.smithi153.stdout:9/340: rename d1/d10/c41 to d1/d10/d16/d22/d2c/d68/c7e 0 2024-04-05T03:02:50.309 INFO:tasks.workunit.client.0.smithi153.stdout:5/441: fsync d0/d25/d4d/f66 0 2024-04-05T03:02:50.310 INFO:tasks.workunit.client.1.smithi203.stdout:8/836: dread d1/d6/d8f/fb0 [0,4194304] 0 2024-04-05T03:02:50.311 INFO:tasks.workunit.client.1.smithi203.stdout:8/837: chown d1/d12/d2f/d1f 1 1 2024-04-05T03:02:50.312 INFO:tasks.workunit.client.1.smithi203.stdout:8/838: mkdir d1/d4b/d5f/d74/df8 0 2024-04-05T03:02:50.315 INFO:tasks.workunit.client.1.smithi203.stdout:8/839: fsync d1/d6/d2e/d5c/d99/dbc/fed 0 2024-04-05T03:02:50.316 INFO:tasks.workunit.client.1.smithi203.stdout:8/840: write d1/d4b/d5f/d74/fb5 [3355478,62322] 0 2024-04-05T03:02:50.317 INFO:tasks.workunit.client.1.smithi203.stdout:8/841: dread - d1/d4b/d68/fcb zero size 2024-04-05T03:02:50.326 INFO:tasks.workunit.client.1.smithi203.stdout:8/842: dwrite d1/d6/d2e/f6b [0,4194304] 0 2024-04-05T03:02:50.328 INFO:tasks.workunit.client.1.smithi203.stdout:8/843: truncate d1/d6/f93 1030293 0 2024-04-05T03:02:50.332 INFO:tasks.workunit.client.1.smithi203.stdout:0/757: dread f5 [0,4194304] 0 2024-04-05T03:02:50.332 INFO:tasks.workunit.client.1.smithi203.stdout:8/844: rename d1/d4b/d68/f95 to d1/d4b/d5f/d74/ff9 0 2024-04-05T03:02:50.336 INFO:tasks.workunit.client.1.smithi203.stdout:8/845: creat d1/d4b/d5f/ffa x:0 0 0 2024-04-05T03:02:50.347 INFO:tasks.workunit.client.1.smithi203.stdout:8/846: dread d1/f58 [0,4194304] 0 2024-04-05T03:02:50.348 INFO:tasks.workunit.client.1.smithi203.stdout:0/758: dread dd/d17/d1d/d26/d28/f2d [0,4194304] 0 2024-04-05T03:02:50.350 INFO:tasks.workunit.client.1.smithi203.stdout:2/879: sync 2024-04-05T03:02:50.351 INFO:tasks.workunit.client.1.smithi203.stdout:0/759: rmdir dd/d17/d1d/d22/d53/d55 39 2024-04-05T03:02:50.351 INFO:tasks.workunit.client.1.smithi203.stdout:8/847: dwrite d1/d6/f7c [0,4194304] 0 2024-04-05T03:02:50.356 INFO:tasks.workunit.client.1.smithi203.stdout:0/760: write dd/d17/d1d/d22/dc1/fd3 [2914446,27525] 0 2024-04-05T03:02:50.357 INFO:tasks.workunit.client.1.smithi203.stdout:8/848: unlink d1/d6/d2e/d5c/dc0/fee 0 2024-04-05T03:02:50.363 INFO:tasks.workunit.client.1.smithi203.stdout:2/880: link d5/d10/d2f/f88 d5/d10/db8/dab/de1/f12f 0 2024-04-05T03:02:50.370 INFO:tasks.workunit.client.1.smithi203.stdout:0/761: dread dd/d17/d1d/d26/d28/f2b [0,4194304] 0 2024-04-05T03:02:50.372 INFO:tasks.workunit.client.1.smithi203.stdout:0/762: symlink dd/d17/d1d/d22/d53/d61/lde 0 2024-04-05T03:02:50.373 INFO:tasks.workunit.client.1.smithi203.stdout:0/763: stat dd/d17/d1d/d22/d53/d61/f9d 0 2024-04-05T03:02:50.375 INFO:tasks.workunit.client.1.smithi203.stdout:0/764: read dd/d17/d1d/d26/fc0 [1102066,103653] 0 2024-04-05T03:02:50.377 INFO:tasks.workunit.client.1.smithi203.stdout:0/765: truncate dd/d17/d1d/d22/d80/f72 687124 0 2024-04-05T03:02:50.379 INFO:tasks.workunit.client.1.smithi203.stdout:0/766: symlink dd/db7/ldf 0 2024-04-05T03:02:50.379 INFO:tasks.workunit.client.1.smithi203.stdout:8/849: dread d1/d4b/d68/d75/d77/f87 [0,4194304] 0 2024-04-05T03:02:50.380 INFO:tasks.workunit.client.1.smithi203.stdout:0/767: chown dd/d17/d1d/d95 38 1 2024-04-05T03:02:50.382 INFO:tasks.workunit.client.1.smithi203.stdout:8/850: mkdir d1/d6/d2e/d5c/dc0/dfb 0 2024-04-05T03:02:50.388 INFO:tasks.workunit.client.1.smithi203.stdout:8/851: dwrite d1/d4b/d5f/ffa [0,4194304] 0 2024-04-05T03:02:50.389 INFO:tasks.workunit.client.1.smithi203.stdout:0/768: dwrite f6 [4194304,4194304] 0 2024-04-05T03:02:50.392 INFO:tasks.workunit.client.1.smithi203.stdout:8/852: creat d1/d6/d2e/d5c/dc0/ffc x:0 0 0 2024-04-05T03:02:50.394 INFO:tasks.workunit.client.1.smithi203.stdout:8/853: mknod d1/d6/d2e/d5c/cfd 0 2024-04-05T03:02:50.396 INFO:tasks.workunit.client.1.smithi203.stdout:0/769: rename dd/d17/d1d/d22/dba to dd/d17/d1d/d22/d53/d55/de0 0 2024-04-05T03:02:50.401 INFO:tasks.workunit.client.1.smithi203.stdout:8/854: dwrite d1/d12/d1a/fdd [0,4194304] 0 2024-04-05T03:02:50.403 INFO:tasks.workunit.client.1.smithi203.stdout:0/770: write dd/d17/d1d/d22/d45/d51/f86 [420745,46597] 0 2024-04-05T03:02:50.405 INFO:tasks.workunit.client.1.smithi203.stdout:8/855: fsync d1/d12/d2f/d55/fd3 0 2024-04-05T03:02:50.407 INFO:tasks.workunit.client.1.smithi203.stdout:0/771: fdatasync dd/d17/d1d/d26/f98 0 2024-04-05T03:02:50.408 INFO:tasks.workunit.client.1.smithi203.stdout:8/856: rmdir d1/d90/da2 39 2024-04-05T03:02:50.409 INFO:tasks.workunit.client.1.smithi203.stdout:0/772: dread - dd/d17/d1d/d22/d53/d7f/f83 zero size 2024-04-05T03:02:50.411 INFO:tasks.workunit.client.1.smithi203.stdout:0/773: chown dd/d17/d1d/d26/d28/c7b 516312 1 2024-04-05T03:02:50.412 INFO:tasks.workunit.client.1.smithi203.stdout:8/857: creat d1/d12/d17/d47/d53/d69/d9c/ffe x:0 0 0 2024-04-05T03:02:50.413 INFO:tasks.workunit.client.1.smithi203.stdout:0/774: unlink dd/d17/d1d/d22/d53/d7f/f83 0 2024-04-05T03:02:50.418 INFO:tasks.workunit.client.1.smithi203.stdout:0/775: dwrite dd/d17/dd8/f35 [4194304,4194304] 0 2024-04-05T03:02:50.420 INFO:tasks.workunit.client.1.smithi203.stdout:0/776: write dd/d17/d1d/d22/d45/d51/d69/f7e [1845856,23115] 0 2024-04-05T03:02:50.428 INFO:tasks.workunit.client.1.smithi203.stdout:8/858: write d1/d12/d1a/f1d [731065,18438] 0 2024-04-05T03:02:50.429 INFO:tasks.workunit.client.0.smithi153.stdout:8/405: link d7/d13/d19/d1b/d33/f3a d7/d13/d19/d1b/d4f/f86 0 2024-04-05T03:02:50.429 INFO:tasks.workunit.client.0.smithi153.stdout:5/442: fdatasync d0/d25/d4d/d62/f9d 0 2024-04-05T03:02:50.430 INFO:tasks.workunit.client.0.smithi153.stdout:9/341: truncate d1/d10/d16/f23 237722 0 2024-04-05T03:02:50.431 INFO:tasks.workunit.client.0.smithi153.stdout:5/443: readlink d0/l7 0 2024-04-05T03:02:50.432 INFO:tasks.workunit.client.1.smithi203.stdout:0/777: dwrite dd/d17/f48 [0,4194304] 0 2024-04-05T03:02:50.433 INFO:tasks.workunit.client.0.smithi153.stdout:9/342: creat d1/d10/d16/f7f x:0 0 0 2024-04-05T03:02:50.434 INFO:tasks.workunit.client.1.smithi203.stdout:8/859: rename d1/d4b/d5f/c7e to d1/d90/cff 0 2024-04-05T03:02:50.435 INFO:tasks.workunit.client.1.smithi203.stdout:8/860: chown d1/d12/d2f/l1b 0 1 2024-04-05T03:02:50.436 INFO:tasks.workunit.client.0.smithi153.stdout:5/444: creat d0/d94/fa1 x:0 0 0 2024-04-05T03:02:50.436 INFO:tasks.workunit.client.0.smithi153.stdout:8/406: dwrite d7/d13/d19/d1b/d21/f69 [0,4194304] 0 2024-04-05T03:02:50.437 INFO:tasks.workunit.client.1.smithi203.stdout:0/778: mkdir dd/d17/d1d/d22/d45/dbe/de1 0 2024-04-05T03:02:50.440 INFO:tasks.workunit.client.1.smithi203.stdout:8/861: symlink d1/d12/d2f/da9/dce/l100 0 2024-04-05T03:02:50.442 INFO:tasks.workunit.client.0.smithi153.stdout:8/407: read d7/d85/f4a [1674560,128659] 0 2024-04-05T03:02:50.445 INFO:tasks.workunit.client.1.smithi203.stdout:0/779: link dd/d17/d1d/d26/f67 dd/d17/d1d/d26/d8c/d6d/dc4/fe2 0 2024-04-05T03:02:50.446 INFO:tasks.workunit.client.0.smithi153.stdout:9/343: dwrite d1/d1d/f5a [0,4194304] 0 2024-04-05T03:02:50.448 INFO:tasks.workunit.client.1.smithi203.stdout:8/862: rename d1/d12/d17/d47/d53/d69/l8c to d1/d12/d17/d47/d53/de0/l101 0 2024-04-05T03:02:50.449 INFO:tasks.workunit.client.0.smithi153.stdout:5/445: rename d0/d25/d4d/d62/c91 to d0/d9/ca2 0 2024-04-05T03:02:50.450 INFO:tasks.workunit.client.1.smithi203.stdout:8/863: write d1/d4b/fb2 [616131,42149] 0 2024-04-05T03:02:50.450 INFO:tasks.workunit.client.1.smithi203.stdout:8/864: stat d1/d4b/d5f/fe9 0 2024-04-05T03:02:50.451 INFO:tasks.workunit.client.1.smithi203.stdout:0/780: creat dd/d17/d1d/d22/d53/d7f/fe3 x:0 0 0 2024-04-05T03:02:50.453 INFO:tasks.workunit.client.1.smithi203.stdout:0/781: fsync dd/d17/d1d/d22/dc1/fd3 0 2024-04-05T03:02:50.453 INFO:tasks.workunit.client.0.smithi153.stdout:2/439: dread d9/da/d19/d2e/d48/f1a [0,4194304] 0 2024-04-05T03:02:50.454 INFO:tasks.workunit.client.0.smithi153.stdout:9/344: mknod d1/d10/d16/d22/d2c/c80 0 2024-04-05T03:02:50.455 INFO:tasks.workunit.client.1.smithi203.stdout:8/865: mknod d1/d12/d17/db8/c102 0 2024-04-05T03:02:50.456 INFO:tasks.workunit.client.1.smithi203.stdout:0/782: rmdir dd/db1 39 2024-04-05T03:02:50.456 INFO:tasks.workunit.client.0.smithi153.stdout:9/345: stat d1/d10/d16/d22/d2c/f46 0 2024-04-05T03:02:50.458 INFO:tasks.workunit.client.0.smithi153.stdout:2/440: rename d9/da/d37/c4a to d9/da/d37/c91 0 2024-04-05T03:02:50.459 INFO:tasks.workunit.client.0.smithi153.stdout:5/446: dwrite d0/f3c [0,4194304] 0 2024-04-05T03:02:50.462 INFO:tasks.workunit.client.0.smithi153.stdout:9/346: rename d1/db to d1/d10/d16/d22/d2c/d68/d81 0 2024-04-05T03:02:50.464 INFO:tasks.workunit.client.0.smithi153.stdout:5/447: symlink d0/d25/d4d/d62/d5f/la3 0 2024-04-05T03:02:50.465 INFO:tasks.workunit.client.0.smithi153.stdout:6/476: sync 2024-04-05T03:02:50.465 INFO:tasks.workunit.client.0.smithi153.stdout:3/446: sync 2024-04-05T03:02:50.465 INFO:tasks.workunit.client.0.smithi153.stdout:0/452: sync 2024-04-05T03:02:50.466 INFO:tasks.workunit.client.0.smithi153.stdout:5/448: stat d0/d2c/d5e 0 2024-04-05T03:02:50.470 INFO:tasks.workunit.client.0.smithi153.stdout:9/347: link d1/d10/d16/d22/d2c/d68/f7c d1/d6c/f82 0 2024-04-05T03:02:50.471 INFO:tasks.workunit.client.0.smithi153.stdout:3/447: write d1d/d32/d93/d3c/f7b [1642648,73059] 0 2024-04-05T03:02:50.473 INFO:tasks.workunit.client.0.smithi153.stdout:2/441: dwrite d9/da/d19/d2e/d48/f33 [4194304,4194304] 0 2024-04-05T03:02:50.473 INFO:tasks.workunit.client.0.smithi153.stdout:6/477: unlink d4/dc/df/d16/d5f/l7f 0 2024-04-05T03:02:50.475 INFO:tasks.workunit.client.0.smithi153.stdout:6/478: chown d4/d38/f93 18604 1 2024-04-05T03:02:50.477 INFO:tasks.workunit.client.0.smithi153.stdout:9/348: creat d1/d3a/f83 x:0 0 0 2024-04-05T03:02:50.479 INFO:tasks.workunit.client.0.smithi153.stdout:3/448: creat d1d/d32/d93/d57/fa0 x:0 0 0 2024-04-05T03:02:50.481 INFO:tasks.workunit.client.0.smithi153.stdout:2/442: chown d9/da/d19/d2e/d7e/d7a/d59/f8d 166986 1 2024-04-05T03:02:50.481 INFO:tasks.workunit.client.0.smithi153.stdout:8/408: dread d7/d13/d19/f4e [0,4194304] 0 2024-04-05T03:02:50.481 INFO:tasks.workunit.client.0.smithi153.stdout:2/443: readlink d9/da/d19/l40 0 2024-04-05T03:02:50.483 INFO:tasks.workunit.client.0.smithi153.stdout:8/409: write d7/d2a/f77 [352774,103212] 0 2024-04-05T03:02:50.484 INFO:tasks.workunit.client.0.smithi153.stdout:8/410: write d7/d2a/d50/f6b [427212,59576] 0 2024-04-05T03:02:50.487 INFO:tasks.workunit.client.0.smithi153.stdout:3/449: creat d1d/d27/fa1 x:0 0 0 2024-04-05T03:02:50.488 INFO:tasks.workunit.client.0.smithi153.stdout:5/449: dread d0/d9/f18 [0,4194304] 0 2024-04-05T03:02:50.494 INFO:tasks.workunit.client.0.smithi153.stdout:8/411: creat d7/d13/d19/d1b/d4f/d5f/f87 x:0 0 0 2024-04-05T03:02:50.496 INFO:tasks.workunit.client.0.smithi153.stdout:0/453: truncate d9/db/f4b 1493958 0 2024-04-05T03:02:50.499 INFO:tasks.workunit.client.0.smithi153.stdout:5/450: rename d0/d94 to d0/d25/d4d/d62/d5f/d86/da4 0 2024-04-05T03:02:50.501 INFO:tasks.workunit.client.0.smithi153.stdout:2/444: truncate d9/da/d37/f3d 1861512 0 2024-04-05T03:02:50.501 INFO:tasks.workunit.client.0.smithi153.stdout:9/349: link d1/d10/d16/d22/c29 d1/d10/d16/d22/d2c/d68/c84 0 2024-04-05T03:02:50.501 INFO:tasks.workunit.client.0.smithi153.stdout:3/450: dread d1d/d32/d93/f4c [0,4194304] 0 2024-04-05T03:02:50.501 INFO:tasks.workunit.client.0.smithi153.stdout:6/479: truncate d4/dc/f3c 1083913 0 2024-04-05T03:02:50.503 INFO:tasks.workunit.client.0.smithi153.stdout:8/412: rename d7/d13/d19/d1b/d4f/d5f/d54/c71 to d7/d2a/d50/c88 0 2024-04-05T03:02:50.504 INFO:tasks.workunit.client.0.smithi153.stdout:5/451: creat d0/d4e/fa5 x:0 0 0 2024-04-05T03:02:50.505 INFO:tasks.workunit.client.1.smithi203.stdout:3/635: sync 2024-04-05T03:02:50.508 INFO:tasks.workunit.client.0.smithi153.stdout:0/454: dread f3 [0,4194304] 0 2024-04-05T03:02:50.509 INFO:tasks.workunit.client.1.smithi203.stdout:0/783: fsync dd/d17/d1d/d22/d53/d61/f6a 0 2024-04-05T03:02:50.509 INFO:tasks.workunit.client.1.smithi203.stdout:8/866: dwrite d1/d12/f6c [0,4194304] 0 2024-04-05T03:02:50.510 INFO:tasks.workunit.client.0.smithi153.stdout:2/445: dread d9/da/d19/f46 [0,4194304] 0 2024-04-05T03:02:50.511 INFO:tasks.workunit.client.1.smithi203.stdout:8/867: fsync d1/d4b/d68/d75/dac/da4/fbd 0 2024-04-05T03:02:50.511 INFO:tasks.workunit.client.1.smithi203.stdout:3/636: mknod d0/dc/d1a/d1f/da2/cb8 0 2024-04-05T03:02:50.511 INFO:tasks.workunit.client.0.smithi153.stdout:9/350: dwrite d1/d9/f5e [0,4194304] 0 2024-04-05T03:02:50.513 INFO:tasks.workunit.client.0.smithi153.stdout:2/446: chown d9/d77/c41 5119493 1 2024-04-05T03:02:50.513 INFO:tasks.workunit.client.1.smithi203.stdout:8/868: dread d1/d12/feb [0,4194304] 0 2024-04-05T03:02:50.520 INFO:tasks.workunit.client.0.smithi153.stdout:9/351: write d1/d9/d1b/f5d [4350745,28578] 0 2024-04-05T03:02:50.532 INFO:tasks.workunit.client.0.smithi153.stdout:8/413: creat d7/d13/d19/d1b/f89 x:0 0 0 2024-04-05T03:02:50.537 INFO:tasks.workunit.client.1.smithi203.stdout:0/784: dwrite f7 [0,4194304] 0 2024-04-05T03:02:50.538 INFO:tasks.workunit.client.0.smithi153.stdout:7/478: sync 2024-04-05T03:02:50.538 INFO:tasks.workunit.client.0.smithi153.stdout:3/451: getdents d1d/d27/d2b 0 2024-04-05T03:02:50.538 INFO:tasks.workunit.client.0.smithi153.stdout:6/480: getdents d4/dc/df/d16/d44 0 2024-04-05T03:02:50.538 INFO:tasks.workunit.client.0.smithi153.stdout:9/352: symlink d1/d10/d16/d22/l85 0 2024-04-05T03:02:50.539 INFO:tasks.workunit.client.0.smithi153.stdout:5/452: rename d0/d25/d4d/d62/f93 to d0/d25/d4d/d62/d37/fa6 0 2024-04-05T03:02:50.540 INFO:tasks.workunit.client.0.smithi153.stdout:2/447: creat d9/da/d78/f92 x:0 0 0 2024-04-05T03:02:50.541 INFO:tasks.workunit.client.1.smithi203.stdout:0/785: fdatasync dd/d17/d1d/d22/d53/d61/fa2 0 2024-04-05T03:02:50.543 INFO:tasks.workunit.client.0.smithi153.stdout:7/479: fsync d2/d51/d93/f74 0 2024-04-05T03:02:50.544 INFO:tasks.workunit.client.0.smithi153.stdout:2/448: stat d9/da/d19/d2e 0 2024-04-05T03:02:50.544 INFO:tasks.workunit.client.1.smithi203.stdout:0/786: mknod dd/d17/d1d/d22/d45/d51/d69/ce4 0 2024-04-05T03:02:50.545 INFO:tasks.workunit.client.0.smithi153.stdout:6/481: fsync d4/dc/df/d16/d36/d52/f9f 0 2024-04-05T03:02:50.545 INFO:tasks.workunit.client.0.smithi153.stdout:5/453: write d0/d25/d4d/d62/d37/f4f [4100483,4503] 0 2024-04-05T03:02:50.546 INFO:tasks.workunit.client.1.smithi203.stdout:3/637: dread d0/dc/d1a/d8d/d96/f51 [0,4194304] 0 2024-04-05T03:02:50.547 INFO:tasks.workunit.client.0.smithi153.stdout:3/452: symlink d1d/d32/d93/d3c/d3f/d9c/la2 0 2024-04-05T03:02:50.547 INFO:tasks.workunit.client.1.smithi203.stdout:0/787: fsync dd/d17/d1d/d22/d53/db0/fda 0 2024-04-05T03:02:50.548 INFO:tasks.workunit.client.0.smithi153.stdout:5/454: write d0/f14 [5515097,3616] 0 2024-04-05T03:02:50.551 INFO:tasks.workunit.client.1.smithi203.stdout:3/638: mknod d0/dc/d38/cb9 0 2024-04-05T03:02:50.552 INFO:tasks.workunit.client.0.smithi153.stdout:8/414: dwrite d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:02:50.552 INFO:tasks.workunit.client.0.smithi153.stdout:5/455: fdatasync d0/d9/d22/f7a 0 2024-04-05T03:02:50.552 INFO:tasks.workunit.client.0.smithi153.stdout:9/353: link d1/d10/d36/f60 d1/d6c/f86 0 2024-04-05T03:02:50.554 INFO:tasks.workunit.client.1.smithi203.stdout:3/639: creat d0/dc/d1a/d1f/dac/fba x:0 0 0 2024-04-05T03:02:50.555 INFO:tasks.workunit.client.0.smithi153.stdout:2/449: creat d9/da/d19/d2e/d48/f93 x:0 0 0 2024-04-05T03:02:50.555 INFO:tasks.workunit.client.0.smithi153.stdout:3/453: write d1d/d32/d93/d3c/d3f/d90/d92/d99/f9a [537320,80442] 0 2024-04-05T03:02:50.557 INFO:tasks.workunit.client.0.smithi153.stdout:6/482: rename d4/dc/df/d16/d36/d52/f9f to d4/dc/df/d16/d5f/da2/fa4 0 2024-04-05T03:02:50.557 INFO:tasks.workunit.client.0.smithi153.stdout:3/454: stat d1d/d27/d2b/f2d 0 2024-04-05T03:02:50.557 INFO:tasks.workunit.client.1.smithi203.stdout:3/640: chown d0/dc/d1a/d8d/d96/f90 0 1 2024-04-05T03:02:50.558 INFO:tasks.workunit.client.0.smithi153.stdout:7/480: getdents d2/d51 0 2024-04-05T03:02:50.558 INFO:tasks.workunit.client.1.smithi203.stdout:3/641: chown d0/dc/f54 700 1 2024-04-05T03:02:50.559 INFO:tasks.workunit.client.0.smithi153.stdout:5/456: mkdir d0/d2c/d5e/da7 0 2024-04-05T03:02:50.560 INFO:tasks.workunit.client.0.smithi153.stdout:9/354: symlink d1/d10/d36/d57/l87 0 2024-04-05T03:02:50.562 INFO:tasks.workunit.client.0.smithi153.stdout:2/450: fdatasync d9/da/d19/d2e/d7e/d7a/f84 0 2024-04-05T03:02:50.565 INFO:tasks.workunit.client.1.smithi203.stdout:3/642: dwrite d0/dc/d34/f6e [0,4194304] 0 2024-04-05T03:02:50.566 INFO:tasks.workunit.client.0.smithi153.stdout:6/483: unlink d4/dc/c18 0 2024-04-05T03:02:50.568 INFO:tasks.workunit.client.0.smithi153.stdout:6/484: write d4/dc/df/d16/d36/d4d/f66 [1719868,41655] 0 2024-04-05T03:02:50.569 INFO:tasks.workunit.client.1.smithi203.stdout:3/643: symlink d0/d21/lbb 0 2024-04-05T03:02:50.569 INFO:tasks.workunit.client.1.smithi203.stdout:0/788: dread dd/d17/d1d/f37 [0,4194304] 0 2024-04-05T03:02:50.570 INFO:tasks.workunit.client.0.smithi153.stdout:2/451: mkdir d9/da/d94 0 2024-04-05T03:02:50.571 INFO:tasks.workunit.client.1.smithi203.stdout:0/789: fsync dd/fd0 0 2024-04-05T03:02:50.572 INFO:tasks.workunit.client.0.smithi153.stdout:2/452: chown d9/da/d19/d2e/d7e/d7a/f57 3295 1 2024-04-05T03:02:50.573 INFO:tasks.workunit.client.0.smithi153.stdout:5/457: mkdir d0/d2c/d3f/da8 0 2024-04-05T03:02:50.575 INFO:tasks.workunit.client.0.smithi153.stdout:9/355: getdents d1/d10/d36/d71 0 2024-04-05T03:02:50.575 INFO:tasks.workunit.client.0.smithi153.stdout:6/485: creat d4/d38/d4c/fa5 x:0 0 0 2024-04-05T03:02:50.575 INFO:tasks.workunit.client.1.smithi203.stdout:0/790: creat dd/d17/d1d/d22/d53/d61/fe5 x:0 0 0 2024-04-05T03:02:50.576 INFO:tasks.workunit.client.0.smithi153.stdout:7/481: truncate d2/da/d11/f2d 1750859 0 2024-04-05T03:02:50.577 INFO:tasks.workunit.client.1.smithi203.stdout:3/644: getdents d0/dc/d1a/d1f/da2 0 2024-04-05T03:02:50.578 INFO:tasks.workunit.client.0.smithi153.stdout:5/458: mknod d0/d25/d4d/d62/ca9 0 2024-04-05T03:02:50.579 INFO:tasks.workunit.client.0.smithi153.stdout:6/486: chown d4/dc/df/d16/d44/l76 163721 1 2024-04-05T03:02:50.579 INFO:tasks.workunit.client.1.smithi203.stdout:3/645: creat d0/d21/d81/fbc x:0 0 0 2024-04-05T03:02:50.580 INFO:tasks.workunit.client.0.smithi153.stdout:8/415: rename d7/d13/d19/d1b/d21/l76 to d7/d13/d19/l8a 0 2024-04-05T03:02:50.581 INFO:tasks.workunit.client.0.smithi153.stdout:2/453: dwrite d9/d77/f69 [0,4194304] 0 2024-04-05T03:02:50.585 INFO:tasks.workunit.client.0.smithi153.stdout:7/482: symlink d2/da/d3a/d41/d7a/d54/la0 0 2024-04-05T03:02:50.592 INFO:tasks.workunit.client.0.smithi153.stdout:9/356: unlink d1/d1d/c33 0 2024-04-05T03:02:50.593 INFO:tasks.workunit.client.0.smithi153.stdout:7/483: rmdir d2/da/d3a/d41/d7a 39 2024-04-05T03:02:50.594 INFO:tasks.workunit.client.0.smithi153.stdout:6/487: dwrite d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:50.595 INFO:tasks.workunit.client.0.smithi153.stdout:7/484: chown d2/da/d47/d5d/d7c/f81 257183 1 2024-04-05T03:02:50.596 INFO:tasks.workunit.client.0.smithi153.stdout:6/488: dread - d4/dc/df/d16/d44/f96 zero size 2024-04-05T03:02:50.598 INFO:tasks.workunit.client.0.smithi153.stdout:9/357: creat d1/d10/d16/d22/f88 x:0 0 0 2024-04-05T03:02:50.599 INFO:tasks.workunit.client.0.smithi153.stdout:9/358: write d1/d10/d36/f56 [293101,73501] 0 2024-04-05T03:02:50.600 INFO:tasks.workunit.client.0.smithi153.stdout:7/485: creat d2/da/d47/fa1 x:0 0 0 2024-04-05T03:02:50.601 INFO:tasks.workunit.client.0.smithi153.stdout:9/359: truncate d1/d9/d1b/f3f 5109968 0 2024-04-05T03:02:50.603 INFO:tasks.workunit.client.0.smithi153.stdout:8/416: getdents d7/d2a 0 2024-04-05T03:02:50.603 INFO:tasks.workunit.client.0.smithi153.stdout:9/360: read d1/d9/f20 [2645938,45959] 0 2024-04-05T03:02:50.606 INFO:tasks.workunit.client.0.smithi153.stdout:8/417: mkdir d7/d2a/d8b 0 2024-04-05T03:02:50.610 INFO:tasks.workunit.client.0.smithi153.stdout:5/459: rmdir d0/d25 39 2024-04-05T03:02:50.613 INFO:tasks.workunit.client.0.smithi153.stdout:7/486: getdents d2/da/d11/d39 0 2024-04-05T03:02:50.614 INFO:tasks.workunit.client.0.smithi153.stdout:5/460: readlink d0/d25/d4d/d9c/l83 0 2024-04-05T03:02:50.614 INFO:tasks.workunit.client.0.smithi153.stdout:2/454: truncate d9/da/d36/f63 966078 0 2024-04-05T03:02:50.615 INFO:tasks.workunit.client.0.smithi153.stdout:8/418: unlink d7/d13/d19/l8a 0 2024-04-05T03:02:50.619 INFO:tasks.workunit.client.0.smithi153.stdout:9/361: write d1/d10/d16/d22/d2c/d68/f7c [659440,68617] 0 2024-04-05T03:02:50.622 INFO:tasks.workunit.client.0.smithi153.stdout:8/419: rename d7/d13/d19/d1b/d4f/d5f/d54/d5b to d7/d13/d19/d1b/d6d/d8c 0 2024-04-05T03:02:50.623 INFO:tasks.workunit.client.0.smithi153.stdout:7/487: creat d2/da/d3a/d41/d7a/d54/fa2 x:0 0 0 2024-04-05T03:02:50.626 INFO:tasks.workunit.client.0.smithi153.stdout:8/420: dread d7/d13/d19/d1b/d21/f69 [0,4194304] 0 2024-04-05T03:02:50.627 INFO:tasks.workunit.client.0.smithi153.stdout:5/461: write d0/d9/f18 [2823217,19058] 0 2024-04-05T03:02:50.627 INFO:tasks.workunit.client.0.smithi153.stdout:5/462: fsync d0/d4e/f9e 0 2024-04-05T03:02:50.627 INFO:tasks.workunit.client.0.smithi153.stdout:7/488: mkdir d2/da/d47/da3 0 2024-04-05T03:02:50.628 INFO:tasks.workunit.client.0.smithi153.stdout:2/455: link d9/da/d19/l28 d9/da/d94/l95 0 2024-04-05T03:02:50.632 INFO:tasks.workunit.client.0.smithi153.stdout:8/421: mknod d7/d85/c8d 0 2024-04-05T03:02:50.633 INFO:tasks.workunit.client.0.smithi153.stdout:7/489: dwrite d2/d51/f79 [0,4194304] 0 2024-04-05T03:02:50.634 INFO:tasks.workunit.client.0.smithi153.stdout:9/362: chown d1/d10/d16/f17 0 1 2024-04-05T03:02:50.634 INFO:tasks.workunit.client.0.smithi153.stdout:8/422: read d7/d13/d19/d1b/d4f/d68/f84 [2511415,38496] 0 2024-04-05T03:02:50.640 INFO:tasks.workunit.client.0.smithi153.stdout:5/463: getdents d0/d2c 0 2024-04-05T03:02:50.640 INFO:tasks.workunit.client.0.smithi153.stdout:2/456: getdents d9/da/d19/d2e/d48 0 2024-04-05T03:02:50.641 INFO:tasks.workunit.client.0.smithi153.stdout:9/363: unlink d1/d9/f20 0 2024-04-05T03:02:50.642 INFO:tasks.workunit.client.0.smithi153.stdout:7/490: dwrite d2/da/d11/f13 [0,4194304] 0 2024-04-05T03:02:50.644 INFO:tasks.workunit.client.0.smithi153.stdout:5/464: creat d0/d25/d4d/d9c/faa x:0 0 0 2024-04-05T03:02:50.644 INFO:tasks.workunit.client.0.smithi153.stdout:9/364: rmdir d1 39 2024-04-05T03:02:50.645 INFO:tasks.workunit.client.0.smithi153.stdout:2/457: symlink d9/da/d19/d2e/d7e/d53/l96 0 2024-04-05T03:02:50.650 INFO:tasks.workunit.client.0.smithi153.stdout:7/491: symlink d2/da/d11/d88/la4 0 2024-04-05T03:02:50.651 INFO:tasks.workunit.client.0.smithi153.stdout:9/365: creat d1/d10/d36/d57/d5c/f89 x:0 0 0 2024-04-05T03:02:50.651 INFO:tasks.workunit.client.0.smithi153.stdout:7/492: truncate d2/da/f7d 1978010 0 2024-04-05T03:02:50.654 INFO:tasks.workunit.client.0.smithi153.stdout:2/458: mknod d9/da/d19/d2e/d7e/d7a/c97 0 2024-04-05T03:02:50.654 INFO:tasks.workunit.client.0.smithi153.stdout:2/459: fsync d9/da/fc 0 2024-04-05T03:02:50.655 INFO:tasks.workunit.client.0.smithi153.stdout:7/493: creat d2/d51/fa5 x:0 0 0 2024-04-05T03:02:50.658 INFO:tasks.workunit.client.0.smithi153.stdout:5/465: dread d0/d25/d4d/d62/d37/f6a [0,4194304] 0 2024-04-05T03:02:50.658 INFO:tasks.workunit.client.0.smithi153.stdout:2/460: mkdir d9/d68/d98 0 2024-04-05T03:02:50.663 INFO:tasks.workunit.client.0.smithi153.stdout:2/461: creat d9/da/d19/d2e/d7e/d7a/d59/f99 x:0 0 0 2024-04-05T03:02:50.664 INFO:tasks.workunit.client.0.smithi153.stdout:9/366: link d1/d10/d16/d22/d2c/d68/d81/c14 d1/d6c/c8a 0 2024-04-05T03:02:50.665 INFO:tasks.workunit.client.0.smithi153.stdout:7/494: dwrite d2/da/d3a/d41/f8a [0,4194304] 0 2024-04-05T03:02:50.666 INFO:tasks.workunit.client.0.smithi153.stdout:9/367: fsync d1/d10/f5b 0 2024-04-05T03:02:50.668 INFO:tasks.workunit.client.0.smithi153.stdout:2/462: symlink d9/da/d19/d2e/d7e/d7a/l9a 0 2024-04-05T03:02:50.669 INFO:tasks.workunit.client.1.smithi203.stdout:9/747: sync 2024-04-05T03:02:50.671 INFO:tasks.workunit.client.1.smithi203.stdout:9/748: write d13/d5c/dd1/d31/f3e [4685024,20693] 0 2024-04-05T03:02:50.672 INFO:tasks.workunit.client.0.smithi153.stdout:5/466: dwrite d0/f75 [0,4194304] 0 2024-04-05T03:02:50.680 INFO:tasks.workunit.client.0.smithi153.stdout:9/368: dwrite d1/d6c/f6e [0,4194304] 0 2024-04-05T03:02:50.682 INFO:tasks.workunit.client.1.smithi203.stdout:9/749: fdatasync d13/d5c/dd1/f4c 0 2024-04-05T03:02:50.683 INFO:tasks.workunit.client.1.smithi203.stdout:9/750: chown d13/d5c/lec 1707092 1 2024-04-05T03:02:50.684 INFO:tasks.workunit.client.0.smithi153.stdout:9/369: fsync d1/d10/d16/d22/d2c/d68/d81/f7a 0 2024-04-05T03:02:50.686 INFO:tasks.workunit.client.0.smithi153.stdout:2/463: dwrite d9/da/f5f [4194304,4194304] 0 2024-04-05T03:02:50.686 INFO:tasks.workunit.client.1.smithi203.stdout:9/751: symlink d13/d1b/d33/d42/d67/d82/lf2 0 2024-04-05T03:02:50.688 INFO:tasks.workunit.client.1.smithi203.stdout:9/752: chown d13/d1b/d6b/d7a/c8c 3 1 2024-04-05T03:02:50.688 INFO:tasks.workunit.client.0.smithi153.stdout:1/414: sync 2024-04-05T03:02:50.690 INFO:tasks.workunit.client.1.smithi203.stdout:9/753: creat d13/ff3 x:0 0 0 2024-04-05T03:02:50.691 INFO:tasks.workunit.client.0.smithi153.stdout:9/370: truncate d1/d10/d16/d22/d2c/f31 4424124 0 2024-04-05T03:02:50.691 INFO:tasks.workunit.client.0.smithi153.stdout:4/535: sync 2024-04-05T03:02:50.691 INFO:tasks.workunit.client.0.smithi153.stdout:1/415: symlink d5/d8/d20/d4b/d18/d16/l80 0 2024-04-05T03:02:50.692 INFO:tasks.workunit.client.0.smithi153.stdout:9/371: readlink d1/d10/l2f 0 2024-04-05T03:02:50.694 INFO:tasks.workunit.client.1.smithi203.stdout:9/754: creat d13/d1b/d33/d42/ff4 x:0 0 0 2024-04-05T03:02:50.696 INFO:tasks.workunit.client.0.smithi153.stdout:2/464: dwrite d9/da/d19/d2e/d4f/f86 [0,4194304] 0 2024-04-05T03:02:50.696 INFO:tasks.workunit.client.0.smithi153.stdout:4/536: rmdir da/db/d20/d35/d38/d76/d90 39 2024-04-05T03:02:50.698 INFO:tasks.workunit.client.0.smithi153.stdout:9/372: mknod d1/d10/c8b 0 2024-04-05T03:02:50.701 INFO:tasks.workunit.client.0.smithi153.stdout:9/373: fsync d1/d9/d1b/f5d 0 2024-04-05T03:02:50.702 INFO:tasks.workunit.client.0.smithi153.stdout:1/416: dwrite d5/f9 [4194304,4194304] 0 2024-04-05T03:02:50.707 INFO:tasks.workunit.client.0.smithi153.stdout:5/467: dread d0/d9/d22/f33 [0,4194304] 0 2024-04-05T03:02:50.707 INFO:tasks.workunit.client.0.smithi153.stdout:4/537: rename da/db/d20/d35/d38/d3e to da/db/d20/d3c/d81/db6 0 2024-04-05T03:02:50.708 INFO:tasks.workunit.client.0.smithi153.stdout:4/538: fsync da/de/d56/d78/d8f/f99 0 2024-04-05T03:02:50.709 INFO:tasks.workunit.client.0.smithi153.stdout:9/374: unlink d1/d10/d16/d22/d2c/c4e 0 2024-04-05T03:02:50.710 INFO:tasks.workunit.client.0.smithi153.stdout:5/468: read d0/f16 [4930717,7214] 0 2024-04-05T03:02:50.713 INFO:tasks.workunit.client.0.smithi153.stdout:4/539: mkdir da/db/d8b/db7 0 2024-04-05T03:02:50.714 INFO:tasks.workunit.client.0.smithi153.stdout:9/375: mknod d1/d9/d1b/c8c 0 2024-04-05T03:02:50.715 INFO:tasks.workunit.client.0.smithi153.stdout:1/417: creat d5/d8/d20/d4b/f81 x:0 0 0 2024-04-05T03:02:50.716 INFO:tasks.workunit.client.0.smithi153.stdout:4/540: dread - da/db/d20/d35/d41/f62 zero size 2024-04-05T03:02:50.719 INFO:tasks.workunit.client.0.smithi153.stdout:1/418: mkdir d5/d8/d20/d4b/d18/d16/d48/d82 0 2024-04-05T03:02:50.724 INFO:tasks.workunit.client.1.smithi203.stdout:9/755: truncate d13/d1b/d6b/d9e/d77/fd3 2041468 0 2024-04-05T03:02:50.724 INFO:tasks.workunit.client.0.smithi153.stdout:1/419: mkdir d5/d8/d20/d4b/d18/d16/d33/d83 0 2024-04-05T03:02:50.724 INFO:tasks.workunit.client.0.smithi153.stdout:9/376: dwrite d1/d9/d1b/f75 [0,4194304] 0 2024-04-05T03:02:50.725 INFO:tasks.workunit.client.1.smithi203.stdout:9/756: symlink d13/d5c/lf5 0 2024-04-05T03:02:50.727 INFO:tasks.workunit.client.0.smithi153.stdout:9/377: dread - d1/d10/f64 zero size 2024-04-05T03:02:50.727 INFO:tasks.workunit.client.1.smithi203.stdout:9/757: mknod d13/d1b/d6b/d9e/d59/d5f/cf6 0 2024-04-05T03:02:50.731 INFO:tasks.workunit.client.0.smithi153.stdout:4/541: dwrite da/de/f1e [0,4194304] 0 2024-04-05T03:02:50.735 INFO:tasks.workunit.client.1.smithi203.stdout:9/758: dread d13/d1b/f47 [0,4194304] 0 2024-04-05T03:02:50.740 INFO:tasks.workunit.client.0.smithi153.stdout:2/465: dread d9/da/d19/f1f [4194304,4194304] 0 2024-04-05T03:02:50.740 INFO:tasks.workunit.client.1.smithi203.stdout:9/759: symlink d13/d1b/d6b/d9e/d59/lf7 0 2024-04-05T03:02:50.743 INFO:tasks.workunit.client.1.smithi203.stdout:9/760: creat d13/d1b/d33/d42/ff8 x:0 0 0 2024-04-05T03:02:50.743 INFO:tasks.workunit.client.0.smithi153.stdout:9/378: dwrite d1/d10/d16/d22/d2c/d68/d81/f76 [0,4194304] 0 2024-04-05T03:02:50.746 INFO:tasks.workunit.client.0.smithi153.stdout:2/466: mkdir d9/d68/d9b 0 2024-04-05T03:02:50.746 INFO:tasks.workunit.client.0.smithi153.stdout:4/542: dwrite da/f4d [0,4194304] 0 2024-04-05T03:02:50.747 INFO:tasks.workunit.client.1.smithi203.stdout:9/761: mknod d13/dc8/dd2/cf9 0 2024-04-05T03:02:50.750 INFO:tasks.workunit.client.0.smithi153.stdout:5/469: dread d0/d25/d4d/d62/d37/f4f [0,4194304] 0 2024-04-05T03:02:50.751 INFO:tasks.workunit.client.0.smithi153.stdout:4/543: chown da/de/d56/d6a/f7f 7 1 2024-04-05T03:02:50.751 INFO:tasks.workunit.client.0.smithi153.stdout:5/470: dread - d0/d2c/f8f zero size 2024-04-05T03:02:50.754 INFO:tasks.workunit.client.1.smithi203.stdout:9/762: dwrite d13/f2a [0,4194304] 0 2024-04-05T03:02:50.754 INFO:tasks.workunit.client.0.smithi153.stdout:5/471: dread d0/d25/d68/f99 [0,4194304] 0 2024-04-05T03:02:50.755 INFO:tasks.workunit.client.0.smithi153.stdout:9/379: link d1/d9/d1b/f75 d1/d9/d1b/f8d 0 2024-04-05T03:02:50.761 INFO:tasks.workunit.client.0.smithi153.stdout:9/380: unlink d1/d10/d16/f38 0 2024-04-05T03:02:50.762 INFO:tasks.workunit.client.0.smithi153.stdout:4/544: creat da/db/d20/d3c/fb8 x:0 0 0 2024-04-05T03:02:50.764 INFO:tasks.workunit.client.0.smithi153.stdout:9/381: unlink d1/d9/d1b/c3d 0 2024-04-05T03:02:50.765 INFO:tasks.workunit.client.0.smithi153.stdout:5/472: link d0/d25/d4d/d62/d37/f4f d0/fab 0 2024-04-05T03:02:50.767 INFO:tasks.workunit.client.0.smithi153.stdout:9/382: chown d1/d9/d1b/f28 6 1 2024-04-05T03:02:50.768 INFO:tasks.workunit.client.0.smithi153.stdout:2/467: truncate d9/da/f64 3076605 0 2024-04-05T03:02:50.769 INFO:tasks.workunit.client.0.smithi153.stdout:5/473: write d0/d9/f20 [3647233,89116] 0 2024-04-05T03:02:50.770 INFO:tasks.workunit.client.0.smithi153.stdout:4/545: creat da/db/d20/fb9 x:0 0 0 2024-04-05T03:02:50.770 INFO:tasks.workunit.client.0.smithi153.stdout:0/455: sync 2024-04-05T03:02:50.770 INFO:tasks.workunit.client.0.smithi153.stdout:3/455: sync 2024-04-05T03:02:50.771 INFO:tasks.workunit.client.0.smithi153.stdout:4/546: readlink da/db/d20/d35/l36 0 2024-04-05T03:02:50.772 INFO:tasks.workunit.client.0.smithi153.stdout:4/547: write f9 [3736128,110766] 0 2024-04-05T03:02:50.775 INFO:tasks.workunit.client.0.smithi153.stdout:9/383: creat d1/d9/f8e x:0 0 0 2024-04-05T03:02:50.775 INFO:tasks.workunit.client.0.smithi153.stdout:5/474: symlink d0/d2c/d3f/da8/lac 0 2024-04-05T03:02:50.777 INFO:tasks.workunit.client.0.smithi153.stdout:2/468: link d9/da/d78/f92 d9/da/d19/d2e/d7e/d53/f9c 0 2024-04-05T03:02:50.782 INFO:tasks.workunit.client.0.smithi153.stdout:5/475: mknod d0/d25/d4d/d62/d5f/d86/da4/cad 0 2024-04-05T03:02:50.782 INFO:tasks.workunit.client.0.smithi153.stdout:4/548: dwrite da/de/d56/f9f [0,4194304] 0 2024-04-05T03:02:50.782 INFO:tasks.workunit.client.0.smithi153.stdout:2/469: write d9/da/d19/f46 [2354151,31101] 0 2024-04-05T03:02:50.783 INFO:tasks.workunit.client.1.smithi203.stdout:9/763: write d13/d1b/d33/d63/f6e [474437,96445] 0 2024-04-05T03:02:50.783 INFO:tasks.workunit.client.0.smithi153.stdout:0/456: getdents d9/df/d1e/d45/d62 0 2024-04-05T03:02:50.785 INFO:tasks.workunit.client.0.smithi153.stdout:3/456: getdents d1d/d32/d93/d3c/d3f/d90/d92 0 2024-04-05T03:02:50.789 INFO:tasks.workunit.client.0.smithi153.stdout:4/549: rmdir da/de/d56/d78 39 2024-04-05T03:02:50.790 INFO:tasks.workunit.client.0.smithi153.stdout:1/420: dread d5/d8/d20/f5f [0,4194304] 0 2024-04-05T03:02:50.790 INFO:tasks.workunit.client.0.smithi153.stdout:5/476: write d0/d25/d4d/d62/d5f/d86/da4/fa1 [456782,7466] 0 2024-04-05T03:02:50.793 INFO:tasks.workunit.client.0.smithi153.stdout:4/550: write da/de/f1e [4776185,115386] 0 2024-04-05T03:02:50.793 INFO:tasks.workunit.client.0.smithi153.stdout:3/457: mknod d1d/d32/d93/d3c/d3f/d9c/ca3 0 2024-04-05T03:02:50.793 INFO:tasks.workunit.client.0.smithi153.stdout:1/421: dread - d5/d8/d20/d4b/d18/d16/d33/f70 zero size 2024-04-05T03:02:50.793 INFO:tasks.workunit.client.0.smithi153.stdout:4/551: fdatasync da/db/f46 0 2024-04-05T03:02:50.797 INFO:tasks.workunit.client.1.smithi203.stdout:9/764: write d13/d1b/d6b/d9e/f5e [4867882,92062] 0 2024-04-05T03:02:50.798 INFO:tasks.workunit.client.1.smithi203.stdout:9/765: stat d13/d1b/d33/d64/fd7 0 2024-04-05T03:02:50.800 INFO:tasks.workunit.client.0.smithi153.stdout:9/384: dread d1/d10/d36/f62 [0,4194304] 0 2024-04-05T03:02:50.802 INFO:tasks.workunit.client.0.smithi153.stdout:3/458: mkdir d1d/d32/d93/d3c/d3f/d90/da4 0 2024-04-05T03:02:50.802 INFO:tasks.workunit.client.0.smithi153.stdout:5/477: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T03:02:50.803 INFO:tasks.workunit.client.0.smithi153.stdout:4/552: symlink da/de/d56/d93/lba 0 2024-04-05T03:02:50.803 INFO:tasks.workunit.client.1.smithi203.stdout:9/766: rename d13/d1b/d33/d42 to d13/d1b/d33/d64/dfa 0 2024-04-05T03:02:50.804 INFO:tasks.workunit.client.0.smithi153.stdout:4/553: dread - da/db/d20/d45/f9e zero size 2024-04-05T03:02:50.805 INFO:tasks.workunit.client.1.smithi203.stdout:9/767: fdatasync d13/d5c/dd1/d51/d81/fa5 0 2024-04-05T03:02:50.806 INFO:tasks.workunit.client.0.smithi153.stdout:5/478: chown d0/d25/f2f 465089 1 2024-04-05T03:02:50.807 INFO:tasks.workunit.client.0.smithi153.stdout:3/459: creat d1d/d32/d93/d57/d6b/fa5 x:0 0 0 2024-04-05T03:02:50.807 INFO:tasks.workunit.client.0.smithi153.stdout:5/479: truncate d0/f75 4896456 0 2024-04-05T03:02:50.808 INFO:tasks.workunit.client.0.smithi153.stdout:1/422: link d5/d8/d20/d4b/d18/d16/d33/f3e d5/f84 0 2024-04-05T03:02:50.811 INFO:tasks.workunit.client.0.smithi153.stdout:0/457: write d9/d14/f23 [4811961,75743] 0 2024-04-05T03:02:50.812 INFO:tasks.workunit.client.1.smithi203.stdout:9/768: write d13/d1b/fbd [21997,8024] 0 2024-04-05T03:02:50.814 INFO:tasks.workunit.client.1.smithi203.stdout:9/769: stat d13/l56 0 2024-04-05T03:02:50.814 INFO:tasks.workunit.client.0.smithi153.stdout:3/460: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/fa6 x:0 0 0 2024-04-05T03:02:50.816 INFO:tasks.workunit.client.0.smithi153.stdout:2/470: truncate d9/da/d19/f2f 197104 0 2024-04-05T03:02:50.817 INFO:tasks.workunit.client.0.smithi153.stdout:0/458: truncate f0 415694 0 2024-04-05T03:02:50.818 INFO:tasks.workunit.client.1.smithi203.stdout:9/770: rename d13/f1d to d13/d5c/dd1/d51/dcb/ffb 0 2024-04-05T03:02:50.820 INFO:tasks.workunit.client.1.smithi203.stdout:9/771: symlink d13/d1b/lfc 0 2024-04-05T03:02:50.835 INFO:tasks.workunit.client.1.smithi203.stdout:1/752: sync 2024-04-05T03:02:50.835 INFO:tasks.workunit.client.1.smithi203.stdout:6/762: sync 2024-04-05T03:02:50.835 INFO:tasks.workunit.client.1.smithi203.stdout:4/577: sync 2024-04-05T03:02:50.839 INFO:tasks.workunit.client.1.smithi203.stdout:6/763: rename d0/da/d54/d4c/f77 to d0/da/d18/d38/d86/fe6 0 2024-04-05T03:02:50.842 INFO:tasks.workunit.client.1.smithi203.stdout:6/764: chown d0/da/c32 376359786 1 2024-04-05T03:02:50.842 INFO:tasks.workunit.client.1.smithi203.stdout:1/753: rename d7/d24/d33/d48/d4b/d6b/db9/dde to d7/d24/d33/d48/d4b/dea/df2 0 2024-04-05T03:02:50.844 INFO:tasks.workunit.client.1.smithi203.stdout:9/772: dwrite d13/d1b/d6b/f87 [0,4194304] 0 2024-04-05T03:02:50.844 INFO:tasks.workunit.client.1.smithi203.stdout:4/578: dwrite da/d29/f42 [4194304,4194304] 0 2024-04-05T03:02:50.844 INFO:tasks.workunit.client.1.smithi203.stdout:6/765: stat d0/da/d18 0 2024-04-05T03:02:50.847 INFO:tasks.workunit.client.1.smithi203.stdout:1/754: creat d7/d14/d9d/ff3 x:0 0 0 2024-04-05T03:02:50.849 INFO:tasks.workunit.client.1.smithi203.stdout:6/766: creat d0/da/d18/d92/fe7 x:0 0 0 2024-04-05T03:02:50.852 INFO:tasks.workunit.client.0.smithi153.stdout:1/423: rename d5/d8/d20/d3f/l6a to d5/dd/l85 0 2024-04-05T03:02:50.852 INFO:tasks.workunit.client.0.smithi153.stdout:0/459: read d9/df/d1e/f50 [3522865,120332] 0 2024-04-05T03:02:50.855 INFO:tasks.workunit.client.0.smithi153.stdout:0/460: chown d9/db/f13 45591 1 2024-04-05T03:02:50.856 INFO:tasks.workunit.client.1.smithi203.stdout:1/755: rename d7/l10 to d7/d24/d33/d48/d4b/dea/df2/lf4 0 2024-04-05T03:02:50.859 INFO:tasks.workunit.client.1.smithi203.stdout:1/756: unlink d7/d14/d2b/f5a 0 2024-04-05T03:02:50.864 INFO:tasks.workunit.client.1.smithi203.stdout:1/757: dwrite d7/d14/d2b/f95 [0,4194304] 0 2024-04-05T03:02:50.868 INFO:tasks.workunit.client.1.smithi203.stdout:1/758: rename d7/d24/d33/f36 to d7/d24/d33/d48/d4b/dea/df2/ff5 0 2024-04-05T03:02:50.869 INFO:tasks.workunit.client.1.smithi203.stdout:1/759: write d7/d24/d33/d48/d4b/d51/fe2 [575023,19715] 0 2024-04-05T03:02:50.874 INFO:tasks.workunit.client.1.smithi203.stdout:1/760: creat d7/d24/d33/d48/d4b/ff6 x:0 0 0 2024-04-05T03:02:50.878 INFO:tasks.workunit.client.1.smithi203.stdout:6/767: write d0/da/d18/d38/d2a/d2c/fa5 [141631,108659] 0 2024-04-05T03:02:50.882 INFO:tasks.workunit.client.1.smithi203.stdout:6/768: creat d0/da/d54/d4c/da1/db7/d95/fe8 x:0 0 0 2024-04-05T03:02:50.882 INFO:tasks.workunit.client.1.smithi203.stdout:9/773: dwrite d13/d1b/d33/d64/dfa/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:50.887 INFO:tasks.workunit.client.1.smithi203.stdout:9/774: read d13/d5c/dd1/d51/d81/fa5 [1057159,116220] 0 2024-04-05T03:02:50.888 INFO:tasks.workunit.client.1.smithi203.stdout:9/775: dread - d13/d1b/d6b/d9e/d59/dbc/fdd zero size 2024-04-05T03:02:50.895 INFO:tasks.workunit.client.1.smithi203.stdout:6/769: write d0/da/d54/f3f [668306,83611] 0 2024-04-05T03:02:50.896 INFO:tasks.workunit.client.1.smithi203.stdout:6/770: chown d0/da/d54/fb0 8373870 1 2024-04-05T03:02:50.898 INFO:tasks.workunit.client.1.smithi203.stdout:9/776: dwrite d13/d1b/d6b/d9e/d77/f7d [0,4194304] 0 2024-04-05T03:02:50.902 INFO:tasks.workunit.client.1.smithi203.stdout:9/777: chown d13/d5c/dd1/d31/f41 1 1 2024-04-05T03:02:50.904 INFO:tasks.workunit.client.1.smithi203.stdout:9/778: chown d13/d1b/d33/d63 57345 1 2024-04-05T03:02:50.907 INFO:tasks.workunit.client.1.smithi203.stdout:9/779: symlink d13/d1b/d6b/d7a/lfd 0 2024-04-05T03:02:50.907 INFO:tasks.workunit.client.1.smithi203.stdout:6/771: dwrite d0/da/d18/d38/d2a/fce [0,4194304] 0 2024-04-05T03:02:50.910 INFO:tasks.workunit.client.1.smithi203.stdout:6/772: fdatasync d0/da/d18/d38/d2a/d2c/f4e 0 2024-04-05T03:02:50.910 INFO:tasks.workunit.client.1.smithi203.stdout:1/761: dread d7/f9c [0,4194304] 0 2024-04-05T03:02:50.912 INFO:tasks.workunit.client.0.smithi153.stdout:6/489: sync 2024-04-05T03:02:50.913 INFO:tasks.workunit.client.1.smithi203.stdout:1/762: dread - d7/d24/d33/d48/d4b/f94 zero size 2024-04-05T03:02:50.914 INFO:tasks.workunit.client.0.smithi153.stdout:1/424: creat d5/d8/d20/d3f/f86 x:0 0 0 2024-04-05T03:02:50.914 INFO:tasks.workunit.client.0.smithi153.stdout:2/471: dwrite d9/da/d19/d2e/d7e/d7a/d6d/f6e [0,4194304] 0 2024-04-05T03:02:50.919 INFO:tasks.workunit.client.0.smithi153.stdout:3/461: rename fd to d1d/fa7 0 2024-04-05T03:02:50.919 INFO:tasks.workunit.client.0.smithi153.stdout:2/472: rmdir d9/da/d36 39 2024-04-05T03:02:50.919 INFO:tasks.workunit.client.1.smithi203.stdout:6/773: fdatasync d0/da/d54/d4c/da1/db7/fa2 0 2024-04-05T03:02:50.919 INFO:tasks.workunit.client.1.smithi203.stdout:9/780: getdents d13/d5c/dd1/d51/dcb 0 2024-04-05T03:02:50.921 INFO:tasks.workunit.client.0.smithi153.stdout:4/554: dread da/de/f11 [0,4194304] 0 2024-04-05T03:02:50.921 INFO:tasks.workunit.client.0.smithi153.stdout:3/462: readlink d1d/d32/d93/d3c/d3f/l95 0 2024-04-05T03:02:50.922 INFO:tasks.workunit.client.0.smithi153.stdout:9/385: dwrite d1/d10/d36/f60 [0,4194304] 0 2024-04-05T03:02:50.922 INFO:tasks.workunit.client.1.smithi203.stdout:1/763: mkdir d7/d24/d33/d48/df7 0 2024-04-05T03:02:50.924 INFO:tasks.workunit.client.0.smithi153.stdout:2/473: creat d9/da/d19/d2e/d7e/d53/f9d x:0 0 0 2024-04-05T03:02:50.925 INFO:tasks.workunit.client.0.smithi153.stdout:9/386: write d1/d9/d1b/f5d [592673,18604] 0 2024-04-05T03:02:50.930 INFO:tasks.workunit.client.0.smithi153.stdout:2/474: unlink d9/da/d19/d2e/d7e/d7a/c97 0 2024-04-05T03:02:50.932 INFO:tasks.workunit.client.0.smithi153.stdout:9/387: rmdir d1/d10/d36/d57 39 2024-04-05T03:02:50.932 INFO:tasks.workunit.client.1.smithi203.stdout:1/764: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:50.934 INFO:tasks.workunit.client.0.smithi153.stdout:4/555: creat da/db/d20/d3c/d91/fbb x:0 0 0 2024-04-05T03:02:50.935 INFO:tasks.workunit.client.0.smithi153.stdout:2/475: truncate d9/da/f15 2174310 0 2024-04-05T03:02:50.935 INFO:tasks.workunit.client.0.smithi153.stdout:4/556: dread - da/db/d20/d45/fa6 zero size 2024-04-05T03:02:50.936 INFO:tasks.workunit.client.0.smithi153.stdout:9/388: symlink d1/d10/d16/d22/d2c/l8f 0 2024-04-05T03:02:50.938 INFO:tasks.workunit.client.0.smithi153.stdout:4/557: write da/db/d20/d35/d41/d67/f80 [4990006,97616] 0 2024-04-05T03:02:50.942 INFO:tasks.workunit.client.0.smithi153.stdout:4/558: creat da/db/d20/d3c/d91/fbc x:0 0 0 2024-04-05T03:02:50.943 INFO:tasks.workunit.client.1.smithi203.stdout:6/774: write d0/da/d54/d53/f6a [439007,62719] 0 2024-04-05T03:02:50.943 INFO:tasks.workunit.client.0.smithi153.stdout:6/490: dread d4/fb [0,4194304] 0 2024-04-05T03:02:50.944 INFO:tasks.workunit.client.0.smithi153.stdout:6/491: chown d4/dc/df/c6f 7525026 1 2024-04-05T03:02:50.944 INFO:tasks.workunit.client.0.smithi153.stdout:4/559: mkdir da/db/dbd 0 2024-04-05T03:02:50.946 INFO:tasks.workunit.client.0.smithi153.stdout:3/463: write d1d/d32/d93/f4c [661215,123945] 0 2024-04-05T03:02:50.948 INFO:tasks.workunit.client.0.smithi153.stdout:3/464: truncate d1d/d27/f54 366587 0 2024-04-05T03:02:50.949 INFO:tasks.workunit.client.1.smithi203.stdout:9/781: dwrite d13/d1b/d6b/d9e/d59/d5f/fb9 [0,4194304] 0 2024-04-05T03:02:50.951 INFO:tasks.workunit.client.0.smithi153.stdout:4/560: rename da/de/f11 to da/db/d20/fbe 0 2024-04-05T03:02:50.952 INFO:tasks.workunit.client.1.smithi203.stdout:9/782: dread - d13/d5c/dd1/d51/da1/dba/fd4 zero size 2024-04-05T03:02:50.953 INFO:tasks.workunit.client.0.smithi153.stdout:3/465: dread d1d/d32/d93/d3c/d3f/d90/d92/d99/f9a [0,4194304] 0 2024-04-05T03:02:50.953 INFO:tasks.workunit.client.1.smithi203.stdout:6/775: dwrite d0/da/d54/d4c/da1/db7/fcb [0,4194304] 0 2024-04-05T03:02:50.954 INFO:tasks.workunit.client.0.smithi153.stdout:4/561: symlink da/de/d56/d6a/d2d/lbf 0 2024-04-05T03:02:50.955 INFO:tasks.workunit.client.0.smithi153.stdout:3/466: read d1d/d27/d2b/f7e [3174425,8775] 0 2024-04-05T03:02:50.956 INFO:tasks.workunit.client.0.smithi153.stdout:4/562: write da/de/d56/d6a/d2d/fa5 [319671,12876] 0 2024-04-05T03:02:50.957 INFO:tasks.workunit.client.0.smithi153.stdout:4/563: readlink l6 0 2024-04-05T03:02:50.957 INFO:tasks.workunit.client.0.smithi153.stdout:6/492: rename d4/dc/df/d16/d44/d90 to d4/dc/df/d16/d5f/da6 0 2024-04-05T03:02:50.958 INFO:tasks.workunit.client.1.smithi203.stdout:9/783: rmdir d13/d5c/dd1/d51/d81 39 2024-04-05T03:02:50.959 INFO:tasks.workunit.client.1.smithi203.stdout:9/784: chown d13/d1b/d6b/d9e/c4b 448101 1 2024-04-05T03:02:50.959 INFO:tasks.workunit.client.1.smithi203.stdout:6/776: rmdir d0/da/d54/d4c/da1/db7/d71 39 2024-04-05T03:02:50.961 INFO:tasks.workunit.client.0.smithi153.stdout:3/467: rename d1d/d27/f6f to d1d/d32/d93/d3c/d3f/d7c/fa8 0 2024-04-05T03:02:50.965 INFO:tasks.workunit.client.1.smithi203.stdout:9/785: link d13/d1b/d33/d63/f6e d13/ffe 0 2024-04-05T03:02:50.966 INFO:tasks.workunit.client.0.smithi153.stdout:3/468: symlink d1d/d32/d93/d3c/d94/la9 0 2024-04-05T03:02:50.966 INFO:tasks.workunit.client.0.smithi153.stdout:6/493: link d4/ce d4/dc/df/d16/d24/d4e/ca7 0 2024-04-05T03:02:50.967 INFO:tasks.workunit.client.1.smithi203.stdout:9/786: fsync d13/d1b/d6b/d9e/f5e 0 2024-04-05T03:02:50.968 INFO:tasks.workunit.client.1.smithi203.stdout:6/777: dwrite d0/da/d54/d4c/da1/db7/d95/fe8 [0,4194304] 0 2024-04-05T03:02:50.968 INFO:tasks.workunit.client.0.smithi153.stdout:6/494: rmdir d4/dc/df/d16/d24 39 2024-04-05T03:02:50.970 INFO:tasks.workunit.client.0.smithi153.stdout:3/469: creat d1d/d32/d93/d3c/d3f/d9c/faa x:0 0 0 2024-04-05T03:02:50.970 INFO:tasks.workunit.client.0.smithi153.stdout:6/495: mkdir d4/dc/d7a/da8 0 2024-04-05T03:02:50.973 INFO:tasks.workunit.client.0.smithi153.stdout:3/470: fsync f3 0 2024-04-05T03:02:50.975 INFO:tasks.workunit.client.0.smithi153.stdout:3/471: chown d1d/f28 53455 1 2024-04-05T03:02:50.975 INFO:tasks.workunit.client.1.smithi203.stdout:9/787: creat d13/d1b/d6b/d9e/d59/d7b/fff x:0 0 0 2024-04-05T03:02:50.979 INFO:tasks.workunit.client.0.smithi153.stdout:3/472: getdents d1d/d32/d93/d57/d6b/d9f 0 2024-04-05T03:02:50.979 INFO:tasks.workunit.client.0.smithi153.stdout:6/496: dread d4/dc/df/d16/d5f/da2/fa4 [0,4194304] 0 2024-04-05T03:02:50.980 INFO:tasks.workunit.client.1.smithi203.stdout:9/788: dread d13/f2a [0,4194304] 0 2024-04-05T03:02:50.984 INFO:tasks.workunit.client.0.smithi153.stdout:6/497: truncate d4/dc/df/d16/d5f/f8c 1044912 0 2024-04-05T03:02:50.985 INFO:tasks.workunit.client.0.smithi153.stdout:0/461: dwrite f0 [0,4194304] 0 2024-04-05T03:02:50.988 INFO:tasks.workunit.client.0.smithi153.stdout:6/498: creat d4/dc/df/d16/d24/d4e/fa9 x:0 0 0 2024-04-05T03:02:50.988 INFO:tasks.workunit.client.0.smithi153.stdout:3/473: dread d1d/d27/d2b/f71 [0,4194304] 0 2024-04-05T03:02:50.989 INFO:tasks.workunit.client.1.smithi203.stdout:8/869: sync 2024-04-05T03:02:50.990 INFO:tasks.workunit.client.1.smithi203.stdout:5/756: sync 2024-04-05T03:02:50.990 INFO:tasks.workunit.client.1.smithi203.stdout:3/646: sync 2024-04-05T03:02:50.990 INFO:tasks.workunit.client.1.smithi203.stdout:2/881: sync 2024-04-05T03:02:50.990 INFO:tasks.workunit.client.1.smithi203.stdout:0/791: sync 2024-04-05T03:02:50.990 INFO:tasks.workunit.client.0.smithi153.stdout:6/499: write d4/dc/df/d16/d17/f3b [2418354,10464] 0 2024-04-05T03:02:50.993 INFO:tasks.workunit.client.1.smithi203.stdout:2/882: chown d5/d10/d35/dee 798982 1 2024-04-05T03:02:50.993 INFO:tasks.workunit.client.1.smithi203.stdout:6/778: write d0/da/d18/d38/d86/f93 [209046,101500] 0 2024-04-05T03:02:50.993 INFO:tasks.workunit.client.1.smithi203.stdout:8/870: chown d1/d4b/d5f/l6f 1 1 2024-04-05T03:02:50.993 INFO:tasks.workunit.client.1.smithi203.stdout:9/789: truncate d13/d5c/d9d/fe4 645545 0 2024-04-05T03:02:50.994 INFO:tasks.workunit.client.0.smithi153.stdout:3/474: rmdir d1d/d32/d93/d57 39 2024-04-05T03:02:50.996 INFO:tasks.workunit.client.1.smithi203.stdout:9/790: write d13/d5c/dd1/d31/f3e [643111,125553] 0 2024-04-05T03:02:50.997 INFO:tasks.workunit.client.1.smithi203.stdout:3/647: creat d0/fbd x:0 0 0 2024-04-05T03:02:50.999 INFO:tasks.workunit.client.1.smithi203.stdout:2/883: unlink d5/d10/d35/d2d/d7c/d29/db1/cf7 0 2024-04-05T03:02:50.999 INFO:tasks.workunit.client.0.smithi153.stdout:3/475: mknod d1d/d32/d93/d57/cab 0 2024-04-05T03:02:51.000 INFO:tasks.workunit.client.1.smithi203.stdout:8/871: creat d1/d4b/d5f/daf/f103 x:0 0 0 2024-04-05T03:02:51.001 INFO:tasks.workunit.client.1.smithi203.stdout:0/792: mknod dd/d17/d1d/d26/ce6 0 2024-04-05T03:02:51.001 INFO:tasks.workunit.client.1.smithi203.stdout:3/648: mknod d0/dc/d1a/d1d/cbe 0 2024-04-05T03:02:51.003 INFO:tasks.workunit.client.1.smithi203.stdout:2/884: fsync d5/d63/f75 0 2024-04-05T03:02:51.004 INFO:tasks.workunit.client.0.smithi153.stdout:3/476: creat d1d/d32/d93/d3c/d3f/d90/d92/fac x:0 0 0 2024-04-05T03:02:51.004 INFO:tasks.workunit.client.1.smithi203.stdout:0/793: write dd/d17/f40 [3748843,19817] 0 2024-04-05T03:02:51.006 INFO:tasks.workunit.client.1.smithi203.stdout:8/872: unlink d1/d4b/d68/d75/dac/cb6 0 2024-04-05T03:02:51.006 INFO:tasks.workunit.client.1.smithi203.stdout:6/779: dwrite d0/da/d18/d38/d2a/fb1 [0,4194304] 0 2024-04-05T03:02:51.007 INFO:tasks.workunit.client.0.smithi153.stdout:3/477: chown d1d/d32/d93/d57/d6b/f67 20 1 2024-04-05T03:02:51.008 INFO:tasks.workunit.client.0.smithi153.stdout:0/462: dwrite d9/db/fd [0,4194304] 0 2024-04-05T03:02:51.011 INFO:tasks.workunit.client.1.smithi203.stdout:0/794: truncate dd/d17/d1d/d22/d53/d61/fe5 788032 0 2024-04-05T03:02:51.013 INFO:tasks.workunit.client.1.smithi203.stdout:3/649: symlink d0/dc/d1a/d1f/lbf 0 2024-04-05T03:02:51.017 INFO:tasks.workunit.client.1.smithi203.stdout:8/873: dwrite d1/d6/f7c [0,4194304] 0 2024-04-05T03:02:51.018 INFO:tasks.workunit.client.1.smithi203.stdout:9/791: link d13/d1b/d6b/d9e/c40 d13/c100 0 2024-04-05T03:02:51.018 INFO:tasks.workunit.client.0.smithi153.stdout:3/478: dwrite d1d/d32/d93/d57/fa0 [0,4194304] 0 2024-04-05T03:02:51.020 INFO:tasks.workunit.client.0.smithi153.stdout:0/463: rename d9/db/d42/c5a to d9/df/d1e/d45/d62/d79/c86 0 2024-04-05T03:02:51.021 INFO:tasks.workunit.client.1.smithi203.stdout:9/792: write d13/d1b/d33/d64/f6d [2047383,96003] 0 2024-04-05T03:02:51.022 INFO:tasks.workunit.client.0.smithi153.stdout:6/500: dread d4/dc/df/d16/d36/d4d/f66 [0,4194304] 0 2024-04-05T03:02:51.025 INFO:tasks.workunit.client.0.smithi153.stdout:0/464: mknod d9/df/d1d/c87 0 2024-04-05T03:02:51.026 INFO:tasks.workunit.client.0.smithi153.stdout:3/479: stat d1d/d32/d93/d3c/d3f/d90/d92/d99/l9d 0 2024-04-05T03:02:51.026 INFO:tasks.workunit.client.1.smithi203.stdout:3/650: unlink d0/dc/d1a/d1d/l39 0 2024-04-05T03:02:51.032 INFO:tasks.workunit.client.1.smithi203.stdout:5/757: dread d3/d1f/d59/d64/f44 [0,4194304] 0 2024-04-05T03:02:51.034 INFO:tasks.workunit.client.0.smithi153.stdout:3/480: getdents d1d/d32/d93/d3c/d3f 0 2024-04-05T03:02:51.034 INFO:tasks.workunit.client.1.smithi203.stdout:8/874: dwrite d1/d12/d1a/f1d [0,4194304] 0 2024-04-05T03:02:51.034 INFO:tasks.workunit.client.0.smithi153.stdout:3/481: fdatasync d1d/d75/f79 0 2024-04-05T03:02:51.035 INFO:tasks.workunit.client.1.smithi203.stdout:9/793: dwrite d13/d5c/f9f [0,4194304] 0 2024-04-05T03:02:51.037 INFO:tasks.workunit.client.1.smithi203.stdout:3/651: creat d0/dc/d1a/d76/d35/fc0 x:0 0 0 2024-04-05T03:02:51.039 INFO:tasks.workunit.client.0.smithi153.stdout:0/465: write d9/f30 [614732,62826] 0 2024-04-05T03:02:51.040 INFO:tasks.workunit.client.0.smithi153.stdout:3/482: dwrite d1d/f21 [4194304,4194304] 0 2024-04-05T03:02:51.041 INFO:tasks.workunit.client.0.smithi153.stdout:0/466: chown d9/db/l31 9 1 2024-04-05T03:02:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:50 smithi203 ceph-mon[39911]: pgmap v10: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 243 MiB/s wr, 446 op/s 2024-04-05T03:02:51.047 INFO:tasks.workunit.client.1.smithi203.stdout:9/794: truncate d13/ffe 790489 0 2024-04-05T03:02:51.047 INFO:tasks.workunit.client.1.smithi203.stdout:6/780: write d0/da/d54/d4c/da1/fa9 [376046,19676] 0 2024-04-05T03:02:51.047 INFO:tasks.workunit.client.1.smithi203.stdout:0/795: write dd/d17/d1d/d26/d8c/d6d/dc4/fe2 [589569,125125] 0 2024-04-05T03:02:51.048 INFO:tasks.workunit.client.0.smithi153.stdout:0/467: symlink d9/db/d42/l88 0 2024-04-05T03:02:51.052 INFO:tasks.workunit.client.0.smithi153.stdout:0/468: mkdir d9/db/d42/d43/d89 0 2024-04-05T03:02:51.053 INFO:tasks.workunit.client.1.smithi203.stdout:9/795: creat d13/f101 x:0 0 0 2024-04-05T03:02:51.057 INFO:tasks.workunit.client.0.smithi153.stdout:0/469: dwrite d9/df/d1e/f80 [0,4194304] 0 2024-04-05T03:02:51.059 INFO:tasks.workunit.client.1.smithi203.stdout:6/781: mknod d0/da/d54/d4c/da1/db7/d95/ce9 0 2024-04-05T03:02:51.059 INFO:tasks.workunit.client.1.smithi203.stdout:5/758: dwrite d3/f4d [0,4194304] 0 2024-04-05T03:02:51.060 INFO:tasks.workunit.client.1.smithi203.stdout:9/796: symlink d13/d5c/dd1/d51/dcb/l102 0 2024-04-05T03:02:51.061 INFO:tasks.workunit.client.1.smithi203.stdout:8/875: dwrite d1/d12/d2f/f2a [0,4194304] 0 2024-04-05T03:02:51.062 INFO:tasks.workunit.client.0.smithi153.stdout:0/470: rename d9/df/d1d/d72 to d9/df/d1d/d61/d8a 0 2024-04-05T03:02:51.066 INFO:tasks.workunit.client.0.smithi153.stdout:8/423: sync 2024-04-05T03:02:51.066 INFO:tasks.workunit.client.0.smithi153.stdout:7/495: sync 2024-04-05T03:02:51.068 INFO:tasks.workunit.client.0.smithi153.stdout:7/496: fsync d2/da/d3a/d41/d7a/f3e 0 2024-04-05T03:02:51.068 INFO:tasks.workunit.client.0.smithi153.stdout:8/424: write d7/d2a/d50/f6b [129028,109411] 0 2024-04-05T03:02:51.069 INFO:tasks.workunit.client.0.smithi153.stdout:8/425: readlink d7/d85/l49 0 2024-04-05T03:02:51.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:50 smithi153 ceph-mon[29381]: pgmap v10: 65 pgs: 65 active+clean; 2.2 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 243 MiB/s wr, 446 op/s 2024-04-05T03:02:51.073 INFO:tasks.workunit.client.0.smithi153.stdout:8/426: rename d7/d13/d19/d1b/d33/f4c to d7/d13/f8e 0 2024-04-05T03:02:51.074 INFO:tasks.workunit.client.1.smithi203.stdout:9/797: dwrite d13/d5c/dd1/d51/da1/dba/fee [0,4194304] 0 2024-04-05T03:02:51.075 INFO:tasks.workunit.client.1.smithi203.stdout:5/759: link d3/d1f/d59/d64/l55 d3/d1f/d59/d64/lfd 0 2024-04-05T03:02:51.077 INFO:tasks.workunit.client.1.smithi203.stdout:9/798: chown d13/d1b/d6b/d9e/c40 490960562 1 2024-04-05T03:02:51.079 INFO:tasks.workunit.client.1.smithi203.stdout:9/799: dread - d13/d5c/dd1/d51/da1/dba/fd4 zero size 2024-04-05T03:02:51.082 INFO:tasks.workunit.client.1.smithi203.stdout:5/760: getdents d3 0 2024-04-05T03:02:51.083 INFO:tasks.workunit.client.1.smithi203.stdout:5/761: fdatasync d3/d4/fd1 0 2024-04-05T03:02:51.086 INFO:tasks.workunit.client.1.smithi203.stdout:9/800: dwrite d13/d1b/d33/d64/dfa/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:51.091 INFO:tasks.workunit.client.1.smithi203.stdout:5/762: dwrite d3/d4/dc/f37 [0,4194304] 0 2024-04-05T03:02:51.092 INFO:tasks.workunit.client.1.smithi203.stdout:5/763: readlink d3/d1f/d59/lfb 0 2024-04-05T03:02:51.095 INFO:tasks.workunit.client.1.smithi203.stdout:5/764: fsync d3/d4/dc/d67/db5/dda/fef 0 2024-04-05T03:02:51.095 INFO:tasks.workunit.client.1.smithi203.stdout:9/801: dwrite d13/d5c/dd1/d31/f3e [4194304,4194304] 0 2024-04-05T03:02:51.096 INFO:tasks.workunit.client.0.smithi153.stdout:8/427: dread d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:51.096 INFO:tasks.workunit.client.1.smithi203.stdout:5/765: dread - d3/d4/dc/f79 zero size 2024-04-05T03:02:51.096 INFO:tasks.workunit.client.1.smithi203.stdout:9/802: stat d13/d5c/dd1/d51/lc9 0 2024-04-05T03:02:51.097 INFO:tasks.workunit.client.1.smithi203.stdout:5/766: stat d3/d1f/d59/d64/d12/le9 0 2024-04-05T03:02:51.098 INFO:tasks.workunit.client.1.smithi203.stdout:9/803: chown d13/d5c/dd1/cb3 9445233 1 2024-04-05T03:02:51.101 INFO:tasks.workunit.client.1.smithi203.stdout:5/767: mkdir d3/d4/dd6/dfe 0 2024-04-05T03:02:51.102 INFO:tasks.workunit.client.0.smithi153.stdout:8/428: dread d7/d13/d19/d1b/d33/f59 [0,4194304] 0 2024-04-05T03:02:51.104 INFO:tasks.workunit.client.1.smithi203.stdout:9/804: mkdir d13/d1b/d6b/d9e/d59/d5f/d103 0 2024-04-05T03:02:51.104 INFO:tasks.workunit.client.1.smithi203.stdout:3/652: dwrite d0/dc/f16 [0,4194304] 0 2024-04-05T03:02:51.104 INFO:tasks.workunit.client.0.smithi153.stdout:7/497: dread d2/da/d3a/d41/f19 [0,4194304] 0 2024-04-05T03:02:51.105 INFO:tasks.workunit.client.1.smithi203.stdout:3/653: stat d0/d69 0 2024-04-05T03:02:51.106 INFO:tasks.workunit.client.1.smithi203.stdout:5/768: mkdir d3/d5f/d92/dff 0 2024-04-05T03:02:51.109 INFO:tasks.workunit.client.0.smithi153.stdout:7/498: mkdir d2/da/d11/d88/da6 0 2024-04-05T03:02:51.110 INFO:tasks.workunit.client.1.smithi203.stdout:5/769: creat d3/d4/dc/f100 x:0 0 0 2024-04-05T03:02:51.112 INFO:tasks.workunit.client.0.smithi153.stdout:7/499: chown d2/da/d47/c65 200344905 1 2024-04-05T03:02:51.114 INFO:tasks.workunit.client.1.smithi203.stdout:5/770: chown d3/d4/dc/d67/f7f 42 1 2024-04-05T03:02:51.115 INFO:tasks.workunit.client.0.smithi153.stdout:7/500: mkdir d2/d51/da7 0 2024-04-05T03:02:51.117 INFO:tasks.workunit.client.1.smithi203.stdout:3/654: dwrite d0/d69/f88 [0,4194304] 0 2024-04-05T03:02:51.118 INFO:tasks.workunit.client.1.smithi203.stdout:3/655: readlink d0/dc/d1a/d1f/lad 0 2024-04-05T03:02:51.118 INFO:tasks.workunit.client.0.smithi153.stdout:7/501: mkdir d2/da/d3a/d41/da8 0 2024-04-05T03:02:51.119 INFO:tasks.workunit.client.1.smithi203.stdout:5/771: creat d3/d4/dd6/dfe/f101 x:0 0 0 2024-04-05T03:02:51.119 INFO:tasks.workunit.client.1.smithi203.stdout:0/796: truncate dd/d17/d1d/d22/d53/db0/f3c 1847023 0 2024-04-05T03:02:51.119 INFO:tasks.workunit.client.0.smithi153.stdout:7/502: fdatasync d2/da/d3a/d41/d7a/d54/fa2 0 2024-04-05T03:02:51.120 INFO:tasks.workunit.client.0.smithi153.stdout:7/503: readlink l1 0 2024-04-05T03:02:51.121 INFO:tasks.workunit.client.1.smithi203.stdout:3/656: read - d0/dc/d38/d5d/d66/f93 zero size 2024-04-05T03:02:51.124 INFO:tasks.workunit.client.1.smithi203.stdout:0/797: mknod dd/d17/d1d/d26/d28/ce7 0 2024-04-05T03:02:51.125 INFO:tasks.workunit.client.1.smithi203.stdout:3/657: write d0/dc/d1a/d8d/d96/f51 [5166962,103837] 0 2024-04-05T03:02:51.127 INFO:tasks.workunit.client.1.smithi203.stdout:6/782: dwrite d0/da/d54/d4c/da1/db7/d71/f79 [4194304,4194304] 0 2024-04-05T03:02:51.127 INFO:tasks.workunit.client.0.smithi153.stdout:7/504: dwrite d2/f5a [0,4194304] 0 2024-04-05T03:02:51.128 INFO:tasks.workunit.client.1.smithi203.stdout:0/798: chown dd/d17/d1d/d22/d53/d7f/caf 15311 1 2024-04-05T03:02:51.133 INFO:tasks.workunit.client.0.smithi153.stdout:7/505: dread d2/d51/f79 [0,4194304] 0 2024-04-05T03:02:51.135 INFO:tasks.workunit.client.1.smithi203.stdout:0/799: symlink dd/d17/d1d/d22/d53/d61/le8 0 2024-04-05T03:02:51.135 INFO:tasks.workunit.client.1.smithi203.stdout:3/658: creat d0/dc/d1a/d1f/da1/daf/fc1 x:0 0 0 2024-04-05T03:02:51.136 INFO:tasks.workunit.client.1.smithi203.stdout:6/783: mknod d0/da/d54/d4c/d7f/dcc/cea 0 2024-04-05T03:02:51.137 INFO:tasks.workunit.client.1.smithi203.stdout:6/784: stat d0/da/d54/f4b 0 2024-04-05T03:02:51.138 INFO:tasks.workunit.client.1.smithi203.stdout:3/659: chown d0/d21/lb5 108639 1 2024-04-05T03:02:51.138 INFO:tasks.workunit.client.1.smithi203.stdout:6/785: dread - d0/da/d18/d92/fe7 zero size 2024-04-05T03:02:51.140 INFO:tasks.workunit.client.1.smithi203.stdout:0/800: mknod dd/db1/ce9 0 2024-04-05T03:02:51.140 INFO:tasks.workunit.client.0.smithi153.stdout:7/506: dwrite d2/da/d3a/d41/f6c [0,4194304] 0 2024-04-05T03:02:51.141 INFO:tasks.workunit.client.1.smithi203.stdout:8/876: truncate d1/d6/d8f/fb0 141579 0 2024-04-05T03:02:51.142 INFO:tasks.workunit.client.1.smithi203.stdout:8/877: write d1/d12/d17/d47/d53/de0/ff1 [2901756,74596] 0 2024-04-05T03:02:51.143 INFO:tasks.workunit.client.1.smithi203.stdout:6/786: unlink d0/db4/fc8 0 2024-04-05T03:02:51.144 INFO:tasks.workunit.client.0.smithi153.stdout:7/507: mkdir d2/da/d47/da9 0 2024-04-05T03:02:51.144 INFO:tasks.workunit.client.1.smithi203.stdout:6/787: dread - d0/da/d54/d4c/daf/fc9 zero size 2024-04-05T03:02:51.146 INFO:tasks.workunit.client.0.smithi153.stdout:7/508: read d2/da/d3a/f46 [231281,77734] 0 2024-04-05T03:02:51.148 INFO:tasks.workunit.client.0.smithi153.stdout:7/509: write d2/da/d11/d39/f63 [3029283,35041] 0 2024-04-05T03:02:51.151 INFO:tasks.workunit.client.1.smithi203.stdout:8/878: dwrite d1/d4b/d5f/daf/f103 [0,4194304] 0 2024-04-05T03:02:51.151 INFO:tasks.workunit.client.1.smithi203.stdout:0/801: link dd/d17/d1d/d26/d8c/d6d/fa9 dd/d17/d1d/d22/d53/d7f/fea 0 2024-04-05T03:02:51.151 INFO:tasks.workunit.client.0.smithi153.stdout:7/510: symlink d2/da/d3a/d41/da8/laa 0 2024-04-05T03:02:51.153 INFO:tasks.workunit.client.1.smithi203.stdout:8/879: write d1/d12/d17/d10/f6e [121455,53527] 0 2024-04-05T03:02:51.153 INFO:tasks.workunit.client.1.smithi203.stdout:6/788: mkdir d0/da/d18/d38/d2a/deb 0 2024-04-05T03:02:51.154 INFO:tasks.workunit.client.0.smithi153.stdout:7/511: rmdir d2/da/d47/d5d/d7c 39 2024-04-05T03:02:51.155 INFO:tasks.workunit.client.1.smithi203.stdout:0/802: unlink dd/d17/d1d/d22/f9c 0 2024-04-05T03:02:51.155 INFO:tasks.workunit.client.0.smithi153.stdout:7/512: write d2/da/d3a/d41/d7a/f4e [3749195,38994] 0 2024-04-05T03:02:51.156 INFO:tasks.workunit.client.1.smithi203.stdout:8/880: chown d1/d12/d2f/d1f/la1 19460 1 2024-04-05T03:02:51.158 INFO:tasks.workunit.client.1.smithi203.stdout:3/660: getdents d0/dc/d38/d5d/d66/d80 0 2024-04-05T03:02:51.160 INFO:tasks.workunit.client.1.smithi203.stdout:6/789: dread d0/da/d18/d38/d86/f93 [0,4194304] 0 2024-04-05T03:02:51.161 INFO:tasks.workunit.client.1.smithi203.stdout:4/579: sync 2024-04-05T03:02:51.163 INFO:tasks.workunit.client.1.smithi203.stdout:0/803: write dd/d17/d1d/d26/d8c/d6d/fa9 [2304947,1392] 0 2024-04-05T03:02:51.165 INFO:tasks.workunit.client.1.smithi203.stdout:3/661: unlink d0/dc/d1a/d76/f9f 0 2024-04-05T03:02:51.167 INFO:tasks.workunit.client.1.smithi203.stdout:9/805: write d13/d5c/d9d/fcd [1612921,23254] 0 2024-04-05T03:02:51.167 INFO:tasks.workunit.client.1.smithi203.stdout:4/580: truncate da/d29/d2e/f7a 861597 0 2024-04-05T03:02:51.167 INFO:tasks.workunit.client.1.smithi203.stdout:8/881: dwrite d1/d4b/d5f/fe9 [0,4194304] 0 2024-04-05T03:02:51.169 INFO:tasks.workunit.client.0.smithi153.stdout:7/513: dread d2/da/d3a/d41/d31/f73 [0,4194304] 0 2024-04-05T03:02:51.170 INFO:tasks.workunit.client.1.smithi203.stdout:3/662: read - d0/d21/d81/f7b zero size 2024-04-05T03:02:51.170 INFO:tasks.workunit.client.0.smithi153.stdout:8/429: fsync d7/d13/f8e 0 2024-04-05T03:02:51.178 INFO:tasks.workunit.client.1.smithi203.stdout:5/772: dread d3/d4/f8e [0,4194304] 0 2024-04-05T03:02:51.178 INFO:tasks.workunit.client.1.smithi203.stdout:0/804: dread dd/d17/d1d/d22/fac [0,4194304] 0 2024-04-05T03:02:51.180 INFO:tasks.workunit.client.1.smithi203.stdout:5/773: chown d3/d1f/d59/d64/f63 5852 1 2024-04-05T03:02:51.180 INFO:tasks.workunit.client.1.smithi203.stdout:4/581: dwrite da/d29/d2e/d53/f97 [0,4194304] 0 2024-04-05T03:02:51.202 INFO:tasks.workunit.client.1.smithi203.stdout:6/790: dread d0/da/d54/d53/f6a [0,4194304] 0 2024-04-05T03:02:51.203 INFO:tasks.workunit.client.0.smithi153.stdout:7/514: dread d2/da/d47/f92 [0,4194304] 0 2024-04-05T03:02:51.207 INFO:tasks.workunit.client.1.smithi203.stdout:8/882: dread d1/d12/d17/d10/f49 [0,4194304] 0 2024-04-05T03:02:51.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/515: dwrite d2/da/d11/f2b [0,4194304] 0 2024-04-05T03:02:51.212 INFO:tasks.workunit.client.0.smithi153.stdout:7/516: link d2/da/d3a/d41/d31/f73 d2/da/d11/fab 0 2024-04-05T03:02:51.213 INFO:tasks.workunit.client.1.smithi203.stdout:1/765: sync 2024-04-05T03:02:51.214 INFO:tasks.workunit.client.1.smithi203.stdout:1/766: read - d7/d24/d33/d48/d4b/f94 zero size 2024-04-05T03:02:51.215 INFO:tasks.workunit.client.0.smithi153.stdout:5/480: sync 2024-04-05T03:02:51.216 INFO:tasks.workunit.client.1.smithi203.stdout:1/767: chown d7/d24/d33/d48/d4b/d6b/cdb 23 1 2024-04-05T03:02:51.218 INFO:tasks.workunit.client.0.smithi153.stdout:5/481: creat d0/d25/d68/fae x:0 0 0 2024-04-05T03:02:51.220 INFO:tasks.workunit.client.0.smithi153.stdout:5/482: unlink d0/d4e/f90 0 2024-04-05T03:02:51.223 INFO:tasks.workunit.client.0.smithi153.stdout:5/483: dread d0/d25/d4d/d62/d37/f4f [0,4194304] 0 2024-04-05T03:02:51.224 INFO:tasks.workunit.client.0.smithi153.stdout:5/484: chown d0/d25/d4d/d62/c5b 5364 1 2024-04-05T03:02:51.225 INFO:tasks.workunit.client.0.smithi153.stdout:7/517: truncate d2/f5 3950859 0 2024-04-05T03:02:51.227 INFO:tasks.workunit.client.0.smithi153.stdout:7/518: symlink d2/da/lac 0 2024-04-05T03:02:51.228 INFO:tasks.workunit.client.0.smithi153.stdout:5/485: dread d0/d25/d4d/d62/d37/f6a [0,4194304] 0 2024-04-05T03:02:51.229 INFO:tasks.workunit.client.0.smithi153.stdout:5/486: write d0/d25/d4d/d62/f80 [351161,45389] 0 2024-04-05T03:02:51.229 INFO:tasks.workunit.client.0.smithi153.stdout:7/519: creat d2/da/d3a/d41/da8/fad x:0 0 0 2024-04-05T03:02:51.230 INFO:tasks.workunit.client.0.smithi153.stdout:7/520: write d2/da/d3a/d41/f1e [2703125,46144] 0 2024-04-05T03:02:51.231 INFO:tasks.workunit.client.0.smithi153.stdout:5/487: write d0/d9/d22/f33 [1748530,97471] 0 2024-04-05T03:02:51.232 INFO:tasks.workunit.client.0.smithi153.stdout:7/521: creat d2/da/d3a/fae x:0 0 0 2024-04-05T03:02:51.233 INFO:tasks.workunit.client.0.smithi153.stdout:7/522: readlink d2/da/l29 0 2024-04-05T03:02:51.235 INFO:tasks.workunit.client.0.smithi153.stdout:5/488: rename d0/d2c/f53 to d0/d25/d4d/faf 0 2024-04-05T03:02:51.236 INFO:tasks.workunit.client.0.smithi153.stdout:7/523: chown d2/l7 14332 1 2024-04-05T03:02:51.237 INFO:tasks.workunit.client.0.smithi153.stdout:7/524: readlink d2/d9b/l91 0 2024-04-05T03:02:51.237 INFO:tasks.workunit.client.1.smithi203.stdout:9/806: truncate d13/d5c/dd1/d51/f53 2832310 0 2024-04-05T03:02:51.238 INFO:tasks.workunit.client.0.smithi153.stdout:5/489: symlink d0/d9/lb0 0 2024-04-05T03:02:51.241 INFO:tasks.workunit.client.0.smithi153.stdout:7/525: dread d2/da/d11/f72 [0,4194304] 0 2024-04-05T03:02:51.242 INFO:tasks.workunit.client.1.smithi203.stdout:9/807: dwrite d13/d5c/dd1/d51/f91 [0,4194304] 0 2024-04-05T03:02:51.243 INFO:tasks.workunit.client.0.smithi153.stdout:7/526: symlink d2/d51/laf 0 2024-04-05T03:02:51.244 INFO:tasks.workunit.client.0.smithi153.stdout:7/527: chown d2/da/d47/fa1 81 1 2024-04-05T03:02:51.247 INFO:tasks.workunit.client.1.smithi203.stdout:9/808: dread d13/d1b/d6b/d9e/d77/f7d [0,4194304] 0 2024-04-05T03:02:51.249 INFO:tasks.workunit.client.0.smithi153.stdout:7/528: rename d2/da/d3a/d41/d7a/f3e to d2/d51/d93/fb0 0 2024-04-05T03:02:51.249 INFO:tasks.workunit.client.1.smithi203.stdout:9/809: read d13/d1b/d33/d64/dfa/f49 [572316,48112] 0 2024-04-05T03:02:51.250 INFO:tasks.workunit.client.1.smithi203.stdout:9/810: chown d13/d5c/dd1/d51/l9a 15576 1 2024-04-05T03:02:51.251 INFO:tasks.workunit.client.1.smithi203.stdout:9/811: readlink d13/d1b/d6b/d9e/d59/d5f/l83 0 2024-04-05T03:02:51.252 INFO:tasks.workunit.client.0.smithi153.stdout:7/529: creat d2/fb1 x:0 0 0 2024-04-05T03:02:51.254 INFO:tasks.workunit.client.0.smithi153.stdout:7/530: creat d2/da/d47/da9/fb2 x:0 0 0 2024-04-05T03:02:51.255 INFO:tasks.workunit.client.0.smithi153.stdout:7/531: truncate d2/d51/d93/f74 481243 0 2024-04-05T03:02:51.256 INFO:tasks.workunit.client.0.smithi153.stdout:7/532: chown d2/d9b/d6f/l84 58041 1 2024-04-05T03:02:51.264 INFO:tasks.workunit.client.1.smithi203.stdout:2/885: sync 2024-04-05T03:02:51.268 INFO:tasks.workunit.client.0.smithi153.stdout:7/533: write d2/da/d3a/d41/d7a/f4e [6541567,125297] 0 2024-04-05T03:02:51.281 INFO:tasks.workunit.client.0.smithi153.stdout:7/534: dread d2/da/d11/f52 [0,4194304] 0 2024-04-05T03:02:51.285 INFO:tasks.workunit.client.0.smithi153.stdout:7/535: rename d2/f5a to d2/da/d3a/d41/d7a/d54/d85/fb3 0 2024-04-05T03:02:51.288 INFO:tasks.workunit.client.0.smithi153.stdout:7/536: dread d2/f1d [0,4194304] 0 2024-04-05T03:02:51.292 INFO:tasks.workunit.client.0.smithi153.stdout:7/537: unlink d2/da/d47/d5d/d7c/f87 0 2024-04-05T03:02:51.292 INFO:tasks.workunit.client.0.smithi153.stdout:7/538: readlink d2/l7 0 2024-04-05T03:02:51.292 INFO:tasks.workunit.client.0.smithi153.stdout:7/539: dread - d2/f40 zero size 2024-04-05T03:02:51.294 INFO:tasks.workunit.client.0.smithi153.stdout:7/540: getdents d2/d51/d59 0 2024-04-05T03:02:51.296 INFO:tasks.workunit.client.0.smithi153.stdout:7/541: chown d2/d51/d93/c9c 4445478 1 2024-04-05T03:02:51.297 INFO:tasks.workunit.client.0.smithi153.stdout:7/542: mknod d2/da/d3a/d41/da8/cb4 0 2024-04-05T03:02:51.299 INFO:tasks.workunit.client.0.smithi153.stdout:7/543: symlink d2/da/d47/da3/lb5 0 2024-04-05T03:02:51.301 INFO:tasks.workunit.client.0.smithi153.stdout:7/544: creat d2/da/d11/d39/fb6 x:0 0 0 2024-04-05T03:02:51.304 INFO:tasks.workunit.client.1.smithi203.stdout:5/774: mknod d3/d1f/d59/d60/c102 0 2024-04-05T03:02:51.304 INFO:tasks.workunit.client.0.smithi153.stdout:7/545: rename d2/d51/d93/c9c to d2/da/d3a/d41/d7a/d54/cb7 0 2024-04-05T03:02:51.306 INFO:tasks.workunit.client.1.smithi203.stdout:4/582: symlink da/d29/d2e/d53/laf 0 2024-04-05T03:02:51.307 INFO:tasks.workunit.client.0.smithi153.stdout:7/546: fsync d2/da/d3a/d41/d7a/f3c 0 2024-04-05T03:02:51.309 INFO:tasks.workunit.client.0.smithi153.stdout:7/547: write d2/da/d11/f52 [4360164,76447] 0 2024-04-05T03:02:51.311 INFO:tasks.workunit.client.0.smithi153.stdout:7/548: symlink d2/da/d47/d5d/d7c/d8d/d95/lb8 0 2024-04-05T03:02:51.315 INFO:tasks.workunit.client.1.smithi203.stdout:9/812: mkdir d13/d5c/dd1/d51/dcb/d104 0 2024-04-05T03:02:51.316 INFO:tasks.workunit.client.0.smithi153.stdout:7/549: dread d2/da/d3a/d41/d31/f9a [0,4194304] 0 2024-04-05T03:02:51.316 INFO:tasks.workunit.client.1.smithi203.stdout:2/886: mknod d5/d10/d35/df4/c130 0 2024-04-05T03:02:51.318 INFO:tasks.workunit.client.1.smithi203.stdout:2/887: write d5/d10/d35/d36/f3b [4649179,58772] 0 2024-04-05T03:02:51.321 INFO:tasks.workunit.client.1.smithi203.stdout:8/883: dread d1/d12/d2f/d5b/fb4 [0,4194304] 0 2024-04-05T03:02:51.321 INFO:tasks.workunit.client.1.smithi203.stdout:1/768: mkdir d7/d14/d83/deb/df8 0 2024-04-05T03:02:51.321 INFO:tasks.workunit.client.1.smithi203.stdout:1/769: readlink d7/le9 0 2024-04-05T03:02:51.323 INFO:tasks.workunit.client.1.smithi203.stdout:9/813: write f12 [1503006,96917] 0 2024-04-05T03:02:51.324 INFO:tasks.workunit.client.1.smithi203.stdout:0/805: creat dd/d17/d1d/d22/d45/d51/feb x:0 0 0 2024-04-05T03:02:51.325 INFO:tasks.workunit.client.1.smithi203.stdout:5/775: dread d3/d1f/d59/d64/d12/f46 [0,4194304] 0 2024-04-05T03:02:51.327 INFO:tasks.workunit.client.1.smithi203.stdout:4/583: fsync da/f28 0 2024-04-05T03:02:51.328 INFO:tasks.workunit.client.1.smithi203.stdout:8/884: dwrite d1/d12/d1a/f1e [4194304,4194304] 0 2024-04-05T03:02:51.330 INFO:tasks.workunit.client.1.smithi203.stdout:8/885: stat d1/d6/fe8 0 2024-04-05T03:02:51.331 INFO:tasks.workunit.client.1.smithi203.stdout:6/791: link d0/da/d54/d4c/da1/fa9 d0/da/d18/d38/d2a/fec 0 2024-04-05T03:02:51.331 INFO:tasks.workunit.client.0.smithi153.stdout:7/550: dread d2/da/d3a/d41/d7a/f4e [0,4194304] 0 2024-04-05T03:02:51.332 INFO:tasks.workunit.client.1.smithi203.stdout:3/663: write d0/d21/d81/f59 [951793,107599] 0 2024-04-05T03:02:51.332 INFO:tasks.workunit.client.1.smithi203.stdout:9/814: dread d13/d1b/d33/d64/fc6 [0,4194304] 0 2024-04-05T03:02:51.333 INFO:tasks.workunit.client.0.smithi153.stdout:7/551: truncate d2/da/d3a/d41/da8/fad 673224 0 2024-04-05T03:02:51.339 INFO:tasks.workunit.client.1.smithi203.stdout:4/584: dwrite da/d29/d6a/d80/f9c [0,4194304] 0 2024-04-05T03:02:51.343 INFO:tasks.workunit.client.1.smithi203.stdout:3/664: dwrite d0/dc/d38/d5d/d66/f7e [0,4194304] 0 2024-04-05T03:02:51.346 INFO:tasks.workunit.client.1.smithi203.stdout:8/886: creat d1/d12/d17/d47/d53/dd0/f104 x:0 0 0 2024-04-05T03:02:51.354 INFO:tasks.workunit.client.0.smithi153.stdout:1/425: sync 2024-04-05T03:02:51.354 INFO:tasks.workunit.client.0.smithi153.stdout:2/476: sync 2024-04-05T03:02:51.354 INFO:tasks.workunit.client.1.smithi203.stdout:9/815: symlink d13/d5c/dd1/d51/dcb/l105 0 2024-04-05T03:02:51.357 INFO:tasks.workunit.client.0.smithi153.stdout:2/477: mknod d9/da/d37/c9e 0 2024-04-05T03:02:51.357 INFO:tasks.workunit.client.0.smithi153.stdout:1/426: symlink d5/d8/d20/d4b/d1e/l87 0 2024-04-05T03:02:51.358 INFO:tasks.workunit.client.0.smithi153.stdout:2/478: write d9/da/d19/d2e/d48/f93 [621992,27599] 0 2024-04-05T03:02:51.359 INFO:tasks.workunit.client.0.smithi153.stdout:1/427: chown d5/d8/d20/d4b/d18/d16/d2c 527 1 2024-04-05T03:02:51.359 INFO:tasks.workunit.client.1.smithi203.stdout:3/665: fdatasync d0/d21/d81/f5b 0 2024-04-05T03:02:51.360 INFO:tasks.workunit.client.1.smithi203.stdout:8/887: rmdir d1/d4b/d68/d75 39 2024-04-05T03:02:51.361 INFO:tasks.workunit.client.0.smithi153.stdout:2/479: mknod d9/c9f 0 2024-04-05T03:02:51.364 INFO:tasks.workunit.client.0.smithi153.stdout:1/428: creat d5/d8/d20/f88 x:0 0 0 2024-04-05T03:02:51.364 INFO:tasks.workunit.client.0.smithi153.stdout:7/552: dread d2/da/d3a/d41/f8c [0,4194304] 0 2024-04-05T03:02:51.365 INFO:tasks.workunit.client.1.smithi203.stdout:4/585: mkdir da/d29/d2e/db0 0 2024-04-05T03:02:51.365 INFO:tasks.workunit.client.1.smithi203.stdout:1/770: write d7/d14/f29 [498837,122031] 0 2024-04-05T03:02:51.366 INFO:tasks.workunit.client.0.smithi153.stdout:2/480: fdatasync d9/da/f64 0 2024-04-05T03:02:51.367 INFO:tasks.workunit.client.1.smithi203.stdout:1/771: read - d7/d14/d83/fd4 zero size 2024-04-05T03:02:51.367 INFO:tasks.workunit.client.1.smithi203.stdout:3/666: unlink d0/d41/f4a 0 2024-04-05T03:02:51.369 INFO:tasks.workunit.client.1.smithi203.stdout:3/667: write d0/d41/f8e [662779,110120] 0 2024-04-05T03:02:51.370 INFO:tasks.workunit.client.1.smithi203.stdout:2/888: write d5/d10/d1a/d57/fac [1759465,69475] 0 2024-04-05T03:02:51.371 INFO:tasks.workunit.client.0.smithi153.stdout:1/429: creat d5/dd/f89 x:0 0 0 2024-04-05T03:02:51.372 INFO:tasks.workunit.client.1.smithi203.stdout:6/792: rename d0/da/c88 to d0/ced 0 2024-04-05T03:02:51.373 INFO:tasks.workunit.client.0.smithi153.stdout:1/430: chown d5/f84 47 1 2024-04-05T03:02:51.373 INFO:tasks.workunit.client.1.smithi203.stdout:2/889: dread - d5/d66/d7d/f116 zero size 2024-04-05T03:02:51.374 INFO:tasks.workunit.client.0.smithi153.stdout:7/553: getdents d2/da/d47/da9 0 2024-04-05T03:02:51.374 INFO:tasks.workunit.client.1.smithi203.stdout:0/806: dread f6 [0,4194304] 0 2024-04-05T03:02:51.375 INFO:tasks.workunit.client.1.smithi203.stdout:2/890: dread - d5/f6f zero size 2024-04-05T03:02:51.376 INFO:tasks.workunit.client.0.smithi153.stdout:2/481: dwrite d9/d77/f58 [0,4194304] 0 2024-04-05T03:02:51.376 INFO:tasks.workunit.client.1.smithi203.stdout:8/888: dread d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:51.379 INFO:tasks.workunit.client.1.smithi203.stdout:6/793: creat d0/da/d18/d38/d2a/d2c/fee x:0 0 0 2024-04-05T03:02:51.380 INFO:tasks.workunit.client.1.smithi203.stdout:9/816: write d13/d1b/d33/d64/f75 [1546040,95105] 0 2024-04-05T03:02:51.381 INFO:tasks.workunit.client.0.smithi153.stdout:2/482: rename d9/da/fc to d9/da/d78/fa0 0 2024-04-05T03:02:51.382 INFO:tasks.workunit.client.0.smithi153.stdout:7/554: mknod d2/da/d11/cb9 0 2024-04-05T03:02:51.383 INFO:tasks.workunit.client.0.smithi153.stdout:2/483: chown d9/da/d94 50397 1 2024-04-05T03:02:51.383 INFO:tasks.workunit.client.1.smithi203.stdout:9/817: chown d13/d1b/d33/f3f 173861731 1 2024-04-05T03:02:51.383 INFO:tasks.workunit.client.0.smithi153.stdout:1/431: dwrite d5/d8/d20/d3f/f5c [4194304,4194304] 0 2024-04-05T03:02:51.384 INFO:tasks.workunit.client.1.smithi203.stdout:1/772: truncate d7/d24/d33/d48/d4b/d6b/f98 1225339 0 2024-04-05T03:02:51.386 INFO:tasks.workunit.client.1.smithi203.stdout:2/891: fsync d5/d10/d35/d36/db6/d87/f89 0 2024-04-05T03:02:51.386 INFO:tasks.workunit.client.1.smithi203.stdout:3/668: link d0/d41/faa d0/dc/d38/fc2 0 2024-04-05T03:02:51.388 INFO:tasks.workunit.client.0.smithi153.stdout:7/555: symlink d2/da/d3a/d41/d31/lba 0 2024-04-05T03:02:51.389 INFO:tasks.workunit.client.1.smithi203.stdout:4/586: dread da/d29/d2e/d57/f66 [0,4194304] 0 2024-04-05T03:02:51.390 INFO:tasks.workunit.client.0.smithi153.stdout:1/432: symlink d5/d8/d20/d4b/d18/d16/d45/d68/l8a 0 2024-04-05T03:02:51.391 INFO:tasks.workunit.client.1.smithi203.stdout:9/818: unlink d13/f2a 0 2024-04-05T03:02:51.391 INFO:tasks.workunit.client.1.smithi203.stdout:3/669: chown d0/dc/d1a/d1d 192136 1 2024-04-05T03:02:51.392 INFO:tasks.workunit.client.0.smithi153.stdout:7/556: creat d2/da/d11/d88/fbb x:0 0 0 2024-04-05T03:02:51.392 INFO:tasks.workunit.client.1.smithi203.stdout:1/773: dread d7/d14/d2b/d58/d7a/f8a [0,4194304] 0 2024-04-05T03:02:51.393 INFO:tasks.workunit.client.1.smithi203.stdout:6/794: dwrite d0/da/d54/d4c/da1/db7/fbf [0,4194304] 0 2024-04-05T03:02:51.396 INFO:tasks.workunit.client.1.smithi203.stdout:2/892: mknod d5/d10/d35/d36/d41/df8/c131 0 2024-04-05T03:02:51.397 INFO:tasks.workunit.client.1.smithi203.stdout:4/587: truncate da/f93 270148 0 2024-04-05T03:02:51.397 INFO:tasks.workunit.client.0.smithi153.stdout:7/557: creat d2/da/d47/fbc x:0 0 0 2024-04-05T03:02:51.399 INFO:tasks.workunit.client.1.smithi203.stdout:4/588: write da/d29/d6a/d80/fa0 [894613,23489] 0 2024-04-05T03:02:51.401 INFO:tasks.workunit.client.1.smithi203.stdout:1/774: mkdir d7/d14/d2b/d58/d86/d91/da6/df9 0 2024-04-05T03:02:51.403 INFO:tasks.workunit.client.1.smithi203.stdout:8/889: getdents d1/d12/d1a/d6d 0 2024-04-05T03:02:51.405 INFO:tasks.workunit.client.0.smithi153.stdout:7/558: dwrite d2/da/d3a/d41/f1e [0,4194304] 0 2024-04-05T03:02:51.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/807: dwrite dd/d17/d1d/d26/d28/f2d [0,4194304] 0 2024-04-05T03:02:51.408 INFO:tasks.workunit.client.0.smithi153.stdout:1/433: truncate d5/d8/d20/d4b/d18/d16/f5b 2830890 0 2024-04-05T03:02:51.411 INFO:tasks.workunit.client.0.smithi153.stdout:1/434: chown d5/dd/c58 427 1 2024-04-05T03:02:51.412 INFO:tasks.workunit.client.0.smithi153.stdout:7/559: dread d2/da/d11/f2b [0,4194304] 0 2024-04-05T03:02:51.412 INFO:tasks.workunit.client.0.smithi153.stdout:1/435: write f4 [1107491,53686] 0 2024-04-05T03:02:51.413 INFO:tasks.workunit.client.0.smithi153.stdout:7/560: chown d2/da/d47/d5d/d7c 3988788 1 2024-04-05T03:02:51.416 INFO:tasks.workunit.client.0.smithi153.stdout:7/561: chown d2/da/d11/f2d 35889 1 2024-04-05T03:02:51.417 INFO:tasks.workunit.client.0.smithi153.stdout:1/436: write d5/d8/d20/d3f/f5c [2176991,1608] 0 2024-04-05T03:02:51.419 INFO:tasks.workunit.client.0.smithi153.stdout:1/437: chown d5/d8/d20/d4b/d18/d16/d45/f4a 24393 1 2024-04-05T03:02:51.419 INFO:tasks.workunit.client.1.smithi203.stdout:1/775: symlink d7/d24/d33/d48/d4b/d6b/db9/lfa 0 2024-04-05T03:02:51.419 INFO:tasks.workunit.client.1.smithi203.stdout:0/808: fdatasync dd/d93/fbc 0 2024-04-05T03:02:51.421 INFO:tasks.workunit.client.1.smithi203.stdout:3/670: getdents d0/dc/d1a/d76/d35 0 2024-04-05T03:02:51.423 INFO:tasks.workunit.client.1.smithi203.stdout:8/890: rename d1/d4b/d68/d75/d77/f87 to d1/d12/d17/d47/d53/dd0/f105 0 2024-04-05T03:02:51.425 INFO:tasks.workunit.client.1.smithi203.stdout:8/891: readlink d1/d12/d1a/la5 0 2024-04-05T03:02:51.427 INFO:tasks.workunit.client.0.smithi153.stdout:1/438: getdents d5/d8/d20/d2f 0 2024-04-05T03:02:51.427 INFO:tasks.workunit.client.0.smithi153.stdout:2/484: dread d9/da/f24 [0,4194304] 0 2024-04-05T03:02:51.427 INFO:tasks.workunit.client.1.smithi203.stdout:8/892: stat d1/d6/d2e/d5c/d99/dbc/fed 0 2024-04-05T03:02:51.429 INFO:tasks.workunit.client.1.smithi203.stdout:9/819: dwrite d13/d1b/d6b/d9e/d77/fa7 [0,4194304] 0 2024-04-05T03:02:51.429 INFO:tasks.workunit.client.1.smithi203.stdout:8/893: read - d1/d4b/d68/d75/d77/fe6 zero size 2024-04-05T03:02:51.430 INFO:tasks.workunit.client.0.smithi153.stdout:1/439: mkdir d5/d8/d8b 0 2024-04-05T03:02:51.431 INFO:tasks.workunit.client.1.smithi203.stdout:9/820: fsync d13/d1b/d6b/d9e/d59/dbc/fc5 0 2024-04-05T03:02:51.431 INFO:tasks.workunit.client.1.smithi203.stdout:8/894: write d1/d6/d2e/f6b [3086596,63742] 0 2024-04-05T03:02:51.433 INFO:tasks.workunit.client.0.smithi153.stdout:2/485: dwrite d9/d68/f7f [4194304,4194304] 0 2024-04-05T03:02:51.435 INFO:tasks.workunit.client.1.smithi203.stdout:8/895: write d1/d12/d17/d47/d53/d69/d9c/ffe [45718,7763] 0 2024-04-05T03:02:51.436 INFO:tasks.workunit.client.0.smithi153.stdout:2/486: write d9/da/d19/d2e/d48/f93 [1589858,140] 0 2024-04-05T03:02:51.437 INFO:tasks.workunit.client.0.smithi153.stdout:1/440: unlink d5/d8/d20/d4b/d18/f1c 0 2024-04-05T03:02:51.439 INFO:tasks.workunit.client.0.smithi153.stdout:1/441: write d5/dd/f89 [887743,114676] 0 2024-04-05T03:02:51.442 INFO:tasks.workunit.client.0.smithi153.stdout:7/562: dread d2/d51/d93/fb0 [0,4194304] 0 2024-04-05T03:02:51.443 INFO:tasks.workunit.client.0.smithi153.stdout:2/487: mkdir d9/da/d19/d2e/d7e/d53/da1 0 2024-04-05T03:02:51.443 INFO:tasks.workunit.client.1.smithi203.stdout:9/821: dwrite d13/d5c/dd1/f4c [0,4194304] 0 2024-04-05T03:02:51.446 INFO:tasks.workunit.client.0.smithi153.stdout:2/488: fsync d9/da/f64 0 2024-04-05T03:02:51.450 INFO:tasks.workunit.client.0.smithi153.stdout:2/489: mknod d9/d68/ca2 0 2024-04-05T03:02:51.450 INFO:tasks.workunit.client.1.smithi203.stdout:9/822: dread d13/d5c/dd1/d51/da1/dba/fee [0,4194304] 0 2024-04-05T03:02:51.452 INFO:tasks.workunit.client.1.smithi203.stdout:6/795: dread d0/da/d54/d4c/f5b [0,4194304] 0 2024-04-05T03:02:51.452 INFO:tasks.workunit.client.0.smithi153.stdout:7/563: getdents d2/da/d11/d88 0 2024-04-05T03:02:51.452 INFO:tasks.workunit.client.1.smithi203.stdout:6/796: fsync d0/da/d18/d38/fbe 0 2024-04-05T03:02:51.453 INFO:tasks.workunit.client.0.smithi153.stdout:7/564: chown d2/d9b/c5c 1094 1 2024-04-05T03:02:51.454 INFO:tasks.workunit.client.0.smithi153.stdout:2/490: truncate d9/da/d19/d2e/d7e/d7a/d59/f99 167962 0 2024-04-05T03:02:51.455 INFO:tasks.workunit.client.0.smithi153.stdout:7/565: write d2/da/d11/d39/f76 [275437,52665] 0 2024-04-05T03:02:51.455 INFO:tasks.workunit.client.0.smithi153.stdout:2/491: chown d9/da/fe 72518443 1 2024-04-05T03:02:51.457 INFO:tasks.workunit.client.0.smithi153.stdout:2/492: mkdir d9/d68/d98/da3 0 2024-04-05T03:02:51.460 INFO:tasks.workunit.client.0.smithi153.stdout:7/566: dwrite d2/da/d11/f2b [0,4194304] 0 2024-04-05T03:02:51.463 INFO:tasks.workunit.client.0.smithi153.stdout:9/389: sync 2024-04-05T03:02:51.463 INFO:tasks.workunit.client.0.smithi153.stdout:2/493: rename d9/da/d19/d2e/d7e/d7a/d6d/c80 to d9/da/d19/d2e/d7e/d7a/ca4 0 2024-04-05T03:02:51.463 INFO:tasks.workunit.client.0.smithi153.stdout:4/564: sync 2024-04-05T03:02:51.465 INFO:tasks.workunit.client.1.smithi203.stdout:3/671: unlink d0/d21/c24 0 2024-04-05T03:02:51.468 INFO:tasks.workunit.client.0.smithi153.stdout:2/494: mknod d9/da/d19/d2e/d7e/d53/ca5 0 2024-04-05T03:02:51.470 INFO:tasks.workunit.client.0.smithi153.stdout:7/567: dread d2/da/d47/f92 [0,4194304] 0 2024-04-05T03:02:51.470 INFO:tasks.workunit.client.0.smithi153.stdout:1/442: dread d5/dd/d1b/f1d [0,4194304] 0 2024-04-05T03:02:51.470 INFO:tasks.workunit.client.1.smithi203.stdout:4/589: link da/f4a da/d29/d2e/fb1 0 2024-04-05T03:02:51.470 INFO:tasks.workunit.client.0.smithi153.stdout:4/565: link da/db/d20/f21 da/de/da1/fc0 0 2024-04-05T03:02:51.475 INFO:tasks.workunit.client.0.smithi153.stdout:2/495: creat d9/da/d19/d2e/d4f/fa6 x:0 0 0 2024-04-05T03:02:51.476 INFO:tasks.workunit.client.0.smithi153.stdout:1/443: write d5/d8/d20/d3f/f5c [413902,84639] 0 2024-04-05T03:02:51.476 INFO:tasks.workunit.client.0.smithi153.stdout:9/390: dwrite d1/d9/d1b/f7d [0,4194304] 0 2024-04-05T03:02:51.477 INFO:tasks.workunit.client.1.smithi203.stdout:9/823: creat d13/d1b/d6b/d9e/d59/d5f/f106 x:0 0 0 2024-04-05T03:02:51.478 INFO:tasks.workunit.client.1.smithi203.stdout:6/797: symlink d0/da/d18/d38/d2a/d2c/lef 0 2024-04-05T03:02:51.481 INFO:tasks.workunit.client.0.smithi153.stdout:7/568: rmdir d2/da/d11/d9d 0 2024-04-05T03:02:51.481 INFO:tasks.workunit.client.0.smithi153.stdout:9/391: write d1/d9/d1b/f7d [3123797,103907] 0 2024-04-05T03:02:51.482 INFO:tasks.workunit.client.1.smithi203.stdout:1/776: dwrite d7/d24/f3e [0,4194304] 0 2024-04-05T03:02:51.484 INFO:tasks.workunit.client.0.smithi153.stdout:1/444: readlink d5/d8/d20/d4b/d18/d16/d45/d68/l8a 0 2024-04-05T03:02:51.485 INFO:tasks.workunit.client.0.smithi153.stdout:4/566: dwrite da/f97 [0,4194304] 0 2024-04-05T03:02:51.486 INFO:tasks.workunit.client.1.smithi203.stdout:0/809: rmdir dd/d17/d1d/d22/d53/d55/de0 0 2024-04-05T03:02:51.487 INFO:tasks.workunit.client.0.smithi153.stdout:2/496: fdatasync d9/da/f49 0 2024-04-05T03:02:51.487 INFO:tasks.workunit.client.1.smithi203.stdout:8/896: dwrite d1/d4b/d5f/d74/ff9 [0,4194304] 0 2024-04-05T03:02:51.491 INFO:tasks.workunit.client.0.smithi153.stdout:1/445: chown d5/d8/d20/d4b/d1e/l87 129507 1 2024-04-05T03:02:51.491 INFO:tasks.workunit.client.0.smithi153.stdout:9/392: dread d1/d9/d1b/f8d [0,4194304] 0 2024-04-05T03:02:51.491 INFO:tasks.workunit.client.1.smithi203.stdout:9/824: symlink d13/d5c/dd1/d51/dcb/l107 0 2024-04-05T03:02:51.492 INFO:tasks.workunit.client.0.smithi153.stdout:7/569: read d2/da/d11/f13 [1440592,10291] 0 2024-04-05T03:02:51.494 INFO:tasks.workunit.client.1.smithi203.stdout:8/897: dread d1/d6/f93 [0,4194304] 0 2024-04-05T03:02:51.495 INFO:tasks.workunit.client.1.smithi203.stdout:1/777: mkdir d7/d14/d83/dfb 0 2024-04-05T03:02:51.495 INFO:tasks.workunit.client.0.smithi153.stdout:1/446: dread - d5/d8/d20/d4b/d18/d16/f7a zero size 2024-04-05T03:02:51.497 INFO:tasks.workunit.client.0.smithi153.stdout:4/567: rename da/de/d56/d93 to da/db/d20/d35/d41/da8/dc1 0 2024-04-05T03:02:51.497 INFO:tasks.workunit.client.1.smithi203.stdout:8/898: fdatasync d1/d6/f7c 0 2024-04-05T03:02:51.498 INFO:tasks.workunit.client.1.smithi203.stdout:1/778: stat d7/d14/d2b/fd9 0 2024-04-05T03:02:51.499 INFO:tasks.workunit.client.0.smithi153.stdout:7/570: mknod d2/da/d47/d5d/d7c/cbd 0 2024-04-05T03:02:51.500 INFO:tasks.workunit.client.1.smithi203.stdout:8/899: chown d1/d12/da8/cf0 32338 1 2024-04-05T03:02:51.500 INFO:tasks.workunit.client.1.smithi203.stdout:3/672: dread d0/d41/f9a [0,4194304] 0 2024-04-05T03:02:51.501 INFO:tasks.workunit.client.0.smithi153.stdout:7/571: write d2/da/d11/f52 [4109210,119707] 0 2024-04-05T03:02:51.502 INFO:tasks.workunit.client.0.smithi153.stdout:2/497: rename d9/da/d19/d2e/f42 to d9/d68/fa7 0 2024-04-05T03:02:51.502 INFO:tasks.workunit.client.1.smithi203.stdout:4/590: creat da/d29/fb2 x:0 0 0 2024-04-05T03:02:51.505 INFO:tasks.workunit.client.1.smithi203.stdout:6/798: rename d0/da/d18/d38/d2a/l5d to d0/da/lf0 0 2024-04-05T03:02:51.506 INFO:tasks.workunit.client.0.smithi153.stdout:7/572: mkdir d2/da/d3a/d41/da8/dbe 0 2024-04-05T03:02:51.508 INFO:tasks.workunit.client.1.smithi203.stdout:1/779: unlink d7/d24/d33/d48/cbd 0 2024-04-05T03:02:51.508 INFO:tasks.workunit.client.1.smithi203.stdout:4/591: getdents da/d29/d2e/d57/da7/dab 0 2024-04-05T03:02:51.509 INFO:tasks.workunit.client.0.smithi153.stdout:4/568: rename da/db/d20/d45/c98 to da/db/d20/d35/d41/da8/dc1/cc2 0 2024-04-05T03:02:51.509 INFO:tasks.workunit.client.0.smithi153.stdout:7/573: rmdir d2/da/d3a/d41/d7a 39 2024-04-05T03:02:51.510 INFO:tasks.workunit.client.1.smithi203.stdout:6/799: readlink d0/da/d54/d4c/da1/db7/l98 0 2024-04-05T03:02:51.510 INFO:tasks.workunit.client.1.smithi203.stdout:6/800: stat d0/l58 0 2024-04-05T03:02:51.511 INFO:tasks.workunit.client.0.smithi153.stdout:7/574: write d2/da/f68 [426122,96263] 0 2024-04-05T03:02:51.511 INFO:tasks.workunit.client.0.smithi153.stdout:4/569: write da/db/d20/d3c/d81/db6/f50 [62699,75449] 0 2024-04-05T03:02:51.511 INFO:tasks.workunit.client.1.smithi203.stdout:8/900: rmdir d1/d6/d2e/d5c/dc0/dfb 0 2024-04-05T03:02:51.512 INFO:tasks.workunit.client.0.smithi153.stdout:2/498: rename d9/da/d19/d2e/d60 to d9/da/d37/d8a/da8 0 2024-04-05T03:02:51.514 INFO:tasks.workunit.client.1.smithi203.stdout:6/801: fdatasync d0/da/d18/d38/d2a/d44/faa 0 2024-04-05T03:02:51.516 INFO:tasks.workunit.client.1.smithi203.stdout:3/673: dread d0/dc/d1a/d76/d35/f47 [0,4194304] 0 2024-04-05T03:02:51.516 INFO:tasks.workunit.client.0.smithi153.stdout:4/570: dread - da/de/d56/d78/d8f/d68/f95 zero size 2024-04-05T03:02:51.516 INFO:tasks.workunit.client.1.smithi203.stdout:8/901: unlink d1/d6/d2e/d5c/dc0/ffc 0 2024-04-05T03:02:51.517 INFO:tasks.workunit.client.0.smithi153.stdout:2/499: unlink d9/da/d19/l40 0 2024-04-05T03:02:51.517 INFO:tasks.workunit.client.0.smithi153.stdout:7/575: getdents d2/d51/da7 0 2024-04-05T03:02:51.518 INFO:tasks.workunit.client.1.smithi203.stdout:9/825: dread d13/d5c/dd1/d31/f99 [0,4194304] 0 2024-04-05T03:02:51.521 INFO:tasks.workunit.client.1.smithi203.stdout:8/902: mknod d1/d4b/d68/d75/d77/c106 0 2024-04-05T03:02:51.523 INFO:tasks.workunit.client.1.smithi203.stdout:4/592: link da/l1d da/d29/d2e/d57/d96/lb3 0 2024-04-05T03:02:51.523 INFO:tasks.workunit.client.1.smithi203.stdout:6/802: truncate d0/da/d54/f45 5189433 0 2024-04-05T03:02:51.523 INFO:tasks.workunit.client.0.smithi153.stdout:7/576: symlink d2/da/d3a/d41/da8/lbf 0 2024-04-05T03:02:51.524 INFO:tasks.workunit.client.1.smithi203.stdout:1/780: dwrite d7/d24/f35 [4194304,4194304] 0 2024-04-05T03:02:51.526 INFO:tasks.workunit.client.1.smithi203.stdout:8/903: fsync d1/d4b/d5f/daf/f103 0 2024-04-05T03:02:51.533 INFO:tasks.workunit.client.0.smithi153.stdout:9/393: dread d1/d9/d1b/f30 [0,4194304] 0 2024-04-05T03:02:51.534 INFO:tasks.workunit.client.1.smithi203.stdout:3/674: creat d0/dc/d1a/fc3 x:0 0 0 2024-04-05T03:02:51.535 INFO:tasks.workunit.client.1.smithi203.stdout:0/810: truncate dd/d17/dd8/fb6 277440 0 2024-04-05T03:02:51.538 INFO:tasks.workunit.client.1.smithi203.stdout:0/811: truncate dd/d17/d1d/d22/d53/d7f/fc7 1086819 0 2024-04-05T03:02:51.542 INFO:tasks.workunit.client.1.smithi203.stdout:1/781: getdents d7/d14/d83/deb 0 2024-04-05T03:02:51.545 INFO:tasks.workunit.client.1.smithi203.stdout:1/782: readlink d7/d14/d2b/d58/d86/ld1 0 2024-04-05T03:02:51.546 INFO:tasks.workunit.client.0.smithi153.stdout:9/394: write d1/f18 [5647221,85861] 0 2024-04-05T03:02:51.547 INFO:tasks.workunit.client.1.smithi203.stdout:0/812: getdents dd/d17/d1d/d22/d80/d49 0 2024-04-05T03:02:51.548 INFO:tasks.workunit.client.0.smithi153.stdout:2/500: write d9/da/d78/f92 [877523,94118] 0 2024-04-05T03:02:51.549 INFO:tasks.workunit.client.1.smithi203.stdout:9/826: dread d13/d1b/d6b/d9e/d59/d5f/f71 [0,4194304] 0 2024-04-05T03:02:51.550 INFO:tasks.workunit.client.0.smithi153.stdout:7/577: rmdir d2 39 2024-04-05T03:02:51.552 INFO:tasks.workunit.client.1.smithi203.stdout:0/813: write dd/d17/d1d/d26/d28/f2d [3877315,50486] 0 2024-04-05T03:02:51.552 INFO:tasks.workunit.client.1.smithi203.stdout:6/803: dread d0/da/d54/f4b [0,4194304] 0 2024-04-05T03:02:51.553 INFO:tasks.workunit.client.1.smithi203.stdout:9/827: read f12 [1761554,107324] 0 2024-04-05T03:02:51.554 INFO:tasks.workunit.client.1.smithi203.stdout:0/814: fdatasync dd/d17/d1d/d26/f4f 0 2024-04-05T03:02:51.555 INFO:tasks.workunit.client.0.smithi153.stdout:9/395: symlink d1/d1d/l90 0 2024-04-05T03:02:51.556 INFO:tasks.workunit.client.1.smithi203.stdout:9/828: chown d13/d5c/dd1/f93 109 1 2024-04-05T03:02:51.557 INFO:tasks.workunit.client.1.smithi203.stdout:1/783: rename d7/d24/d33/d48/d4b/f94 to d7/d24/d33/d48/ffc 0 2024-04-05T03:02:51.558 INFO:tasks.workunit.client.1.smithi203.stdout:1/784: write d7/d14/d9d/ff3 [49323,3850] 0 2024-04-05T03:02:51.560 INFO:tasks.workunit.client.0.smithi153.stdout:7/578: read d2/da/d3a/d41/d7a/f3c [1100351,57231] 0 2024-04-05T03:02:51.561 INFO:tasks.workunit.client.1.smithi203.stdout:1/785: read - d7/d90/fd7 zero size 2024-04-05T03:02:51.563 INFO:tasks.workunit.client.1.smithi203.stdout:3/675: dread d0/dc/d1a/d1f/f32 [0,4194304] 0 2024-04-05T03:02:51.564 INFO:tasks.workunit.client.1.smithi203.stdout:9/829: truncate d13/d5c/dd1/d51/d81/fa5 4881548 0 2024-04-05T03:02:51.565 INFO:tasks.workunit.client.1.smithi203.stdout:3/676: stat d0/dc/d1a/d76 0 2024-04-05T03:02:51.566 INFO:tasks.workunit.client.0.smithi153.stdout:9/396: mkdir d1/d3a/d91 0 2024-04-05T03:02:51.566 INFO:tasks.workunit.client.0.smithi153.stdout:7/579: dread d2/da/d3a/d41/f6c [0,4194304] 0 2024-04-05T03:02:51.566 INFO:tasks.workunit.client.1.smithi203.stdout:8/904: dread d1/d12/d1a/f2c [8388608,4194304] 0 2024-04-05T03:02:51.567 INFO:tasks.workunit.client.0.smithi153.stdout:2/501: dread d9/da/d36/f55 [0,4194304] 0 2024-04-05T03:02:51.569 INFO:tasks.workunit.client.1.smithi203.stdout:9/830: write d13/d5c/f89 [5068160,21929] 0 2024-04-05T03:02:51.572 INFO:tasks.workunit.client.0.smithi153.stdout:2/502: truncate d9/da/d19/d2e/d48/f1a 2433672 0 2024-04-05T03:02:51.574 INFO:tasks.workunit.client.0.smithi153.stdout:9/397: creat d1/d10/f92 x:0 0 0 2024-04-05T03:02:51.574 INFO:tasks.workunit.client.0.smithi153.stdout:7/580: dwrite d2/da/d3a/d41/f8a [4194304,4194304] 0 2024-04-05T03:02:51.575 INFO:tasks.workunit.client.1.smithi203.stdout:9/831: symlink d13/dc8/l108 0 2024-04-05T03:02:51.577 INFO:tasks.workunit.client.1.smithi203.stdout:1/786: dwrite d7/d14/d2b/fd9 [0,4194304] 0 2024-04-05T03:02:51.578 INFO:tasks.workunit.client.0.smithi153.stdout:7/581: creat d2/d9b/d6f/fc0 x:0 0 0 2024-04-05T03:02:51.579 INFO:tasks.workunit.client.1.smithi203.stdout:6/804: truncate d0/da/d18/d38/d2a/d2c/f4f 1426200 0 2024-04-05T03:02:51.579 INFO:tasks.workunit.client.0.smithi153.stdout:7/582: readlink d2/d9b/l91 0 2024-04-05T03:02:51.581 INFO:tasks.workunit.client.1.smithi203.stdout:9/832: rmdir d13/d5c/dd1/d51/dcb 39 2024-04-05T03:02:51.582 INFO:tasks.workunit.client.0.smithi153.stdout:9/398: dwrite d1/d9/d1b/f3f [0,4194304] 0 2024-04-05T03:02:51.582 INFO:tasks.workunit.client.1.smithi203.stdout:0/815: dread dd/d17/d1d/d26/f3d [0,4194304] 0 2024-04-05T03:02:51.589 INFO:tasks.workunit.client.0.smithi153.stdout:7/583: dwrite d2/da/d47/d5d/f69 [0,4194304] 0 2024-04-05T03:02:51.590 INFO:tasks.workunit.client.0.smithi153.stdout:7/584: readlink d2/d51/laf 0 2024-04-05T03:02:51.591 INFO:tasks.workunit.client.1.smithi203.stdout:6/805: dwrite d0/f9 [0,4194304] 0 2024-04-05T03:02:51.592 INFO:tasks.workunit.client.1.smithi203.stdout:1/787: dwrite d7/d24/d33/d48/d4b/d51/fe2 [0,4194304] 0 2024-04-05T03:02:51.597 INFO:tasks.workunit.client.1.smithi203.stdout:1/788: fdatasync d7/d14/d2b/d58/ff0 0 2024-04-05T03:02:51.598 INFO:tasks.workunit.client.1.smithi203.stdout:9/833: truncate d13/d5c/dd1/d31/f99 87142 0 2024-04-05T03:02:51.601 INFO:tasks.workunit.client.1.smithi203.stdout:9/834: chown f12 440579 1 2024-04-05T03:02:51.604 INFO:tasks.workunit.client.1.smithi203.stdout:1/789: mknod d7/d24/d33/d48/d4b/dea/def/cfd 0 2024-04-05T03:02:51.605 INFO:tasks.workunit.client.0.smithi153.stdout:3/483: sync 2024-04-05T03:02:51.605 INFO:tasks.workunit.client.0.smithi153.stdout:6/501: sync 2024-04-05T03:02:51.605 INFO:tasks.workunit.client.1.smithi203.stdout:9/835: symlink d13/d1b/d6b/l109 0 2024-04-05T03:02:51.608 INFO:tasks.workunit.client.1.smithi203.stdout:3/677: dread d0/dc/d1a/d8d/d96/f5e [0,4194304] 0 2024-04-05T03:02:51.609 INFO:tasks.workunit.client.1.smithi203.stdout:6/806: dwrite d0/f9c [4194304,4194304] 0 2024-04-05T03:02:51.610 INFO:tasks.workunit.client.0.smithi153.stdout:3/484: truncate d1d/d27/d2b/f35 1456919 0 2024-04-05T03:02:51.611 INFO:tasks.workunit.client.1.smithi203.stdout:8/905: dread d1/d12/d1a/fdd [0,4194304] 0 2024-04-05T03:02:51.612 INFO:tasks.workunit.client.0.smithi153.stdout:7/585: dwrite d2/d51/d93/f4b [4194304,4194304] 0 2024-04-05T03:02:51.612 INFO:tasks.workunit.client.1.smithi203.stdout:8/906: chown d1/d6/f7c 65304595 1 2024-04-05T03:02:51.614 INFO:tasks.workunit.client.1.smithi203.stdout:8/907: readlink d1/d12/d2f/d1f/l43 0 2024-04-05T03:02:51.615 INFO:tasks.workunit.client.0.smithi153.stdout:3/485: write d1d/d32/d93/d57/f76 [1248393,88294] 0 2024-04-05T03:02:51.615 INFO:tasks.workunit.client.1.smithi203.stdout:8/908: readlink d1/d12/d2f/d55/ld9 0 2024-04-05T03:02:51.615 INFO:tasks.workunit.client.0.smithi153.stdout:3/486: readlink d1d/d32/d93/l6e 0 2024-04-05T03:02:51.616 INFO:tasks.workunit.client.0.smithi153.stdout:7/586: mknod d2/da/d47/d5d/d7c/d8d/cc1 0 2024-04-05T03:02:51.617 INFO:tasks.workunit.client.1.smithi203.stdout:9/836: rmdir d13/d1b/d33/d64/dfa/d67 39 2024-04-05T03:02:51.617 INFO:tasks.workunit.client.1.smithi203.stdout:1/790: fdatasync d7/d14/f21 0 2024-04-05T03:02:51.618 INFO:tasks.workunit.client.0.smithi153.stdout:7/587: write d2/d51/fa5 [756965,77786] 0 2024-04-05T03:02:51.619 INFO:tasks.workunit.client.1.smithi203.stdout:3/678: creat d0/dc/d1a/d76/fc4 x:0 0 0 2024-04-05T03:02:51.620 INFO:tasks.workunit.client.1.smithi203.stdout:8/909: mknod d1/d90/c107 0 2024-04-05T03:02:51.622 INFO:tasks.workunit.client.1.smithi203.stdout:6/807: mknod d0/da/d18/cf1 0 2024-04-05T03:02:51.623 INFO:tasks.workunit.client.0.smithi153.stdout:7/588: creat d2/da/d11/d88/da6/fc2 x:0 0 0 2024-04-05T03:02:51.624 INFO:tasks.workunit.client.0.smithi153.stdout:7/589: dread - d2/da/d3a/fae zero size 2024-04-05T03:02:51.625 INFO:tasks.workunit.client.1.smithi203.stdout:3/679: creat d0/dc/d1a/d1f/dac/fc5 x:0 0 0 2024-04-05T03:02:51.626 INFO:tasks.workunit.client.1.smithi203.stdout:8/910: dread d1/d6/f7c [0,4194304] 0 2024-04-05T03:02:51.627 INFO:tasks.workunit.client.1.smithi203.stdout:1/791: creat d7/d14/d2b/d58/d86/d91/da6/dee/ffe x:0 0 0 2024-04-05T03:02:51.627 INFO:tasks.workunit.client.1.smithi203.stdout:3/680: fdatasync d0/dc/d1a/d8d/d96/f51 0 2024-04-05T03:02:51.628 INFO:tasks.workunit.client.1.smithi203.stdout:9/837: link d13/d1b/d6b/d7a/lfd d13/d1b/d6b/d9e/df0/l10a 0 2024-04-05T03:02:51.629 INFO:tasks.workunit.client.1.smithi203.stdout:6/808: write d0/da/d18/d38/d86/f93 [488225,124721] 0 2024-04-05T03:02:51.629 INFO:tasks.workunit.client.0.smithi153.stdout:7/590: link d2/da/d47/l4a d2/da/d3a/d41/lc3 0 2024-04-05T03:02:51.631 INFO:tasks.workunit.client.0.smithi153.stdout:7/591: fsync d2/da/d11/d39/fb6 0 2024-04-05T03:02:51.631 INFO:tasks.workunit.client.1.smithi203.stdout:9/838: mkdir d13/de9/d10b 0 2024-04-05T03:02:51.632 INFO:tasks.workunit.client.1.smithi203.stdout:1/792: mkdir d7/d14/d83/deb/dff 0 2024-04-05T03:02:51.632 INFO:tasks.workunit.client.0.smithi153.stdout:3/487: dread d1d/d32/d93/d57/d6b/f67 [0,4194304] 0 2024-04-05T03:02:51.633 INFO:tasks.workunit.client.1.smithi203.stdout:3/681: symlink d0/dc/d1a/d1f/da1/daf/lc6 0 2024-04-05T03:02:51.635 INFO:tasks.workunit.client.1.smithi203.stdout:8/911: rename d1/d12/d17/d47/d53/de0/ff1 to d1/d6/d2e/d5c/df5/f108 0 2024-04-05T03:02:51.635 INFO:tasks.workunit.client.0.smithi153.stdout:3/488: unlink d1d/d27/f4d 0 2024-04-05T03:02:51.637 INFO:tasks.workunit.client.0.smithi153.stdout:3/489: rmdir d1d/d32/d93/d57/d6b 39 2024-04-05T03:02:51.638 INFO:tasks.workunit.client.1.smithi203.stdout:8/912: write d1/d12/d2f/d5b/fb3 [676448,28039] 0 2024-04-05T03:02:51.639 INFO:tasks.workunit.client.1.smithi203.stdout:1/793: dwrite d7/d14/d2b/d58/f60 [4194304,4194304] 0 2024-04-05T03:02:51.641 INFO:tasks.workunit.client.1.smithi203.stdout:1/794: stat d7/d14/f52 0 2024-04-05T03:02:51.642 INFO:tasks.workunit.client.0.smithi153.stdout:7/592: rename d2/da/d3a/d41/lc3 to d2/lc4 0 2024-04-05T03:02:51.642 INFO:tasks.workunit.client.1.smithi203.stdout:1/795: chown d7/d14/f76 444163 1 2024-04-05T03:02:51.642 INFO:tasks.workunit.client.0.smithi153.stdout:7/593: chown d2/da/d47 184085 1 2024-04-05T03:02:51.643 INFO:tasks.workunit.client.1.smithi203.stdout:1/796: fsync d7/d14/f52 0 2024-04-05T03:02:51.643 INFO:tasks.workunit.client.0.smithi153.stdout:7/594: chown d2/da/d3a/d41/d31/f9a 17 1 2024-04-05T03:02:51.644 INFO:tasks.workunit.client.1.smithi203.stdout:8/913: mknod d1/d4b/d5f/daf/c109 0 2024-04-05T03:02:51.644 INFO:tasks.workunit.client.0.smithi153.stdout:3/490: chown d1d/d32/d93/d3c/c8a 112 1 2024-04-05T03:02:51.645 INFO:tasks.workunit.client.1.smithi203.stdout:8/914: readlink d1/d6/l64 0 2024-04-05T03:02:51.645 INFO:tasks.workunit.client.0.smithi153.stdout:7/595: readlink d2/d51/d93/l67 0 2024-04-05T03:02:51.646 INFO:tasks.workunit.client.1.smithi203.stdout:1/797: creat d7/d14/d2b/d58/d86/d91/f100 x:0 0 0 2024-04-05T03:02:51.649 INFO:tasks.workunit.client.0.smithi153.stdout:3/491: rename d1d/d32/d93/d57/fa0 to d1d/d32/d93/d3c/d3f/d7c/fad 0 2024-04-05T03:02:51.654 INFO:tasks.workunit.client.1.smithi203.stdout:1/798: dwrite d7/d24/d33/d48/d4b/d6b/fb3 [0,4194304] 0 2024-04-05T03:02:51.655 INFO:tasks.workunit.client.0.smithi153.stdout:7/596: creat d2/d9b/fc5 x:0 0 0 2024-04-05T03:02:51.656 INFO:tasks.workunit.client.0.smithi153.stdout:9/399: dwrite d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:02:51.657 INFO:tasks.workunit.client.0.smithi153.stdout:6/502: dwrite d4/d38/f75 [0,4194304] 0 2024-04-05T03:02:51.658 INFO:tasks.workunit.client.0.smithi153.stdout:6/503: dread - d4/f89 zero size 2024-04-05T03:02:51.662 INFO:tasks.workunit.client.0.smithi153.stdout:3/492: creat d1d/d32/d93/d57/d6b/d9f/fae x:0 0 0 2024-04-05T03:02:51.665 INFO:tasks.workunit.client.0.smithi153.stdout:6/504: read d4/dc/df/f39 [1878195,8544] 0 2024-04-05T03:02:51.666 INFO:tasks.workunit.client.1.smithi203.stdout:6/809: fsync d0/f9 0 2024-04-05T03:02:51.666 INFO:tasks.workunit.client.0.smithi153.stdout:7/597: mkdir d2/d9b/d6f/dc6 0 2024-04-05T03:02:51.667 INFO:tasks.workunit.client.1.smithi203.stdout:1/799: write d7/f9c [1070457,122935] 0 2024-04-05T03:02:51.668 INFO:tasks.workunit.client.0.smithi153.stdout:9/400: creat d1/d3a/f93 x:0 0 0 2024-04-05T03:02:51.669 INFO:tasks.workunit.client.1.smithi203.stdout:6/810: chown d0/da/d54/d82 45 1 2024-04-05T03:02:51.671 INFO:tasks.workunit.client.0.smithi153.stdout:3/493: dwrite d1d/d27/fa1 [0,4194304] 0 2024-04-05T03:02:51.677 INFO:tasks.workunit.client.0.smithi153.stdout:6/505: chown d4/c27 46 1 2024-04-05T03:02:51.678 INFO:tasks.workunit.client.0.smithi153.stdout:6/506: chown d4/l95 614 1 2024-04-05T03:02:51.678 INFO:tasks.workunit.client.0.smithi153.stdout:7/598: creat d2/da/d3a/d41/d7a/d54/d85/fc7 x:0 0 0 2024-04-05T03:02:51.682 INFO:tasks.workunit.client.0.smithi153.stdout:9/401: rename d1/d9/d1b/f5d to d1/d6c/f94 0 2024-04-05T03:02:51.684 INFO:tasks.workunit.client.0.smithi153.stdout:3/494: readlink lf 0 2024-04-05T03:02:51.686 INFO:tasks.workunit.client.0.smithi153.stdout:6/507: creat d4/dc/df/d16/d17/d54/faa x:0 0 0 2024-04-05T03:02:51.687 INFO:tasks.workunit.client.1.smithi203.stdout:9/839: truncate d13/d1b/d6b/d9e/d59/d5f/fb9 454339 0 2024-04-05T03:02:51.687 INFO:tasks.workunit.client.0.smithi153.stdout:9/402: symlink d1/d10/d16/l95 0 2024-04-05T03:02:51.688 INFO:tasks.workunit.client.0.smithi153.stdout:7/599: dwrite d2/da/d3a/d41/d7a/d54/d85/fc7 [0,4194304] 0 2024-04-05T03:02:51.689 INFO:tasks.workunit.client.0.smithi153.stdout:3/495: truncate d1d/d32/d93/d57/f59 877264 0 2024-04-05T03:02:51.692 INFO:tasks.workunit.client.0.smithi153.stdout:3/496: dread - d1d/d32/d93/d3c/d3f/d90/d92/fac zero size 2024-04-05T03:02:51.693 INFO:tasks.workunit.client.0.smithi153.stdout:3/497: stat d1d/d32/d93/d3c/d3f 0 2024-04-05T03:02:51.694 INFO:tasks.workunit.client.0.smithi153.stdout:9/403: dread d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:02:51.695 INFO:tasks.workunit.client.1.smithi203.stdout:3/682: dwrite d0/f7f [0,4194304] 0 2024-04-05T03:02:51.696 INFO:tasks.workunit.client.0.smithi153.stdout:9/404: dread - d1/d10/d16/d22/d2c/d68/d81/f7a zero size 2024-04-05T03:02:51.697 INFO:tasks.workunit.client.0.smithi153.stdout:7/600: fdatasync d2/da/d3a/d41/f8c 0 2024-04-05T03:02:51.698 INFO:tasks.workunit.client.1.smithi203.stdout:8/915: dread d1/d12/d2f/d1f/f23 [0,4194304] 0 2024-04-05T03:02:51.698 INFO:tasks.workunit.client.0.smithi153.stdout:7/601: fsync d2/da/d11/d88/fbb 0 2024-04-05T03:02:51.701 INFO:tasks.workunit.client.1.smithi203.stdout:5/776: sync 2024-04-05T03:02:51.701 INFO:tasks.workunit.client.1.smithi203.stdout:8/916: chown d1/d6/d2e/d5c/fd6 16967484 1 2024-04-05T03:02:51.701 INFO:tasks.workunit.client.0.smithi153.stdout:3/498: chown d1d/d27/d2b/c60 0 1 2024-04-05T03:02:51.702 INFO:tasks.workunit.client.1.smithi203.stdout:9/840: dwrite d13/d5c/dd1/d51/da1/dba/fd4 [0,4194304] 0 2024-04-05T03:02:51.702 INFO:tasks.workunit.client.0.smithi153.stdout:3/499: write d1d/d75/f79 [800839,115549] 0 2024-04-05T03:02:51.703 INFO:tasks.workunit.client.1.smithi203.stdout:3/683: write d0/dc/d1a/d8d/f9c [288052,121104] 0 2024-04-05T03:02:51.704 INFO:tasks.workunit.client.0.smithi153.stdout:6/508: rename d4/dc/df/d16/d24/d8d/c69 to d4/dc/d7a/da8/cab 0 2024-04-05T03:02:51.704 INFO:tasks.workunit.client.1.smithi203.stdout:8/917: write d1/d6/f45 [3773031,92381] 0 2024-04-05T03:02:51.705 INFO:tasks.workunit.client.0.smithi153.stdout:6/509: fdatasync d4/d38/d4c/fa5 0 2024-04-05T03:02:51.707 INFO:tasks.workunit.client.0.smithi153.stdout:9/405: mkdir d1/d3a/d91/d96 0 2024-04-05T03:02:51.707 INFO:tasks.workunit.client.0.smithi153.stdout:6/510: dread - d4/f7 zero size 2024-04-05T03:02:51.708 INFO:tasks.workunit.client.0.smithi153.stdout:9/406: readlink d1/d10/d16/l95 0 2024-04-05T03:02:51.711 INFO:tasks.workunit.client.1.smithi203.stdout:1/800: stat d7/d24/d33/d48/d4b/l6f 0 2024-04-05T03:02:51.713 INFO:tasks.workunit.client.1.smithi203.stdout:8/918: mknod d1/d12/d17/d47/d53/c10a 0 2024-04-05T03:02:51.715 INFO:tasks.workunit.client.1.smithi203.stdout:6/811: mknod d0/da/d54/cf2 0 2024-04-05T03:02:51.715 INFO:tasks.workunit.client.1.smithi203.stdout:5/777: mkdir d3/d5f/d92/dff/d103 0 2024-04-05T03:02:51.717 INFO:tasks.workunit.client.1.smithi203.stdout:5/778: mkdir d3/d5f/d92/dff/d104 0 2024-04-05T03:02:51.718 INFO:tasks.workunit.client.1.smithi203.stdout:8/919: mkdir d1/d12/d17/d10/d10b 0 2024-04-05T03:02:51.719 INFO:tasks.workunit.client.0.smithi153.stdout:6/511: unlink d4/ce 0 2024-04-05T03:02:51.720 INFO:tasks.workunit.client.0.smithi153.stdout:9/407: mkdir d1/d10/d36/d57/d97 0 2024-04-05T03:02:51.721 INFO:tasks.workunit.client.1.smithi203.stdout:1/801: getdents d7/d24/d33 0 2024-04-05T03:02:51.722 INFO:tasks.workunit.client.1.smithi203.stdout:5/779: write d3/d4/dc/f25 [3569343,100727] 0 2024-04-05T03:02:51.722 INFO:tasks.workunit.client.1.smithi203.stdout:3/684: read d0/dc/d1a/d76/d35/f3c [596603,47596] 0 2024-04-05T03:02:51.722 INFO:tasks.workunit.client.1.smithi203.stdout:8/920: symlink d1/d4b/d68/d75/l10c 0 2024-04-05T03:02:51.723 INFO:tasks.workunit.client.0.smithi153.stdout:9/408: creat d1/d10/d16/f98 x:0 0 0 2024-04-05T03:02:51.723 INFO:tasks.workunit.client.1.smithi203.stdout:1/802: creat d7/d90/f101 x:0 0 0 2024-04-05T03:02:51.724 INFO:tasks.workunit.client.1.smithi203.stdout:3/685: dread - d0/d21/d81/f7b zero size 2024-04-05T03:02:51.725 INFO:tasks.workunit.client.0.smithi153.stdout:9/409: write d1/d10/d16/d22/d2c/f58 [70317,103073] 0 2024-04-05T03:02:51.725 INFO:tasks.workunit.client.0.smithi153.stdout:6/512: dwrite d4/dc/df/d16/d17/f84 [0,4194304] 0 2024-04-05T03:02:51.727 INFO:tasks.workunit.client.1.smithi203.stdout:8/921: creat d1/d4b/d5f/daf/f10d x:0 0 0 2024-04-05T03:02:51.727 INFO:tasks.workunit.client.1.smithi203.stdout:5/780: readlink d3/d1f/d59/d64/l80 0 2024-04-05T03:02:51.729 INFO:tasks.workunit.client.1.smithi203.stdout:5/781: chown d3/d5f/d92/dff 3010423 1 2024-04-05T03:02:51.730 INFO:tasks.workunit.client.1.smithi203.stdout:5/782: chown d3/d4/dc/f37 557435 1 2024-04-05T03:02:51.732 INFO:tasks.workunit.client.0.smithi153.stdout:6/513: dwrite d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:02:51.734 INFO:tasks.workunit.client.0.smithi153.stdout:9/410: dwrite d1/d10/d16/d22/f74 [0,4194304] 0 2024-04-05T03:02:51.734 INFO:tasks.workunit.client.1.smithi203.stdout:3/686: symlink d0/dc/d38/d5d/d66/lc7 0 2024-04-05T03:02:51.734 INFO:tasks.workunit.client.1.smithi203.stdout:5/783: dread - d3/d4/dc/f100 zero size 2024-04-05T03:02:51.737 INFO:tasks.workunit.client.1.smithi203.stdout:3/687: readlink d0/d21/lb4 0 2024-04-05T03:02:51.738 INFO:tasks.workunit.client.1.smithi203.stdout:1/803: dwrite d7/d90/fd7 [0,4194304] 0 2024-04-05T03:02:51.741 INFO:tasks.workunit.client.0.smithi153.stdout:0/471: sync 2024-04-05T03:02:51.745 INFO:tasks.workunit.client.1.smithi203.stdout:8/922: getdents d1/d12/d17/d47/d53/d69/d9c 0 2024-04-05T03:02:51.745 INFO:tasks.workunit.client.0.smithi153.stdout:0/472: mkdir d9/db/d42/d7e/d8b 0 2024-04-05T03:02:51.746 INFO:tasks.workunit.client.0.smithi153.stdout:0/473: chown f3 12052000 1 2024-04-05T03:02:51.746 INFO:tasks.workunit.client.1.smithi203.stdout:8/923: chown d1/d4b/d68/d75/dac/da4 2429 1 2024-04-05T03:02:51.747 INFO:tasks.workunit.client.1.smithi203.stdout:8/924: readlink d1/d6/l29 0 2024-04-05T03:02:51.749 INFO:tasks.workunit.client.1.smithi203.stdout:5/784: rename d3/d4/l7d to d3/d4/dc/d67/db5/l105 0 2024-04-05T03:02:51.750 INFO:tasks.workunit.client.1.smithi203.stdout:1/804: write d7/d14/d9d/fc4 [740226,123713] 0 2024-04-05T03:02:51.752 INFO:tasks.workunit.client.1.smithi203.stdout:5/785: write d3/d4/dc/d67/db5/dda/fef [784741,102917] 0 2024-04-05T03:02:51.753 INFO:tasks.workunit.client.0.smithi153.stdout:6/514: rename d4/dc/df/d16/d24/d5e/f62 to d4/dc/df/d16/d5f/fac 0 2024-04-05T03:02:51.755 INFO:tasks.workunit.client.0.smithi153.stdout:0/474: creat d9/db/d42/d7e/f8c x:0 0 0 2024-04-05T03:02:51.756 INFO:tasks.workunit.client.0.smithi153.stdout:9/411: link d1/d10/d36/d57/d5c/l70 d1/d10/l99 0 2024-04-05T03:02:51.756 INFO:tasks.workunit.client.1.smithi203.stdout:6/812: dread d0/da/d54/d4c/da1/db7/d71/f79 [0,4194304] 0 2024-04-05T03:02:51.758 INFO:tasks.workunit.client.1.smithi203.stdout:5/786: creat d3/d5f/d92/dff/d103/f106 x:0 0 0 2024-04-05T03:02:51.758 INFO:tasks.workunit.client.1.smithi203.stdout:9/841: truncate d13/d1b/d33/d64/dfa/fc3 10520 0 2024-04-05T03:02:51.759 INFO:tasks.workunit.client.0.smithi153.stdout:6/515: mkdir d4/dc/df/d16/d36/d4d/dad 0 2024-04-05T03:02:51.760 INFO:tasks.workunit.client.1.smithi203.stdout:5/787: truncate d3/d4/dc/d67/dac/fe7 5226288 0 2024-04-05T03:02:51.761 INFO:tasks.workunit.client.1.smithi203.stdout:9/842: symlink d13/d1b/d6b/d9e/d59/d5f/l10c 0 2024-04-05T03:02:51.763 INFO:tasks.workunit.client.1.smithi203.stdout:5/788: creat d3/d5f/d92/dff/f107 x:0 0 0 2024-04-05T03:02:51.764 INFO:tasks.workunit.client.0.smithi153.stdout:9/412: write d1/f2e [696943,56433] 0 2024-04-05T03:02:51.764 INFO:tasks.workunit.client.0.smithi153.stdout:6/516: chown d4/dc/df/c1c 3530 1 2024-04-05T03:02:51.765 INFO:tasks.workunit.client.1.smithi203.stdout:6/813: getdents d0/da/d54/d4c/da1/db7 0 2024-04-05T03:02:51.765 INFO:tasks.workunit.client.1.smithi203.stdout:9/843: truncate d13/d1b/d6b/d9e/d77/f7d 4669347 0 2024-04-05T03:02:51.765 INFO:tasks.workunit.client.0.smithi153.stdout:6/517: dread - d4/f7 zero size 2024-04-05T03:02:51.766 INFO:tasks.workunit.client.0.smithi153.stdout:9/413: write d1/d10/d16/d22/f3e [754047,46790] 0 2024-04-05T03:02:51.767 INFO:tasks.workunit.client.0.smithi153.stdout:0/475: rmdir d9/df/d5f 0 2024-04-05T03:02:51.769 INFO:tasks.workunit.client.0.smithi153.stdout:6/518: symlink d4/dc/df/d16/d36/d52/lae 0 2024-04-05T03:02:51.773 INFO:tasks.workunit.client.0.smithi153.stdout:9/414: creat d1/d9/f9a x:0 0 0 2024-04-05T03:02:51.774 INFO:tasks.workunit.client.0.smithi153.stdout:9/415: write d1/d6c/f82 [984516,65416] 0 2024-04-05T03:02:51.780 INFO:tasks.workunit.client.1.smithi203.stdout:6/814: dwrite d0/da/d54/f45 [0,4194304] 0 2024-04-05T03:02:51.783 INFO:tasks.workunit.client.1.smithi203.stdout:9/844: dwrite d13/d1b/d6b/d9e/f5e [0,4194304] 0 2024-04-05T03:02:51.784 INFO:tasks.workunit.client.0.smithi153.stdout:6/519: write d4/dc/df/d16/d24/d4e/f5a [4894044,97503] 0 2024-04-05T03:02:51.784 INFO:tasks.workunit.client.1.smithi203.stdout:8/925: dread d1/d4b/d5f/d74/fb5 [0,4194304] 0 2024-04-05T03:02:51.788 INFO:tasks.workunit.client.0.smithi153.stdout:0/476: write d9/df/d1e/f52 [936466,111443] 0 2024-04-05T03:02:51.792 INFO:tasks.workunit.client.0.smithi153.stdout:6/520: write d4/dc/df/d16/d24/d4e/f8a [1197632,50022] 0 2024-04-05T03:02:51.792 INFO:tasks.workunit.client.1.smithi203.stdout:8/926: symlink d1/d12/d2f/d5b/l10e 0 2024-04-05T03:02:51.793 INFO:tasks.workunit.client.1.smithi203.stdout:3/688: dread d0/dc/d1a/d1d/d53/f99 [0,4194304] 0 2024-04-05T03:02:51.793 INFO:tasks.workunit.client.0.smithi153.stdout:6/521: write d4/d38/f8b [879367,72900] 0 2024-04-05T03:02:51.795 INFO:tasks.workunit.client.0.smithi153.stdout:0/477: write d9/d14/f40 [1068908,58648] 0 2024-04-05T03:02:51.796 INFO:tasks.workunit.client.1.smithi203.stdout:9/845: dwrite d13/d1b/d6b/d9e/d59/d5f/f71 [0,4194304] 0 2024-04-05T03:02:51.798 INFO:tasks.workunit.client.1.smithi203.stdout:8/927: creat d1/d12/d17/d10/f10f x:0 0 0 2024-04-05T03:02:51.799 INFO:tasks.workunit.client.0.smithi153.stdout:6/522: rmdir d4/dc/df/d16/d5f/d73 39 2024-04-05T03:02:51.799 INFO:tasks.workunit.client.1.smithi203.stdout:3/689: truncate d0/dc/d1a/d1d/d53/f98 997914 0 2024-04-05T03:02:51.806 INFO:tasks.workunit.client.0.smithi153.stdout:6/523: dwrite d4/dc/f97 [0,4194304] 0 2024-04-05T03:02:51.806 INFO:tasks.workunit.client.1.smithi203.stdout:9/846: write d13/d1b/d6b/d9e/d59/dbc/fc4 [858300,87668] 0 2024-04-05T03:02:51.807 INFO:tasks.workunit.client.0.smithi153.stdout:0/478: dread d9/db/d42/d43/f85 [0,4194304] 0 2024-04-05T03:02:51.808 INFO:tasks.workunit.client.1.smithi203.stdout:1/805: dread d7/fa [0,4194304] 0 2024-04-05T03:02:51.813 INFO:tasks.workunit.client.1.smithi203.stdout:8/928: dwrite d1/d6/fe8 [0,4194304] 0 2024-04-05T03:02:51.815 INFO:tasks.workunit.client.0.smithi153.stdout:6/524: dwrite d4/dc/df/d16/f23 [0,4194304] 0 2024-04-05T03:02:51.817 INFO:tasks.workunit.client.0.smithi153.stdout:6/525: fsync d4/dc/df/d16/d24/d4e/f8a 0 2024-04-05T03:02:51.817 INFO:tasks.workunit.client.1.smithi203.stdout:1/806: mknod d7/d14/d83/dfb/c102 0 2024-04-05T03:02:51.819 INFO:tasks.workunit.client.1.smithi203.stdout:3/690: dwrite d0/dc/d1a/d1d/d53/f73 [0,4194304] 0 2024-04-05T03:02:51.820 INFO:tasks.workunit.client.1.smithi203.stdout:3/691: chown d0/d21/f45 112253 1 2024-04-05T03:02:51.821 INFO:tasks.workunit.client.0.smithi153.stdout:6/526: write d4/dc/df/d16/d5f/d73/f9e [851850,119810] 0 2024-04-05T03:02:51.822 INFO:tasks.workunit.client.0.smithi153.stdout:6/527: fsync d4/dc/df/d16/d5f/f7c 0 2024-04-05T03:02:51.828 INFO:tasks.workunit.client.1.smithi203.stdout:3/692: write d0/dc/d1a/d8d/d96/f5e [917158,13669] 0 2024-04-05T03:02:51.839 INFO:tasks.workunit.client.1.smithi203.stdout:6/815: dread d0/da/d18/d38/d86/fe6 [4194304,4194304] 0 2024-04-05T03:02:51.839 INFO:tasks.workunit.client.1.smithi203.stdout:3/693: dwrite d0/dc/d1a/fc3 [0,4194304] 0 2024-04-05T03:02:51.843 INFO:tasks.workunit.client.1.smithi203.stdout:9/847: dwrite d13/d1b/d33/d64/dfa/f49 [0,4194304] 0 2024-04-05T03:02:51.852 INFO:tasks.workunit.client.1.smithi203.stdout:9/848: rename d13/d5c/dd1/d51/dcb/l102 to d13/d5c/dd1/d51/dcb/d104/l10d 0 2024-04-05T03:02:51.854 INFO:tasks.workunit.client.1.smithi203.stdout:6/816: write d0/da/f61 [1809532,45850] 0 2024-04-05T03:02:51.856 INFO:tasks.workunit.client.1.smithi203.stdout:9/849: creat d13/d1b/d6b/d9e/d59/d5f/d103/f10e x:0 0 0 2024-04-05T03:02:51.859 INFO:tasks.workunit.client.1.smithi203.stdout:9/850: mkdir d13/d1b/d6b/d10f 0 2024-04-05T03:02:51.860 INFO:tasks.workunit.client.1.smithi203.stdout:3/694: dwrite d0/dc/d1a/d1d/f5c [0,4194304] 0 2024-04-05T03:02:51.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/695: dread - d0/dc/d1a/d1f/dac/fb6 zero size 2024-04-05T03:02:51.864 INFO:tasks.workunit.client.1.smithi203.stdout:9/851: chown d13/d1b/d33/d64/dfa/d67/d82/ce3 0 1 2024-04-05T03:02:51.865 INFO:tasks.workunit.client.1.smithi203.stdout:6/817: dwrite d0/da/d54/d53/f6a [0,4194304] 0 2024-04-05T03:02:51.865 INFO:tasks.workunit.client.1.smithi203.stdout:3/696: write d0/dc/d1a/d76/d35/f3d [1243480,44294] 0 2024-04-05T03:02:51.867 INFO:tasks.workunit.client.0.smithi153.stdout:0/479: write d9/df/f46 [1244768,78357] 0 2024-04-05T03:02:51.870 INFO:tasks.workunit.client.0.smithi153.stdout:6/528: write d4/dc/df/d16/d5f/f74 [4454999,53503] 0 2024-04-05T03:02:51.870 INFO:tasks.workunit.client.0.smithi153.stdout:0/480: write d9/df/d1e/d45/f6a [1490954,52922] 0 2024-04-05T03:02:51.871 INFO:tasks.workunit.client.1.smithi203.stdout:6/818: unlink d0/da/d18/d38/d86/f7c 0 2024-04-05T03:02:51.871 INFO:tasks.workunit.client.0.smithi153.stdout:6/529: read d4/d38/f75 [1620910,44681] 0 2024-04-05T03:02:51.873 INFO:tasks.workunit.client.1.smithi203.stdout:6/819: write d0/da/d18/d38/d2a/fce [2913937,86963] 0 2024-04-05T03:02:51.875 INFO:tasks.workunit.client.1.smithi203.stdout:9/852: unlink d13/d1b/d6b/d7a/c8c 0 2024-04-05T03:02:51.876 INFO:tasks.workunit.client.0.smithi153.stdout:0/481: symlink d9/df/d1d/d61/l8d 0 2024-04-05T03:02:51.877 INFO:tasks.workunit.client.0.smithi153.stdout:6/530: dread d4/dc/df/d16/d5f/da2/fa4 [0,4194304] 0 2024-04-05T03:02:51.877 INFO:tasks.workunit.client.1.smithi203.stdout:6/820: stat d0/da/l50 0 2024-04-05T03:02:51.877 INFO:tasks.workunit.client.0.smithi153.stdout:0/482: write d9/df/f59 [3164880,35095] 0 2024-04-05T03:02:51.880 INFO:tasks.workunit.client.0.smithi153.stdout:6/531: mknod d4/dc/df/d16/d24/caf 0 2024-04-05T03:02:51.880 INFO:tasks.workunit.client.1.smithi203.stdout:3/697: rename d0/dc/d38/d5d/d66/f7e to d0/dc/d38/d5d/d66/fc8 0 2024-04-05T03:02:51.881 INFO:tasks.workunit.client.1.smithi203.stdout:3/698: chown d0/d41/f9a 0 1 2024-04-05T03:02:51.883 INFO:tasks.workunit.client.0.smithi153.stdout:6/532: creat d4/d38/d4c/fb0 x:0 0 0 2024-04-05T03:02:51.884 INFO:tasks.workunit.client.0.smithi153.stdout:6/533: fdatasync d4/dc/df/d16/d5f/f74 0 2024-04-05T03:02:51.885 INFO:tasks.workunit.client.1.smithi203.stdout:1/807: dread d7/f4f [0,4194304] 0 2024-04-05T03:02:51.887 INFO:tasks.workunit.client.1.smithi203.stdout:9/853: getdents d13/d1b/d6b/d9e/d59/d7b 0 2024-04-05T03:02:51.889 INFO:tasks.workunit.client.1.smithi203.stdout:2/893: sync 2024-04-05T03:02:51.890 INFO:tasks.workunit.client.1.smithi203.stdout:6/821: getdents d0/da/d18/d38/d2a 0 2024-04-05T03:02:51.890 INFO:tasks.workunit.client.0.smithi153.stdout:6/534: rename d4/dc/df/d16/d5f/d73/f7e to d4/dc/df/d16/d5f/d73/fb1 0 2024-04-05T03:02:51.891 INFO:tasks.workunit.client.1.smithi203.stdout:9/854: symlink d13/d5c/dd1/d51/d81/l110 0 2024-04-05T03:02:51.892 INFO:tasks.workunit.client.0.smithi153.stdout:8/430: sync 2024-04-05T03:02:51.892 INFO:tasks.workunit.client.0.smithi153.stdout:5/490: sync 2024-04-05T03:02:51.893 INFO:tasks.workunit.client.1.smithi203.stdout:6/822: mknod d0/da/d18/d38/d2a/d44/d78/cf3 0 2024-04-05T03:02:51.897 INFO:tasks.workunit.client.1.smithi203.stdout:2/894: dwrite d5/d66/f10a [4194304,4194304] 0 2024-04-05T03:02:51.897 INFO:tasks.workunit.client.0.smithi153.stdout:5/491: unlink d0/d25/d4d/d62/l30 0 2024-04-05T03:02:51.898 INFO:tasks.workunit.client.0.smithi153.stdout:6/535: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:51.899 INFO:tasks.workunit.client.1.smithi203.stdout:6/823: fsync d0/da/d18/d38/f47 0 2024-04-05T03:02:51.902 INFO:tasks.workunit.client.0.smithi153.stdout:5/492: write d0/d25/d4d/d62/d37/f4b [3579656,97873] 0 2024-04-05T03:02:51.903 INFO:tasks.workunit.client.0.smithi153.stdout:5/493: chown d0/d25/d68 658322 1 2024-04-05T03:02:51.904 INFO:tasks.workunit.client.1.smithi203.stdout:8/929: dread d1/d4b/d68/d75/dac/da4/fc9 [0,4194304] 0 2024-04-05T03:02:51.905 INFO:tasks.workunit.client.0.smithi153.stdout:6/536: creat d4/dc/df/d16/d36/fb2 x:0 0 0 2024-04-05T03:02:51.905 INFO:tasks.workunit.client.1.smithi203.stdout:3/699: write d0/dc/d1a/d8d/d96/f90 [2256729,69856] 0 2024-04-05T03:02:51.905 INFO:tasks.workunit.client.1.smithi203.stdout:8/930: read d1/d12/d17/d10/f40 [4605839,116720] 0 2024-04-05T03:02:51.907 INFO:tasks.workunit.client.0.smithi153.stdout:8/431: dwrite d7/d85/f41 [0,4194304] 0 2024-04-05T03:02:51.911 INFO:tasks.workunit.client.1.smithi203.stdout:1/808: write d7/d24/fc5 [1003867,103680] 0 2024-04-05T03:02:51.911 INFO:tasks.workunit.client.1.smithi203.stdout:2/895: mkdir d5/d10/d35/d36/d41/d132 0 2024-04-05T03:02:51.911 INFO:tasks.workunit.client.0.smithi153.stdout:5/494: creat d0/d25/fb1 x:0 0 0 2024-04-05T03:02:51.912 INFO:tasks.workunit.client.0.smithi153.stdout:0/483: dread d9/db/f49 [0,4194304] 0 2024-04-05T03:02:51.912 INFO:tasks.workunit.client.0.smithi153.stdout:6/537: write d4/dc/df/d16/d24/d4e/fa9 [586969,37570] 0 2024-04-05T03:02:51.913 INFO:tasks.workunit.client.0.smithi153.stdout:8/432: write d7/f23 [10577000,67838] 0 2024-04-05T03:02:51.915 INFO:tasks.workunit.client.1.smithi203.stdout:9/855: link d13/d1b/d33/d64/dfa/d67/l94 d13/d1b/d6b/d9e/d59/d7b/l111 0 2024-04-05T03:02:51.916 INFO:tasks.workunit.client.0.smithi153.stdout:8/433: write d7/f16 [368509,54186] 0 2024-04-05T03:02:51.920 INFO:tasks.workunit.client.1.smithi203.stdout:2/896: creat d5/d66/d7d/f133 x:0 0 0 2024-04-05T03:02:51.920 INFO:tasks.workunit.client.1.smithi203.stdout:5/789: sync 2024-04-05T03:02:51.920 INFO:tasks.workunit.client.0.smithi153.stdout:5/495: mkdir d0/d25/d4d/d62/d5f/db2 0 2024-04-05T03:02:51.921 INFO:tasks.workunit.client.1.smithi203.stdout:0/816: sync 2024-04-05T03:02:51.921 INFO:tasks.workunit.client.1.smithi203.stdout:4/593: sync 2024-04-05T03:02:51.921 INFO:tasks.workunit.client.1.smithi203.stdout:8/931: rename d1/d12/d17/d47/d53/de0/l101 to d1/d6/d2e/d5c/d99/dbc/l110 0 2024-04-05T03:02:51.922 INFO:tasks.workunit.client.0.smithi153.stdout:5/496: write d0/d25/d4d/d62/d5f/f7e [887045,4262] 0 2024-04-05T03:02:51.922 INFO:tasks.workunit.client.1.smithi203.stdout:6/824: getdents d0/db4 0 2024-04-05T03:02:51.923 INFO:tasks.workunit.client.0.smithi153.stdout:0/484: getdents d9/db/d42/d7e/d8b 0 2024-04-05T03:02:51.923 INFO:tasks.workunit.client.1.smithi203.stdout:9/856: creat d13/d1b/d33/d63/f112 x:0 0 0 2024-04-05T03:02:51.923 INFO:tasks.workunit.client.1.smithi203.stdout:5/790: chown d3/d1f/d49/lae 672920 1 2024-04-05T03:02:51.925 INFO:tasks.workunit.client.0.smithi153.stdout:6/538: creat d4/dc/d7a/d9c/fb3 x:0 0 0 2024-04-05T03:02:51.925 INFO:tasks.workunit.client.0.smithi153.stdout:5/497: rmdir d0/d25/d4d 39 2024-04-05T03:02:51.927 INFO:tasks.workunit.client.0.smithi153.stdout:8/434: creat d7/d13/d19/f8f x:0 0 0 2024-04-05T03:02:51.928 INFO:tasks.workunit.client.1.smithi203.stdout:2/897: rmdir d5/d10/d35/d2d/d7c/d29/d67/d104 0 2024-04-05T03:02:51.929 INFO:tasks.workunit.client.0.smithi153.stdout:8/435: write d7/d13/d19/d1b/f35 [3322509,101977] 0 2024-04-05T03:02:51.929 INFO:tasks.workunit.client.1.smithi203.stdout:5/791: creat d3/d5f/d92/dff/d104/f108 x:0 0 0 2024-04-05T03:02:51.930 INFO:tasks.workunit.client.0.smithi153.stdout:6/539: mknod d4/dc/df/d16/d33/cb4 0 2024-04-05T03:02:51.932 INFO:tasks.workunit.client.0.smithi153.stdout:5/498: symlink d0/d25/d4d/d62/d37/d38/lb3 0 2024-04-05T03:02:51.932 INFO:tasks.workunit.client.1.smithi203.stdout:0/817: dread dd/d17/d1d/d26/d28/f87 [0,4194304] 0 2024-04-05T03:02:51.933 INFO:tasks.workunit.client.0.smithi153.stdout:5/499: chown d0/d25/d4d/d62/d37/c82 86 1 2024-04-05T03:02:51.934 INFO:tasks.workunit.client.0.smithi153.stdout:8/436: read d7/f16 [1265477,52539] 0 2024-04-05T03:02:51.934 INFO:tasks.workunit.client.1.smithi203.stdout:4/594: mknod da/d29/d2e/d57/da5/cb4 0 2024-04-05T03:02:51.935 INFO:tasks.workunit.client.1.smithi203.stdout:0/818: write dd/d17/d1d/d22/d53/d7f/fe3 [69376,1252] 0 2024-04-05T03:02:51.936 INFO:tasks.workunit.client.1.smithi203.stdout:5/792: write d3/d1f/d59/d60/fb4 [9099532,26875] 0 2024-04-05T03:02:51.937 INFO:tasks.workunit.client.1.smithi203.stdout:0/819: write dd/d17/d1d/d22/d53/d7f/fc7 [1517082,125692] 0 2024-04-05T03:02:51.937 INFO:tasks.workunit.client.1.smithi203.stdout:2/898: creat d5/d10/d35/d36/db6/d71/f134 x:0 0 0 2024-04-05T03:02:51.939 INFO:tasks.workunit.client.1.smithi203.stdout:6/825: symlink d0/da/d18/d38/d2a/deb/lf4 0 2024-04-05T03:02:51.940 INFO:tasks.workunit.client.0.smithi153.stdout:5/500: dwrite d0/d9/f1e [0,4194304] 0 2024-04-05T03:02:51.940 INFO:tasks.workunit.client.1.smithi203.stdout:9/857: rmdir d13/d1b/d6b/d9e/d59 39 2024-04-05T03:02:51.941 INFO:tasks.workunit.client.1.smithi203.stdout:6/826: readlink d0/da/d54/d4c/da1/db7/l76 0 2024-04-05T03:02:51.948 INFO:tasks.workunit.client.0.smithi153.stdout:8/437: creat d7/d13/d19/d1b/d6c/f90 x:0 0 0 2024-04-05T03:02:51.948 INFO:tasks.workunit.client.0.smithi153.stdout:8/438: fsync f6 0 2024-04-05T03:02:51.948 INFO:tasks.workunit.client.0.smithi153.stdout:8/439: chown f6 823 1 2024-04-05T03:02:51.949 INFO:tasks.workunit.client.0.smithi153.stdout:6/540: dwrite d4/dc/df/f39 [0,4194304] 0 2024-04-05T03:02:51.950 INFO:tasks.workunit.client.1.smithi203.stdout:1/809: dread d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:51.951 INFO:tasks.workunit.client.0.smithi153.stdout:8/440: chown d7/d13/d19/d1b/d4f/d68/f84 84 1 2024-04-05T03:02:51.951 INFO:tasks.workunit.client.0.smithi153.stdout:5/501: dread d0/d25/f31 [0,4194304] 0 2024-04-05T03:02:51.953 INFO:tasks.workunit.client.1.smithi203.stdout:0/820: dwrite dd/d17/d1d/d22/d53/db0/fd4 [0,4194304] 0 2024-04-05T03:02:51.955 INFO:tasks.workunit.client.0.smithi153.stdout:8/441: creat d7/d13/f91 x:0 0 0 2024-04-05T03:02:51.955 INFO:tasks.workunit.client.1.smithi203.stdout:9/858: dread d13/d5c/dd1/f4c [0,4194304] 0 2024-04-05T03:02:51.958 INFO:tasks.workunit.client.0.smithi153.stdout:6/541: rename d4/dc/df/d16/d5f/da2/ca3 to d4/dc/d7a/d9c/cb5 0 2024-04-05T03:02:51.959 INFO:tasks.workunit.client.1.smithi203.stdout:8/932: getdents d1/d6/d2e/d5c/d99 0 2024-04-05T03:02:51.960 INFO:tasks.workunit.client.0.smithi153.stdout:6/542: truncate d4/d38/f8b 1080610 0 2024-04-05T03:02:51.961 INFO:tasks.workunit.client.1.smithi203.stdout:3/700: dwrite d0/dc/d38/f8f [0,4194304] 0 2024-04-05T03:02:51.961 INFO:tasks.workunit.client.1.smithi203.stdout:4/595: unlink da/d3c/d4c/l4f 0 2024-04-05T03:02:51.964 INFO:tasks.workunit.client.1.smithi203.stdout:4/596: write da/d3c/d4c/f87 [2193227,11567] 0 2024-04-05T03:02:51.964 INFO:tasks.workunit.client.1.smithi203.stdout:5/793: write d3/d1f/d59/d64/f91 [806395,94497] 0 2024-04-05T03:02:51.968 INFO:tasks.workunit.client.1.smithi203.stdout:8/933: rmdir d1/d12/da8 39 2024-04-05T03:02:51.968 INFO:tasks.workunit.client.1.smithi203.stdout:1/810: fsync d7/d14/f16 0 2024-04-05T03:02:51.972 INFO:tasks.workunit.client.0.smithi153.stdout:6/543: rename d4/dc/df/d16/d17/f3b to d4/dc/df/fb6 0 2024-04-05T03:02:51.974 INFO:tasks.workunit.client.1.smithi203.stdout:5/794: mknod d3/d4/dc/c109 0 2024-04-05T03:02:51.976 INFO:tasks.workunit.client.1.smithi203.stdout:8/934: fsync d1/d4b/d5f/daf/fba 0 2024-04-05T03:02:51.976 INFO:tasks.workunit.client.1.smithi203.stdout:9/859: getdents d13/d5c/dd1 0 2024-04-05T03:02:51.979 INFO:tasks.workunit.client.0.smithi153.stdout:8/442: dwrite d7/d2a/f44 [0,4194304] 0 2024-04-05T03:02:51.979 INFO:tasks.workunit.client.1.smithi203.stdout:8/935: fdatasync d1/d12/d1a/f1e 0 2024-04-05T03:02:51.979 INFO:tasks.workunit.client.1.smithi203.stdout:5/795: fdatasync d3/d4/d94/fed 0 2024-04-05T03:02:51.980 INFO:tasks.workunit.client.0.smithi153.stdout:8/443: fsync d7/d13/d19/f4e 0 2024-04-05T03:02:51.981 INFO:tasks.workunit.client.0.smithi153.stdout:8/444: chown f0 1269484 1 2024-04-05T03:02:51.981 INFO:tasks.workunit.client.0.smithi153.stdout:6/544: rename d4/dc/df/d16/d5f/f74 to d4/dc/df/d16/d5f/fb7 0 2024-04-05T03:02:51.982 INFO:tasks.workunit.client.1.smithi203.stdout:2/899: dwrite d5/d10/d35/fad [0,4194304] 0 2024-04-05T03:02:51.983 INFO:tasks.workunit.client.1.smithi203.stdout:3/701: dwrite d0/f7f [0,4194304] 0 2024-04-05T03:02:51.983 INFO:tasks.workunit.client.0.smithi153.stdout:6/545: read d4/dc/df/d16/f23 [3894369,53112] 0 2024-04-05T03:02:51.984 INFO:tasks.workunit.client.1.smithi203.stdout:9/860: chown d13/d1b/d33/d64/dfa/d67/c84 86887224 1 2024-04-05T03:02:51.985 INFO:tasks.workunit.client.1.smithi203.stdout:1/811: symlink d7/d14/d2b/l103 0 2024-04-05T03:02:51.986 INFO:tasks.workunit.client.1.smithi203.stdout:5/796: fdatasync d3/f7a 0 2024-04-05T03:02:51.987 INFO:tasks.workunit.client.1.smithi203.stdout:1/812: truncate d7/d14/f93 13123856 0 2024-04-05T03:02:51.988 INFO:tasks.workunit.client.0.smithi153.stdout:8/445: dwrite d7/f23 [0,4194304] 0 2024-04-05T03:02:51.989 INFO:tasks.workunit.client.1.smithi203.stdout:6/827: dread d0/da/d54/d4c/da1/db7/d95/fe8 [0,4194304] 0 2024-04-05T03:02:51.992 INFO:tasks.workunit.client.1.smithi203.stdout:9/861: mkdir d13/d5c/dd1/d51/dcb/d104/d113 0 2024-04-05T03:02:51.992 INFO:tasks.workunit.client.1.smithi203.stdout:2/900: creat d5/d66/d7d/f135 x:0 0 0 2024-04-05T03:02:51.992 INFO:tasks.workunit.client.1.smithi203.stdout:4/597: getdents da/d29/d2e/d8b 0 2024-04-05T03:02:51.994 INFO:tasks.workunit.client.1.smithi203.stdout:9/862: chown d13/d1b/fbd 3050686 1 2024-04-05T03:02:51.995 INFO:tasks.workunit.client.0.smithi153.stdout:6/546: dwrite d4/dc/df/d16/d24/d5e/f60 [0,4194304] 0 2024-04-05T03:02:51.995 INFO:tasks.workunit.client.1.smithi203.stdout:3/702: read d0/dc/d1a/d1d/f29 [148253,48579] 0 2024-04-05T03:02:51.995 INFO:tasks.workunit.client.0.smithi153.stdout:8/446: readlink d7/d13/d19/d1b/d6d/d8c/l7d 0 2024-04-05T03:02:51.997 INFO:tasks.workunit.client.0.smithi153.stdout:8/447: chown d7/f23 1048 1 2024-04-05T03:02:52.001 INFO:tasks.workunit.client.0.smithi153.stdout:6/547: rmdir d4/dc/df 39 2024-04-05T03:02:52.002 INFO:tasks.workunit.client.1.smithi203.stdout:8/936: creat d1/d12/d2f/d55/f111 x:0 0 0 2024-04-05T03:02:52.002 INFO:tasks.workunit.client.1.smithi203.stdout:6/828: dwrite d0/da/d54/d4c/f5b [0,4194304] 0 2024-04-05T03:02:52.003 INFO:tasks.workunit.client.1.smithi203.stdout:1/813: mknod d7/d14/d83/deb/c104 0 2024-04-05T03:02:52.003 INFO:tasks.workunit.client.1.smithi203.stdout:9/863: read d13/d5c/dd1/d31/f3e [2742685,16176] 0 2024-04-05T03:02:52.005 INFO:tasks.workunit.client.0.smithi153.stdout:6/548: write d4/dc/df/d16/d44/f96 [991904,27914] 0 2024-04-05T03:02:52.009 INFO:tasks.workunit.client.1.smithi203.stdout:8/937: dread d1/f58 [0,4194304] 0 2024-04-05T03:02:52.012 INFO:tasks.workunit.client.1.smithi203.stdout:3/703: unlink d0/f52 0 2024-04-05T03:02:52.015 INFO:tasks.workunit.client.1.smithi203.stdout:4/598: creat da/d29/d2e/db0/fb5 x:0 0 0 2024-04-05T03:02:52.016 INFO:tasks.workunit.client.0.smithi153.stdout:6/549: dwrite d4/dc/df/d16/d24/f32 [0,4194304] 0 2024-04-05T03:02:52.019 INFO:tasks.workunit.client.1.smithi203.stdout:2/901: write d5/d10/d35/d2d/d7c/d29/d67/f77 [3561951,17682] 0 2024-04-05T03:02:52.019 INFO:tasks.workunit.client.1.smithi203.stdout:8/938: rmdir d1/d90 39 2024-04-05T03:02:52.021 INFO:tasks.workunit.client.1.smithi203.stdout:2/902: read - d5/d10/d35/d2d/f12c zero size 2024-04-05T03:02:52.022 INFO:tasks.workunit.client.1.smithi203.stdout:3/704: mkdir d0/dc/d1a/d8d/dc9 0 2024-04-05T03:02:52.022 INFO:tasks.workunit.client.1.smithi203.stdout:9/864: getdents d13/d1b/d6b/d10f 0 2024-04-05T03:02:52.024 INFO:tasks.workunit.client.1.smithi203.stdout:9/865: truncate d13/fe6 437070 0 2024-04-05T03:02:52.024 INFO:tasks.workunit.client.1.smithi203.stdout:5/797: truncate d3/d1f/d59/d64/f63 1237637 0 2024-04-05T03:02:52.025 INFO:tasks.workunit.client.0.smithi153.stdout:6/550: dwrite d4/dc/df/d16/d5f/d73/fb1 [0,4194304] 0 2024-04-05T03:02:52.026 INFO:tasks.workunit.client.1.smithi203.stdout:2/903: unlink d5/c129 0 2024-04-05T03:02:52.027 INFO:tasks.workunit.client.0.smithi153.stdout:6/551: write d4/dc/df/d16/d24/d5e/f60 [3971532,93926] 0 2024-04-05T03:02:52.029 INFO:tasks.workunit.client.1.smithi203.stdout:1/814: write d7/f4f [3730661,8581] 0 2024-04-05T03:02:52.031 INFO:tasks.workunit.client.1.smithi203.stdout:3/705: unlink d0/dc/d1a/d76/d35/f3c 0 2024-04-05T03:02:52.037 INFO:tasks.workunit.client.1.smithi203.stdout:1/815: rename d7/d24/d33/d48/d4b/d6b/db9/fce to d7/d14/d83/f105 0 2024-04-05T03:02:52.039 INFO:tasks.workunit.client.1.smithi203.stdout:9/866: truncate d13/d5c/dd1/d51/f53 2404441 0 2024-04-05T03:02:52.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:51 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:52.051 INFO:tasks.workunit.client.0.smithi153.stdout:8/448: dwrite d7/d13/d19/f22 [0,4194304] 0 2024-04-05T03:02:52.052 INFO:tasks.workunit.client.1.smithi203.stdout:4/599: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:51 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:02:52.085 INFO:tasks.workunit.client.0.smithi153.stdout:4/571: sync 2024-04-05T03:02:52.085 INFO:tasks.workunit.client.0.smithi153.stdout:1/447: sync 2024-04-05T03:02:52.085 INFO:tasks.workunit.client.0.smithi153.stdout:6/552: dwrite d4/f7 [0,4194304] 0 2024-04-05T03:02:52.086 INFO:tasks.workunit.client.1.smithi203.stdout:1/816: mkdir d7/d14/d2b/d58/d86/d91/da6/df9/d106 0 2024-04-05T03:02:52.087 INFO:tasks.workunit.client.0.smithi153.stdout:4/572: fdatasync da/de/d56/f70 0 2024-04-05T03:02:52.088 INFO:tasks.workunit.client.1.smithi203.stdout:9/867: truncate d13/d1b/d6b/d9e/d59/dbc/fc5 156654 0 2024-04-05T03:02:52.088 INFO:tasks.workunit.client.0.smithi153.stdout:4/573: dread - da/db/d20/d3c/d91/fbb zero size 2024-04-05T03:02:52.089 INFO:tasks.workunit.client.1.smithi203.stdout:9/868: stat d13/d5c/dd1/d51/d81/lae 0 2024-04-05T03:02:52.092 INFO:tasks.workunit.client.0.smithi153.stdout:6/553: fsync d4/f15 0 2024-04-05T03:02:52.093 INFO:tasks.workunit.client.0.smithi153.stdout:6/554: readlink d4/dc/df/d16/d44/l76 0 2024-04-05T03:02:52.094 INFO:tasks.workunit.client.0.smithi153.stdout:4/574: mkdir da/db/d20/d45/dc3 0 2024-04-05T03:02:52.095 INFO:tasks.workunit.client.0.smithi153.stdout:4/575: stat da/d2a/f47 0 2024-04-05T03:02:52.096 INFO:tasks.workunit.client.0.smithi153.stdout:6/555: creat d4/dc/df/d16/d17/d54/fb8 x:0 0 0 2024-04-05T03:02:52.097 INFO:tasks.workunit.client.1.smithi203.stdout:4/600: creat da/d29/d2e/d57/da7/dab/fb6 x:0 0 0 2024-04-05T03:02:52.099 INFO:tasks.workunit.client.0.smithi153.stdout:4/576: fdatasync da/db/f1a 0 2024-04-05T03:02:52.101 INFO:tasks.workunit.client.1.smithi203.stdout:4/601: rmdir da/d71 39 2024-04-05T03:02:52.101 INFO:tasks.workunit.client.0.smithi153.stdout:6/556: creat d4/dc/df/d16/d24/fb9 x:0 0 0 2024-04-05T03:02:52.102 INFO:tasks.workunit.client.1.smithi203.stdout:4/602: chown da/d29/d2e/c58 801583340 1 2024-04-05T03:02:52.102 INFO:tasks.workunit.client.1.smithi203.stdout:3/706: read d0/dc/d34/f6e [2539221,103148] 0 2024-04-05T03:02:52.102 INFO:tasks.workunit.client.0.smithi153.stdout:6/557: chown d4/dc/df/d16/l31 2434581 1 2024-04-05T03:02:52.103 INFO:tasks.workunit.client.0.smithi153.stdout:4/577: mknod da/db/d8b/db7/cc4 0 2024-04-05T03:02:52.103 INFO:tasks.workunit.client.0.smithi153.stdout:6/558: chown d4/dc/df/f26 5767 1 2024-04-05T03:02:52.105 INFO:tasks.workunit.client.1.smithi203.stdout:4/603: creat da/d3c/d4c/fb7 x:0 0 0 2024-04-05T03:02:52.109 INFO:tasks.workunit.client.0.smithi153.stdout:6/559: dread d4/dc/df/d16/d24/d4e/fa9 [0,4194304] 0 2024-04-05T03:02:52.112 INFO:tasks.workunit.client.1.smithi203.stdout:2/904: dread d5/d10/db8/dab/fa8 [0,4194304] 0 2024-04-05T03:02:52.112 INFO:tasks.workunit.client.1.smithi203.stdout:9/869: dread d13/d1b/d33/d64/f6d [0,4194304] 0 2024-04-05T03:02:52.112 INFO:tasks.workunit.client.1.smithi203.stdout:5/798: dwrite d3/d1f/d59/d64/f78 [0,4194304] 0 2024-04-05T03:02:52.113 INFO:tasks.workunit.client.0.smithi153.stdout:6/560: dread d4/d38/d4c/f6a [0,4194304] 0 2024-04-05T03:02:52.115 INFO:tasks.workunit.client.1.smithi203.stdout:4/604: rename da/d3c/l3f to da/d29/d2e/d57/da5/lb8 0 2024-04-05T03:02:52.115 INFO:tasks.workunit.client.0.smithi153.stdout:6/561: chown d4/dc/df/d16/d33/l49 117510700 1 2024-04-05T03:02:52.119 INFO:tasks.workunit.client.1.smithi203.stdout:5/799: unlink d3/d4/dc/f2d 0 2024-04-05T03:02:52.121 INFO:tasks.workunit.client.0.smithi153.stdout:6/562: mknod d4/dc/df/cba 0 2024-04-05T03:02:52.122 INFO:tasks.workunit.client.1.smithi203.stdout:9/870: write d13/d1b/d6b/d9e/d77/f95 [869585,113562] 0 2024-04-05T03:02:52.124 INFO:tasks.workunit.client.1.smithi203.stdout:3/707: dwrite d0/dc/d1a/d76/f94 [0,4194304] 0 2024-04-05T03:02:52.125 INFO:tasks.workunit.client.1.smithi203.stdout:9/871: stat d13/d1b/d33/d64/dfa/d67/d82/ce3 0 2024-04-05T03:02:52.126 INFO:tasks.workunit.client.0.smithi153.stdout:6/563: rename d4/d38/f93 to d4/d38/fbb 0 2024-04-05T03:02:52.127 INFO:tasks.workunit.client.1.smithi203.stdout:8/939: dread d1/d6/fab [0,4194304] 0 2024-04-05T03:02:52.127 INFO:tasks.workunit.client.1.smithi203.stdout:9/872: truncate d13/fe6 835608 0 2024-04-05T03:02:52.127 INFO:tasks.workunit.client.1.smithi203.stdout:2/905: link d5/d10/d35/d2d/d7c/d72/d8f/lc8 d5/d10/d2f/l136 0 2024-04-05T03:02:52.128 INFO:tasks.workunit.client.0.smithi153.stdout:4/578: dread da/db/f96 [0,4194304] 0 2024-04-05T03:02:52.128 INFO:tasks.workunit.client.0.smithi153.stdout:4/579: readlink da/db/d20/d35/l3a 0 2024-04-05T03:02:52.130 INFO:tasks.workunit.client.1.smithi203.stdout:2/906: readlink d5/d10/d35/df4/lff 0 2024-04-05T03:02:52.131 INFO:tasks.workunit.client.1.smithi203.stdout:9/873: fsync d13/d1b/d6b/d9e/d59/dbc/fc4 0 2024-04-05T03:02:52.132 INFO:tasks.workunit.client.0.smithi153.stdout:6/564: symlink d4/dc/df/d16/d36/d4d/dad/lbc 0 2024-04-05T03:02:52.133 INFO:tasks.workunit.client.1.smithi203.stdout:3/708: mkdir d0/dc/d38/dca 0 2024-04-05T03:02:52.135 INFO:tasks.workunit.client.1.smithi203.stdout:1/817: dread d7/d24/d33/d48/d4b/d51/fb2 [0,4194304] 0 2024-04-05T03:02:52.135 INFO:tasks.workunit.client.1.smithi203.stdout:5/800: write d3/f36 [3012164,11892] 0 2024-04-05T03:02:52.136 INFO:tasks.workunit.client.0.smithi153.stdout:4/580: link c2 da/db/d20/d3c/d91/cc5 0 2024-04-05T03:02:52.139 INFO:tasks.workunit.client.1.smithi203.stdout:9/874: symlink d13/d1b/d33/l114 0 2024-04-05T03:02:52.140 INFO:tasks.workunit.client.0.smithi153.stdout:6/565: dwrite d4/d38/d4c/fa0 [0,4194304] 0 2024-04-05T03:02:52.141 INFO:tasks.workunit.client.0.smithi153.stdout:6/566: dread - d4/d38/d4c/fb0 zero size 2024-04-05T03:02:52.142 INFO:tasks.workunit.client.0.smithi153.stdout:6/567: chown d4/dc/df/d16/d36/d4d/d64 0 1 2024-04-05T03:02:52.142 INFO:tasks.workunit.client.0.smithi153.stdout:4/581: creat da/db/d20/d35/d38/d76/fc6 x:0 0 0 2024-04-05T03:02:52.143 INFO:tasks.workunit.client.1.smithi203.stdout:8/940: dwrite d1/f58 [0,4194304] 0 2024-04-05T03:02:52.145 INFO:tasks.workunit.client.0.smithi153.stdout:6/568: write d4/fb [4020564,91580] 0 2024-04-05T03:02:52.146 INFO:tasks.workunit.client.0.smithi153.stdout:4/582: mkdir da/db/d20/d3c/d81/db6/dc7 0 2024-04-05T03:02:52.149 INFO:tasks.workunit.client.0.smithi153.stdout:6/569: creat d4/dc/df/d16/d36/d4d/d64/fbd x:0 0 0 2024-04-05T03:02:52.150 INFO:tasks.workunit.client.1.smithi203.stdout:4/605: dread da/d29/d2e/d57/f7f [0,4194304] 0 2024-04-05T03:02:52.150 INFO:tasks.workunit.client.1.smithi203.stdout:3/709: symlink d0/dc/d38/d5d/d66/lcb 0 2024-04-05T03:02:52.153 INFO:tasks.workunit.client.0.smithi153.stdout:4/583: symlink da/d2a/lc8 0 2024-04-05T03:02:52.154 INFO:tasks.workunit.client.0.smithi153.stdout:6/570: dread d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:52.155 INFO:tasks.workunit.client.1.smithi203.stdout:5/801: unlink d3/d1f/d59/cf6 0 2024-04-05T03:02:52.156 INFO:tasks.workunit.client.0.smithi153.stdout:4/584: mknod da/db/d8b/cc9 0 2024-04-05T03:02:52.157 INFO:tasks.workunit.client.0.smithi153.stdout:4/585: dread - da/de/d56/d6a/d2d/f7a zero size 2024-04-05T03:02:52.157 INFO:tasks.workunit.client.1.smithi203.stdout:3/710: creat d0/dc/d1a/d8d/d96/fcc x:0 0 0 2024-04-05T03:02:52.159 INFO:tasks.workunit.client.1.smithi203.stdout:4/606: rename da/f56 to da/d29/d6a/d80/fb9 0 2024-04-05T03:02:52.159 INFO:tasks.workunit.client.1.smithi203.stdout:5/802: stat d3/d4/dc/l2b 0 2024-04-05T03:02:52.161 INFO:tasks.workunit.client.1.smithi203.stdout:8/941: rmdir d1/d12/d17/d10/d10b 0 2024-04-05T03:02:52.161 INFO:tasks.workunit.client.1.smithi203.stdout:9/875: dwrite d13/d1b/d6b/fbf [0,4194304] 0 2024-04-05T03:02:52.163 INFO:tasks.workunit.client.0.smithi153.stdout:4/586: creat da/db/d20/d35/d38/d76/d90/fca x:0 0 0 2024-04-05T03:02:52.163 INFO:tasks.workunit.client.1.smithi203.stdout:3/711: fdatasync d0/dc/d1a/d8d/d96/fcc 0 2024-04-05T03:02:52.165 INFO:tasks.workunit.client.1.smithi203.stdout:8/942: chown d1/d4b 216569539 1 2024-04-05T03:02:52.166 INFO:tasks.workunit.client.1.smithi203.stdout:5/803: rmdir d3/d5f/d92/dff/d104 39 2024-04-05T03:02:52.166 INFO:tasks.workunit.client.0.smithi153.stdout:4/587: rename da/de/d56/d6a/cac to da/de/d56/d6a/ccb 0 2024-04-05T03:02:52.168 INFO:tasks.workunit.client.1.smithi203.stdout:9/876: fsync d13/d5c/dd1/d51/f52 0 2024-04-05T03:02:52.168 INFO:tasks.workunit.client.1.smithi203.stdout:2/907: dwrite d5/d66/d7d/f94 [0,4194304] 0 2024-04-05T03:02:52.170 INFO:tasks.workunit.client.0.smithi153.stdout:4/588: mknod da/db/d20/d35/d38/ccc 0 2024-04-05T03:02:52.171 INFO:tasks.workunit.client.1.smithi203.stdout:8/943: creat d1/d12/d1a/da0/f112 x:0 0 0 2024-04-05T03:02:52.174 INFO:tasks.workunit.client.1.smithi203.stdout:1/818: dwrite d7/d24/f66 [0,4194304] 0 2024-04-05T03:02:52.174 INFO:tasks.workunit.client.1.smithi203.stdout:2/908: dread - d5/d10/d35/d36/db6/d71/f134 zero size 2024-04-05T03:02:52.174 INFO:tasks.workunit.client.1.smithi203.stdout:5/804: creat d3/d4/d94/f10a x:0 0 0 2024-04-05T03:02:52.175 INFO:tasks.workunit.client.0.smithi153.stdout:4/589: unlink da/db/d20/fb5 0 2024-04-05T03:02:52.177 INFO:tasks.workunit.client.1.smithi203.stdout:5/805: read - d3/d5f/d92/fee zero size 2024-04-05T03:02:52.178 INFO:tasks.workunit.client.1.smithi203.stdout:9/877: write d13/d1b/d6b/d9e/d77/f95 [1789467,39629] 0 2024-04-05T03:02:52.179 INFO:tasks.workunit.client.0.smithi153.stdout:4/590: read da/de/d56/d78/f7e [1997490,7074] 0 2024-04-05T03:02:52.181 INFO:tasks.workunit.client.0.smithi153.stdout:4/591: creat da/d2a/fcd x:0 0 0 2024-04-05T03:02:52.183 INFO:tasks.workunit.client.1.smithi203.stdout:2/909: mkdir d5/d10/d137 0 2024-04-05T03:02:52.183 INFO:tasks.workunit.client.0.smithi153.stdout:4/592: readlink da/de/d56/d78/d8f/d68/l85 0 2024-04-05T03:02:52.187 INFO:tasks.workunit.client.1.smithi203.stdout:4/607: write da/d29/d6a/f6b [44741,66880] 0 2024-04-05T03:02:52.188 INFO:tasks.workunit.client.1.smithi203.stdout:9/878: creat d13/d5c/d9d/f115 x:0 0 0 2024-04-05T03:02:52.189 INFO:tasks.workunit.client.1.smithi203.stdout:2/910: fsync d5/d66/fde 0 2024-04-05T03:02:52.190 INFO:tasks.workunit.client.1.smithi203.stdout:1/819: creat d7/d14/d2b/d58/d86/d91/da6/df9/d106/f107 x:0 0 0 2024-04-05T03:02:52.190 INFO:tasks.workunit.client.1.smithi203.stdout:9/879: stat d13/d1b/d33/d64/dfa/fb0 0 2024-04-05T03:02:52.190 INFO:tasks.workunit.client.0.smithi153.stdout:6/571: dread d4/f20 [0,4194304] 0 2024-04-05T03:02:52.193 INFO:tasks.workunit.client.1.smithi203.stdout:8/944: write d1/d6/d8f/fb0 [424163,66226] 0 2024-04-05T03:02:52.193 INFO:tasks.workunit.client.1.smithi203.stdout:1/820: creat d7/d24/d33/d48/d4b/d51/f108 x:0 0 0 2024-04-05T03:02:52.196 INFO:tasks.workunit.client.1.smithi203.stdout:5/806: write d3/d1f/d49/f84 [842278,125245] 0 2024-04-05T03:02:52.196 INFO:tasks.workunit.client.0.smithi153.stdout:6/572: link d4/dc/d7a/d9c/fb3 d4/d38/fbe 0 2024-04-05T03:02:52.197 INFO:tasks.workunit.client.1.smithi203.stdout:4/608: mknod da/d71/cba 0 2024-04-05T03:02:52.197 INFO:tasks.workunit.client.1.smithi203.stdout:8/945: read d1/d4b/d5f/daf/fbf [1121791,77893] 0 2024-04-05T03:02:52.197 INFO:tasks.workunit.client.1.smithi203.stdout:1/821: write d7/d14/d9d/fc4 [462305,129874] 0 2024-04-05T03:02:52.200 INFO:tasks.workunit.client.0.smithi153.stdout:6/573: symlink d4/dc/df/d16/d24/d8d/lbf 0 2024-04-05T03:02:52.201 INFO:tasks.workunit.client.1.smithi203.stdout:2/911: mkdir d5/d10/d35/d2d/d7c/d72/d10b/d138 0 2024-04-05T03:02:52.201 INFO:tasks.workunit.client.1.smithi203.stdout:8/946: read d1/d6/d2e/d5c/fd6 [1139624,20943] 0 2024-04-05T03:02:52.201 INFO:tasks.workunit.client.0.smithi153.stdout:4/593: dread da/db/d20/d3c/d81/db6/f8c [0,4194304] 0 2024-04-05T03:02:52.202 INFO:tasks.workunit.client.1.smithi203.stdout:5/807: truncate f2 775701 0 2024-04-05T03:02:52.203 INFO:tasks.workunit.client.0.smithi153.stdout:4/594: read da/de/f14 [8263,71983] 0 2024-04-05T03:02:52.206 INFO:tasks.workunit.client.1.smithi203.stdout:1/822: symlink d7/d14/d2b/d58/de8/l109 0 2024-04-05T03:02:52.207 INFO:tasks.workunit.client.1.smithi203.stdout:5/808: readlink d3/d4/dc/l38 0 2024-04-05T03:02:52.208 INFO:tasks.workunit.client.1.smithi203.stdout:5/809: chown d3/d4/d94/fed 123416 1 2024-04-05T03:02:52.209 INFO:tasks.workunit.client.1.smithi203.stdout:9/880: getdents d13/d1b/d33/d64/dfa 0 2024-04-05T03:02:52.209 INFO:tasks.workunit.client.0.smithi153.stdout:6/574: dwrite d4/dc/df/d16/d24/d4e/f5a [4194304,4194304] 0 2024-04-05T03:02:52.210 INFO:tasks.workunit.client.1.smithi203.stdout:2/912: mknod d5/d10/d35/d2d/d7c/d29/c139 0 2024-04-05T03:02:52.211 INFO:tasks.workunit.client.0.smithi153.stdout:4/595: dwrite da/de/f88 [0,4194304] 0 2024-04-05T03:02:52.214 INFO:tasks.workunit.client.1.smithi203.stdout:8/947: truncate d1/d6/f7c 3835832 0 2024-04-05T03:02:52.217 INFO:tasks.workunit.client.1.smithi203.stdout:5/810: dwrite d3/d1f/fc0 [0,4194304] 0 2024-04-05T03:02:52.217 INFO:tasks.workunit.client.0.smithi153.stdout:6/575: getdents d4 0 2024-04-05T03:02:52.218 INFO:tasks.workunit.client.0.smithi153.stdout:6/576: write d4/dc/df/d16/d5f/f7c [3045628,98921] 0 2024-04-05T03:02:52.218 INFO:tasks.workunit.client.1.smithi203.stdout:4/609: rename da/d3c/c51 to da/d3c/cbb 0 2024-04-05T03:02:52.219 INFO:tasks.workunit.client.1.smithi203.stdout:8/948: stat d1/d12/da8/cf0 0 2024-04-05T03:02:52.221 INFO:tasks.workunit.client.1.smithi203.stdout:8/949: write d1/d12/d2f/d1f/f50 [7616094,57840] 0 2024-04-05T03:02:52.223 INFO:tasks.workunit.client.1.smithi203.stdout:9/881: truncate d13/d1b/d6b/d9e/d77/fd3 3012159 0 2024-04-05T03:02:52.225 INFO:tasks.workunit.client.0.smithi153.stdout:4/596: rename da/de/d56/d6a/f44 to da/de/d56/d78/fce 0 2024-04-05T03:02:52.226 INFO:tasks.workunit.client.1.smithi203.stdout:5/811: mknod d3/d4/dd6/dfe/c10b 0 2024-04-05T03:02:52.226 INFO:tasks.workunit.client.1.smithi203.stdout:4/610: mkdir da/d29/d2e/d57/d96/dbc 0 2024-04-05T03:02:52.228 INFO:tasks.workunit.client.1.smithi203.stdout:5/812: write d3/d4/dc/d67/f7f [359227,125066] 0 2024-04-05T03:02:52.228 INFO:tasks.workunit.client.0.smithi153.stdout:6/577: rename d4/dc/df/d16/d36/d52/lae to d4/lc0 0 2024-04-05T03:02:52.229 INFO:tasks.workunit.client.1.smithi203.stdout:9/882: read - d13/d1b/d33/d64/fd7 zero size 2024-04-05T03:02:52.229 INFO:tasks.workunit.client.1.smithi203.stdout:1/823: creat d7/d14/f10a x:0 0 0 2024-04-05T03:02:52.232 INFO:tasks.workunit.client.1.smithi203.stdout:5/813: mkdir d3/d1f/d49/d10c 0 2024-04-05T03:02:52.232 INFO:tasks.workunit.client.1.smithi203.stdout:4/611: creat da/d29/d2e/d57/d96/dbc/fbd x:0 0 0 2024-04-05T03:02:52.234 INFO:tasks.workunit.client.1.smithi203.stdout:9/883: mknod d13/d5c/dd1/d51/dcb/d104/c116 0 2024-04-05T03:02:52.234 INFO:tasks.workunit.client.1.smithi203.stdout:5/814: mkdir d3/d1f/d49/d10c/d10d 0 2024-04-05T03:02:52.236 INFO:tasks.workunit.client.1.smithi203.stdout:5/815: dread - d3/d4/dc/f79 zero size 2024-04-05T03:02:52.237 INFO:tasks.workunit.client.1.smithi203.stdout:9/884: creat d13/d1b/d33/f117 x:0 0 0 2024-04-05T03:02:52.239 INFO:tasks.workunit.client.1.smithi203.stdout:9/885: fdatasync d13/d5c/f9f 0 2024-04-05T03:02:52.240 INFO:tasks.workunit.client.1.smithi203.stdout:5/816: getdents d3/d4/d9e/dde 0 2024-04-05T03:02:52.241 INFO:tasks.workunit.client.0.smithi153.stdout:2/503: sync 2024-04-05T03:02:52.242 INFO:tasks.workunit.client.1.smithi203.stdout:4/612: dwrite da/d3c/d4c/f73 [0,4194304] 0 2024-04-05T03:02:52.243 INFO:tasks.workunit.client.0.smithi153.stdout:2/504: dread - d9/da/d19/d2e/d48/f6b zero size 2024-04-05T03:02:52.244 INFO:tasks.workunit.client.1.smithi203.stdout:4/613: readlink da/d29/d2e/d8b/la1 0 2024-04-05T03:02:52.245 INFO:tasks.workunit.client.0.smithi153.stdout:6/578: write d4/dc/df/d16/d24/f29 [1256800,86285] 0 2024-04-05T03:02:52.248 INFO:tasks.workunit.client.0.smithi153.stdout:6/579: truncate d4/dc/df/d16/d44/fa1 242920 0 2024-04-05T03:02:52.248 INFO:tasks.workunit.client.1.smithi203.stdout:5/817: rename d3/d1f/d59/d60/dc1 to d3/d1f/d10e 0 2024-04-05T03:02:52.250 INFO:tasks.workunit.client.0.smithi153.stdout:2/505: unlink d9/da/d19/f46 0 2024-04-05T03:02:52.251 INFO:tasks.workunit.client.1.smithi203.stdout:9/886: fdatasync d13/d1b/d6b/d9e/d59/fb2 0 2024-04-05T03:02:52.252 INFO:tasks.workunit.client.1.smithi203.stdout:9/887: write d13/d5c/dd1/d51/fab [476564,14055] 0 2024-04-05T03:02:52.258 INFO:tasks.workunit.client.1.smithi203.stdout:2/913: dread d5/d10/d35/d36/db6/d71/f97 [0,4194304] 0 2024-04-05T03:02:52.258 INFO:tasks.workunit.client.1.smithi203.stdout:1/824: write d7/d14/f21 [1454493,47108] 0 2024-04-05T03:02:52.260 INFO:tasks.workunit.client.0.smithi153.stdout:4/597: dread da/de/f1e [0,4194304] 0 2024-04-05T03:02:52.260 INFO:tasks.workunit.client.1.smithi203.stdout:1/825: chown d7/f99 414 1 2024-04-05T03:02:52.261 INFO:tasks.workunit.client.1.smithi203.stdout:9/888: dwrite d13/d5c/dd1/d51/d81/fa5 [0,4194304] 0 2024-04-05T03:02:52.261 INFO:tasks.workunit.client.1.smithi203.stdout:5/818: getdents d3/d4/d94 0 2024-04-05T03:02:52.261 INFO:tasks.workunit.client.1.smithi203.stdout:8/950: write d1/d6/f7c [4677147,14445] 0 2024-04-05T03:02:52.263 INFO:tasks.workunit.client.1.smithi203.stdout:5/819: stat d3/d1f/d49/l65 0 2024-04-05T03:02:52.267 INFO:tasks.workunit.client.1.smithi203.stdout:1/826: creat d7/d24/d33/d48/d4b/dea/def/f10b x:0 0 0 2024-04-05T03:02:52.267 INFO:tasks.workunit.client.0.smithi153.stdout:4/598: dwrite da/db/d20/d3c/fb8 [0,4194304] 0 2024-04-05T03:02:52.268 INFO:tasks.workunit.client.1.smithi203.stdout:5/820: unlink d3/d4/dd6/dfe/c10b 0 2024-04-05T03:02:52.269 INFO:tasks.workunit.client.0.smithi153.stdout:4/599: write da/d2a/f9c [975824,24886] 0 2024-04-05T03:02:52.271 INFO:tasks.workunit.client.1.smithi203.stdout:4/614: dwrite da/f10 [0,4194304] 0 2024-04-05T03:02:52.271 INFO:tasks.workunit.client.0.smithi153.stdout:4/600: mkdir da/db/dcf 0 2024-04-05T03:02:52.273 INFO:tasks.workunit.client.1.smithi203.stdout:5/821: mknod d3/d1f/d59/c10f 0 2024-04-05T03:02:52.277 INFO:tasks.workunit.client.1.smithi203.stdout:5/822: mkdir d3/d4/dd6/d110 0 2024-04-05T03:02:52.278 INFO:tasks.workunit.client.0.smithi153.stdout:4/601: mknod da/db/d20/d3c/cd0 0 2024-04-05T03:02:52.279 INFO:tasks.workunit.client.0.smithi153.stdout:6/580: dwrite d4/d38/fbe [0,4194304] 0 2024-04-05T03:02:52.280 INFO:tasks.workunit.client.0.smithi153.stdout:2/506: dwrite d9/da/d19/f2f [0,4194304] 0 2024-04-05T03:02:52.283 INFO:tasks.workunit.client.0.smithi153.stdout:6/581: chown d4/dc/df/d16/d24/d5e/f60 1468243 1 2024-04-05T03:02:52.283 INFO:tasks.workunit.client.0.smithi153.stdout:2/507: chown d9/da/d19/d2e/d4f 0 1 2024-04-05T03:02:52.284 INFO:tasks.workunit.client.1.smithi203.stdout:5/823: getdents d3/d1f/d59/d60 0 2024-04-05T03:02:52.286 INFO:tasks.workunit.client.0.smithi153.stdout:4/602: symlink da/de/d66/ld1 0 2024-04-05T03:02:52.288 INFO:tasks.workunit.client.1.smithi203.stdout:9/889: truncate d13/d1b/d6b/d9e/d59/fc1 4749579 0 2024-04-05T03:02:52.291 INFO:tasks.workunit.client.1.smithi203.stdout:1/827: read d7/d24/d33/d48/d4b/d6b/f75 [2493035,129797] 0 2024-04-05T03:02:52.293 INFO:tasks.workunit.client.1.smithi203.stdout:5/824: creat d3/d4/d9e/dde/f111 x:0 0 0 2024-04-05T03:02:52.296 INFO:tasks.workunit.client.1.smithi203.stdout:1/828: symlink d7/d14/d2b/d58/d86/d91/l10c 0 2024-04-05T03:02:52.297 INFO:tasks.workunit.client.1.smithi203.stdout:9/890: dwrite d13/d1b/d33/d64/dfa/d67/d82/f8e [0,4194304] 0 2024-04-05T03:02:52.298 INFO:tasks.workunit.client.1.smithi203.stdout:5/825: unlink d3/d4/dc/l5c 0 2024-04-05T03:02:52.298 INFO:tasks.workunit.client.0.smithi153.stdout:6/582: dwrite d4/dc/df/d16/d24/d4e/f8a [0,4194304] 0 2024-04-05T03:02:52.299 INFO:tasks.workunit.client.0.smithi153.stdout:2/508: dwrite d9/da/d19/d2e/d7e/d7a/d6d/f6e [0,4194304] 0 2024-04-05T03:02:52.302 INFO:tasks.workunit.client.0.smithi153.stdout:2/509: fdatasync d9/da/d19/f6a 0 2024-04-05T03:02:52.302 INFO:tasks.workunit.client.1.smithi203.stdout:9/891: creat d13/d1b/d6b/d9e/d59/d7b/f118 x:0 0 0 2024-04-05T03:02:52.303 INFO:tasks.workunit.client.0.smithi153.stdout:4/603: dwrite da/db/d20/fad [0,4194304] 0 2024-04-05T03:02:52.304 INFO:tasks.workunit.client.1.smithi203.stdout:4/615: dwrite da/f39 [4194304,4194304] 0 2024-04-05T03:02:52.305 INFO:tasks.workunit.client.0.smithi153.stdout:6/583: creat d4/dc/df/d16/d5f/d73/fc1 x:0 0 0 2024-04-05T03:02:52.306 INFO:tasks.workunit.client.1.smithi203.stdout:9/892: dread d13/fe6 [0,4194304] 0 2024-04-05T03:02:52.308 INFO:tasks.workunit.client.0.smithi153.stdout:6/584: chown d4/dc/df/d16/d5f/d73/fc1 338324885 1 2024-04-05T03:02:52.309 INFO:tasks.workunit.client.1.smithi203.stdout:1/829: unlink d7/lf 0 2024-04-05T03:02:52.310 INFO:tasks.workunit.client.1.smithi203.stdout:5/826: symlink d3/d5f/d92/dff/d104/l112 0 2024-04-05T03:02:52.311 INFO:tasks.workunit.client.0.smithi153.stdout:4/604: creat da/de/fd2 x:0 0 0 2024-04-05T03:02:52.313 INFO:tasks.workunit.client.0.smithi153.stdout:4/605: read - da/de/d56/d78/d8f/d68/f95 zero size 2024-04-05T03:02:52.313 INFO:tasks.workunit.client.1.smithi203.stdout:9/893: truncate d13/f20 5837669 0 2024-04-05T03:02:52.314 INFO:tasks.workunit.client.0.smithi153.stdout:2/510: creat d9/da/d19/d2e/d7e/d7a/fa9 x:0 0 0 2024-04-05T03:02:52.316 INFO:tasks.workunit.client.0.smithi153.stdout:4/606: rmdir da/db/d20/d35/d41/da8 39 2024-04-05T03:02:52.317 INFO:tasks.workunit.client.0.smithi153.stdout:2/511: write d9/da/d19/d2e/d48/f93 [263037,73888] 0 2024-04-05T03:02:52.321 INFO:tasks.workunit.client.1.smithi203.stdout:5/827: rename d3/d5f/d92/lc3 to d3/d5f/d92/dff/d103/l113 0 2024-04-05T03:02:52.324 INFO:tasks.workunit.client.0.smithi153.stdout:4/607: rename da/de/fd2 to da/d2a/fd3 0 2024-04-05T03:02:52.326 INFO:tasks.workunit.client.1.smithi203.stdout:8/951: dread d1/d12/d1a/f1e [4194304,4194304] 0 2024-04-05T03:02:52.329 INFO:tasks.workunit.client.0.smithi153.stdout:4/608: symlink da/de/d56/d6a/ld4 0 2024-04-05T03:02:52.330 INFO:tasks.workunit.client.1.smithi203.stdout:9/894: getdents d13/d1b/d6b/d7a 0 2024-04-05T03:02:52.330 INFO:tasks.workunit.client.0.smithi153.stdout:2/512: rename d9/da/d19/d2e/d7e/d53/c82 to d9/d68/d9b/caa 0 2024-04-05T03:02:52.331 INFO:tasks.workunit.client.1.smithi203.stdout:8/952: creat d1/d4b/d68/d75/f113 x:0 0 0 2024-04-05T03:02:52.334 INFO:tasks.workunit.client.0.smithi153.stdout:2/513: creat d9/d68/fab x:0 0 0 2024-04-05T03:02:52.336 INFO:tasks.workunit.client.1.smithi203.stdout:8/953: mkdir d1/d12/d17/d47/d114 0 2024-04-05T03:02:52.338 INFO:tasks.workunit.client.0.smithi153.stdout:4/609: dwrite da/db/d20/d45/f9e [0,4194304] 0 2024-04-05T03:02:52.341 INFO:tasks.workunit.client.1.smithi203.stdout:9/895: getdents d13/d1b/d6b/d9e/d59/d9b/dd8 0 2024-04-05T03:02:52.341 INFO:tasks.workunit.client.0.smithi153.stdout:2/514: rename d9/da/d19/d2e/d7e/d53/f9d to d9/d68/d98/fac 0 2024-04-05T03:02:52.342 INFO:tasks.workunit.client.0.smithi153.stdout:2/515: stat d9/da/d19/d2e/d7e/d7a 0 2024-04-05T03:02:52.344 INFO:tasks.workunit.client.1.smithi203.stdout:1/830: dread d7/d24/f3e [0,4194304] 0 2024-04-05T03:02:52.344 INFO:tasks.workunit.client.1.smithi203.stdout:8/954: creat d1/d6/d2e/d5c/df5/f115 x:0 0 0 2024-04-05T03:02:52.345 INFO:tasks.workunit.client.1.smithi203.stdout:1/831: dread - d7/d14/fdd zero size 2024-04-05T03:02:52.346 INFO:tasks.workunit.client.0.smithi153.stdout:2/516: mkdir d9/d77/dad 0 2024-04-05T03:02:52.348 INFO:tasks.workunit.client.1.smithi203.stdout:4/616: dread da/f22 [0,4194304] 0 2024-04-05T03:02:52.348 INFO:tasks.workunit.client.1.smithi203.stdout:8/955: mknod d1/d12/d17/d47/c116 0 2024-04-05T03:02:52.351 INFO:tasks.workunit.client.0.smithi153.stdout:2/517: link d9/da/c1d d9/da/d19/d2e/d4f/cae 0 2024-04-05T03:02:52.352 INFO:tasks.workunit.client.1.smithi203.stdout:1/832: chown d7/d14/d2b/d58/lac 270234 1 2024-04-05T03:02:52.353 INFO:tasks.workunit.client.0.smithi153.stdout:2/518: truncate d9/da/d36/f55 2415089 0 2024-04-05T03:02:52.353 INFO:tasks.workunit.client.1.smithi203.stdout:4/617: mkdir da/d29/d2e/db0/dbe 0 2024-04-05T03:02:52.355 INFO:tasks.workunit.client.1.smithi203.stdout:8/956: rename d1/d12/f31 to d1/d12/d1a/d6d/d97/f117 0 2024-04-05T03:02:52.356 INFO:tasks.workunit.client.0.smithi153.stdout:2/519: mknod d9/da/d19/d2e/d7e/d7a/d6d/caf 0 2024-04-05T03:02:52.356 INFO:tasks.workunit.client.1.smithi203.stdout:1/833: creat d7/d14/d9d/dbf/f10d x:0 0 0 2024-04-05T03:02:52.361 INFO:tasks.workunit.client.1.smithi203.stdout:4/618: dwrite da/d29/f9e [0,4194304] 0 2024-04-05T03:02:52.363 INFO:tasks.workunit.client.0.smithi153.stdout:2/520: dread d9/da/fb [0,4194304] 0 2024-04-05T03:02:52.364 INFO:tasks.workunit.client.1.smithi203.stdout:8/957: rename d1/d12/d1a/d6d/fcc to d1/d90/dcd/f118 0 2024-04-05T03:02:52.364 INFO:tasks.workunit.client.1.smithi203.stdout:9/896: write d13/d5c/dd1/d51/f52 [298852,56534] 0 2024-04-05T03:02:52.365 INFO:tasks.workunit.client.1.smithi203.stdout:9/897: chown d13/d1b/d6b 612179964 1 2024-04-05T03:02:52.365 INFO:tasks.workunit.client.0.smithi153.stdout:2/521: symlink d9/da/d37/lb0 0 2024-04-05T03:02:52.366 INFO:tasks.workunit.client.1.smithi203.stdout:5/828: dread d3/f7a [0,4194304] 0 2024-04-05T03:02:52.368 INFO:tasks.workunit.client.1.smithi203.stdout:4/619: fdatasync da/d29/f47 0 2024-04-05T03:02:52.370 INFO:tasks.workunit.client.1.smithi203.stdout:1/834: creat d7/d24/d33/d48/d4b/d6b/f10e x:0 0 0 2024-04-05T03:02:52.371 INFO:tasks.workunit.client.1.smithi203.stdout:1/835: stat d7/d14/d2b/d58/de8/l109 0 2024-04-05T03:02:52.372 INFO:tasks.workunit.client.1.smithi203.stdout:9/898: rmdir d13/d5c/dd1/d31 39 2024-04-05T03:02:52.372 INFO:tasks.workunit.client.1.smithi203.stdout:8/958: dwrite d1/d12/d2f/d5b/fb3 [0,4194304] 0 2024-04-05T03:02:52.373 INFO:tasks.workunit.client.1.smithi203.stdout:5/829: symlink d3/d1f/d59/d60/dca/l114 0 2024-04-05T03:02:52.375 INFO:tasks.workunit.client.1.smithi203.stdout:3/712: sync 2024-04-05T03:02:52.375 INFO:tasks.workunit.client.1.smithi203.stdout:0/821: sync 2024-04-05T03:02:52.375 INFO:tasks.workunit.client.1.smithi203.stdout:1/836: chown d7/d14/d2b/d58/d7a/fda 199476113 1 2024-04-05T03:02:52.375 INFO:tasks.workunit.client.1.smithi203.stdout:6/829: sync 2024-04-05T03:02:52.380 INFO:tasks.workunit.client.1.smithi203.stdout:5/830: unlink d3/d1f/d59/d64/d12/l27 0 2024-04-05T03:02:52.380 INFO:tasks.workunit.client.1.smithi203.stdout:5/831: chown d3/d4/dc/f25 0 1 2024-04-05T03:02:52.381 INFO:tasks.workunit.client.1.smithi203.stdout:5/832: stat d3/d1f/d59/d60 0 2024-04-05T03:02:52.384 INFO:tasks.workunit.client.1.smithi203.stdout:6/830: creat d0/da/d18/d38/d2a/ff5 x:0 0 0 2024-04-05T03:02:52.385 INFO:tasks.workunit.client.1.smithi203.stdout:0/822: dread dd/d17/dd8/fb6 [0,4194304] 0 2024-04-05T03:02:52.385 INFO:tasks.workunit.client.1.smithi203.stdout:1/837: fdatasync d7/d14/d83/f105 0 2024-04-05T03:02:52.386 INFO:tasks.workunit.client.1.smithi203.stdout:8/959: write d1/d12/d17/d10/f15 [3143505,124607] 0 2024-04-05T03:02:52.388 INFO:tasks.workunit.client.1.smithi203.stdout:5/833: creat d3/d1f/d59/d60/dca/f115 x:0 0 0 2024-04-05T03:02:52.390 INFO:tasks.workunit.client.1.smithi203.stdout:9/899: rename d13/d1b/d6b/d7a/lfd to d13/d1b/d6b/d10f/l119 0 2024-04-05T03:02:52.391 INFO:tasks.workunit.client.1.smithi203.stdout:3/713: write d0/dc/d38/f3b [1608262,88132] 0 2024-04-05T03:02:52.392 INFO:tasks.workunit.client.1.smithi203.stdout:0/823: rmdir dd/db7 39 2024-04-05T03:02:52.392 INFO:tasks.workunit.client.0.smithi153.stdout:2/522: dread d9/d77/f69 [0,4194304] 0 2024-04-05T03:02:52.395 INFO:tasks.workunit.client.1.smithi203.stdout:4/620: dwrite da/d29/d2e/fb1 [0,4194304] 0 2024-04-05T03:02:52.397 INFO:tasks.workunit.client.1.smithi203.stdout:8/960: dwrite d1/d12/d1a/f1e [0,4194304] 0 2024-04-05T03:02:52.398 INFO:tasks.workunit.client.1.smithi203.stdout:6/831: creat d0/da/d18/ff6 x:0 0 0 2024-04-05T03:02:52.401 INFO:tasks.workunit.client.0.smithi153.stdout:2/523: dwrite d9/da/d19/d2e/d7e/d7a/d59/f8d [0,4194304] 0 2024-04-05T03:02:52.406 INFO:tasks.workunit.client.1.smithi203.stdout:9/900: readlink d13/d5c/dd1/d51/dcb/l105 0 2024-04-05T03:02:52.406 INFO:tasks.workunit.client.1.smithi203.stdout:0/824: mkdir dd/d17/d1d/d22/d53/dec 0 2024-04-05T03:02:52.407 INFO:tasks.workunit.client.0.smithi153.stdout:2/524: mknod d9/cb1 0 2024-04-05T03:02:52.409 INFO:tasks.workunit.client.1.smithi203.stdout:6/832: mknod d0/da/d18/d38/d86/cf7 0 2024-04-05T03:02:52.411 INFO:tasks.workunit.client.1.smithi203.stdout:5/834: dwrite d3/d4/f87 [0,4194304] 0 2024-04-05T03:02:52.411 INFO:tasks.workunit.client.0.smithi153.stdout:2/525: symlink d9/da/d19/d2e/d7e/d53/lb2 0 2024-04-05T03:02:52.417 INFO:tasks.workunit.client.0.smithi153.stdout:2/526: dwrite d9/da/d36/f81 [0,4194304] 0 2024-04-05T03:02:52.418 INFO:tasks.workunit.client.0.smithi153.stdout:2/527: write d9/d77/f88 [288104,2185] 0 2024-04-05T03:02:52.418 INFO:tasks.workunit.client.1.smithi203.stdout:2/914: sync 2024-04-05T03:02:52.423 INFO:tasks.workunit.client.1.smithi203.stdout:3/714: dwrite d0/dc/d1a/d8d/d96/fcc [0,4194304] 0 2024-04-05T03:02:52.423 INFO:tasks.workunit.client.1.smithi203.stdout:4/621: dwrite da/f9f [0,4194304] 0 2024-04-05T03:02:52.424 INFO:tasks.workunit.client.0.smithi153.stdout:3/500: sync 2024-04-05T03:02:52.424 INFO:tasks.workunit.client.0.smithi153.stdout:7/602: sync 2024-04-05T03:02:52.428 INFO:tasks.workunit.client.1.smithi203.stdout:1/838: dwrite d7/d14/d2b/fa3 [0,4194304] 0 2024-04-05T03:02:52.428 INFO:tasks.workunit.client.1.smithi203.stdout:2/915: write d5/d10/d35/d2d/d7c/d72/d8f/fe4 [4329847,126886] 0 2024-04-05T03:02:52.429 INFO:tasks.workunit.client.0.smithi153.stdout:3/501: unlink d1d/d32/d93/d3c/f46 0 2024-04-05T03:02:52.431 INFO:tasks.workunit.client.1.smithi203.stdout:0/825: mknod dd/d17/d1d/d26/d28/ced 0 2024-04-05T03:02:52.431 INFO:tasks.workunit.client.0.smithi153.stdout:7/603: mkdir d2/d51/da7/dc8 0 2024-04-05T03:02:52.433 INFO:tasks.workunit.client.1.smithi203.stdout:2/916: symlink d5/d10/db8/dbe/l13a 0 2024-04-05T03:02:52.435 INFO:tasks.workunit.client.1.smithi203.stdout:6/833: dread d0/da/f94 [0,4194304] 0 2024-04-05T03:02:52.437 INFO:tasks.workunit.client.0.smithi153.stdout:7/604: dwrite d2/d51/d93/f44 [0,4194304] 0 2024-04-05T03:02:52.440 INFO:tasks.workunit.client.0.smithi153.stdout:7/605: fsync d2/da/d3a/d41/d7a/f82 0 2024-04-05T03:02:52.441 INFO:tasks.workunit.client.1.smithi203.stdout:9/901: creat d13/d1b/d6b/d9e/d59/d9b/dd8/f11a x:0 0 0 2024-04-05T03:02:52.441 INFO:tasks.workunit.client.1.smithi203.stdout:4/622: unlink da/d29/d2e/l3b 0 2024-04-05T03:02:52.442 INFO:tasks.workunit.client.1.smithi203.stdout:3/715: chown d0/dc/d1a/d1f/l26 1513577 1 2024-04-05T03:02:52.443 INFO:tasks.workunit.client.1.smithi203.stdout:9/902: chown d13/d1b/d6b/d9e/d77/fa7 23 1 2024-04-05T03:02:52.444 INFO:tasks.workunit.client.1.smithi203.stdout:4/623: write da/d29/d2e/d57/d96/fa2 [572253,27380] 0 2024-04-05T03:02:52.445 INFO:tasks.workunit.client.0.smithi153.stdout:7/606: creat d2/da/d47/d5d/fc9 x:0 0 0 2024-04-05T03:02:52.447 INFO:tasks.workunit.client.0.smithi153.stdout:7/607: write d2/da/d3a/d41/d31/f4d [239438,127158] 0 2024-04-05T03:02:52.449 INFO:tasks.workunit.client.0.smithi153.stdout:7/608: write d2/da/d47/d5d/d7c/f5b [2327784,15514] 0 2024-04-05T03:02:52.451 INFO:tasks.workunit.client.0.smithi153.stdout:7/609: creat d2/da/d11/d88/fca x:0 0 0 2024-04-05T03:02:52.452 INFO:tasks.workunit.client.0.smithi153.stdout:7/610: write d2/da/d11/f13 [53568,29119] 0 2024-04-05T03:02:52.453 INFO:tasks.workunit.client.0.smithi153.stdout:7/611: chown d2/da/d11/d39/c9f 6850 1 2024-04-05T03:02:52.454 INFO:tasks.workunit.client.0.smithi153.stdout:7/612: dread - d2/da/d47/fa1 zero size 2024-04-05T03:02:52.455 INFO:tasks.workunit.client.0.smithi153.stdout:7/613: truncate d2/f40 472841 0 2024-04-05T03:02:52.461 INFO:tasks.workunit.client.0.smithi153.stdout:7/614: dwrite d2/da/d47/d5d/f69 [4194304,4194304] 0 2024-04-05T03:02:52.463 INFO:tasks.workunit.client.0.smithi153.stdout:7/615: symlink d2/da/d11/lcb 0 2024-04-05T03:02:52.465 INFO:tasks.workunit.client.0.smithi153.stdout:7/616: fsync d2/da/d3a/d41/f19 0 2024-04-05T03:02:52.465 INFO:tasks.workunit.client.0.smithi153.stdout:7/617: chown d2/da/d11/f13 19579841 1 2024-04-05T03:02:52.467 INFO:tasks.workunit.client.0.smithi153.stdout:7/618: symlink d2/da/d11/d39/lcc 0 2024-04-05T03:02:52.468 INFO:tasks.workunit.client.0.smithi153.stdout:7/619: mknod d2/da/d47/da9/ccd 0 2024-04-05T03:02:52.470 INFO:tasks.workunit.client.0.smithi153.stdout:7/620: rmdir d2/d9b/d6f/dc6 0 2024-04-05T03:02:52.471 INFO:tasks.workunit.client.0.smithi153.stdout:7/621: rmdir d2/da/d11/d39 39 2024-04-05T03:02:52.475 INFO:tasks.workunit.client.0.smithi153.stdout:2/528: write d9/da/d19/d2e/d7e/d53/f83 [428490,6279] 0 2024-04-05T03:02:52.477 INFO:tasks.workunit.client.0.smithi153.stdout:7/622: rmdir d2/da/d3a/d41/da8/dbe 0 2024-04-05T03:02:52.477 INFO:tasks.workunit.client.0.smithi153.stdout:2/529: truncate d9/da/d19/d2e/d7e/d7a/fa9 530464 0 2024-04-05T03:02:52.486 INFO:tasks.workunit.client.0.smithi153.stdout:7/623: rename d2/d51/da7 to d2/da/d47/d5d/d7c/d8d/d95/dce 0 2024-04-05T03:02:52.487 INFO:tasks.workunit.client.0.smithi153.stdout:3/502: write f3 [52545,63115] 0 2024-04-05T03:02:52.490 INFO:tasks.workunit.client.1.smithi203.stdout:6/834: write d0/da/d54/f4b [4612929,78603] 0 2024-04-05T03:02:52.491 INFO:tasks.workunit.client.0.smithi153.stdout:2/530: rename d9/da/d19/d2e/d4f/c5c to d9/da/d37/d8a/da8/cb3 0 2024-04-05T03:02:52.491 INFO:tasks.workunit.client.0.smithi153.stdout:2/531: readlink l2 0 2024-04-05T03:02:52.492 INFO:tasks.workunit.client.0.smithi153.stdout:2/532: chown d9/da/d37/lb0 846 1 2024-04-05T03:02:52.493 INFO:tasks.workunit.client.0.smithi153.stdout:7/624: unlink d2/da/d47/da3/lb5 0 2024-04-05T03:02:52.494 INFO:tasks.workunit.client.1.smithi203.stdout:8/961: truncate d1/d12/d1a/f1e 1161616 0 2024-04-05T03:02:52.495 INFO:tasks.workunit.client.0.smithi153.stdout:3/503: mknod d1d/d32/caf 0 2024-04-05T03:02:52.498 INFO:tasks.workunit.client.1.smithi203.stdout:5/835: getdents d3/d1f/d49/d10c/d10d 0 2024-04-05T03:02:52.498 INFO:tasks.workunit.client.0.smithi153.stdout:3/504: dread d1d/d27/f54 [0,4194304] 0 2024-04-05T03:02:52.499 INFO:tasks.workunit.client.0.smithi153.stdout:3/505: chown d1d/d27/d2b/l65 27 1 2024-04-05T03:02:52.499 INFO:tasks.workunit.client.0.smithi153.stdout:2/533: dwrite d9/da/d19/d2e/f43 [4194304,4194304] 0 2024-04-05T03:02:52.501 INFO:tasks.workunit.client.1.smithi203.stdout:0/826: symlink dd/lee 0 2024-04-05T03:02:52.501 INFO:tasks.workunit.client.0.smithi153.stdout:7/625: rename d2/da/d3a/d41/d7a/d54/d85/fb3 to d2/d51/d59/fcf 0 2024-04-05T03:02:52.503 INFO:tasks.workunit.client.0.smithi153.stdout:3/506: rmdir d1d/d75 39 2024-04-05T03:02:52.504 INFO:tasks.workunit.client.1.smithi203.stdout:8/962: dwrite d1/d12/d17/d47/d53/dd0/f104 [0,4194304] 0 2024-04-05T03:02:52.505 INFO:tasks.workunit.client.0.smithi153.stdout:2/534: write d9/da/f24 [887656,104373] 0 2024-04-05T03:02:52.505 INFO:tasks.workunit.client.0.smithi153.stdout:7/626: write d2/d51/d93/fb0 [165478,49394] 0 2024-04-05T03:02:52.509 INFO:tasks.workunit.client.1.smithi203.stdout:9/903: mkdir d13/d1b/d11b 0 2024-04-05T03:02:52.510 INFO:tasks.workunit.client.0.smithi153.stdout:3/507: creat d1d/d32/d93/d3c/d3f/d90/da4/fb0 x:0 0 0 2024-04-05T03:02:52.510 INFO:tasks.workunit.client.1.smithi203.stdout:9/904: readlink d13/d1b/d33/d64/l6c 0 2024-04-05T03:02:52.511 INFO:tasks.workunit.client.1.smithi203.stdout:4/624: truncate da/f26 1080288 0 2024-04-05T03:02:52.511 INFO:tasks.workunit.client.0.smithi153.stdout:2/535: creat d9/d68/d9b/fb4 x:0 0 0 2024-04-05T03:02:52.514 INFO:tasks.workunit.client.0.smithi153.stdout:2/536: readlink d9/da/d19/d2e/d7e/d7a/d59/l7d 0 2024-04-05T03:02:52.515 INFO:tasks.workunit.client.1.smithi203.stdout:0/827: rmdir dd/d17/d1d/d22/d45/dbe 39 2024-04-05T03:02:52.516 INFO:tasks.workunit.client.1.smithi203.stdout:4/625: chown da/d29/d2e/f61 56093060 1 2024-04-05T03:02:52.516 INFO:tasks.workunit.client.0.smithi153.stdout:7/627: dwrite d2/da/d3a/f94 [0,4194304] 0 2024-04-05T03:02:52.516 INFO:tasks.workunit.client.0.smithi153.stdout:3/508: creat d1d/d32/d93/fb1 x:0 0 0 2024-04-05T03:02:52.517 INFO:tasks.workunit.client.0.smithi153.stdout:2/537: read d9/da/d78/f92 [760549,29675] 0 2024-04-05T03:02:52.518 INFO:tasks.workunit.client.1.smithi203.stdout:2/917: truncate d5/d10/d35/fad 1417704 0 2024-04-05T03:02:52.519 INFO:tasks.workunit.client.0.smithi153.stdout:7/628: stat d2/da/d3a/d41/d31/f45 0 2024-04-05T03:02:52.519 INFO:tasks.workunit.client.0.smithi153.stdout:3/509: read - d1d/d27/d2b/f86 zero size 2024-04-05T03:02:52.519 INFO:tasks.workunit.client.1.smithi203.stdout:1/839: dread d7/d24/f34 [0,4194304] 0 2024-04-05T03:02:52.519 INFO:tasks.workunit.client.1.smithi203.stdout:3/716: dwrite d0/dc/d1a/d8d/d96/f4c [0,4194304] 0 2024-04-05T03:02:52.519 INFO:tasks.workunit.client.1.smithi203.stdout:5/836: symlink d3/d1f/d10e/de0/l116 0 2024-04-05T03:02:52.522 INFO:tasks.workunit.client.1.smithi203.stdout:8/963: link d1/d12/d2f/f2a d1/d12/d17/d47/d53/dd0/f119 0 2024-04-05T03:02:52.523 INFO:tasks.workunit.client.0.smithi153.stdout:7/629: write d2/da/d47/f92 [355612,14098] 0 2024-04-05T03:02:52.527 INFO:tasks.workunit.client.1.smithi203.stdout:6/835: dwrite d0/da/d18/d38/f6d [4194304,4194304] 0 2024-04-05T03:02:52.527 INFO:tasks.workunit.client.1.smithi203.stdout:9/905: rename d13/d1b/l4d to d13/d1b/d33/d64/dd6/l11c 0 2024-04-05T03:02:52.534 INFO:tasks.workunit.client.1.smithi203.stdout:8/964: fsync d1/d4b/d5f/daf/fbf 0 2024-04-05T03:02:52.536 INFO:tasks.workunit.client.1.smithi203.stdout:9/906: creat d13/d1b/d33/d64/dfa/d67/f11d x:0 0 0 2024-04-05T03:02:52.541 INFO:tasks.workunit.client.1.smithi203.stdout:8/965: creat d1/d12/d17/d10/f11a x:0 0 0 2024-04-05T03:02:52.542 INFO:tasks.workunit.client.1.smithi203.stdout:4/626: link da/f1b da/d29/fbf 0 2024-04-05T03:02:52.542 INFO:tasks.workunit.client.0.smithi153.stdout:5/502: sync 2024-04-05T03:02:52.542 INFO:tasks.workunit.client.0.smithi153.stdout:0/485: sync 2024-04-05T03:02:52.542 INFO:tasks.workunit.client.0.smithi153.stdout:9/416: sync 2024-04-05T03:02:52.542 INFO:tasks.workunit.client.0.smithi153.stdout:8/449: sync 2024-04-05T03:02:52.543 INFO:tasks.workunit.client.0.smithi153.stdout:8/450: stat d7/d13/d19/f22 0 2024-04-05T03:02:52.544 INFO:tasks.workunit.client.1.smithi203.stdout:5/837: getdents d3/d5f/d92 0 2024-04-05T03:02:52.544 INFO:tasks.workunit.client.0.smithi153.stdout:5/503: chown d0/d25/d4d/d62/c1b 77 1 2024-04-05T03:02:52.544 INFO:tasks.workunit.client.1.smithi203.stdout:6/836: creat d0/da/d18/d38/d2a/d44/d78/ff8 x:0 0 0 2024-04-05T03:02:52.545 INFO:tasks.workunit.client.1.smithi203.stdout:8/966: truncate d1/d6/f45 4839477 0 2024-04-05T03:02:52.545 INFO:tasks.workunit.client.1.smithi203.stdout:0/828: rename dd/d17/d1d/d22/d53/c5c to dd/d17/cef 0 2024-04-05T03:02:52.546 INFO:tasks.workunit.client.1.smithi203.stdout:4/627: symlink da/d71/lc0 0 2024-04-05T03:02:52.548 INFO:tasks.workunit.client.1.smithi203.stdout:9/907: dwrite d13/d1b/d6b/d9e/d77/f95 [0,4194304] 0 2024-04-05T03:02:52.550 INFO:tasks.workunit.client.1.smithi203.stdout:6/837: creat d0/da/d18/d92/ff9 x:0 0 0 2024-04-05T03:02:52.553 INFO:tasks.workunit.client.1.smithi203.stdout:4/628: creat da/d3c/d4c/d99/fc1 x:0 0 0 2024-04-05T03:02:52.554 INFO:tasks.workunit.client.1.smithi203.stdout:9/908: creat d13/d1b/d6b/d9e/df0/f11e x:0 0 0 2024-04-05T03:02:52.557 INFO:tasks.workunit.client.1.smithi203.stdout:0/829: dwrite f5 [0,4194304] 0 2024-04-05T03:02:52.559 INFO:tasks.workunit.client.0.smithi153.stdout:2/538: rename d9/d62/l8c to d9/da/d36/lb5 0 2024-04-05T03:02:52.559 INFO:tasks.workunit.client.1.smithi203.stdout:6/838: creat d0/da/d18/d38/d2a/d44/ffa x:0 0 0 2024-04-05T03:02:52.561 INFO:tasks.workunit.client.0.smithi153.stdout:7/630: write d2/da/d47/f9e [1015520,104647] 0 2024-04-05T03:02:52.561 INFO:tasks.workunit.client.0.smithi153.stdout:2/539: chown d9/da/fb 0 1 2024-04-05T03:02:52.561 INFO:tasks.workunit.client.1.smithi203.stdout:0/830: truncate dd/d17/d1d/d26/d28/fad 510359 0 2024-04-05T03:02:52.562 INFO:tasks.workunit.client.1.smithi203.stdout:4/629: rename da/d29/d2e/d57/f66 to da/d71/fc2 0 2024-04-05T03:02:52.563 INFO:tasks.workunit.client.1.smithi203.stdout:9/909: fdatasync d13/d1b/d6b/d9e/d59/d7b/f8a 0 2024-04-05T03:02:52.563 INFO:tasks.workunit.client.0.smithi153.stdout:3/510: dwrite d1d/f1e [0,4194304] 0 2024-04-05T03:02:52.565 INFO:tasks.workunit.client.1.smithi203.stdout:9/910: write d13/d1b/d6b/d9e/d59/dbc/fc4 [1653550,24312] 0 2024-04-05T03:02:52.566 INFO:tasks.workunit.client.0.smithi153.stdout:8/451: rmdir d7/d13/d19/d1b 39 2024-04-05T03:02:52.566 INFO:tasks.workunit.client.1.smithi203.stdout:0/831: creat dd/d17/d1d/d26/d8c/ff0 x:0 0 0 2024-04-05T03:02:52.566 INFO:tasks.workunit.client.1.smithi203.stdout:6/839: truncate d0/da/d54/d53/f63 398112 0 2024-04-05T03:02:52.568 INFO:tasks.workunit.client.0.smithi153.stdout:2/540: creat d9/da/d37/fb6 x:0 0 0 2024-04-05T03:02:52.568 INFO:tasks.workunit.client.0.smithi153.stdout:5/504: write d0/d9/f76 [690182,96842] 0 2024-04-05T03:02:52.569 INFO:tasks.workunit.client.1.smithi203.stdout:4/630: symlink da/d29/d2e/db0/lc3 0 2024-04-05T03:02:52.570 INFO:tasks.workunit.client.0.smithi153.stdout:0/486: getdents d9/db/d42/d43/d89 0 2024-04-05T03:02:52.571 INFO:tasks.workunit.client.0.smithi153.stdout:7/631: truncate d2/da/d11/f24 4720440 0 2024-04-05T03:02:52.574 INFO:tasks.workunit.client.1.smithi203.stdout:0/832: dwrite dd/d17/d1d/d22/d53/d7f/fea [0,4194304] 0 2024-04-05T03:02:52.574 INFO:tasks.workunit.client.1.smithi203.stdout:8/967: dread d1/d4b/d5f/fe9 [0,4194304] 0 2024-04-05T03:02:52.575 INFO:tasks.workunit.client.0.smithi153.stdout:3/511: mknod d1d/d32/d93/d3c/cb2 0 2024-04-05T03:02:52.578 INFO:tasks.workunit.client.1.smithi203.stdout:6/840: rename d0/da/fd to d0/da/d54/d4c/d7f/dcc/ffb 0 2024-04-05T03:02:52.579 INFO:tasks.workunit.client.1.smithi203.stdout:4/631: creat da/d29/d2e/d57/da5/fc4 x:0 0 0 2024-04-05T03:02:52.579 INFO:tasks.workunit.client.0.smithi153.stdout:7/632: write d2/da/d47/f9e [1801303,112382] 0 2024-04-05T03:02:52.580 INFO:tasks.workunit.client.0.smithi153.stdout:9/417: dwrite d1/d9/d1b/f30 [8388608,4194304] 0 2024-04-05T03:02:52.581 INFO:tasks.workunit.client.0.smithi153.stdout:0/487: symlink d9/d68/d6d/l8e 0 2024-04-05T03:02:52.581 INFO:tasks.workunit.client.1.smithi203.stdout:8/968: fsync d1/d4b/d5f/fe9 0 2024-04-05T03:02:52.585 INFO:tasks.workunit.client.0.smithi153.stdout:8/452: dwrite f6 [0,4194304] 0 2024-04-05T03:02:52.586 INFO:tasks.workunit.client.0.smithi153.stdout:5/505: creat d0/d25/d4d/d62/d5f/db2/fb4 x:0 0 0 2024-04-05T03:02:52.587 INFO:tasks.workunit.client.1.smithi203.stdout:0/833: write dd/d17/d1d/d26/fc0 [824746,62134] 0 2024-04-05T03:02:52.590 INFO:tasks.workunit.client.0.smithi153.stdout:0/488: mknod d9/db/d42/d7e/c8f 0 2024-04-05T03:02:52.590 INFO:tasks.workunit.client.0.smithi153.stdout:9/418: readlink d1/d10/d36/d57/l87 0 2024-04-05T03:02:52.590 INFO:tasks.workunit.client.0.smithi153.stdout:5/506: chown d0/d25/d4d/d62/d5f/d86/f77 5039894 1 2024-04-05T03:02:52.591 INFO:tasks.workunit.client.0.smithi153.stdout:5/507: fdatasync d0/f44 0 2024-04-05T03:02:52.591 INFO:tasks.workunit.client.0.smithi153.stdout:9/419: truncate d1/d3a/f93 504193 0 2024-04-05T03:02:52.592 INFO:tasks.workunit.client.0.smithi153.stdout:8/453: fdatasync d7/d2a/f2d 0 2024-04-05T03:02:52.593 INFO:tasks.workunit.client.1.smithi203.stdout:9/911: getdents d13/d1b/d6b/d9e/d59/d9b 0 2024-04-05T03:02:52.595 INFO:tasks.workunit.client.1.smithi203.stdout:8/969: mkdir d1/d12/d17/d47/d11b 0 2024-04-05T03:02:52.598 INFO:tasks.workunit.client.0.smithi153.stdout:7/633: rename d2/d51/l70 to d2/da/d11/d88/da6/ld0 0 2024-04-05T03:02:52.599 INFO:tasks.workunit.client.0.smithi153.stdout:0/489: rmdir d9/df/d1d/d61 39 2024-04-05T03:02:52.602 INFO:tasks.workunit.client.0.smithi153.stdout:3/512: dwrite d1d/d32/d93/d3c/d3f/d7c/fad [0,4194304] 0 2024-04-05T03:02:52.602 INFO:tasks.workunit.client.1.smithi203.stdout:4/632: creat da/d29/d2e/db0/dbe/fc5 x:0 0 0 2024-04-05T03:02:52.604 INFO:tasks.workunit.client.0.smithi153.stdout:5/508: unlink d0/d25/d4d/d62/d5f/d86/f9a 0 2024-04-05T03:02:52.606 INFO:tasks.workunit.client.0.smithi153.stdout:9/420: read - d1/d10/d16/d22/d2c/f46 zero size 2024-04-05T03:02:52.608 INFO:tasks.workunit.client.1.smithi203.stdout:0/834: mkdir dd/d17/d1d/d22/d53/dec/df1 0 2024-04-05T03:02:52.613 INFO:tasks.workunit.client.0.smithi153.stdout:7/634: dwrite d2/da/d3a/f71 [4194304,4194304] 0 2024-04-05T03:02:52.615 INFO:tasks.workunit.client.1.smithi203.stdout:2/918: dwrite d5/d10/d35/d2d/d7c/d29/f53 [0,4194304] 0 2024-04-05T03:02:52.617 INFO:tasks.workunit.client.0.smithi153.stdout:2/541: dread d9/da/f49 [0,4194304] 0 2024-04-05T03:02:52.618 INFO:tasks.workunit.client.0.smithi153.stdout:8/454: symlink d7/d13/d19/d1b/d4f/d72/l92 0 2024-04-05T03:02:52.620 INFO:tasks.workunit.client.1.smithi203.stdout:8/970: dwrite d1/d6/fe8 [0,4194304] 0 2024-04-05T03:02:52.622 INFO:tasks.workunit.client.0.smithi153.stdout:3/513: dwrite d1d/d75/f77 [0,4194304] 0 2024-04-05T03:02:52.623 INFO:tasks.workunit.client.1.smithi203.stdout:3/717: truncate d0/dc/f14 2798753 0 2024-04-05T03:02:52.623 INFO:tasks.workunit.client.1.smithi203.stdout:8/971: fdatasync d1/d4b/d68/fcb 0 2024-04-05T03:02:52.624 INFO:tasks.workunit.client.0.smithi153.stdout:9/421: creat d1/d3a/d91/f9b x:0 0 0 2024-04-05T03:02:52.624 INFO:tasks.workunit.client.0.smithi153.stdout:3/514: fdatasync d1d/d27/d2b/f62 0 2024-04-05T03:02:52.625 INFO:tasks.workunit.client.1.smithi203.stdout:1/840: dwrite d7/d24/d33/f45 [0,4194304] 0 2024-04-05T03:02:52.628 INFO:tasks.workunit.client.0.smithi153.stdout:0/490: mknod d9/df/d1d/c90 0 2024-04-05T03:02:52.628 INFO:tasks.workunit.client.1.smithi203.stdout:4/633: unlink da/d71/c8c 0 2024-04-05T03:02:52.631 INFO:tasks.workunit.client.0.smithi153.stdout:2/542: dwrite d9/da/d19/d2e/d7e/d7a/d59/f8d [0,4194304] 0 2024-04-05T03:02:52.632 INFO:tasks.workunit.client.0.smithi153.stdout:7/635: readlink d2/da/l21 0 2024-04-05T03:02:52.633 INFO:tasks.workunit.client.0.smithi153.stdout:2/543: fdatasync d9/d77/f88 0 2024-04-05T03:02:52.642 INFO:tasks.workunit.client.1.smithi203.stdout:6/841: getdents d0/da 0 2024-04-05T03:02:52.642 INFO:tasks.workunit.client.1.smithi203.stdout:8/972: unlink d1/d12/d17/d47/d53/cd1 0 2024-04-05T03:02:52.645 INFO:tasks.workunit.client.1.smithi203.stdout:5/838: dwrite d3/d4/d9e/fab [0,4194304] 0 2024-04-05T03:02:52.646 INFO:tasks.workunit.client.1.smithi203.stdout:3/718: rename d0/d21/d81/fbc to d0/dc/d1a/d1d/d53/fcd 0 2024-04-05T03:02:52.647 INFO:tasks.workunit.client.0.smithi153.stdout:3/515: rename d1d/d32/f83 to d1d/d32/d93/d3c/d3f/d90/da4/fb3 0 2024-04-05T03:02:52.662 INFO:tasks.workunit.client.0.smithi153.stdout:7/636: rename d2/d9b/c5c to d2/da/d47/d5d/d7c/d8d/d95/dce/cd1 0 2024-04-05T03:02:52.666 INFO:tasks.workunit.client.0.smithi153.stdout:3/516: rename d1d/f36 to d1d/d75/fb4 0 2024-04-05T03:02:52.667 INFO:tasks.workunit.client.0.smithi153.stdout:7/637: creat d2/da/d3a/fd2 x:0 0 0 2024-04-05T03:02:52.667 INFO:tasks.workunit.client.0.smithi153.stdout:7/638: chown d2/da/c7e 139611612 1 2024-04-05T03:02:52.668 INFO:tasks.workunit.client.0.smithi153.stdout:3/517: symlink d1d/d32/d93/d3c/d94/lb5 0 2024-04-05T03:02:52.669 INFO:tasks.workunit.client.0.smithi153.stdout:3/518: write f3 [2711650,67565] 0 2024-04-05T03:02:52.670 INFO:tasks.workunit.client.0.smithi153.stdout:7/639: write d2/da/d3a/d41/d31/f45 [785555,109383] 0 2024-04-05T03:02:52.670 INFO:tasks.workunit.client.0.smithi153.stdout:0/491: getdents d9 0 2024-04-05T03:02:52.672 INFO:tasks.workunit.client.0.smithi153.stdout:0/492: mknod d9/c91 0 2024-04-05T03:02:52.672 INFO:tasks.workunit.client.0.smithi153.stdout:7/640: chown d2/da/d11/d39/f76 469 1 2024-04-05T03:02:52.673 INFO:tasks.workunit.client.0.smithi153.stdout:7/641: readlink d2/da/l21 0 2024-04-05T03:02:52.673 INFO:tasks.workunit.client.0.smithi153.stdout:3/519: mkdir d1d/d32/d93/d3c/d3f/d90/db6 0 2024-04-05T03:02:52.674 INFO:tasks.workunit.client.0.smithi153.stdout:6/585: sync 2024-04-05T03:02:52.674 INFO:tasks.workunit.client.0.smithi153.stdout:1/448: sync 2024-04-05T03:02:52.674 INFO:tasks.workunit.client.0.smithi153.stdout:4/610: sync 2024-04-05T03:02:52.674 INFO:tasks.workunit.client.0.smithi153.stdout:0/493: fdatasync d9/df/d1d/d61/d8a/f7b 0 2024-04-05T03:02:52.676 INFO:tasks.workunit.client.0.smithi153.stdout:7/642: creat d2/da/d3a/d41/d7a/fd3 x:0 0 0 2024-04-05T03:02:52.677 INFO:tasks.workunit.client.0.smithi153.stdout:4/611: mkdir da/db/d20/d3c/d91/dd5 0 2024-04-05T03:02:52.678 INFO:tasks.workunit.client.0.smithi153.stdout:3/520: rename d1d/d75/f79 to d1d/d32/d93/d3c/d3f/d90/db6/fb7 0 2024-04-05T03:02:52.679 INFO:tasks.workunit.client.0.smithi153.stdout:6/586: mknod d4/dc/df/d16/d24/d5e/cc2 0 2024-04-05T03:02:52.679 INFO:tasks.workunit.client.0.smithi153.stdout:7/643: write d2/d51/d93/f74 [816604,66889] 0 2024-04-05T03:02:52.680 INFO:tasks.workunit.client.0.smithi153.stdout:4/612: dread - da/db/d20/d3c/d91/fbc zero size 2024-04-05T03:02:52.682 INFO:tasks.workunit.client.0.smithi153.stdout:3/521: write d1d/d27/d2b/f2f [4299493,56507] 0 2024-04-05T03:02:52.683 INFO:tasks.workunit.client.0.smithi153.stdout:7/644: unlink d2/da/l29 0 2024-04-05T03:02:52.685 INFO:tasks.workunit.client.0.smithi153.stdout:6/587: rmdir d4/dc/df/d16 39 2024-04-05T03:02:52.685 INFO:tasks.workunit.client.0.smithi153.stdout:1/449: dwrite d5/dd/d1b/f1d [4194304,4194304] 0 2024-04-05T03:02:52.687 INFO:tasks.workunit.client.0.smithi153.stdout:3/522: creat d1d/d32/d93/d3c/d3f/fb8 x:0 0 0 2024-04-05T03:02:52.687 INFO:tasks.workunit.client.0.smithi153.stdout:5/509: write d0/d25/d4d/faf [551771,55942] 0 2024-04-05T03:02:52.691 INFO:tasks.workunit.client.0.smithi153.stdout:3/523: write d1d/d27/fa1 [1027253,64589] 0 2024-04-05T03:02:52.693 INFO:tasks.workunit.client.0.smithi153.stdout:7/645: link d2/da/f68 d2/da/d3a/fd4 0 2024-04-05T03:02:52.697 INFO:tasks.workunit.client.0.smithi153.stdout:1/450: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:52.698 INFO:tasks.workunit.client.0.smithi153.stdout:7/646: write d2/d9b/fc5 [445198,42038] 0 2024-04-05T03:02:52.702 INFO:tasks.workunit.client.0.smithi153.stdout:5/510: symlink d0/d9/lb5 0 2024-04-05T03:02:52.702 INFO:tasks.workunit.client.0.smithi153.stdout:8/455: dwrite d7/fc [0,4194304] 0 2024-04-05T03:02:52.706 INFO:tasks.workunit.client.0.smithi153.stdout:6/588: link d4/d38/f75 d4/dc/d7a/d9c/fc3 0 2024-04-05T03:02:52.706 INFO:tasks.workunit.client.0.smithi153.stdout:9/422: dwrite d1/d10/d16/d22/d2c/d68/d81/f45 [0,4194304] 0 2024-04-05T03:02:52.710 INFO:tasks.workunit.client.0.smithi153.stdout:1/451: fdatasync d5/f7 0 2024-04-05T03:02:52.711 INFO:tasks.workunit.client.0.smithi153.stdout:5/511: mkdir d0/d4e/db6 0 2024-04-05T03:02:52.712 INFO:tasks.workunit.client.0.smithi153.stdout:2/544: dwrite d9/da/fb [0,4194304] 0 2024-04-05T03:02:52.713 INFO:tasks.workunit.client.0.smithi153.stdout:7/647: rename d2/d9b/d6f/fc0 to d2/da/d3a/d41/d31/fd5 0 2024-04-05T03:02:52.715 INFO:tasks.workunit.client.0.smithi153.stdout:6/589: dread - d4/dc/df/d16/d36/fb2 zero size 2024-04-05T03:02:52.715 INFO:tasks.workunit.client.0.smithi153.stdout:5/512: chown d0/d2c/d5e/c8c 7733 1 2024-04-05T03:02:52.722 INFO:tasks.workunit.client.0.smithi153.stdout:0/494: truncate d9/df/d1e/d45/f6a 1108953 0 2024-04-05T03:02:52.728 INFO:tasks.workunit.client.0.smithi153.stdout:2/545: dread d9/da/d36/f63 [0,4194304] 0 2024-04-05T03:02:52.730 INFO:tasks.workunit.client.0.smithi153.stdout:9/423: write d1/d9/f49 [933534,82498] 0 2024-04-05T03:02:52.731 INFO:tasks.workunit.client.0.smithi153.stdout:7/648: dwrite d2/da/d3a/d41/d31/f45 [0,4194304] 0 2024-04-05T03:02:52.731 INFO:tasks.workunit.client.0.smithi153.stdout:5/513: dwrite d0/d25/d4d/d62/d37/d38/f98 [0,4194304] 0 2024-04-05T03:02:52.733 INFO:tasks.workunit.client.0.smithi153.stdout:1/452: mknod d5/d8/d20/d4b/d18/d16/d48/d82/c8c 0 2024-04-05T03:02:52.735 INFO:tasks.workunit.client.0.smithi153.stdout:8/456: getdents d7/d13/d19/d1b/d6d/d8c 0 2024-04-05T03:02:52.738 INFO:tasks.workunit.client.0.smithi153.stdout:5/514: dread d0/d9/f21 [0,4194304] 0 2024-04-05T03:02:52.740 INFO:tasks.workunit.client.0.smithi153.stdout:7/649: creat d2/fd6 x:0 0 0 2024-04-05T03:02:52.742 INFO:tasks.workunit.client.0.smithi153.stdout:1/453: write d5/d8/d20/d4b/d18/d16/d45/f4a [1460346,57573] 0 2024-04-05T03:02:52.742 INFO:tasks.workunit.client.0.smithi153.stdout:3/524: dread d1d/d32/f45 [0,4194304] 0 2024-04-05T03:02:52.744 INFO:tasks.workunit.client.0.smithi153.stdout:2/546: rename d9/da/d19/l28 to d9/da/d19/lb7 0 2024-04-05T03:02:52.744 INFO:tasks.workunit.client.0.smithi153.stdout:9/424: rename d1/d10/d16/d22/f88 to d1/d3a/d91/f9c 0 2024-04-05T03:02:52.747 INFO:tasks.workunit.client.0.smithi153.stdout:8/457: mknod d7/d13/d19/d1b/d33/c93 0 2024-04-05T03:02:52.747 INFO:tasks.workunit.client.0.smithi153.stdout:9/425: truncate d1/d10/d16/d22/d2c/d68/d81/f45 9394954 0 2024-04-05T03:02:52.749 INFO:tasks.workunit.client.0.smithi153.stdout:7/650: read d2/da/d11/fab [1037482,31139] 0 2024-04-05T03:02:52.749 INFO:tasks.workunit.client.0.smithi153.stdout:8/458: read d7/fe [4617565,24210] 0 2024-04-05T03:02:52.750 INFO:tasks.workunit.client.0.smithi153.stdout:5/515: fsync d0/d25/d4d/d62/d5f/d86/f74 0 2024-04-05T03:02:52.751 INFO:tasks.workunit.client.0.smithi153.stdout:5/516: read d0/d9/f21 [1547187,108845] 0 2024-04-05T03:02:52.752 INFO:tasks.workunit.client.0.smithi153.stdout:1/454: creat d5/f8d x:0 0 0 2024-04-05T03:02:52.754 INFO:tasks.workunit.client.0.smithi153.stdout:2/547: fsync d9/da/fe 0 2024-04-05T03:02:52.755 INFO:tasks.workunit.client.1.smithi203.stdout:2/919: mknod d5/d10/d35/d2d/d7c/d29/db1/c13b 0 2024-04-05T03:02:52.756 INFO:tasks.workunit.client.0.smithi153.stdout:2/548: stat d9/da/d78 0 2024-04-05T03:02:52.757 INFO:tasks.workunit.client.1.smithi203.stdout:8/973: mknod d1/d12/d17/db8/c11c 0 2024-04-05T03:02:52.759 INFO:tasks.workunit.client.1.smithi203.stdout:8/974: write d1/d6/f56 [2414320,70856] 0 2024-04-05T03:02:52.761 INFO:tasks.workunit.client.0.smithi153.stdout:8/459: dwrite d7/d13/d19/f4e [0,4194304] 0 2024-04-05T03:02:52.762 INFO:tasks.workunit.client.0.smithi153.stdout:5/517: unlink d0/d25/d4d/c72 0 2024-04-05T03:02:52.765 INFO:tasks.workunit.client.0.smithi153.stdout:6/590: dwrite d4/f15 [0,4194304] 0 2024-04-05T03:02:52.766 INFO:tasks.workunit.client.0.smithi153.stdout:5/518: fdatasync d0/d9/f20 0 2024-04-05T03:02:52.767 INFO:tasks.workunit.client.1.smithi203.stdout:0/835: dwrite dd/d17/d1d/f37 [0,4194304] 0 2024-04-05T03:02:52.767 INFO:tasks.workunit.client.0.smithi153.stdout:9/426: mknod d1/d9/c9d 0 2024-04-05T03:02:52.767 INFO:tasks.workunit.client.0.smithi153.stdout:2/549: dread d9/da/d19/d2e/d7e/d7a/d59/f99 [0,4194304] 0 2024-04-05T03:02:52.767 INFO:tasks.workunit.client.0.smithi153.stdout:3/525: dread - d1d/d32/d93/d57/d6b/f97 zero size 2024-04-05T03:02:52.770 INFO:tasks.workunit.client.0.smithi153.stdout:7/651: symlink d2/d9b/ld7 0 2024-04-05T03:02:52.771 INFO:tasks.workunit.client.0.smithi153.stdout:2/550: dread - d9/da/d19/d2e/d4f/fa6 zero size 2024-04-05T03:02:52.771 INFO:tasks.workunit.client.0.smithi153.stdout:1/455: dread d5/dd/f26 [0,4194304] 0 2024-04-05T03:02:52.772 INFO:tasks.workunit.client.1.smithi203.stdout:9/912: dread d13/d1b/d6b/d9e/f5e [0,4194304] 0 2024-04-05T03:02:52.773 INFO:tasks.workunit.client.0.smithi153.stdout:6/591: truncate d4/dc/df/d16/d36/d52/f55 279984 0 2024-04-05T03:02:52.774 INFO:tasks.workunit.client.1.smithi203.stdout:9/913: dread - d13/d5c/d9d/f115 zero size 2024-04-05T03:02:52.774 INFO:tasks.workunit.client.0.smithi153.stdout:1/456: truncate d5/f76 5348227 0 2024-04-05T03:02:52.775 INFO:tasks.workunit.client.0.smithi153.stdout:1/457: stat d5/dd/f89 0 2024-04-05T03:02:52.781 INFO:tasks.workunit.client.0.smithi153.stdout:3/526: mkdir d1d/d32/d93/d3c/d3f/db9 0 2024-04-05T03:02:52.781 INFO:tasks.workunit.client.0.smithi153.stdout:5/519: mkdir d0/d25/d4d/d9c/db7 0 2024-04-05T03:02:52.782 INFO:tasks.workunit.client.0.smithi153.stdout:1/458: dwrite d5/d8/d20/d4b/f11 [0,4194304] 0 2024-04-05T03:02:52.784 INFO:tasks.workunit.client.0.smithi153.stdout:3/527: chown d1d/d32/d93/d3c/d3f/l95 1 1 2024-04-05T03:02:52.784 INFO:tasks.workunit.client.0.smithi153.stdout:0/495: dread d9/df/d1d/d61/f2b [0,4194304] 0 2024-04-05T03:02:52.785 INFO:tasks.workunit.client.0.smithi153.stdout:5/520: write d0/d9/d22/f7a [958558,18487] 0 2024-04-05T03:02:52.787 INFO:tasks.workunit.client.0.smithi153.stdout:2/551: creat d9/da/d19/d2e/d7e/d53/da1/fb8 x:0 0 0 2024-04-05T03:02:52.789 INFO:tasks.workunit.client.0.smithi153.stdout:3/528: write d1d/d32/d93/f4c [1485406,28051] 0 2024-04-05T03:02:52.789 INFO:tasks.workunit.client.0.smithi153.stdout:5/521: write d0/d25/d68/fa0 [706130,80881] 0 2024-04-05T03:02:52.790 INFO:tasks.workunit.client.0.smithi153.stdout:2/552: dread - d9/da/d19/d2e/d7e/d7a/f57 zero size 2024-04-05T03:02:52.792 INFO:tasks.workunit.client.0.smithi153.stdout:3/529: truncate d1d/d32/f56 1385447 0 2024-04-05T03:02:52.793 INFO:tasks.workunit.client.0.smithi153.stdout:2/553: write d9/da/d19/d2e/d7e/d7a/f84 [1387135,26979] 0 2024-04-05T03:02:52.794 INFO:tasks.workunit.client.0.smithi153.stdout:9/427: getdents d1/d1d 0 2024-04-05T03:02:52.794 INFO:tasks.workunit.client.0.smithi153.stdout:9/428: fdatasync d1/d10/d36/f56 0 2024-04-05T03:02:52.795 INFO:tasks.workunit.client.0.smithi153.stdout:1/459: creat d5/d8/d20/d4b/d18/d16/d33/d83/f8e x:0 0 0 2024-04-05T03:02:52.797 INFO:tasks.workunit.client.0.smithi153.stdout:1/460: chown d5/d8/d20/d4b/d18/d16/d33 1 1 2024-04-05T03:02:52.797 INFO:tasks.workunit.client.0.smithi153.stdout:0/496: symlink d9/db/l92 0 2024-04-05T03:02:52.798 INFO:tasks.workunit.client.0.smithi153.stdout:2/554: chown c7 2021391 1 2024-04-05T03:02:52.802 INFO:tasks.workunit.client.0.smithi153.stdout:2/555: fsync d9/da/f49 0 2024-04-05T03:02:52.803 INFO:tasks.workunit.client.0.smithi153.stdout:9/429: symlink d1/d3a/d91/d96/l9e 0 2024-04-05T03:02:52.806 INFO:tasks.workunit.client.0.smithi153.stdout:0/497: mknod d9/df/d1d/d61/c93 0 2024-04-05T03:02:52.812 INFO:tasks.workunit.client.0.smithi153.stdout:9/430: dwrite d1/d10/d16/d22/f3e [0,4194304] 0 2024-04-05T03:02:52.820 INFO:tasks.workunit.client.0.smithi153.stdout:9/431: write d1/d10/d16/d22/d2c/d68/d81/f76 [114639,25724] 0 2024-04-05T03:02:52.820 INFO:tasks.workunit.client.0.smithi153.stdout:0/498: dwrite d9/db/d42/f74 [0,4194304] 0 2024-04-05T03:02:52.823 INFO:tasks.workunit.client.0.smithi153.stdout:3/530: dread d1d/d32/d93/d57/d6b/f81 [0,4194304] 0 2024-04-05T03:02:52.823 INFO:tasks.workunit.client.0.smithi153.stdout:1/461: truncate f4 12161488 0 2024-04-05T03:02:52.826 INFO:tasks.workunit.client.0.smithi153.stdout:2/556: write d9/da/d78/f92 [1197379,83169] 0 2024-04-05T03:02:52.826 INFO:tasks.workunit.client.0.smithi153.stdout:3/531: read - d1d/d32/d93/d3c/d3f/fb8 zero size 2024-04-05T03:02:52.834 INFO:tasks.workunit.client.0.smithi153.stdout:9/432: dwrite d1/d10/d16/d22/d2c/d68/d81/f7a [0,4194304] 0 2024-04-05T03:02:52.834 INFO:tasks.workunit.client.0.smithi153.stdout:2/557: fsync d9/da/d19/d2e/d7e/d7a/f57 0 2024-04-05T03:02:52.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/462: creat d5/d8/d20/d4b/d1e/f8f x:0 0 0 2024-04-05T03:02:52.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/463: fdatasync d5/d8/d20/d4b/d18/d16/f22 0 2024-04-05T03:02:52.834 INFO:tasks.workunit.client.0.smithi153.stdout:2/558: readlink d9/da/d37/l5d 0 2024-04-05T03:02:52.836 INFO:tasks.workunit.client.0.smithi153.stdout:3/532: mknod d1d/d32/d93/d3c/d3f/d90/db6/cba 0 2024-04-05T03:02:52.836 INFO:tasks.workunit.client.0.smithi153.stdout:1/464: truncate d5/d8/d20/d4b/d18/d16/f7a 917703 0 2024-04-05T03:02:52.838 INFO:tasks.workunit.client.0.smithi153.stdout:2/559: rename d9/da/d78 to d9/d68/db9 0 2024-04-05T03:02:52.841 INFO:tasks.workunit.client.0.smithi153.stdout:1/465: symlink d5/d8/d20/d4b/d18/d16/d33/d83/l90 0 2024-04-05T03:02:52.841 INFO:tasks.workunit.client.0.smithi153.stdout:2/560: creat d9/d68/fba x:0 0 0 2024-04-05T03:02:52.843 INFO:tasks.workunit.client.0.smithi153.stdout:1/466: write d5/d8/d20/d4b/fe [1364671,126276] 0 2024-04-05T03:02:52.844 INFO:tasks.workunit.client.0.smithi153.stdout:3/533: dwrite d1d/d75/f77 [0,4194304] 0 2024-04-05T03:02:52.845 INFO:tasks.workunit.client.0.smithi153.stdout:2/561: creat d9/d77/dad/fbb x:0 0 0 2024-04-05T03:02:52.846 INFO:tasks.workunit.client.0.smithi153.stdout:2/562: chown d9/da 14221539 1 2024-04-05T03:02:52.846 INFO:tasks.workunit.client.0.smithi153.stdout:3/534: chown d1d/d27/c5a 13434 1 2024-04-05T03:02:52.851 INFO:tasks.workunit.client.0.smithi153.stdout:1/467: rename d5/d8/d20/d4b/d18/d16/d45/f54 to d5/d8/d20/d3f/f91 0 2024-04-05T03:02:52.852 INFO:tasks.workunit.client.0.smithi153.stdout:2/563: truncate d9/da/f64 3485675 0 2024-04-05T03:02:52.853 INFO:tasks.workunit.client.0.smithi153.stdout:3/535: creat d1d/d32/d93/d3c/fbb x:0 0 0 2024-04-05T03:02:52.856 INFO:tasks.workunit.client.0.smithi153.stdout:3/536: mknod d1d/d75/cbc 0 2024-04-05T03:02:52.858 INFO:tasks.workunit.client.0.smithi153.stdout:1/468: dwrite d5/dd/d1b/f66 [4194304,4194304] 0 2024-04-05T03:02:52.859 INFO:tasks.workunit.client.0.smithi153.stdout:3/537: dread - d1d/d32/d93/d57/d6b/d9f/fae zero size 2024-04-05T03:02:52.860 INFO:tasks.workunit.client.0.smithi153.stdout:2/564: dwrite d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:02:52.861 INFO:tasks.workunit.client.0.smithi153.stdout:3/538: dread - d1d/d27/d2b/f86 zero size 2024-04-05T03:02:52.863 INFO:tasks.workunit.client.0.smithi153.stdout:1/469: rename d5/d8/d20/d4b/f81 to d5/d8/d20/d4b/d18/d16/d48/d82/f92 0 2024-04-05T03:02:52.867 INFO:tasks.workunit.client.0.smithi153.stdout:2/565: dwrite d9/da/f24 [0,4194304] 0 2024-04-05T03:02:52.868 INFO:tasks.workunit.client.0.smithi153.stdout:2/566: dread - d9/da/d19/d2e/f8f zero size 2024-04-05T03:02:52.869 INFO:tasks.workunit.client.0.smithi153.stdout:2/567: stat d9/da/d19/d2e/d7e/d7a/d6d/f6e 0 2024-04-05T03:02:52.885 INFO:tasks.workunit.client.0.smithi153.stdout:1/470: mkdir d5/d8/d20/d4b/d18/d16/d45/d93 0 2024-04-05T03:02:52.886 INFO:tasks.workunit.client.0.smithi153.stdout:2/568: chown d9/d62/c76 18979 1 2024-04-05T03:02:52.888 INFO:tasks.workunit.client.0.smithi153.stdout:3/539: mkdir d1d/d32/d93/d57/d6b/dbd 0 2024-04-05T03:02:52.893 INFO:tasks.workunit.client.0.smithi153.stdout:1/471: write d5/d8/d20/f5f [3090135,7485] 0 2024-04-05T03:02:52.894 INFO:tasks.workunit.client.1.smithi203.stdout:5/839: unlink d3/d4/dc/d67/le3 0 2024-04-05T03:02:52.895 INFO:tasks.workunit.client.1.smithi203.stdout:2/920: truncate d5/d10/db8/ffd 379081 0 2024-04-05T03:02:52.898 INFO:tasks.workunit.client.0.smithi153.stdout:1/472: symlink d5/d8/d20/d4b/d18/l94 0 2024-04-05T03:02:52.899 INFO:tasks.workunit.client.0.smithi153.stdout:1/473: readlink d5/dd/d1b/l4e 0 2024-04-05T03:02:52.902 INFO:tasks.workunit.client.0.smithi153.stdout:3/540: link d1d/d32/d93/d57/d6b/c63 d1d/cbe 0 2024-04-05T03:02:52.902 INFO:tasks.workunit.client.0.smithi153.stdout:1/474: write d5/dd/f13 [229138,99341] 0 2024-04-05T03:02:52.903 INFO:tasks.workunit.client.1.smithi203.stdout:0/836: mkdir dd/d17/d1d/d26/d8c/dcf/df2 0 2024-04-05T03:02:52.903 INFO:tasks.workunit.client.0.smithi153.stdout:3/541: fdatasync d1d/d27/f64 0 2024-04-05T03:02:52.904 INFO:tasks.workunit.client.0.smithi153.stdout:3/542: chown d1d/d32/d93/d3c/f73 2 1 2024-04-05T03:02:52.905 INFO:tasks.workunit.client.1.smithi203.stdout:9/914: creat d13/d5c/dd1/d51/d81/f11f x:0 0 0 2024-04-05T03:02:52.909 INFO:tasks.workunit.client.1.smithi203.stdout:5/840: unlink d3/d4/d9e/lec 0 2024-04-05T03:02:52.909 INFO:tasks.workunit.client.0.smithi153.stdout:3/543: rmdir d1d/d32/d93/d3c/d3f/db9 0 2024-04-05T03:02:52.910 INFO:tasks.workunit.client.1.smithi203.stdout:5/841: dread - d3/f82 zero size 2024-04-05T03:02:52.910 INFO:tasks.workunit.client.0.smithi153.stdout:1/475: dwrite d5/dd/d1b/f56 [0,4194304] 0 2024-04-05T03:02:52.912 INFO:tasks.workunit.client.1.smithi203.stdout:0/837: rmdir dd/d17/d1d/d26/d8c/d6d/dc4 39 2024-04-05T03:02:52.913 INFO:tasks.workunit.client.1.smithi203.stdout:0/838: fdatasync dd/d17/d1d/d26/d28/f3f 0 2024-04-05T03:02:52.914 INFO:tasks.workunit.client.0.smithi153.stdout:3/544: creat d1d/d32/d93/d57/d6b/d9f/fbf x:0 0 0 2024-04-05T03:02:52.916 INFO:tasks.workunit.client.0.smithi153.stdout:1/476: rename d5/dd/d1b/l4e to d5/d8/d20/d4b/d18/d16/d33/l95 0 2024-04-05T03:02:52.917 INFO:tasks.workunit.client.1.smithi203.stdout:9/915: creat d13/d1b/d6b/d9e/d77/f120 x:0 0 0 2024-04-05T03:02:52.918 INFO:tasks.workunit.client.0.smithi153.stdout:1/477: chown d5/d8/d20/d4b/d18/d16/f22 253244 1 2024-04-05T03:02:52.920 INFO:tasks.workunit.client.0.smithi153.stdout:1/478: read d5/f84 [2850921,40366] 0 2024-04-05T03:02:52.920 INFO:tasks.workunit.client.1.smithi203.stdout:2/921: mkdir d5/d10/db8/d13c 0 2024-04-05T03:02:52.922 INFO:tasks.workunit.client.0.smithi153.stdout:3/545: creat d1d/d32/d93/fc0 x:0 0 0 2024-04-05T03:02:52.922 INFO:tasks.workunit.client.1.smithi203.stdout:0/839: dwrite dd/d17/d1d/d22/d45/d51/feb [0,4194304] 0 2024-04-05T03:02:52.926 INFO:tasks.workunit.client.1.smithi203.stdout:9/916: mknod d13/dc8/dd2/c121 0 2024-04-05T03:02:52.927 INFO:tasks.workunit.client.0.smithi153.stdout:3/546: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/lc1 0 2024-04-05T03:02:52.929 INFO:tasks.workunit.client.1.smithi203.stdout:8/975: getdents d1/d4b 0 2024-04-05T03:02:52.931 INFO:tasks.workunit.client.0.smithi153.stdout:3/547: mkdir d1d/d32/d93/dc2 0 2024-04-05T03:02:52.931 INFO:tasks.workunit.client.1.smithi203.stdout:5/842: mkdir d3/d4/dc/d67/db5/dda/df9/d117 0 2024-04-05T03:02:52.931 INFO:tasks.workunit.client.1.smithi203.stdout:0/840: fsync dd/d17/d1d/d22/fac 0 2024-04-05T03:02:52.932 INFO:tasks.workunit.client.1.smithi203.stdout:5/843: fsync d3/d5f/d92/fee 0 2024-04-05T03:02:52.933 INFO:tasks.workunit.client.0.smithi153.stdout:3/548: fdatasync d1d/d32/d93/d3c/d3f/d7c/fa8 0 2024-04-05T03:02:52.937 INFO:tasks.workunit.client.0.smithi153.stdout:3/549: getdents d1d/d32/d93/d3c/d3f/d90 0 2024-04-05T03:02:52.941 INFO:tasks.workunit.client.0.smithi153.stdout:3/550: dread d1d/d27/d2b/f35 [0,4194304] 0 2024-04-05T03:02:52.944 INFO:tasks.workunit.client.0.smithi153.stdout:3/551: mknod d1d/d32/d93/dc2/cc3 0 2024-04-05T03:02:52.946 INFO:tasks.workunit.client.0.smithi153.stdout:3/552: creat d1d/d32/d93/d3c/d3f/d9c/fc4 x:0 0 0 2024-04-05T03:02:52.948 INFO:tasks.workunit.client.0.smithi153.stdout:3/553: mkdir d1d/d32/d93/d3c/dc5 0 2024-04-05T03:02:52.949 INFO:tasks.workunit.client.0.smithi153.stdout:3/554: chown d1d/f7a 438714 1 2024-04-05T03:02:52.950 INFO:tasks.workunit.client.0.smithi153.stdout:3/555: mknod d1d/d32/cc6 0 2024-04-05T03:02:52.950 INFO:tasks.workunit.client.0.smithi153.stdout:3/556: stat d1d/d75/f84 0 2024-04-05T03:02:52.951 INFO:tasks.workunit.client.0.smithi153.stdout:3/557: write d1d/d27/d2b/f62 [407753,64638] 0 2024-04-05T03:02:52.952 INFO:tasks.workunit.client.1.smithi203.stdout:1/841: rename d7/d14/f76 to d7/d24/d33/d48/d4b/d6b/f10f 0 2024-04-05T03:02:52.952 INFO:tasks.workunit.client.0.smithi153.stdout:3/558: stat d1d/d32/d93/d3c/d94 0 2024-04-05T03:02:52.953 INFO:tasks.workunit.client.1.smithi203.stdout:1/842: fsync d7/d14/d9d/dbf/fcd 0 2024-04-05T03:02:52.954 INFO:tasks.workunit.client.1.smithi203.stdout:5/844: dread d3/d4/dc/f95 [0,4194304] 0 2024-04-05T03:02:52.958 INFO:tasks.workunit.client.0.smithi153.stdout:3/559: symlink d1d/d32/d93/d57/d6b/d9f/lc7 0 2024-04-05T03:02:52.960 INFO:tasks.workunit.client.0.smithi153.stdout:3/560: write d1d/d27/d2b/f2f [3010076,86265] 0 2024-04-05T03:02:52.961 INFO:tasks.workunit.client.0.smithi153.stdout:3/561: stat d1d/d27/f8f 0 2024-04-05T03:02:52.962 INFO:tasks.workunit.client.0.smithi153.stdout:2/569: truncate d9/da/f15 389928 0 2024-04-05T03:02:52.963 INFO:tasks.workunit.client.0.smithi153.stdout:3/562: fdatasync d1d/d27/d2b/f7e 0 2024-04-05T03:02:52.965 INFO:tasks.workunit.client.0.smithi153.stdout:2/570: creat d9/d68/db9/fbc x:0 0 0 2024-04-05T03:02:52.966 INFO:tasks.workunit.client.1.smithi203.stdout:6/842: rename d0/da/d18/d38/d2a/d2c/fb9 to d0/da/d18/d38/d2a/ffc 0 2024-04-05T03:02:52.967 INFO:tasks.workunit.client.1.smithi203.stdout:5/845: read d3/d1f/d59/d64/f44 [2793704,91978] 0 2024-04-05T03:02:52.967 INFO:tasks.workunit.client.0.smithi153.stdout:3/563: symlink d1d/d32/d93/lc8 0 2024-04-05T03:02:52.969 INFO:tasks.workunit.client.1.smithi203.stdout:9/917: rename d13/d1b/d6b/d9e/d59/d9b/fd9 to d13/d1b/d33/d64/dfa/d67/d82/f122 0 2024-04-05T03:02:52.969 INFO:tasks.workunit.client.0.smithi153.stdout:2/571: mknod d9/d77/cbd 0 2024-04-05T03:02:52.969 INFO:tasks.workunit.client.0.smithi153.stdout:3/564: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/fa6 zero size 2024-04-05T03:02:52.969 INFO:tasks.workunit.client.1.smithi203.stdout:6/843: stat d0/da/d18/d38/d2a/d44 0 2024-04-05T03:02:52.970 INFO:tasks.workunit.client.1.smithi203.stdout:6/844: readlink d0/da/d54/d4c/da1/db7/l98 0 2024-04-05T03:02:52.971 INFO:tasks.workunit.client.1.smithi203.stdout:6/845: stat d0/da/d54/cf2 0 2024-04-05T03:02:52.972 INFO:tasks.workunit.client.1.smithi203.stdout:5/846: chown d3/d1f/d59/d60/f99 12 1 2024-04-05T03:02:52.972 INFO:tasks.workunit.client.0.smithi153.stdout:2/572: stat d9/da/d19/d2e/d48/l3f 0 2024-04-05T03:02:52.973 INFO:tasks.workunit.client.1.smithi203.stdout:6/846: write d0/da/d54/d4c/d7f/f91 [1813231,117701] 0 2024-04-05T03:02:52.981 INFO:tasks.workunit.client.1.smithi203.stdout:5/847: mkdir d3/d4/d9e/d118 0 2024-04-05T03:02:52.984 INFO:tasks.workunit.client.1.smithi203.stdout:0/841: rename dd/d17/d1d/d22/d45/d51/d69/l9e to dd/d17/d1d/d22/d53/d7f/dc5/lf3 0 2024-04-05T03:02:52.985 INFO:tasks.workunit.client.0.smithi153.stdout:2/573: getdents d9/da/d37/d8a 0 2024-04-05T03:02:52.986 INFO:tasks.workunit.client.1.smithi203.stdout:9/918: readlink d13/d5c/dd1/d31/l37 0 2024-04-05T03:02:52.987 INFO:tasks.workunit.client.1.smithi203.stdout:2/922: write d5/d10/f1d [3572764,113543] 0 2024-04-05T03:02:52.988 INFO:tasks.workunit.client.0.smithi153.stdout:1/479: dwrite d5/d8/d20/d4b/d18/d16/d48/f50 [0,4194304] 0 2024-04-05T03:02:52.988 INFO:tasks.workunit.client.0.smithi153.stdout:2/574: write d9/d77/f58 [707103,79085] 0 2024-04-05T03:02:52.989 INFO:tasks.workunit.client.1.smithi203.stdout:6/847: dwrite d0/da/d54/d4c/da1/db7/d71/f79 [0,4194304] 0 2024-04-05T03:02:52.989 INFO:tasks.workunit.client.1.smithi203.stdout:1/843: rename d7/d24/d33/d48/d4b/dea/def/cfd to d7/d24/d33/c110 0 2024-04-05T03:02:52.991 INFO:tasks.workunit.client.0.smithi153.stdout:3/565: dread d1d/d75/f84 [0,4194304] 0 2024-04-05T03:02:52.991 INFO:tasks.workunit.client.0.smithi153.stdout:1/480: truncate d5/d8/d20/f88 3015 0 2024-04-05T03:02:52.994 INFO:tasks.workunit.client.1.smithi203.stdout:0/842: unlink dd/d17/d1d/d22/d53/l9b 0 2024-04-05T03:02:52.998 INFO:tasks.workunit.client.0.smithi153.stdout:3/566: creat d1d/d27/d2b/fc9 x:0 0 0 2024-04-05T03:02:52.999 INFO:tasks.workunit.client.1.smithi203.stdout:9/919: rename d13/d1b/d6b/d7a/fbe to d13/dc8/f123 0 2024-04-05T03:02:53.000 INFO:tasks.workunit.client.0.smithi153.stdout:4/613: sync 2024-04-05T03:02:53.000 INFO:tasks.workunit.client.0.smithi153.stdout:1/481: mkdir d5/d8/d20/d4b/d18/d16/d48/d96 0 2024-04-05T03:02:53.001 INFO:tasks.workunit.client.1.smithi203.stdout:6/848: symlink d0/da/d54/d53/lfd 0 2024-04-05T03:02:53.002 INFO:tasks.workunit.client.0.smithi153.stdout:4/614: dread - da/de/d56/fa0 zero size 2024-04-05T03:02:53.002 INFO:tasks.workunit.client.1.smithi203.stdout:9/920: write d13/d1b/d6b/d9e/d77/fed [1353965,79979] 0 2024-04-05T03:02:53.004 INFO:tasks.workunit.client.1.smithi203.stdout:9/921: chown d13/d5c/dd1/d51/dcb/cd5 104289 1 2024-04-05T03:02:53.005 INFO:tasks.workunit.client.1.smithi203.stdout:8/976: dwrite d1/d12/d17/d10/f40 [4194304,4194304] 0 2024-04-05T03:02:53.005 INFO:tasks.workunit.client.0.smithi153.stdout:1/482: chown d5/d8/d20/d4b/d18/d16/d33/l95 4 1 2024-04-05T03:02:53.006 INFO:tasks.workunit.client.1.smithi203.stdout:0/843: symlink dd/d17/d1d/d22/d53/d7f/lf4 0 2024-04-05T03:02:53.007 INFO:tasks.workunit.client.0.smithi153.stdout:4/615: creat da/db/d20/d35/d38/d76/d90/fd6 x:0 0 0 2024-04-05T03:02:53.007 INFO:tasks.workunit.client.1.smithi203.stdout:8/977: write d1/d12/d1a/f42 [1997521,41476] 0 2024-04-05T03:02:53.009 INFO:tasks.workunit.client.1.smithi203.stdout:2/923: creat d5/d10/d35/d36/db6/f13d x:0 0 0 2024-04-05T03:02:53.010 INFO:tasks.workunit.client.1.smithi203.stdout:8/978: write d1/d4b/d5f/d74/ff9 [3927394,64558] 0 2024-04-05T03:02:53.013 INFO:tasks.workunit.client.1.smithi203.stdout:3/719: write d0/dc/f14 [2495772,96501] 0 2024-04-05T03:02:53.013 INFO:tasks.workunit.client.1.smithi203.stdout:8/979: write d1/d4b/d68/d75/f113 [1039338,93080] 0 2024-04-05T03:02:53.013 INFO:tasks.workunit.client.1.smithi203.stdout:2/924: fdatasync d5/d10/f20 0 2024-04-05T03:02:53.014 INFO:tasks.workunit.client.1.smithi203.stdout:0/844: symlink dd/d17/d1d/d22/d53/dec/df1/lf5 0 2024-04-05T03:02:53.015 INFO:tasks.workunit.client.1.smithi203.stdout:9/922: dread d13/fe6 [0,4194304] 0 2024-04-05T03:02:53.015 INFO:tasks.workunit.client.1.smithi203.stdout:2/925: stat d5/d10/d44/df6 0 2024-04-05T03:02:53.019 INFO:tasks.workunit.client.1.smithi203.stdout:4/634: sync 2024-04-05T03:02:53.020 INFO:tasks.workunit.client.1.smithi203.stdout:3/720: write d0/dc/d38/d5d/d66/fc8 [2876027,8088] 0 2024-04-05T03:02:53.020 INFO:tasks.workunit.client.1.smithi203.stdout:2/926: chown d5/d10/db8/dab/de1/dbd/lcd 3 1 2024-04-05T03:02:53.024 INFO:tasks.workunit.client.0.smithi153.stdout:1/483: write f4 [6385847,92713] 0 2024-04-05T03:02:53.024 INFO:tasks.workunit.client.1.smithi203.stdout:6/849: read d0/da/d18/f30 [569752,89828] 0 2024-04-05T03:02:53.024 INFO:tasks.workunit.client.0.smithi153.stdout:4/616: write f8 [3907534,34986] 0 2024-04-05T03:02:53.025 INFO:tasks.workunit.client.1.smithi203.stdout:5/848: dread d3/d1f/d59/d60/f74 [0,4194304] 0 2024-04-05T03:02:53.025 INFO:tasks.workunit.client.1.smithi203.stdout:8/980: link d1/d6/f45 d1/d6/d2e/d5c/dc0/f11d 0 2024-04-05T03:02:53.027 INFO:tasks.workunit.client.1.smithi203.stdout:0/845: dwrite dd/d17/f48 [4194304,4194304] 0 2024-04-05T03:02:53.028 INFO:tasks.workunit.client.1.smithi203.stdout:4/635: mknod da/d3c/d4c/d99/cc6 0 2024-04-05T03:02:53.029 INFO:tasks.workunit.client.0.smithi153.stdout:4/617: symlink da/d2a/ld7 0 2024-04-05T03:02:53.030 INFO:tasks.workunit.client.0.smithi153.stdout:2/575: dread d9/da/f64 [0,4194304] 0 2024-04-05T03:02:53.031 INFO:tasks.workunit.client.0.smithi153.stdout:1/484: dread d5/d8/d20/d4b/d18/d16/d45/d68/f77 [0,4194304] 0 2024-04-05T03:02:53.031 INFO:tasks.workunit.client.0.smithi153.stdout:2/576: chown d9/da/l25 311575410 1 2024-04-05T03:02:53.031 INFO:tasks.workunit.client.1.smithi203.stdout:2/927: creat d5/d63/f13e x:0 0 0 2024-04-05T03:02:53.032 INFO:tasks.workunit.client.1.smithi203.stdout:0/846: fsync dd/d93/fbc 0 2024-04-05T03:02:53.034 INFO:tasks.workunit.client.1.smithi203.stdout:3/721: mknod d0/dc/d1a/d76/d35/cce 0 2024-04-05T03:02:53.038 INFO:tasks.workunit.client.0.smithi153.stdout:2/577: rmdir d9/da/d19 39 2024-04-05T03:02:53.038 INFO:tasks.workunit.client.1.smithi203.stdout:8/981: dwrite d1/d6/d2e/fea [0,4194304] 0 2024-04-05T03:02:53.039 INFO:tasks.workunit.client.0.smithi153.stdout:1/485: creat d5/d8/d20/d4b/d18/d16/d48/f97 x:0 0 0 2024-04-05T03:02:53.040 INFO:tasks.workunit.client.0.smithi153.stdout:4/618: getdents da/db/d20/d45/db1 0 2024-04-05T03:02:53.041 INFO:tasks.workunit.client.0.smithi153.stdout:4/619: stat da/de/f88 0 2024-04-05T03:02:53.041 INFO:tasks.workunit.client.0.smithi153.stdout:2/578: fdatasync d9/da/d19/d2e/d48/f6b 0 2024-04-05T03:02:53.043 INFO:tasks.workunit.client.1.smithi203.stdout:2/928: dread d5/d10/d35/d36/db6/d71/f97 [0,4194304] 0 2024-04-05T03:02:53.043 INFO:tasks.workunit.client.0.smithi153.stdout:4/620: dread da/de/f1e [4194304,4194304] 0 2024-04-05T03:02:53.044 INFO:tasks.workunit.client.1.smithi203.stdout:1/844: creat d7/d24/d33/d48/d4b/dea/def/f111 x:0 0 0 2024-04-05T03:02:53.045 INFO:tasks.workunit.client.1.smithi203.stdout:5/849: creat d3/d4/dc/d67/f119 x:0 0 0 2024-04-05T03:02:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:52 smithi203 ceph-mon[39911]: pgmap v11: 65 pgs: 65 active+clean; 2.3 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 82 MiB/s rd, 263 MiB/s wr, 480 op/s 2024-04-05T03:02:53.049 INFO:tasks.workunit.client.1.smithi203.stdout:2/929: dread - d5/d10/d35/f119 zero size 2024-04-05T03:02:53.050 INFO:tasks.workunit.client.1.smithi203.stdout:0/847: creat dd/d17/dd8/ff6 x:0 0 0 2024-04-05T03:02:53.050 INFO:tasks.workunit.client.0.smithi153.stdout:1/486: dwrite d5/d8/d20/d3f/f91 [0,4194304] 0 2024-04-05T03:02:53.051 INFO:tasks.workunit.client.1.smithi203.stdout:9/923: dread d13/d1b/d33/d64/dfa/f49 [0,4194304] 0 2024-04-05T03:02:53.051 INFO:tasks.workunit.client.1.smithi203.stdout:1/845: fdatasync d7/d14/d83/fa4 0 2024-04-05T03:02:53.052 INFO:tasks.workunit.client.0.smithi153.stdout:2/579: dwrite d9/da/d19/d2e/f8f [0,4194304] 0 2024-04-05T03:02:53.052 INFO:tasks.workunit.client.1.smithi203.stdout:5/850: dread - d3/d1f/d59/d64/fcd zero size 2024-04-05T03:02:53.053 INFO:tasks.workunit.client.1.smithi203.stdout:0/848: write dd/d17/d1d/d26/fd1 [478002,76537] 0 2024-04-05T03:02:53.055 INFO:tasks.workunit.client.1.smithi203.stdout:5/851: write d3/d4/dc/d67/dac/ff0 [997118,10449] 0 2024-04-05T03:02:53.059 INFO:tasks.workunit.client.1.smithi203.stdout:9/924: rename d13/dc8 to d13/d1b/d33/d64/dfa/d124 0 2024-04-05T03:02:53.060 INFO:tasks.workunit.client.0.smithi153.stdout:2/580: symlink d9/da/d36/lbe 0 2024-04-05T03:02:53.061 INFO:tasks.workunit.client.1.smithi203.stdout:9/925: write d13/d1b/d6b/fbf [1138971,107205] 0 2024-04-05T03:02:53.062 INFO:tasks.workunit.client.1.smithi203.stdout:1/846: creat d7/d14/d2b/d58/d7a/f112 x:0 0 0 2024-04-05T03:02:53.064 INFO:tasks.workunit.client.1.smithi203.stdout:0/849: creat dd/d17/d1d/d26/d28/ff7 x:0 0 0 2024-04-05T03:02:53.065 INFO:tasks.workunit.client.0.smithi153.stdout:2/581: mkdir d9/da/d94/dbf 0 2024-04-05T03:02:53.066 INFO:tasks.workunit.client.0.smithi153.stdout:4/621: dwrite da/db/d20/f4f [0,4194304] 0 2024-04-05T03:02:53.069 INFO:tasks.workunit.client.1.smithi203.stdout:4/636: dread da/d29/f47 [0,4194304] 0 2024-04-05T03:02:53.070 INFO:tasks.workunit.client.0.smithi153.stdout:4/622: symlink da/db/d20/d3c/d81/ld8 0 2024-04-05T03:02:53.070 INFO:tasks.workunit.client.1.smithi203.stdout:5/852: dwrite d3/f36 [0,4194304] 0 2024-04-05T03:02:53.072 INFO:tasks.workunit.client.0.smithi153.stdout:2/582: dwrite d9/da/d36/f81 [4194304,4194304] 0 2024-04-05T03:02:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:52 smithi153 ceph-mon[29381]: pgmap v11: 65 pgs: 65 active+clean; 2.3 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 82 MiB/s rd, 263 MiB/s wr, 480 op/s 2024-04-05T03:02:53.074 INFO:tasks.workunit.client.0.smithi153.stdout:4/623: mknod da/de/d56/d78/d8f/d68/cd9 0 2024-04-05T03:02:53.075 INFO:tasks.workunit.client.0.smithi153.stdout:4/624: chown da/db/f1a 3820 1 2024-04-05T03:02:53.081 INFO:tasks.workunit.client.0.smithi153.stdout:4/625: dwrite da/db/d20/d45/f43 [0,4194304] 0 2024-04-05T03:02:53.087 INFO:tasks.workunit.client.1.smithi203.stdout:8/982: dwrite d1/d4b/d5f/daf/fbf [4194304,4194304] 0 2024-04-05T03:02:53.089 INFO:tasks.workunit.client.0.smithi153.stdout:2/583: link d9/da/d19/d2e/d7e/d53/f83 d9/d77/dad/fc0 0 2024-04-05T03:02:53.090 INFO:tasks.workunit.client.1.smithi203.stdout:1/847: dwrite d7/f1c [0,4194304] 0 2024-04-05T03:02:53.091 INFO:tasks.workunit.client.1.smithi203.stdout:2/930: dwrite d5/d63/f75 [0,4194304] 0 2024-04-05T03:02:53.093 INFO:tasks.workunit.client.0.smithi153.stdout:2/584: rmdir d9/da/d19/d2e/d4f 39 2024-04-05T03:02:53.098 INFO:tasks.workunit.client.0.smithi153.stdout:2/585: dread d9/d68/db9/f92 [0,4194304] 0 2024-04-05T03:02:53.098 INFO:tasks.workunit.client.0.smithi153.stdout:4/626: creat da/db/d20/fda x:0 0 0 2024-04-05T03:02:53.100 INFO:tasks.workunit.client.0.smithi153.stdout:4/627: truncate da/db/d20/d45/fa6 112666 0 2024-04-05T03:02:53.103 INFO:tasks.workunit.client.0.smithi153.stdout:4/628: dread - da/de/d56/d6a/d2d/f7a zero size 2024-04-05T03:02:53.103 INFO:tasks.workunit.client.1.smithi203.stdout:4/637: creat da/d29/d6a/d80/fc7 x:0 0 0 2024-04-05T03:02:53.105 INFO:tasks.workunit.client.1.smithi203.stdout:5/853: dwrite d3/d4/d94/fed [0,4194304] 0 2024-04-05T03:02:53.105 INFO:tasks.workunit.client.0.smithi153.stdout:4/629: readlink da/de/d56/d6a/d2d/lbf 0 2024-04-05T03:02:53.107 INFO:tasks.workunit.client.0.smithi153.stdout:2/586: dwrite d9/d68/db9/fbc [0,4194304] 0 2024-04-05T03:02:53.107 INFO:tasks.workunit.client.1.smithi203.stdout:4/638: dread - da/d29/fb2 zero size 2024-04-05T03:02:53.107 INFO:tasks.workunit.client.0.smithi153.stdout:4/630: dread - da/db/d20/d35/d38/d76/fc6 zero size 2024-04-05T03:02:53.108 INFO:tasks.workunit.client.1.smithi203.stdout:0/850: unlink dd/d17/d1d/d26/d8c/d6d/dc4/fe2 0 2024-04-05T03:02:53.112 INFO:tasks.workunit.client.0.smithi153.stdout:4/631: mkdir da/de/d56/d6a/d2d/ddb 0 2024-04-05T03:02:53.112 INFO:tasks.workunit.client.1.smithi203.stdout:5/854: truncate d3/d1f/d59/d64/f23 1230455 0 2024-04-05T03:02:53.113 INFO:tasks.workunit.client.1.smithi203.stdout:2/931: getdents d5/d10/d35/d2d/d7c/d29/d82/d102 0 2024-04-05T03:02:53.114 INFO:tasks.workunit.client.0.smithi153.stdout:4/632: chown da/de/d56/d6a/d2d/fa5 0 1 2024-04-05T03:02:53.114 INFO:tasks.workunit.client.1.smithi203.stdout:1/848: mknod d7/d24/d33/d48/d4b/d6b/c113 0 2024-04-05T03:02:53.115 INFO:tasks.workunit.client.0.smithi153.stdout:2/587: rename d9/da/d19/d2e/d7e/d53/l96 to d9/da/d19/d2e/d7e/d7a/d6d/lc1 0 2024-04-05T03:02:53.117 INFO:tasks.workunit.client.1.smithi203.stdout:5/855: mknod d3/d1f/d59/d60/c11a 0 2024-04-05T03:02:53.119 INFO:tasks.workunit.client.0.smithi153.stdout:4/633: mkdir da/db/d20/d35/d38/d76/d90/da7/ddc 0 2024-04-05T03:02:53.120 INFO:tasks.workunit.client.0.smithi153.stdout:4/634: chown da/db/f96 1 1 2024-04-05T03:02:53.120 INFO:tasks.workunit.client.1.smithi203.stdout:2/932: creat d5/d10/d44/df6/f13f x:0 0 0 2024-04-05T03:02:53.121 INFO:tasks.workunit.client.0.smithi153.stdout:4/635: fdatasync da/de/d56/d6a/d2d/fa5 0 2024-04-05T03:02:53.122 INFO:tasks.workunit.client.1.smithi203.stdout:0/851: rmdir dd/d17/d1d/d22/d53/daa 0 2024-04-05T03:02:53.123 INFO:tasks.workunit.client.0.smithi153.stdout:4/636: stat da/de/d56/d6a/ccb 0 2024-04-05T03:02:53.123 INFO:tasks.workunit.client.1.smithi203.stdout:4/639: getdents da/d29/d2e/d57/da5 0 2024-04-05T03:02:53.124 INFO:tasks.workunit.client.1.smithi203.stdout:4/640: fdatasync da/d29/d2e/f7a 0 2024-04-05T03:02:53.126 INFO:tasks.workunit.client.0.smithi153.stdout:4/637: rename da/db/d20/f4f to da/de/d56/d6a/d2d/ddb/fdd 0 2024-04-05T03:02:53.128 INFO:tasks.workunit.client.0.smithi153.stdout:4/638: symlink da/db/d20/d3c/d81/db6/lde 0 2024-04-05T03:02:53.129 INFO:tasks.workunit.client.1.smithi203.stdout:4/641: mkdir da/d29/d2e/d8b/dc8 0 2024-04-05T03:02:53.129 INFO:tasks.workunit.client.1.smithi203.stdout:5/856: link d3/d1f/d59/d64/d12/l97 d3/d4/dc/df2/l11b 0 2024-04-05T03:02:53.129 INFO:tasks.workunit.client.0.smithi153.stdout:4/639: chown da/db/d8b/c9d 1 1 2024-04-05T03:02:53.131 INFO:tasks.workunit.client.1.smithi203.stdout:5/857: symlink d3/d5f/d92/dff/d103/l11c 0 2024-04-05T03:02:53.132 INFO:tasks.workunit.client.1.smithi203.stdout:9/926: write d13/d1b/d6b/d9e/d59/fc1 [3164688,108966] 0 2024-04-05T03:02:53.132 INFO:tasks.workunit.client.1.smithi203.stdout:4/642: mkdir da/d29/d2e/dc9 0 2024-04-05T03:02:53.133 INFO:tasks.workunit.client.1.smithi203.stdout:5/858: unlink d3/d4/l7c 0 2024-04-05T03:02:53.134 INFO:tasks.workunit.client.1.smithi203.stdout:5/859: readlink d3/d1f/d10e/de0/l116 0 2024-04-05T03:02:53.135 INFO:tasks.workunit.client.1.smithi203.stdout:4/643: rename da/d3c/lad to da/d29/d2e/d57/lca 0 2024-04-05T03:02:53.136 INFO:tasks.workunit.client.1.smithi203.stdout:5/860: creat d3/d4/dc/df2/f11d x:0 0 0 2024-04-05T03:02:53.140 INFO:tasks.workunit.client.1.smithi203.stdout:5/861: rename d3/d4/dc/d67/f119 to d3/d1f/d59/f11e 0 2024-04-05T03:02:53.146 INFO:tasks.workunit.client.1.smithi203.stdout:8/983: truncate d1/d6/d2e/f6b 2684363 0 2024-04-05T03:02:53.152 INFO:tasks.workunit.client.1.smithi203.stdout:0/852: truncate dd/d17/d1d/d26/d28/f87 1151579 0 2024-04-05T03:02:53.154 INFO:tasks.workunit.client.1.smithi203.stdout:9/927: write d13/d1b/d6b/d9e/d59/fb2 [1962009,62384] 0 2024-04-05T03:02:53.155 INFO:tasks.workunit.client.1.smithi203.stdout:2/933: dwrite d5/d10/d35/d73/f95 [0,4194304] 0 2024-04-05T03:02:53.155 INFO:tasks.workunit.client.1.smithi203.stdout:0/853: read dd/d17/d1d/d22/d53/d61/f9d [353459,106169] 0 2024-04-05T03:02:53.158 INFO:tasks.workunit.client.1.smithi203.stdout:1/849: dread d7/f4f [0,4194304] 0 2024-04-05T03:02:53.158 INFO:tasks.workunit.client.1.smithi203.stdout:4/644: write da/f74 [980860,70151] 0 2024-04-05T03:02:53.160 INFO:tasks.workunit.client.1.smithi203.stdout:5/862: write d3/d1f/d59/d64/f44 [1679174,105181] 0 2024-04-05T03:02:53.160 INFO:tasks.workunit.client.1.smithi203.stdout:0/854: chown dd/d17/d1d/d22/d80/cb9 24797357 1 2024-04-05T03:02:53.161 INFO:tasks.workunit.client.1.smithi203.stdout:0/855: stat dd/d17/lb4 0 2024-04-05T03:02:53.168 INFO:tasks.workunit.client.0.smithi153.stdout:8/460: sync 2024-04-05T03:02:53.168 INFO:tasks.workunit.client.1.smithi203.stdout:9/928: symlink d13/d1b/d6b/d9e/d59/d9b/dd8/l125 0 2024-04-05T03:02:53.169 INFO:tasks.workunit.client.0.smithi153.stdout:4/640: truncate da/f4d 490730 0 2024-04-05T03:02:53.170 INFO:tasks.workunit.client.1.smithi203.stdout:1/850: creat d7/d24/d33/d48/d4b/dea/df2/f114 x:0 0 0 2024-04-05T03:02:53.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/461: creat d7/d13/d19/d1b/d4f/f94 x:0 0 0 2024-04-05T03:02:53.172 INFO:tasks.workunit.client.0.smithi153.stdout:4/641: mknod da/db/d20/d35/d38/d76/d90/da7/cdf 0 2024-04-05T03:02:53.173 INFO:tasks.workunit.client.0.smithi153.stdout:2/588: dwrite d9/da/d19/d2e/d7e/f5e [0,4194304] 0 2024-04-05T03:02:53.173 INFO:tasks.workunit.client.1.smithi203.stdout:5/863: creat d3/d4/d9e/dde/f11f x:0 0 0 2024-04-05T03:02:53.175 INFO:tasks.workunit.client.1.smithi203.stdout:5/864: dread - d3/d4/d9e/dde/f11f zero size 2024-04-05T03:02:53.178 INFO:tasks.workunit.client.1.smithi203.stdout:8/984: dwrite d1/d6/d2e/f30 [0,4194304] 0 2024-04-05T03:02:53.178 INFO:tasks.workunit.client.1.smithi203.stdout:4/645: rename da/d29/d6a to da/d29/d2e/db0/dcb 0 2024-04-05T03:02:53.178 INFO:tasks.workunit.client.0.smithi153.stdout:4/642: getdents da/de/d56/d78/d8f 0 2024-04-05T03:02:53.179 INFO:tasks.workunit.client.1.smithi203.stdout:4/646: read - da/d29/d2e/db0/fb5 zero size 2024-04-05T03:02:53.181 INFO:tasks.workunit.client.0.smithi153.stdout:2/589: rename d9/da/d19/d2e/d48/f1a to d9/da/d37/fc2 0 2024-04-05T03:02:53.182 INFO:tasks.workunit.client.0.smithi153.stdout:8/462: dwrite d7/fc [0,4194304] 0 2024-04-05T03:02:53.185 INFO:tasks.workunit.client.0.smithi153.stdout:2/590: write d9/d68/d9b/fb4 [610755,14802] 0 2024-04-05T03:02:53.185 INFO:tasks.workunit.client.0.smithi153.stdout:4/643: creat da/db/d8b/db7/fe0 x:0 0 0 2024-04-05T03:02:53.186 INFO:tasks.workunit.client.0.smithi153.stdout:8/463: fdatasync d7/d13/f8e 0 2024-04-05T03:02:53.188 INFO:tasks.workunit.client.0.smithi153.stdout:2/591: truncate d9/da/d19/d2e/d48/f5b 1044742 0 2024-04-05T03:02:53.189 INFO:tasks.workunit.client.0.smithi153.stdout:4/644: chown da/db/d20/d35/d41/d67/caa 856 1 2024-04-05T03:02:53.190 INFO:tasks.workunit.client.1.smithi203.stdout:1/851: mknod d7/d14/d83/deb/c115 0 2024-04-05T03:02:53.193 INFO:tasks.workunit.client.1.smithi203.stdout:5/865: symlink d3/d5f/d92/dff/d103/l120 0 2024-04-05T03:02:53.194 INFO:tasks.workunit.client.1.smithi203.stdout:8/985: creat d1/d4b/d5f/d74/f11e x:0 0 0 2024-04-05T03:02:53.195 INFO:tasks.workunit.client.1.smithi203.stdout:4/647: mknod da/d29/d2e/d57/d96/ccc 0 2024-04-05T03:02:53.198 INFO:tasks.workunit.client.1.smithi203.stdout:9/929: write d13/d5c/d9d/fde [2731964,85836] 0 2024-04-05T03:02:53.200 INFO:tasks.workunit.client.1.smithi203.stdout:2/934: dwrite d5/d10/d35/d2d/f50 [4194304,4194304] 0 2024-04-05T03:02:53.201 INFO:tasks.workunit.client.1.smithi203.stdout:2/935: chown d5/d10/d35/d2d/l32 5908 1 2024-04-05T03:02:53.202 INFO:tasks.workunit.client.1.smithi203.stdout:0/856: getdents dd/d17/dd8 0 2024-04-05T03:02:53.205 INFO:tasks.workunit.client.1.smithi203.stdout:1/852: rename d7/d24/d33/d48/d4b/d6b/c113 to d7/d24/d33/d48/d4b/d6b/c116 0 2024-04-05T03:02:53.208 INFO:tasks.workunit.client.1.smithi203.stdout:5/866: creat d3/d1f/d59/d60/dca/f121 x:0 0 0 2024-04-05T03:02:53.211 INFO:tasks.workunit.client.1.smithi203.stdout:2/936: mkdir d5/d10/d35/d2d/d7c/d72/d10b/d138/d140 0 2024-04-05T03:02:53.211 INFO:tasks.workunit.client.1.smithi203.stdout:4/648: link da/d29/d2e/db0/dcb/d80/c8e da/d29/d2e/db0/dcb/ccd 0 2024-04-05T03:02:53.211 INFO:tasks.workunit.client.1.smithi203.stdout:9/930: dread d13/d1b/d33/d64/dfa/f49 [0,4194304] 0 2024-04-05T03:02:53.213 INFO:tasks.workunit.client.1.smithi203.stdout:5/867: chown d3/d1f/d59/d64/d12/leb 30763364 1 2024-04-05T03:02:53.214 INFO:tasks.workunit.client.1.smithi203.stdout:0/857: rename dd/d17/d1d/d22/f65 to dd/d17/ff8 0 2024-04-05T03:02:53.215 INFO:tasks.workunit.client.1.smithi203.stdout:1/853: dwrite d7/d14/d83/fb4 [0,4194304] 0 2024-04-05T03:02:53.215 INFO:tasks.workunit.client.1.smithi203.stdout:4/649: mkdir da/d29/d2e/db0/dbe/dce 0 2024-04-05T03:02:53.216 INFO:tasks.workunit.client.1.smithi203.stdout:1/854: chown d7/f2a 1561347 1 2024-04-05T03:02:53.217 INFO:tasks.workunit.client.1.smithi203.stdout:2/937: creat d5/d10/db8/dab/f141 x:0 0 0 2024-04-05T03:02:53.217 INFO:tasks.workunit.client.1.smithi203.stdout:1/855: stat d7/d14/d2b/cb7 0 2024-04-05T03:02:53.219 INFO:tasks.workunit.client.1.smithi203.stdout:9/931: truncate d13/f4a 5582076 0 2024-04-05T03:02:53.225 INFO:tasks.workunit.client.1.smithi203.stdout:5/868: rename d3/d4/d94/cd0 to d3/d1f/d59/d64/d12/c122 0 2024-04-05T03:02:53.226 INFO:tasks.workunit.client.1.smithi203.stdout:1/856: mkdir d7/d14/d83/d117 0 2024-04-05T03:02:53.227 INFO:tasks.workunit.client.1.smithi203.stdout:5/869: chown d3/d1f/d59/d64/lfd 248 1 2024-04-05T03:02:53.227 INFO:tasks.workunit.client.1.smithi203.stdout:2/938: read - d5/d10/d35/d36/db6/d71/fe7 zero size 2024-04-05T03:02:53.227 INFO:tasks.workunit.client.1.smithi203.stdout:5/870: fsync d3/d4/d9e/dde/f11f 0 2024-04-05T03:02:53.228 INFO:tasks.workunit.client.1.smithi203.stdout:9/932: truncate d13/d1b/d6b/d7a/fb6 956071 0 2024-04-05T03:02:53.229 INFO:tasks.workunit.client.1.smithi203.stdout:2/939: read - d5/d10/db8/dbe/fe0 zero size 2024-04-05T03:02:53.229 INFO:tasks.workunit.client.1.smithi203.stdout:0/858: mknod dd/d17/d1d/d22/d45/cf9 0 2024-04-05T03:02:53.231 INFO:tasks.workunit.client.1.smithi203.stdout:4/650: dread da/f26 [0,4194304] 0 2024-04-05T03:02:53.231 INFO:tasks.workunit.client.1.smithi203.stdout:5/871: creat d3/d5f/f123 x:0 0 0 2024-04-05T03:02:53.232 INFO:tasks.workunit.client.1.smithi203.stdout:2/940: creat d5/d10/d35/df4/f142 x:0 0 0 2024-04-05T03:02:53.235 INFO:tasks.workunit.client.1.smithi203.stdout:0/859: truncate dd/d17/d1d/d26/d8c/fc9 1535826 0 2024-04-05T03:02:53.235 INFO:tasks.workunit.client.1.smithi203.stdout:4/651: rename da/d29/f50 to da/d29/d2e/d53/d78/fcf 0 2024-04-05T03:02:53.240 INFO:tasks.workunit.client.1.smithi203.stdout:8/986: dread d1/d12/d17/d47/d53/dd0/f105 [0,4194304] 0 2024-04-05T03:02:53.241 INFO:tasks.workunit.client.1.smithi203.stdout:0/860: fsync dd/d17/d1d/d95/fb2 0 2024-04-05T03:02:53.243 INFO:tasks.workunit.client.1.smithi203.stdout:2/941: rename d5/d10/d35/d2d/l3d to d5/d10/d35/d36/d109/l143 0 2024-04-05T03:02:53.244 INFO:tasks.workunit.client.1.smithi203.stdout:8/987: chown d1/d12/d1a/f1d 7557596 1 2024-04-05T03:02:53.246 INFO:tasks.workunit.client.1.smithi203.stdout:5/872: dwrite d3/d1f/fc0 [0,4194304] 0 2024-04-05T03:02:53.246 INFO:tasks.workunit.client.1.smithi203.stdout:4/652: rename da/d29/d2e/d57/da7/dab to da/d29/d2e/db0/dbe/dce/dd0 0 2024-04-05T03:02:53.250 INFO:tasks.workunit.client.1.smithi203.stdout:5/873: chown d3/l3c 9287791 1 2024-04-05T03:02:53.250 INFO:tasks.workunit.client.1.smithi203.stdout:0/861: unlink dd/d17/d1d/d22/d53/db0/c31 0 2024-04-05T03:02:53.250 INFO:tasks.workunit.client.1.smithi203.stdout:4/653: chown da/d29/d2e/d57/da5 30418 1 2024-04-05T03:02:53.251 INFO:tasks.workunit.client.1.smithi203.stdout:1/857: dread d7/f2a [0,4194304] 0 2024-04-05T03:02:53.251 INFO:tasks.workunit.client.1.smithi203.stdout:8/988: dread d1/d12/d17/d47/d53/d69/d9c/ffe [0,4194304] 0 2024-04-05T03:02:53.253 INFO:tasks.workunit.client.1.smithi203.stdout:9/933: dread d13/d5c/dd1/d31/f41 [0,4194304] 0 2024-04-05T03:02:53.254 INFO:tasks.workunit.client.1.smithi203.stdout:4/654: creat da/d29/d2e/db0/fd1 x:0 0 0 2024-04-05T03:02:53.255 INFO:tasks.workunit.client.1.smithi203.stdout:2/942: creat d5/d10/d35/d2d/d7c/d72/f144 x:0 0 0 2024-04-05T03:02:53.256 INFO:tasks.workunit.client.1.smithi203.stdout:0/862: symlink dd/d17/lfa 0 2024-04-05T03:02:53.257 INFO:tasks.workunit.client.1.smithi203.stdout:8/989: mkdir d1/d12/d2f/d1f/d11f 0 2024-04-05T03:02:53.258 INFO:tasks.workunit.client.1.smithi203.stdout:5/874: getdents d3/d4/dc/df2 0 2024-04-05T03:02:53.258 INFO:tasks.workunit.client.1.smithi203.stdout:4/655: truncate da/d29/d2e/db0/dcb/d80/fa6 668638 0 2024-04-05T03:02:53.259 INFO:tasks.workunit.client.1.smithi203.stdout:2/943: mkdir d5/d10/d35/d36/db6/d87/d145 0 2024-04-05T03:02:53.260 INFO:tasks.workunit.client.1.smithi203.stdout:0/863: rmdir dd/d17/d1d/d26/d8c 39 2024-04-05T03:02:53.261 INFO:tasks.workunit.client.1.smithi203.stdout:4/656: fdatasync da/f13 0 2024-04-05T03:02:53.261 INFO:tasks.workunit.client.1.smithi203.stdout:9/934: truncate d13/d1b/d33/d63/f6e 1159950 0 2024-04-05T03:02:53.264 INFO:tasks.workunit.client.1.smithi203.stdout:0/864: mknod dd/d17/d1d/d22/d53/d55/cfb 0 2024-04-05T03:02:53.265 INFO:tasks.workunit.client.1.smithi203.stdout:4/657: unlink da/f9f 0 2024-04-05T03:02:53.266 INFO:tasks.workunit.client.1.smithi203.stdout:8/990: creat d1/d12/d2f/d55/f120 x:0 0 0 2024-04-05T03:02:53.270 INFO:tasks.workunit.client.1.smithi203.stdout:0/865: mknod dd/d17/d1d/d26/d28/cfc 0 2024-04-05T03:02:53.273 INFO:tasks.workunit.client.1.smithi203.stdout:1/858: truncate d7/d24/d33/d48/d4b/d51/f67 1701826 0 2024-04-05T03:02:53.276 INFO:tasks.workunit.client.1.smithi203.stdout:8/991: dwrite d1/d4b/d5f/d74/f11e [0,4194304] 0 2024-04-05T03:02:53.278 INFO:tasks.workunit.client.1.smithi203.stdout:0/866: read - dd/d17/d1d/d22/d45/dbe/fd2 zero size 2024-04-05T03:02:53.280 INFO:tasks.workunit.client.1.smithi203.stdout:4/658: dwrite da/d29/d2e/d57/f86 [0,4194304] 0 2024-04-05T03:02:53.281 INFO:tasks.workunit.client.1.smithi203.stdout:1/859: rename d7/l70 to d7/d14/d83/dfb/l118 0 2024-04-05T03:02:53.283 INFO:tasks.workunit.client.1.smithi203.stdout:8/992: read d1/d4b/d5f/daf/fba [377610,51405] 0 2024-04-05T03:02:53.284 INFO:tasks.workunit.client.1.smithi203.stdout:9/935: truncate d13/d1b/d6b/d9e/d59/dbc/fc4 889052 0 2024-04-05T03:02:53.285 INFO:tasks.workunit.client.1.smithi203.stdout:2/944: dwrite d5/d10/d1a/f65 [0,4194304] 0 2024-04-05T03:02:53.285 INFO:tasks.workunit.client.0.smithi153.stdout:7/652: sync 2024-04-05T03:02:53.288 INFO:tasks.workunit.client.1.smithi203.stdout:4/659: mknod da/d71/cd2 0 2024-04-05T03:02:53.289 INFO:tasks.workunit.client.1.smithi203.stdout:9/936: chown d13/d1b/d33/d64/dfa/f49 68 1 2024-04-05T03:02:53.291 INFO:tasks.workunit.client.0.smithi153.stdout:7/653: dread d2/da/d3a/d41/d31/f4d [0,4194304] 0 2024-04-05T03:02:53.292 INFO:tasks.workunit.client.1.smithi203.stdout:0/867: dwrite dd/d17/d1d/d26/d28/fad [0,4194304] 0 2024-04-05T03:02:53.293 INFO:tasks.workunit.client.1.smithi203.stdout:2/945: creat d5/d10/d2f/d5d/f146 x:0 0 0 2024-04-05T03:02:53.295 INFO:tasks.workunit.client.1.smithi203.stdout:8/993: truncate d1/d12/da8/fe1 254721 0 2024-04-05T03:02:53.295 INFO:tasks.workunit.client.1.smithi203.stdout:0/868: readlink dd/d17/d1d/d22/d53/d61/lde 0 2024-04-05T03:02:53.296 INFO:tasks.workunit.client.1.smithi203.stdout:2/946: write d5/d10/d35/d2d/d7c/d29/f53 [1536151,85809] 0 2024-04-05T03:02:53.297 INFO:tasks.workunit.client.1.smithi203.stdout:8/994: stat d1/d12/d1a/d6d/d97/f117 0 2024-04-05T03:02:53.298 INFO:tasks.workunit.client.1.smithi203.stdout:4/660: symlink da/d29/d2e/d8b/dc8/ld3 0 2024-04-05T03:02:53.300 INFO:tasks.workunit.client.1.smithi203.stdout:0/869: mknod dd/d17/d1d/d22/d53/d7f/dc5/cfd 0 2024-04-05T03:02:53.301 INFO:tasks.workunit.client.1.smithi203.stdout:9/937: rmdir d13/de9/d10b 0 2024-04-05T03:02:53.303 INFO:tasks.workunit.client.1.smithi203.stdout:8/995: creat d1/d12/d1a/f121 x:0 0 0 2024-04-05T03:02:53.305 INFO:tasks.workunit.client.1.smithi203.stdout:4/661: unlink da/d29/ca9 0 2024-04-05T03:02:53.308 INFO:tasks.workunit.client.1.smithi203.stdout:9/938: creat d13/d1b/d6b/d9e/d59/d5f/d103/f126 x:0 0 0 2024-04-05T03:02:53.310 INFO:tasks.workunit.client.1.smithi203.stdout:2/947: mkdir d5/d10/d35/d2d/d7c/d72/d8f/d147 0 2024-04-05T03:02:53.313 INFO:tasks.workunit.client.1.smithi203.stdout:4/662: mknod da/d29/d2e/db0/dbe/cd4 0 2024-04-05T03:02:53.316 INFO:tasks.workunit.client.1.smithi203.stdout:1/860: truncate d7/d14/d83/fb4 3116935 0 2024-04-05T03:02:53.317 INFO:tasks.workunit.client.1.smithi203.stdout:4/663: unlink da/d71/c90 0 2024-04-05T03:02:53.320 INFO:tasks.workunit.client.1.smithi203.stdout:1/861: mknod d7/d14/d83/dfb/c119 0 2024-04-05T03:02:53.322 INFO:tasks.workunit.client.1.smithi203.stdout:9/939: getdents d13/d5c/dd1/d51/dcb 0 2024-04-05T03:02:53.327 INFO:tasks.workunit.client.1.smithi203.stdout:9/940: getdents d13/d5c 0 2024-04-05T03:02:53.329 INFO:tasks.workunit.client.1.smithi203.stdout:8/996: dread d1/d12/d17/d47/d53/f82 [0,4194304] 0 2024-04-05T03:02:53.334 INFO:tasks.workunit.client.1.smithi203.stdout:9/941: fsync d13/d5c/dd1/d51/d81/f11f 0 2024-04-05T03:02:53.335 INFO:tasks.workunit.client.1.smithi203.stdout:0/870: dwrite dd/d17/d1d/d22/d80/f72 [0,4194304] 0 2024-04-05T03:02:53.336 INFO:tasks.workunit.client.0.smithi153.stdout:2/592: mknod d9/d77/dad/cc3 0 2024-04-05T03:02:53.337 INFO:tasks.workunit.client.1.smithi203.stdout:2/948: dwrite d5/d10/d35/d73/f110 [0,4194304] 0 2024-04-05T03:02:53.342 INFO:tasks.workunit.client.0.smithi153.stdout:7/654: mknod d2/da/d47/d5d/d7c/d8d/d95/dce/cd8 0 2024-04-05T03:02:53.344 INFO:tasks.workunit.client.1.smithi203.stdout:4/664: dread da/d29/fbf [0,4194304] 0 2024-04-05T03:02:53.345 INFO:tasks.workunit.client.0.smithi153.stdout:8/464: mkdir d7/d13/d19/d95 0 2024-04-05T03:02:53.345 INFO:tasks.workunit.client.1.smithi203.stdout:8/997: fdatasync d1/d90/dcd/f118 0 2024-04-05T03:02:53.345 INFO:tasks.workunit.client.1.smithi203.stdout:1/862: dread d7/d14/d2b/d58/fba [0,4194304] 0 2024-04-05T03:02:53.348 INFO:tasks.workunit.client.0.smithi153.stdout:4/645: mkdir da/db/d20/d3c/d81/db6/dc7/de1 0 2024-04-05T03:02:53.349 INFO:tasks.workunit.client.1.smithi203.stdout:0/871: mkdir dd/d17/d1d/d22/d45/d51/d69/dfe 0 2024-04-05T03:02:53.351 INFO:tasks.workunit.client.1.smithi203.stdout:1/863: symlink d7/d24/d33/d48/d4b/dea/df2/l11a 0 2024-04-05T03:02:53.353 INFO:tasks.workunit.client.0.smithi153.stdout:4/646: getdents da/db/d20/d3c/d91/dd5 0 2024-04-05T03:02:53.353 INFO:tasks.workunit.client.1.smithi203.stdout:1/864: write d7/d90/f101 [221512,9819] 0 2024-04-05T03:02:53.353 INFO:tasks.workunit.client.1.smithi203.stdout:0/872: fdatasync dd/d17/f79 0 2024-04-05T03:02:53.354 INFO:tasks.workunit.client.0.smithi153.stdout:2/593: getdents d9/da/d36 0 2024-04-05T03:02:53.355 INFO:tasks.workunit.client.1.smithi203.stdout:4/665: link da/d29/d2e/db0/dcb/d80/c85 da/d29/cd5 0 2024-04-05T03:02:53.356 INFO:tasks.workunit.client.1.smithi203.stdout:8/998: getdents d1/d12/d17/d47/d53/dd0 0 2024-04-05T03:02:53.359 INFO:tasks.workunit.client.1.smithi203.stdout:1/865: mknod d7/d24/d33/d48/d4b/d6b/c11b 0 2024-04-05T03:02:53.362 INFO:tasks.workunit.client.1.smithi203.stdout:0/873: creat dd/d17/d1d/d22/d53/fff x:0 0 0 2024-04-05T03:02:53.363 INFO:tasks.workunit.client.1.smithi203.stdout:4/666: rmdir da/d29 39 2024-04-05T03:02:53.366 INFO:tasks.workunit.client.1.smithi203.stdout:0/874: rmdir dd/d17/d1d/d22/d45/dbe 39 2024-04-05T03:02:53.369 INFO:tasks.workunit.client.1.smithi203.stdout:4/667: getdents da/d3c 0 2024-04-05T03:02:53.370 INFO:tasks.workunit.client.1.smithi203.stdout:1/866: dwrite d7/d24/d33/d48/d4b/d51/f108 [0,4194304] 0 2024-04-05T03:02:53.371 INFO:tasks.workunit.client.1.smithi203.stdout:0/875: write dd/d17/d1d/d26/d28/f2d [3957753,120883] 0 2024-04-05T03:02:53.372 INFO:tasks.workunit.client.1.smithi203.stdout:2/949: dread d5/d10/d35/d2d/f50 [0,4194304] 0 2024-04-05T03:02:53.374 INFO:tasks.workunit.client.1.smithi203.stdout:1/867: fsync d7/d14/d2b/f95 0 2024-04-05T03:02:53.374 INFO:tasks.workunit.client.1.smithi203.stdout:4/668: rmdir da/d29/d2e/d57/d96 39 2024-04-05T03:02:53.377 INFO:tasks.workunit.client.1.smithi203.stdout:8/999: dread d1/d12/d17/d47/d53/dd0/f119 [0,4194304] 0 2024-04-05T03:02:53.377 INFO:tasks.workunit.client.1.smithi203.stdout:0/876: chown dd/d17/d1d/d22/d45/d51/lce 17258736 1 2024-04-05T03:02:53.377 INFO:tasks.workunit.client.1.smithi203.stdout:1/868: fsync d7/d14/d2b/d58/d86/d91/da6/fe1 0 2024-04-05T03:02:53.377 INFO:tasks.workunit.client.1.smithi203.stdout:2/950: mkdir d5/d148 0 2024-04-05T03:02:53.379 INFO:tasks.workunit.client.1.smithi203.stdout:1/869: read d7/d24/fc5 [28039,19033] 0 2024-04-05T03:02:53.380 INFO:tasks.workunit.client.1.smithi203.stdout:2/951: rmdir d5/d10/d35/d36/d109 39 2024-04-05T03:02:53.381 INFO:tasks.workunit.client.1.smithi203.stdout:2/952: fsync d5/d10/db8/f127 0 2024-04-05T03:02:53.383 INFO:tasks.workunit.client.1.smithi203.stdout:4/669: link da/d29/d2e/d57/f7f da/d29/d2e/d8b/dc8/fd6 0 2024-04-05T03:02:53.388 INFO:tasks.workunit.client.1.smithi203.stdout:1/870: link d7/d24/la9 d7/d24/d33/d48/d4b/d6b/db9/l11c 0 2024-04-05T03:02:53.391 INFO:tasks.workunit.client.1.smithi203.stdout:1/871: creat d7/d24/d33/d48/d4b/dea/def/f11d x:0 0 0 2024-04-05T03:02:53.391 INFO:tasks.workunit.client.1.smithi203.stdout:1/872: truncate d7/d14/f21 1704153 0 2024-04-05T03:02:53.392 INFO:tasks.workunit.client.1.smithi203.stdout:2/953: getdents d5/d10/d35/d36/db6/d87 0 2024-04-05T03:02:53.393 INFO:tasks.workunit.client.1.smithi203.stdout:1/873: fsync d7/d14/d2b/d58/d86/d91/da6/df9/d106/f107 0 2024-04-05T03:02:53.394 INFO:tasks.workunit.client.1.smithi203.stdout:1/874: write d7/d14/d2b/d58/ff0 [80679,102110] 0 2024-04-05T03:02:53.410 INFO:tasks.workunit.client.0.smithi153.stdout:7/655: getdents d2/da/d47/da9 0 2024-04-05T03:02:53.410 INFO:tasks.workunit.client.0.smithi153.stdout:4/647: unlink da/db/d20/d35/d41/d67/c6f 0 2024-04-05T03:02:53.413 INFO:tasks.workunit.client.0.smithi153.stdout:4/648: creat da/d2a/d86/fe2 x:0 0 0 2024-04-05T03:02:53.414 INFO:tasks.workunit.client.0.smithi153.stdout:2/594: creat d9/da/d19/d2e/d4f/fc4 x:0 0 0 2024-04-05T03:02:53.414 INFO:tasks.workunit.client.0.smithi153.stdout:4/649: stat da/db/d20/d45/ca9 0 2024-04-05T03:02:53.421 INFO:tasks.workunit.client.0.smithi153.stdout:4/650: dwrite da/de/d56/d78/f7e [0,4194304] 0 2024-04-05T03:02:53.422 INFO:tasks.workunit.client.0.smithi153.stdout:7/656: truncate d2/d9b/d6f/f78 3945628 0 2024-04-05T03:02:53.424 INFO:tasks.workunit.client.0.smithi153.stdout:2/595: truncate d9/da/f5f 1544797 0 2024-04-05T03:02:53.425 INFO:tasks.workunit.client.0.smithi153.stdout:2/596: rmdir d9/d62 39 2024-04-05T03:02:53.427 INFO:tasks.workunit.client.0.smithi153.stdout:7/657: dread d2/da/d3a/f46 [0,4194304] 0 2024-04-05T03:02:53.428 INFO:tasks.workunit.client.0.smithi153.stdout:2/597: unlink d9/da/d19/d2e/d7e/d53/ca5 0 2024-04-05T03:02:53.429 INFO:tasks.workunit.client.0.smithi153.stdout:7/658: chown d2/da/d3a/d41/d31/fd5 841071 1 2024-04-05T03:02:53.432 INFO:tasks.workunit.client.0.smithi153.stdout:2/598: dread d9/d68/d9b/fb4 [0,4194304] 0 2024-04-05T03:02:53.433 INFO:tasks.workunit.client.0.smithi153.stdout:7/659: rename d2/da/d11/d88/fbb to d2/da/d11/d88/da6/fd9 0 2024-04-05T03:02:53.433 INFO:tasks.workunit.client.0.smithi153.stdout:2/599: mknod d9/d68/cc5 0 2024-04-05T03:02:53.434 INFO:tasks.workunit.client.0.smithi153.stdout:7/660: rmdir d2/d51/d93 39 2024-04-05T03:02:53.436 INFO:tasks.workunit.client.0.smithi153.stdout:2/600: dread d9/da/f64 [0,4194304] 0 2024-04-05T03:02:53.437 INFO:tasks.workunit.client.0.smithi153.stdout:7/661: rmdir d2/da/d47/da3 0 2024-04-05T03:02:53.438 INFO:tasks.workunit.client.0.smithi153.stdout:2/601: symlink d9/da/d37/d8a/lc6 0 2024-04-05T03:02:53.438 INFO:tasks.workunit.client.0.smithi153.stdout:7/662: chown d2/da/d3a/d41/d7a/d54/d85 24 1 2024-04-05T03:02:53.440 INFO:tasks.workunit.client.0.smithi153.stdout:7/663: creat d2/da/d47/fda x:0 0 0 2024-04-05T03:02:53.441 INFO:tasks.workunit.client.0.smithi153.stdout:7/664: fsync d2/da/d3a/d41/f8a 0 2024-04-05T03:02:53.442 INFO:tasks.workunit.client.0.smithi153.stdout:2/602: fdatasync d9/da/d19/d2e/d7e/d53/f9c 0 2024-04-05T03:02:53.442 INFO:tasks.workunit.client.0.smithi153.stdout:7/665: stat d2/da/d3a/d41/d31/fd5 0 2024-04-05T03:02:53.443 INFO:tasks.workunit.client.0.smithi153.stdout:2/603: chown d9/da/d19/d2e/d7e/d7a 68 1 2024-04-05T03:02:53.452 INFO:tasks.workunit.client.0.smithi153.stdout:2/604: truncate d9/d77/f58 3659994 0 2024-04-05T03:02:53.456 INFO:tasks.workunit.client.0.smithi153.stdout:7/666: dwrite d2/da/d3a/d41/d31/f4d [0,4194304] 0 2024-04-05T03:02:53.457 INFO:tasks.workunit.client.0.smithi153.stdout:2/605: rename d9/d77/dad to d9/da/d19/d2e/dc7 0 2024-04-05T03:02:53.460 INFO:tasks.workunit.client.0.smithi153.stdout:2/606: mknod d9/da/d19/d2e/d7e/d53/cc8 0 2024-04-05T03:02:53.464 INFO:tasks.workunit.client.0.smithi153.stdout:6/592: sync 2024-04-05T03:02:53.464 INFO:tasks.workunit.client.0.smithi153.stdout:7/667: link d2/f5 d2/da/d47/fdb 0 2024-04-05T03:02:53.467 INFO:tasks.workunit.client.0.smithi153.stdout:7/668: truncate d2/da/f68 158334 0 2024-04-05T03:02:53.469 INFO:tasks.workunit.client.0.smithi153.stdout:2/607: dwrite d9/d77/f69 [0,4194304] 0 2024-04-05T03:02:53.473 INFO:tasks.workunit.client.0.smithi153.stdout:7/669: symlink d2/da/d3a/d41/da8/ldc 0 2024-04-05T03:02:53.475 INFO:tasks.workunit.client.0.smithi153.stdout:6/593: dwrite d4/dc/df/d16/d5f/fac [0,4194304] 0 2024-04-05T03:02:53.477 INFO:tasks.workunit.client.1.smithi203.stdout:0/877: truncate dd/d17/d1d/d22/d53/d7f/fc7 1394094 0 2024-04-05T03:02:53.478 INFO:tasks.workunit.client.1.smithi203.stdout:1/875: symlink d7/l11e 0 2024-04-05T03:02:53.479 INFO:tasks.workunit.client.1.smithi203.stdout:4/670: dread da/d29/d2e/db0/dcb/d80/fb9 [0,4194304] 0 2024-04-05T03:02:53.480 INFO:tasks.workunit.client.0.smithi153.stdout:7/670: mkdir d2/da/d3a/d41/ddd 0 2024-04-05T03:02:53.481 INFO:tasks.workunit.client.0.smithi153.stdout:6/594: symlink d4/dc/df/d16/d33/lc4 0 2024-04-05T03:02:53.481 INFO:tasks.workunit.client.1.smithi203.stdout:1/876: stat d7/c18 0 2024-04-05T03:02:53.484 INFO:tasks.workunit.client.1.smithi203.stdout:0/878: truncate dd/d17/d1d/d22/d53/d61/fe5 1197504 0 2024-04-05T03:02:53.485 INFO:tasks.workunit.client.1.smithi203.stdout:4/671: creat da/d29/d2e/d8b/fd7 x:0 0 0 2024-04-05T03:02:53.485 INFO:tasks.workunit.client.0.smithi153.stdout:6/595: creat d4/dc/d7a/da8/fc5 x:0 0 0 2024-04-05T03:02:53.485 INFO:tasks.workunit.client.1.smithi203.stdout:2/954: dwrite d5/d10/d35/d36/db6/d87/f117 [0,4194304] 0 2024-04-05T03:02:53.487 INFO:tasks.workunit.client.0.smithi153.stdout:7/671: dwrite d2/da/d3a/d41/d31/f4d [0,4194304] 0 2024-04-05T03:02:53.488 INFO:tasks.workunit.client.1.smithi203.stdout:1/877: mkdir d7/d24/d11f 0 2024-04-05T03:02:53.489 INFO:tasks.workunit.client.1.smithi203.stdout:1/878: stat d7/d24/f34 0 2024-04-05T03:02:53.490 INFO:tasks.workunit.client.1.smithi203.stdout:4/672: creat da/d3c/fd8 x:0 0 0 2024-04-05T03:02:53.492 INFO:tasks.workunit.client.1.smithi203.stdout:2/955: rename d5/d10/d35/d2d/d7c/d29/d67/c90 to d5/d10/db8/dab/de1/dbd/c149 0 2024-04-05T03:02:53.495 INFO:tasks.workunit.client.1.smithi203.stdout:2/956: fdatasync d5/d10/db8/dbe/fe0 0 2024-04-05T03:02:53.495 INFO:tasks.workunit.client.0.smithi153.stdout:6/596: read d4/d38/fbe [3310701,66887] 0 2024-04-05T03:02:53.496 INFO:tasks.workunit.client.0.smithi153.stdout:7/672: mkdir d2/da/d47/d5d/d7c/d8d/d95/dce/dc8/dde 0 2024-04-05T03:02:53.496 INFO:tasks.workunit.client.1.smithi203.stdout:2/957: dread - d5/d10/d2f/d5d/f146 zero size 2024-04-05T03:02:53.497 INFO:tasks.workunit.client.0.smithi153.stdout:2/608: dwrite d9/da/d36/f63 [0,4194304] 0 2024-04-05T03:02:53.498 INFO:tasks.workunit.client.1.smithi203.stdout:1/879: dwrite d7/d14/f1e [4194304,4194304] 0 2024-04-05T03:02:53.499 INFO:tasks.workunit.client.1.smithi203.stdout:2/958: chown d5/d66/d7d/c7e 1154617 1 2024-04-05T03:02:53.500 INFO:tasks.workunit.client.0.smithi153.stdout:6/597: write d4/dc/df/d16/d36/d52/f55 [101474,98482] 0 2024-04-05T03:02:53.503 INFO:tasks.workunit.client.1.smithi203.stdout:1/880: mkdir d7/d14/d2b/d58/d86/d120 0 2024-04-05T03:02:53.505 INFO:tasks.workunit.client.0.smithi153.stdout:7/673: creat d2/d51/d77/fdf x:0 0 0 2024-04-05T03:02:53.506 INFO:tasks.workunit.client.0.smithi153.stdout:6/598: creat d4/dc/d7a/da8/fc6 x:0 0 0 2024-04-05T03:02:53.507 INFO:tasks.workunit.client.0.smithi153.stdout:7/674: write d2/da/d47/d5d/d7c/f81 [942211,6674] 0 2024-04-05T03:02:53.509 INFO:tasks.workunit.client.0.smithi153.stdout:6/599: fdatasync d4/dc/df/fb6 0 2024-04-05T03:02:53.510 INFO:tasks.workunit.client.1.smithi203.stdout:1/881: rmdir d7/d24/d33/d48/d4b 39 2024-04-05T03:02:53.513 INFO:tasks.workunit.client.1.smithi203.stdout:1/882: mknod d7/c121 0 2024-04-05T03:02:53.514 INFO:tasks.workunit.client.0.smithi153.stdout:6/600: link d4/dc/df/c34 d4/dc/d7a/cc7 0 2024-04-05T03:02:53.515 INFO:tasks.workunit.client.1.smithi203.stdout:1/883: dread - d7/d24/d33/d48/d4b/dea/df2/f114 zero size 2024-04-05T03:02:53.515 INFO:tasks.workunit.client.1.smithi203.stdout:4/673: truncate da/f74 949040 0 2024-04-05T03:02:53.516 INFO:tasks.workunit.client.1.smithi203.stdout:6/850: sync 2024-04-05T03:02:53.518 INFO:tasks.workunit.client.0.smithi153.stdout:6/601: rmdir d4/dc/df/d16/d17/d54 39 2024-04-05T03:02:53.519 INFO:tasks.workunit.client.1.smithi203.stdout:6/851: write d0/da/d54/d4c/da1/db7/fcb [2162250,105260] 0 2024-04-05T03:02:53.519 INFO:tasks.workunit.client.1.smithi203.stdout:2/959: write d5/d10/d35/d2d/d7c/d29/d82/f105 [935322,49351] 0 2024-04-05T03:02:53.520 INFO:tasks.workunit.client.0.smithi153.stdout:6/602: symlink d4/d38/d4c/lc8 0 2024-04-05T03:02:53.520 INFO:tasks.workunit.client.1.smithi203.stdout:1/884: mkdir d7/d14/d2b/d58/d86/d91/da6/df9/d106/d122 0 2024-04-05T03:02:53.521 INFO:tasks.workunit.client.0.smithi153.stdout:6/603: stat d4/dc/df/d16/d36/d52/f55 0 2024-04-05T03:02:53.521 INFO:tasks.workunit.client.0.smithi153.stdout:6/604: chown d4/dc/df/d16/d33 197 1 2024-04-05T03:02:53.522 INFO:tasks.workunit.client.1.smithi203.stdout:4/674: write da/d3c/d4c/d99/fc1 [543757,77451] 0 2024-04-05T03:02:53.523 INFO:tasks.workunit.client.0.smithi153.stdout:6/605: chown d4/dc/df/d16/d24/c92 0 1 2024-04-05T03:02:53.525 INFO:tasks.workunit.client.1.smithi203.stdout:2/960: creat d5/d10/d35/d2d/d7c/f14a x:0 0 0 2024-04-05T03:02:53.525 INFO:tasks.workunit.client.0.smithi153.stdout:6/606: symlink d4/dc/d7a/lc9 0 2024-04-05T03:02:53.526 INFO:tasks.workunit.client.1.smithi203.stdout:4/675: chown da/d3c/f44 30290 1 2024-04-05T03:02:53.530 INFO:tasks.workunit.client.1.smithi203.stdout:2/961: chown d5/d10/d35/d2d/d7c/c30 341014 1 2024-04-05T03:02:53.531 INFO:tasks.workunit.client.1.smithi203.stdout:6/852: dread d0/da/d54/d4c/d7f/f91 [0,4194304] 0 2024-04-05T03:02:53.531 INFO:tasks.workunit.client.1.smithi203.stdout:1/885: dwrite d7/d14/d9d/ff3 [0,4194304] 0 2024-04-05T03:02:53.531 INFO:tasks.workunit.client.0.smithi153.stdout:6/607: rename d4/dc/df/d16/d5f/d73/fc1 to d4/dc/d7a/da8/fca 0 2024-04-05T03:02:53.533 INFO:tasks.workunit.client.1.smithi203.stdout:4/676: unlink da/d29/d2e/d8b/dc8/ld3 0 2024-04-05T03:02:53.533 INFO:tasks.workunit.client.1.smithi203.stdout:2/962: symlink d5/d10/db8/dbe/l14b 0 2024-04-05T03:02:53.535 INFO:tasks.workunit.client.1.smithi203.stdout:1/886: mknod d7/d14/c123 0 2024-04-05T03:02:53.535 INFO:tasks.workunit.client.1.smithi203.stdout:4/677: stat da/d29/d2e/db0/dcb/d80 0 2024-04-05T03:02:53.538 INFO:tasks.workunit.client.1.smithi203.stdout:4/678: truncate da/d29/d2e/db0/dcb/d80/fc7 502929 0 2024-04-05T03:02:53.538 INFO:tasks.workunit.client.1.smithi203.stdout:2/963: creat d5/d10/db8/dbe/f14c x:0 0 0 2024-04-05T03:02:53.540 INFO:tasks.workunit.client.0.smithi153.stdout:2/609: dwrite d9/da/d37/f3d [0,4194304] 0 2024-04-05T03:02:53.540 INFO:tasks.workunit.client.1.smithi203.stdout:6/853: truncate d0/da/d18/d38/d2a/d44/d78/da7/fc5 1406409 0 2024-04-05T03:02:53.543 INFO:tasks.workunit.client.1.smithi203.stdout:2/964: mknod d5/d63/d86/c14d 0 2024-04-05T03:02:53.543 INFO:tasks.workunit.client.1.smithi203.stdout:6/854: mkdir d0/da/d18/d92/dfe 0 2024-04-05T03:02:53.546 INFO:tasks.workunit.client.1.smithi203.stdout:4/679: unlink da/d3c/d4c/c94 0 2024-04-05T03:02:53.547 INFO:tasks.workunit.client.1.smithi203.stdout:2/965: dread d5/d10/db8/f123 [0,4194304] 0 2024-04-05T03:02:53.547 INFO:tasks.workunit.client.1.smithi203.stdout:6/855: stat d0/da/d18/d38/d2a/fd8 0 2024-04-05T03:02:53.549 INFO:tasks.workunit.client.1.smithi203.stdout:1/887: rename d7/d24/d33/ld8 to d7/d24/d33/d48/d4b/l124 0 2024-04-05T03:02:53.549 INFO:tasks.workunit.client.0.smithi153.stdout:2/610: creat d9/da/d94/fc9 x:0 0 0 2024-04-05T03:02:53.550 INFO:tasks.workunit.client.0.smithi153.stdout:6/608: mkdir d4/dc/df/dcb 0 2024-04-05T03:02:53.550 INFO:tasks.workunit.client.1.smithi203.stdout:4/680: creat da/d29/d2e/d57/da7/fd9 x:0 0 0 2024-04-05T03:02:53.551 INFO:tasks.workunit.client.0.smithi153.stdout:2/611: mkdir d9/da/d94/dca 0 2024-04-05T03:02:53.551 INFO:tasks.workunit.client.1.smithi203.stdout:6/856: symlink d0/da/d18/d92/dfe/lff 0 2024-04-05T03:02:53.552 INFO:tasks.workunit.client.1.smithi203.stdout:2/966: symlink d5/d10/d35/d36/d109/l14e 0 2024-04-05T03:02:53.553 INFO:tasks.workunit.client.0.smithi153.stdout:2/612: creat d9/da/d37/fcb x:0 0 0 2024-04-05T03:02:53.554 INFO:tasks.workunit.client.1.smithi203.stdout:4/681: mknod da/d29/d2e/d53/cda 0 2024-04-05T03:02:53.554 INFO:tasks.workunit.client.1.smithi203.stdout:4/682: fsync da/f13 0 2024-04-05T03:02:53.555 INFO:tasks.workunit.client.0.smithi153.stdout:2/613: fsync d9/da/d19/d2e/f8f 0 2024-04-05T03:02:53.555 INFO:tasks.workunit.client.1.smithi203.stdout:2/967: fdatasync d5/d10/d2f/d5d/ff5 0 2024-04-05T03:02:53.556 INFO:tasks.workunit.client.1.smithi203.stdout:2/968: readlink d5/d10/d35/d2d/d7c/d29/d82/lcc 0 2024-04-05T03:02:53.557 INFO:tasks.workunit.client.0.smithi153.stdout:6/609: creat d4/dc/df/d16/fcc x:0 0 0 2024-04-05T03:02:53.557 INFO:tasks.workunit.client.1.smithi203.stdout:2/969: write d5/d10/f1d [3514902,26949] 0 2024-04-05T03:02:53.558 INFO:tasks.workunit.client.1.smithi203.stdout:6/857: dread d0/da/d54/d53/fb3 [0,4194304] 0 2024-04-05T03:02:53.559 INFO:tasks.workunit.client.1.smithi203.stdout:2/970: chown d5/fb3 24110 1 2024-04-05T03:02:53.559 INFO:tasks.workunit.client.0.smithi153.stdout:2/614: unlink d9/da/d19/d2e/d48/f93 0 2024-04-05T03:02:53.562 INFO:tasks.workunit.client.1.smithi203.stdout:2/971: mknod d5/c14f 0 2024-04-05T03:02:53.563 INFO:tasks.workunit.client.0.smithi153.stdout:2/615: creat d9/da/d19/d2e/dc7/fcc x:0 0 0 2024-04-05T03:02:53.564 INFO:tasks.workunit.client.0.smithi153.stdout:6/610: fdatasync d4/dc/d7a/d9c/fc3 0 2024-04-05T03:02:53.565 INFO:tasks.workunit.client.0.smithi153.stdout:2/616: mknod d9/da/d94/ccd 0 2024-04-05T03:02:53.565 INFO:tasks.workunit.client.1.smithi203.stdout:2/972: dread d5/d10/d2f/f88 [0,4194304] 0 2024-04-05T03:02:53.571 INFO:tasks.workunit.client.0.smithi153.stdout:6/611: creat d4/dc/df/d16/d33/fcd x:0 0 0 2024-04-05T03:02:53.573 INFO:tasks.workunit.client.1.smithi203.stdout:6/858: write d0/da/d18/f9f [3725169,9531] 0 2024-04-05T03:02:53.574 INFO:tasks.workunit.client.1.smithi203.stdout:2/973: write d5/d10/d44/df6/f113 [1033191,58734] 0 2024-04-05T03:02:53.575 INFO:tasks.workunit.client.1.smithi203.stdout:6/859: readlink d0/da/d54/d4c/da1/db7/l76 0 2024-04-05T03:02:53.577 INFO:tasks.workunit.client.0.smithi153.stdout:6/612: getdents d4/dc/df/d16 0 2024-04-05T03:02:53.578 INFO:tasks.workunit.client.1.smithi203.stdout:1/888: dwrite d7/d24/d33/d48/d4b/d6b/db9/fd2 [0,4194304] 0 2024-04-05T03:02:53.579 INFO:tasks.workunit.client.1.smithi203.stdout:4/683: dread da/f4a [0,4194304] 0 2024-04-05T03:02:53.581 INFO:tasks.workunit.client.0.smithi153.stdout:2/617: truncate d9/d68/db9/fbc 1602455 0 2024-04-05T03:02:53.583 INFO:tasks.workunit.client.0.smithi153.stdout:6/613: dread d4/dc/df/d16/d33/f83 [0,4194304] 0 2024-04-05T03:02:53.586 INFO:tasks.workunit.client.1.smithi203.stdout:6/860: dread d0/da/d54/f3f [0,4194304] 0 2024-04-05T03:02:53.586 INFO:tasks.workunit.client.0.smithi153.stdout:6/614: write d4/dc/f97 [624798,102762] 0 2024-04-05T03:02:53.586 INFO:tasks.workunit.client.0.smithi153.stdout:2/618: dread d9/da/fb [0,4194304] 0 2024-04-05T03:02:53.587 INFO:tasks.workunit.client.1.smithi203.stdout:2/974: dwrite d5/d10/d1a/f34 [0,4194304] 0 2024-04-05T03:02:53.587 INFO:tasks.workunit.client.1.smithi203.stdout:6/861: unlink d0/da/d18/lcf 0 2024-04-05T03:02:53.588 INFO:tasks.workunit.client.0.smithi153.stdout:2/619: creat d9/d68/fce x:0 0 0 2024-04-05T03:02:53.589 INFO:tasks.workunit.client.1.smithi203.stdout:4/684: link da/f26 da/d29/d2e/d8b/fdb 0 2024-04-05T03:02:53.589 INFO:tasks.workunit.client.1.smithi203.stdout:1/889: write d7/d14/d83/fa4 [1243083,39896] 0 2024-04-05T03:02:53.590 INFO:tasks.workunit.client.1.smithi203.stdout:2/975: mknod d5/d10/d35/d2d/c150 0 2024-04-05T03:02:53.593 INFO:tasks.workunit.client.0.smithi153.stdout:2/620: mknod d9/da/d19/d2e/d7e/d53/da1/ccf 0 2024-04-05T03:02:53.594 INFO:tasks.workunit.client.1.smithi203.stdout:6/862: creat d0/da/d18/d92/dfe/f100 x:0 0 0 2024-04-05T03:02:53.594 INFO:tasks.workunit.client.0.smithi153.stdout:6/615: dwrite d4/dc/df/d16/d33/fcd [0,4194304] 0 2024-04-05T03:02:53.595 INFO:tasks.workunit.client.1.smithi203.stdout:4/685: truncate da/d29/f47 3664372 0 2024-04-05T03:02:53.599 INFO:tasks.workunit.client.1.smithi203.stdout:6/863: creat d0/da/d54/f101 x:0 0 0 2024-04-05T03:02:53.600 INFO:tasks.workunit.client.0.smithi153.stdout:2/621: creat d9/da/fd0 x:0 0 0 2024-04-05T03:02:53.601 INFO:tasks.workunit.client.1.smithi203.stdout:4/686: mkdir da/d29/d2e/d57/d96/ddc 0 2024-04-05T03:02:53.601 INFO:tasks.workunit.client.0.smithi153.stdout:2/622: write d9/da/d19/f2f [4367081,67620] 0 2024-04-05T03:02:53.602 INFO:tasks.workunit.client.1.smithi203.stdout:4/687: dread - da/d29/fb2 zero size 2024-04-05T03:02:53.603 INFO:tasks.workunit.client.1.smithi203.stdout:6/864: truncate d0/da/d18/d38/d2a/d44/faa 429255 0 2024-04-05T03:02:53.610 INFO:tasks.workunit.client.1.smithi203.stdout:6/865: dwrite d0/da/d54/d4c/da1/db7/fcb [0,4194304] 0 2024-04-05T03:02:53.613 INFO:tasks.workunit.client.0.smithi153.stdout:9/433: sync 2024-04-05T03:02:53.613 INFO:tasks.workunit.client.0.smithi153.stdout:0/499: sync 2024-04-05T03:02:53.613 INFO:tasks.workunit.client.0.smithi153.stdout:5/522: sync 2024-04-05T03:02:53.614 INFO:tasks.workunit.client.0.smithi153.stdout:9/434: fsync d1/d10/d16/f7f 0 2024-04-05T03:02:53.615 INFO:tasks.workunit.client.1.smithi203.stdout:6/866: truncate d0/da/d54/d4c/d7f/dcc/ffb 242135 0 2024-04-05T03:02:53.617 INFO:tasks.workunit.client.0.smithi153.stdout:5/523: mknod d0/d25/d4d/d62/d37/d38/cb8 0 2024-04-05T03:02:53.619 INFO:tasks.workunit.client.1.smithi203.stdout:6/867: creat d0/db4/f102 x:0 0 0 2024-04-05T03:02:53.621 INFO:tasks.workunit.client.1.smithi203.stdout:2/976: dread d5/d10/db8/dab/de1/f9a [0,4194304] 0 2024-04-05T03:02:53.621 INFO:tasks.workunit.client.0.smithi153.stdout:9/435: creat d1/d3a/d91/f9f x:0 0 0 2024-04-05T03:02:53.623 INFO:tasks.workunit.client.0.smithi153.stdout:5/524: mknod d0/d25/d4d/d62/cb9 0 2024-04-05T03:02:53.624 INFO:tasks.workunit.client.0.smithi153.stdout:6/616: dread d4/dc/f3c [0,4194304] 0 2024-04-05T03:02:53.626 INFO:tasks.workunit.client.0.smithi153.stdout:9/436: dread d1/d10/d16/d22/d2c/d68/d81/f7a [0,4194304] 0 2024-04-05T03:02:53.627 INFO:tasks.workunit.client.1.smithi203.stdout:1/890: dwrite d7/fe [0,4194304] 0 2024-04-05T03:02:53.628 INFO:tasks.workunit.client.0.smithi153.stdout:2/623: write d9/da/d19/d2e/d7e/d7a/f84 [1756767,115447] 0 2024-04-05T03:02:53.629 INFO:tasks.workunit.client.1.smithi203.stdout:1/891: chown d7/d14/d83/fd4 50551338 1 2024-04-05T03:02:53.629 INFO:tasks.workunit.client.0.smithi153.stdout:2/624: read d9/da/f15 [110233,49520] 0 2024-04-05T03:02:53.630 INFO:tasks.workunit.client.0.smithi153.stdout:6/617: dread d4/dc/df/d16/d33/fcd [0,4194304] 0 2024-04-05T03:02:53.631 INFO:tasks.workunit.client.1.smithi203.stdout:4/688: truncate da/d29/f9e 2613224 0 2024-04-05T03:02:53.632 INFO:tasks.workunit.client.1.smithi203.stdout:6/868: link d0/da/d18/d38/c22 d0/da/d54/d4c/da1/db7/d95/c103 0 2024-04-05T03:02:53.632 INFO:tasks.workunit.client.0.smithi153.stdout:6/618: read d4/dc/df/d16/d5f/da2/fa4 [1900023,61884] 0 2024-04-05T03:02:53.633 INFO:tasks.workunit.client.0.smithi153.stdout:5/525: rename d0/d25/c6b to d0/d4e/db6/cba 0 2024-04-05T03:02:53.634 INFO:tasks.workunit.client.0.smithi153.stdout:5/526: chown d0/d9/f1e 8730287 1 2024-04-05T03:02:53.635 INFO:tasks.workunit.client.0.smithi153.stdout:9/437: write d1/d10/d36/f62 [1998543,66872] 0 2024-04-05T03:02:53.636 INFO:tasks.workunit.client.0.smithi153.stdout:6/619: chown d4/d38/d4c/fa5 7219335 1 2024-04-05T03:02:53.636 INFO:tasks.workunit.client.1.smithi203.stdout:6/869: truncate d0/da/d54/d53/fbd 5112911 0 2024-04-05T03:02:53.638 INFO:tasks.workunit.client.1.smithi203.stdout:4/689: fdatasync da/d3c/d4c/f79 0 2024-04-05T03:02:53.639 INFO:tasks.workunit.client.1.smithi203.stdout:2/977: write d5/d10/d35/d2d/d7c/d72/d8f/fa5 [1624648,92503] 0 2024-04-05T03:02:53.640 INFO:tasks.workunit.client.1.smithi203.stdout:1/892: dread d7/d24/fc5 [0,4194304] 0 2024-04-05T03:02:53.640 INFO:tasks.workunit.client.0.smithi153.stdout:2/625: mknod d9/da/d19/d2e/d7e/d7a/d6d/cd1 0 2024-04-05T03:02:53.640 INFO:tasks.workunit.client.0.smithi153.stdout:0/500: dwrite d9/d14/f3f [0,4194304] 0 2024-04-05T03:02:53.645 INFO:tasks.workunit.client.1.smithi203.stdout:1/893: chown d7/d24/d33/c8f 61949852 1 2024-04-05T03:02:53.647 INFO:tasks.workunit.client.0.smithi153.stdout:5/527: truncate d0/d9/f1f 916484 0 2024-04-05T03:02:53.648 INFO:tasks.workunit.client.1.smithi203.stdout:4/690: rename da/d29/d2e/f45 to da/d29/d2e/d53/d78/fdd 0 2024-04-05T03:02:53.648 INFO:tasks.workunit.client.1.smithi203.stdout:2/978: dwrite d5/d10/f1d [0,4194304] 0 2024-04-05T03:02:53.648 INFO:tasks.workunit.client.0.smithi153.stdout:6/620: stat d4/dc/df/d16/d36/d52/c5d 0 2024-04-05T03:02:53.649 INFO:tasks.workunit.client.0.smithi153.stdout:5/528: dread - d0/d4e/f9f zero size 2024-04-05T03:02:53.650 INFO:tasks.workunit.client.0.smithi153.stdout:2/626: symlink d9/da/d94/ld2 0 2024-04-05T03:02:53.650 INFO:tasks.workunit.client.0.smithi153.stdout:2/627: chown d9/da/d36/f63 3 1 2024-04-05T03:02:53.651 INFO:tasks.workunit.client.0.smithi153.stdout:0/501: unlink d9/df/d1d/f24 0 2024-04-05T03:02:53.652 INFO:tasks.workunit.client.1.smithi203.stdout:1/894: truncate d7/d14/d2b/d58/d86/d91/f97 262916 0 2024-04-05T03:02:53.652 INFO:tasks.workunit.client.0.smithi153.stdout:0/502: stat d9/db/d42/d7e/c83 0 2024-04-05T03:02:53.657 INFO:tasks.workunit.client.1.smithi203.stdout:4/691: mknod da/d29/d2e/dc9/cde 0 2024-04-05T03:02:53.658 INFO:tasks.workunit.client.1.smithi203.stdout:1/895: mknod d7/d24/d33/d48/d4b/dea/df2/c125 0 2024-04-05T03:02:53.659 INFO:tasks.workunit.client.0.smithi153.stdout:5/529: mkdir d0/d25/d4d/d62/dbb 0 2024-04-05T03:02:53.659 INFO:tasks.workunit.client.1.smithi203.stdout:4/692: fsync da/d29/d2e/db0/dcb/d80/f9c 0 2024-04-05T03:02:53.660 INFO:tasks.workunit.client.0.smithi153.stdout:2/628: dread d9/da/d19/d2e/d7e/d53/f83 [0,4194304] 0 2024-04-05T03:02:53.662 INFO:tasks.workunit.client.1.smithi203.stdout:1/896: fsync d7/d24/d33/d48/d4b/dea/df2/ff5 0 2024-04-05T03:02:53.665 INFO:tasks.workunit.client.0.smithi153.stdout:5/530: symlink d0/d2c/d3f/lbc 0 2024-04-05T03:02:53.665 INFO:tasks.workunit.client.1.smithi203.stdout:1/897: mkdir d7/d126 0 2024-04-05T03:02:53.666 INFO:tasks.workunit.client.0.smithi153.stdout:5/531: chown d0/d25/d4d/d62/d37/d38/cb8 3531682 1 2024-04-05T03:02:53.667 INFO:tasks.workunit.client.0.smithi153.stdout:2/629: rename d9/da/d19/d2e/d4f/c85 to d9/da/d94/dca/cd3 0 2024-04-05T03:02:53.668 INFO:tasks.workunit.client.1.smithi203.stdout:4/693: rename da/d29/f42 to da/fdf 0 2024-04-05T03:02:53.669 INFO:tasks.workunit.client.0.smithi153.stdout:2/630: write d9/da/d19/d2e/d7e/d7a/d6d/f6e [680487,43509] 0 2024-04-05T03:02:53.671 INFO:tasks.workunit.client.1.smithi203.stdout:2/979: dwrite d5/d10/d35/d2d/d7c/d29/d67/d9e/fd6 [0,4194304] 0 2024-04-05T03:02:53.671 INFO:tasks.workunit.client.0.smithi153.stdout:9/438: truncate d1/d9/d1b/f1c 7695303 0 2024-04-05T03:02:53.672 INFO:tasks.workunit.client.0.smithi153.stdout:9/439: chown d1/d10/f64 60202 1 2024-04-05T03:02:53.673 INFO:tasks.workunit.client.0.smithi153.stdout:6/621: write d4/dc/df/d16/d36/d4d/f91 [260606,101930] 0 2024-04-05T03:02:53.674 INFO:tasks.workunit.client.1.smithi203.stdout:1/898: rename d7/d24/d33/d48/d4b/d51/la7 to d7/d14/d2b/d58/d86/d91/da6/df9/d106/d122/l127 0 2024-04-05T03:02:53.675 INFO:tasks.workunit.client.1.smithi203.stdout:2/980: mknod d5/d10/db8/dab/de1/c151 0 2024-04-05T03:02:53.676 INFO:tasks.workunit.client.0.smithi153.stdout:5/532: symlink d0/d25/d4d/d62/dbb/lbd 0 2024-04-05T03:02:53.677 INFO:tasks.workunit.client.1.smithi203.stdout:2/981: chown d5/lbc 1391 1 2024-04-05T03:02:53.680 INFO:tasks.workunit.client.0.smithi153.stdout:5/533: dread d0/d9/f21 [0,4194304] 0 2024-04-05T03:02:53.680 INFO:tasks.workunit.client.0.smithi153.stdout:0/503: dread d9/f30 [0,4194304] 0 2024-04-05T03:02:53.681 INFO:tasks.workunit.client.0.smithi153.stdout:2/631: rename d9/da/d37/d8a/da8/cb3 to d9/d68/db9/cd4 0 2024-04-05T03:02:53.682 INFO:tasks.workunit.client.1.smithi203.stdout:1/899: rename d7/d24/la9 to d7/d14/d2b/d58/d86/d91/da6/dee/l128 0 2024-04-05T03:02:53.683 INFO:tasks.workunit.client.0.smithi153.stdout:3/567: sync 2024-04-05T03:02:53.683 INFO:tasks.workunit.client.0.smithi153.stdout:1/487: sync 2024-04-05T03:02:53.684 INFO:tasks.workunit.client.1.smithi203.stdout:2/982: mkdir d5/d10/d35/d152 0 2024-04-05T03:02:53.685 INFO:tasks.workunit.client.1.smithi203.stdout:1/900: fdatasync d7/d14/d2b/d58/d7a/f8a 0 2024-04-05T03:02:53.686 INFO:tasks.workunit.client.0.smithi153.stdout:9/440: dread d1/d10/d16/f40 [0,4194304] 0 2024-04-05T03:02:53.687 INFO:tasks.workunit.client.1.smithi203.stdout:1/901: chown d7/d24/d33/d48/d4b/d6b/db9/lfa 1857 1 2024-04-05T03:02:53.688 INFO:tasks.workunit.client.1.smithi203.stdout:2/983: rename d5/d10/d7f/cce to d5/d63/d86/c153 0 2024-04-05T03:02:53.690 INFO:tasks.workunit.client.0.smithi153.stdout:3/568: write d1d/d32/d93/d3c/d3f/d90/db6/fb7 [605078,44127] 0 2024-04-05T03:02:53.690 INFO:tasks.workunit.client.1.smithi203.stdout:2/984: write d5/d10/d35/d2d/d7c/d72/d8f/fa5 [1648652,89276] 0 2024-04-05T03:02:53.690 INFO:tasks.workunit.client.1.smithi203.stdout:1/902: rmdir d7/d24 39 2024-04-05T03:02:53.691 INFO:tasks.workunit.client.0.smithi153.stdout:1/488: mknod d5/d8/d20/d4b/d18/d16/d45/c98 0 2024-04-05T03:02:53.692 INFO:tasks.workunit.client.0.smithi153.stdout:2/632: creat d9/da/fd5 x:0 0 0 2024-04-05T03:02:53.693 INFO:tasks.workunit.client.0.smithi153.stdout:3/569: truncate d1d/d32/d93/d3c/d3f/d7c/f8e 1003530 0 2024-04-05T03:02:53.694 INFO:tasks.workunit.client.0.smithi153.stdout:9/441: rename d1/d10/d16/f40 to d1/d10/d36/d71/fa0 0 2024-04-05T03:02:53.695 INFO:tasks.workunit.client.0.smithi153.stdout:2/633: write d9/da/d19/d2e/d4f/fa6 [952163,92392] 0 2024-04-05T03:02:53.695 INFO:tasks.workunit.client.0.smithi153.stdout:9/442: write d1/d6c/f82 [2023348,16617] 0 2024-04-05T03:02:53.696 INFO:tasks.workunit.client.1.smithi203.stdout:2/985: mkdir d5/d10/d35/d154 0 2024-04-05T03:02:53.697 INFO:tasks.workunit.client.0.smithi153.stdout:1/489: creat d5/d8/d20/d4b/d18/d16/d48/f99 x:0 0 0 2024-04-05T03:02:53.699 INFO:tasks.workunit.client.0.smithi153.stdout:6/622: dread d4/dc/f97 [0,4194304] 0 2024-04-05T03:02:53.702 INFO:tasks.workunit.client.0.smithi153.stdout:6/623: fdatasync d4/d38/d4c/fa0 0 2024-04-05T03:02:53.707 INFO:tasks.workunit.client.0.smithi153.stdout:3/570: dwrite d1d/d32/d93/d57/d6b/fa5 [0,4194304] 0 2024-04-05T03:02:53.709 INFO:tasks.workunit.client.0.smithi153.stdout:2/634: dwrite d9/da/d19/d2e/d7e/d53/f83 [0,4194304] 0 2024-04-05T03:02:53.709 INFO:tasks.workunit.client.0.smithi153.stdout:9/443: link d1/d9/f9a d1/d10/d16/d22/d2c/d68/fa1 0 2024-04-05T03:02:53.709 INFO:tasks.workunit.client.0.smithi153.stdout:5/534: write d0/d25/d4d/d62/d5f/d86/f74 [56537,36360] 0 2024-04-05T03:02:53.711 INFO:tasks.workunit.client.0.smithi153.stdout:0/504: write d9/db/f4b [454628,120501] 0 2024-04-05T03:02:53.713 INFO:tasks.workunit.client.1.smithi203.stdout:2/986: dwrite d5/d10/d35/d36/ffa [0,4194304] 0 2024-04-05T03:02:53.715 INFO:tasks.workunit.client.1.smithi203.stdout:1/903: dread d7/d14/f16 [0,4194304] 0 2024-04-05T03:02:53.716 INFO:tasks.workunit.client.1.smithi203.stdout:2/987: fdatasync d5/d10/d35/d36/db6/d87/f89 0 2024-04-05T03:02:53.717 INFO:tasks.workunit.client.0.smithi153.stdout:2/635: dwrite d9/da/d19/d2e/d7e/f5e [0,4194304] 0 2024-04-05T03:02:53.727 INFO:tasks.workunit.client.1.smithi203.stdout:2/988: unlink d5/d10/d1a/d57/fac 0 2024-04-05T03:02:53.727 INFO:tasks.workunit.client.0.smithi153.stdout:5/535: creat d0/d2c/fbe x:0 0 0 2024-04-05T03:02:53.728 INFO:tasks.workunit.client.0.smithi153.stdout:1/490: getdents d5/d8/d20/d4b/d18/d16/d48 0 2024-04-05T03:02:53.729 INFO:tasks.workunit.client.1.smithi203.stdout:2/989: chown d5/d10/d35/d36/d41/l48 9560 1 2024-04-05T03:02:53.729 INFO:tasks.workunit.client.1.smithi203.stdout:1/904: dwrite d7/d14/f52 [0,4194304] 0 2024-04-05T03:02:53.734 INFO:tasks.workunit.client.1.smithi203.stdout:1/905: chown d7/d24 1096175 1 2024-04-05T03:02:53.739 INFO:tasks.workunit.client.0.smithi153.stdout:5/536: creat d0/d25/d4d/d62/d5f/fbf x:0 0 0 2024-04-05T03:02:53.740 INFO:tasks.workunit.client.1.smithi203.stdout:1/906: creat d7/d24/d33/d48/d4b/d6b/f129 x:0 0 0 2024-04-05T03:02:53.740 INFO:tasks.workunit.client.0.smithi153.stdout:5/537: dread - d0/d4e/f9f zero size 2024-04-05T03:02:53.741 INFO:tasks.workunit.client.1.smithi203.stdout:1/907: dread - d7/d14/d2b/d58/d86/d91/da6/dee/ffe zero size 2024-04-05T03:02:53.742 INFO:tasks.workunit.client.0.smithi153.stdout:6/624: dwrite d4/dc/df/d16/d17/f25 [0,4194304] 0 2024-04-05T03:02:53.742 INFO:tasks.workunit.client.0.smithi153.stdout:3/571: link d1d/d32/d93/d3c/d3f/d9c/la2 d1d/d32/d93/d3c/d3f/d90/d92/lca 0 2024-04-05T03:02:53.742 INFO:tasks.workunit.client.0.smithi153.stdout:8/465: sync 2024-04-05T03:02:53.742 INFO:tasks.workunit.client.0.smithi153.stdout:7/675: sync 2024-04-05T03:02:53.742 INFO:tasks.workunit.client.0.smithi153.stdout:4/651: sync 2024-04-05T03:02:53.746 INFO:tasks.workunit.client.0.smithi153.stdout:2/636: link d9/c9f d9/d68/d98/da3/cd6 0 2024-04-05T03:02:53.747 INFO:tasks.workunit.client.0.smithi153.stdout:1/491: rmdir d5/d8/d8b 0 2024-04-05T03:02:53.747 INFO:tasks.workunit.client.0.smithi153.stdout:7/676: truncate d2/d9b/fc5 858668 0 2024-04-05T03:02:53.748 INFO:tasks.workunit.client.0.smithi153.stdout:5/538: symlink d0/d25/d4d/d62/d5f/d86/da4/lc0 0 2024-04-05T03:02:53.749 INFO:tasks.workunit.client.1.smithi203.stdout:1/908: dwrite d7/d24/d33/d48/d4b/dea/def/f11d [0,4194304] 0 2024-04-05T03:02:53.750 INFO:tasks.workunit.client.0.smithi153.stdout:3/572: dread d1d/d27/f54 [0,4194304] 0 2024-04-05T03:02:53.751 INFO:tasks.workunit.client.0.smithi153.stdout:5/539: dread - d0/d4e/f9e zero size 2024-04-05T03:02:53.755 INFO:tasks.workunit.client.0.smithi153.stdout:4/652: dwrite f8 [0,4194304] 0 2024-04-05T03:02:53.755 INFO:tasks.workunit.client.0.smithi153.stdout:6/625: link d4/dc/df/d16/d5f/da2/fa4 d4/dc/fce 0 2024-04-05T03:02:53.757 INFO:tasks.workunit.client.0.smithi153.stdout:6/626: chown d4/dc/d7a/lc9 772331 1 2024-04-05T03:02:53.758 INFO:tasks.workunit.client.0.smithi153.stdout:6/627: stat d4/dc/df/d16/d5f/f7c 0 2024-04-05T03:02:53.758 INFO:tasks.workunit.client.0.smithi153.stdout:3/573: creat d1d/d32/d93/d57/d6b/d9f/fcb x:0 0 0 2024-04-05T03:02:53.760 INFO:tasks.workunit.client.0.smithi153.stdout:1/492: creat d5/d8/d20/d4b/d18/d16/d45/d93/f9a x:0 0 0 2024-04-05T03:02:53.768 INFO:tasks.workunit.client.0.smithi153.stdout:7/677: link d2/da/d3a/d41/da8/cb4 d2/da/d3a/d41/d7a/d54/ce0 0 2024-04-05T03:02:53.770 INFO:tasks.workunit.client.0.smithi153.stdout:4/653: creat da/db/d20/d35/d38/fe3 x:0 0 0 2024-04-05T03:02:53.774 INFO:tasks.workunit.client.0.smithi153.stdout:5/540: rename d0/d25/d4d/d62/l5c to d0/d25/lc1 0 2024-04-05T03:02:53.774 INFO:tasks.workunit.client.0.smithi153.stdout:9/444: truncate d1/d10/d36/f48 3175579 0 2024-04-05T03:02:53.775 INFO:tasks.workunit.client.0.smithi153.stdout:0/505: truncate d9/db/f4b 767148 0 2024-04-05T03:02:53.776 INFO:tasks.workunit.client.0.smithi153.stdout:4/654: rmdir da/de/d56/d6a/d2d/ddb 39 2024-04-05T03:02:53.777 INFO:tasks.workunit.client.0.smithi153.stdout:3/574: link d1d/d32/d93/d3c/d3f/d9c/la2 d1d/d32/d93/d3c/d3f/d90/d92/lcc 0 2024-04-05T03:02:53.779 INFO:tasks.workunit.client.0.smithi153.stdout:6/628: rename d4/dc/df/f39 to d4/dc/df/d16/fcf 0 2024-04-05T03:02:53.783 INFO:tasks.workunit.client.0.smithi153.stdout:8/466: dread d7/d85/f4a [0,4194304] 0 2024-04-05T03:02:53.784 INFO:tasks.workunit.client.0.smithi153.stdout:1/493: rename d5/d8/d20/d4b/c17 to d5/d8/d20/d4b/d18/d16/d45/c9b 0 2024-04-05T03:02:53.786 INFO:tasks.workunit.client.0.smithi153.stdout:1/494: stat d5/d8/d20/d4b/d51/l53 0 2024-04-05T03:02:53.787 INFO:tasks.workunit.client.1.smithi203.stdout:2/990: truncate d5/d10/db8/dab/fa8 582614 0 2024-04-05T03:02:53.788 INFO:tasks.workunit.client.0.smithi153.stdout:3/575: dwrite d1d/d32/f45 [0,4194304] 0 2024-04-05T03:02:53.788 INFO:tasks.workunit.client.1.smithi203.stdout:2/991: chown d5/d10/d35/d2d/d7c/c30 1071509 1 2024-04-05T03:02:53.789 INFO:tasks.workunit.client.0.smithi153.stdout:4/655: link da/db/d20/d45/fa6 da/db/fe4 0 2024-04-05T03:02:53.790 INFO:tasks.workunit.client.0.smithi153.stdout:8/467: fsync d7/d13/f18 0 2024-04-05T03:02:53.793 INFO:tasks.workunit.client.0.smithi153.stdout:5/541: rename d0/d9/f18 to d0/d9/d22/fc2 0 2024-04-05T03:02:53.795 INFO:tasks.workunit.client.0.smithi153.stdout:2/637: truncate d9/da/d19/d2e/d7e/d53/f83 1966131 0 2024-04-05T03:02:53.795 INFO:tasks.workunit.client.0.smithi153.stdout:1/495: dread d5/dd/f89 [0,4194304] 0 2024-04-05T03:02:53.796 INFO:tasks.workunit.client.0.smithi153.stdout:6/629: creat d4/dc/df/d16/d5f/d73/fd0 x:0 0 0 2024-04-05T03:02:53.797 INFO:tasks.workunit.client.1.smithi203.stdout:2/992: dwrite d5/d10/d44/df6/f13f [0,4194304] 0 2024-04-05T03:02:53.797 INFO:tasks.workunit.client.1.smithi203.stdout:1/909: dwrite d7/f77 [0,4194304] 0 2024-04-05T03:02:53.801 INFO:tasks.workunit.client.1.smithi203.stdout:2/993: write d5/d10/f1d [3860627,82468] 0 2024-04-05T03:02:53.805 INFO:tasks.workunit.client.1.smithi203.stdout:2/994: creat d5/d10/d35/d2d/d7c/d72/d8f/d147/f155 x:0 0 0 2024-04-05T03:02:53.807 INFO:tasks.workunit.client.1.smithi203.stdout:1/910: dread d7/d24/d33/d48/d4b/d51/fb2 [4194304,4194304] 0 2024-04-05T03:02:53.811 INFO:tasks.workunit.client.0.smithi153.stdout:8/468: mkdir d7/d13/d19/d1b/d4f/d96 0 2024-04-05T03:02:53.812 INFO:tasks.workunit.client.0.smithi153.stdout:7/678: write d2/da/d3a/d41/d7a/f38 [803520,76001] 0 2024-04-05T03:02:53.813 INFO:tasks.workunit.client.1.smithi203.stdout:2/995: rename d5/d63/d86/fb0 to d5/d10/db8/dbe/f156 0 2024-04-05T03:02:53.814 INFO:tasks.workunit.client.0.smithi153.stdout:0/506: rename d9/db/l92 to d9/df/d1d/d61/d8a/l94 0 2024-04-05T03:02:53.815 INFO:tasks.workunit.client.1.smithi203.stdout:2/996: chown d5/d10/d35/d36/db6/cd2 116 1 2024-04-05T03:02:53.815 INFO:tasks.workunit.client.0.smithi153.stdout:0/507: readlink d9/df/d1d/d61/l8d 0 2024-04-05T03:02:53.816 INFO:tasks.workunit.client.0.smithi153.stdout:4/656: dread da/de/d56/d6a/f7f [0,4194304] 0 2024-04-05T03:02:53.816 INFO:tasks.workunit.client.0.smithi153.stdout:2/638: symlink d9/da/d19/d2e/d4f/ld7 0 2024-04-05T03:02:53.817 INFO:tasks.workunit.client.1.smithi203.stdout:2/997: read d5/d10/db8/f123 [940248,122563] 0 2024-04-05T03:02:53.819 INFO:tasks.workunit.client.0.smithi153.stdout:6/630: rename d4/dc/d7a/da8/fc6 to d4/dc/d7a/da8/fd1 0 2024-04-05T03:02:53.819 INFO:tasks.workunit.client.0.smithi153.stdout:9/445: getdents d1/d10/d36/d71 0 2024-04-05T03:02:53.821 INFO:tasks.workunit.client.0.smithi153.stdout:5/542: mknod d0/cc3 0 2024-04-05T03:02:53.821 INFO:tasks.workunit.client.1.smithi203.stdout:2/998: unlink d5/d10/db8/dbe/l13a 0 2024-04-05T03:02:53.822 INFO:tasks.workunit.client.1.smithi203.stdout:1/911: link d7/d14/l57 d7/d14/d83/deb/l12a 0 2024-04-05T03:02:53.823 INFO:tasks.workunit.client.0.smithi153.stdout:4/657: creat da/db/d20/d3c/d81/db6/fe5 x:0 0 0 2024-04-05T03:02:53.824 INFO:tasks.workunit.client.1.smithi203.stdout:1/912: symlink d7/d14/d83/l12b 0 2024-04-05T03:02:53.825 INFO:tasks.workunit.client.0.smithi153.stdout:6/631: mkdir d4/dc/d7a/dd2 0 2024-04-05T03:02:53.825 INFO:tasks.workunit.client.0.smithi153.stdout:5/543: dread d0/d25/d4d/d62/d5f/d86/da4/fa1 [0,4194304] 0 2024-04-05T03:02:53.827 INFO:tasks.workunit.client.0.smithi153.stdout:7/679: mkdir d2/da/d3a/d41/ddd/de1 0 2024-04-05T03:02:53.827 INFO:tasks.workunit.client.1.smithi203.stdout:1/913: creat d7/d24/d33/d48/d4b/dea/f12c x:0 0 0 2024-04-05T03:02:53.830 INFO:tasks.workunit.client.0.smithi153.stdout:4/658: mkdir da/db/de6 0 2024-04-05T03:02:53.830 INFO:tasks.workunit.client.0.smithi153.stdout:3/576: link l16 d1d/d32/d93/d3c/d94/lcd 0 2024-04-05T03:02:53.831 INFO:tasks.workunit.client.1.smithi203.stdout:1/914: creat d7/d126/f12d x:0 0 0 2024-04-05T03:02:53.831 INFO:tasks.workunit.client.0.smithi153.stdout:4/659: chown da/de/d56/f61 21 1 2024-04-05T03:02:53.833 INFO:tasks.workunit.client.0.smithi153.stdout:8/469: rename d7/d2a/f6e to d7/d13/d19/d1b/f97 0 2024-04-05T03:02:53.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/496: link d5/dd/c23 d5/d8/d20/d4b/d18/c9c 0 2024-04-05T03:02:53.837 INFO:tasks.workunit.client.0.smithi153.stdout:7/680: creat d2/da/d47/d5d/d7c/d8d/d95/fe2 x:0 0 0 2024-04-05T03:02:53.838 INFO:tasks.workunit.client.0.smithi153.stdout:6/632: getdents d4/dc/df/d16/d5f/da6 0 2024-04-05T03:02:53.838 INFO:tasks.workunit.client.0.smithi153.stdout:9/446: dread d1/d9/d1b/f26 [0,4194304] 0 2024-04-05T03:02:53.840 INFO:tasks.workunit.client.0.smithi153.stdout:3/577: chown d1d/d32/d93/d57/d6b/f67 1871287925 1 2024-04-05T03:02:53.842 INFO:tasks.workunit.client.0.smithi153.stdout:3/578: dread - d1d/d32/d93/d57/d6b/d9f/fcb zero size 2024-04-05T03:02:53.844 INFO:tasks.workunit.client.0.smithi153.stdout:2/639: rename d9/da/d19/d2e/d48 to d9/d68/db9/dd8 0 2024-04-05T03:02:53.845 INFO:tasks.workunit.client.1.smithi203.stdout:2/999: dwrite d5/d63/f9c [0,4194304] 0 2024-04-05T03:02:53.845 INFO:tasks.workunit.client.0.smithi153.stdout:7/681: dwrite d2/da/d3a/f71 [4194304,4194304] 0 2024-04-05T03:02:53.848 INFO:tasks.workunit.client.0.smithi153.stdout:9/447: symlink d1/d6c/la2 0 2024-04-05T03:02:53.852 INFO:tasks.workunit.client.0.smithi153.stdout:6/633: rename d4/dc/df/d16/d5f/d73/fd0 to d4/d38/d4c/fd3 0 2024-04-05T03:02:53.855 INFO:tasks.workunit.client.0.smithi153.stdout:9/448: creat d1/d3a/d91/d96/fa3 x:0 0 0 2024-04-05T03:02:53.856 INFO:tasks.workunit.client.0.smithi153.stdout:1/497: creat d5/d8/f9d x:0 0 0 2024-04-05T03:02:53.857 INFO:tasks.workunit.client.0.smithi153.stdout:9/449: dread - d1/d3a/d91/f9f zero size 2024-04-05T03:02:53.861 INFO:tasks.workunit.client.0.smithi153.stdout:3/579: rename c10 to d1d/d32/d93/d3c/d3f/d7c/cce 0 2024-04-05T03:02:53.862 INFO:tasks.workunit.client.0.smithi153.stdout:7/682: dwrite d2/da/d3a/d41/f6c [0,4194304] 0 2024-04-05T03:02:53.862 INFO:tasks.workunit.client.0.smithi153.stdout:6/634: symlink d4/ld4 0 2024-04-05T03:02:53.862 INFO:tasks.workunit.client.1.smithi203.stdout:9/942: sync 2024-04-05T03:02:53.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/722: sync 2024-04-05T03:02:53.862 INFO:tasks.workunit.client.1.smithi203.stdout:9/943: readlink d13/d1b/d6b/l109 0 2024-04-05T03:02:53.863 INFO:tasks.workunit.client.1.smithi203.stdout:3/723: chown d0/d21/d83/l95 23 1 2024-04-05T03:02:53.865 INFO:tasks.workunit.client.0.smithi153.stdout:3/580: write d1d/d32/d93/d3c/d3f/d7c/f8e [101220,8703] 0 2024-04-05T03:02:53.867 INFO:tasks.workunit.client.0.smithi153.stdout:2/640: rename d9/da/d37/d8a/da8/c6f to d9/d68/d98/da3/cd9 0 2024-04-05T03:02:53.868 INFO:tasks.workunit.client.1.smithi203.stdout:3/724: symlink d0/dc/d38/d5d/d66/d80/lcf 0 2024-04-05T03:02:53.868 INFO:tasks.workunit.client.1.smithi203.stdout:3/725: chown d0/dc/d1a/d1d/c1e 14486674 1 2024-04-05T03:02:53.868 INFO:tasks.workunit.client.0.smithi153.stdout:0/508: truncate d9/df/d1e/f50 815767 0 2024-04-05T03:02:53.869 INFO:tasks.workunit.client.0.smithi153.stdout:1/498: dwrite d5/f9 [0,4194304] 0 2024-04-05T03:02:53.869 INFO:tasks.workunit.client.0.smithi153.stdout:4/660: dread da/db/d20/d45/f69 [0,4194304] 0 2024-04-05T03:02:53.869 INFO:tasks.workunit.client.1.smithi203.stdout:9/944: write d13/d5c/dd1/d31/f62 [174429,102531] 0 2024-04-05T03:02:53.871 INFO:tasks.workunit.client.1.smithi203.stdout:3/726: read d0/dc/d38/d92/fb1 [4044511,79702] 0 2024-04-05T03:02:53.875 INFO:tasks.workunit.client.0.smithi153.stdout:3/581: write d1d/d32/d93/d3c/d3f/d9c/fc4 [561070,13617] 0 2024-04-05T03:02:53.876 INFO:tasks.workunit.client.1.smithi203.stdout:3/727: creat d0/dc/d1a/d1f/da2/fd0 x:0 0 0 2024-04-05T03:02:53.877 INFO:tasks.workunit.client.0.smithi153.stdout:4/661: dread - da/d2a/fd3 zero size 2024-04-05T03:02:53.878 INFO:tasks.workunit.client.0.smithi153.stdout:4/662: dread - da/de/fb2 zero size 2024-04-05T03:02:53.879 INFO:tasks.workunit.client.1.smithi203.stdout:3/728: read d0/dc/d1a/d1d/d53/f73 [465262,51651] 0 2024-04-05T03:02:53.881 INFO:tasks.workunit.client.1.smithi203.stdout:9/945: dread d13/d1b/d33/d64/dfa/f58 [0,4194304] 0 2024-04-05T03:02:53.883 INFO:tasks.workunit.client.0.smithi153.stdout:0/509: symlink d9/db/d42/d7e/d8b/l95 0 2024-04-05T03:02:53.883 INFO:tasks.workunit.client.1.smithi203.stdout:9/946: symlink d13/d1b/d6b/d7a/l127 0 2024-04-05T03:02:53.887 INFO:tasks.workunit.client.0.smithi153.stdout:0/510: mknod d9/db/d42/d7e/c96 0 2024-04-05T03:02:53.888 INFO:tasks.workunit.client.0.smithi153.stdout:0/511: stat d9/db/fd 0 2024-04-05T03:02:53.888 INFO:tasks.workunit.client.1.smithi203.stdout:9/947: link d13/d5c/dd1/d51/f91 d13/d1b/d6b/d7a/f128 0 2024-04-05T03:02:53.889 INFO:tasks.workunit.client.0.smithi153.stdout:0/512: fsync d9/df/d1e/f80 0 2024-04-05T03:02:53.891 INFO:tasks.workunit.client.0.smithi153.stdout:4/663: link da/db/f79 da/db/d20/d3c/d91/fe7 0 2024-04-05T03:02:53.891 INFO:tasks.workunit.client.1.smithi203.stdout:9/948: read d13/d5c/dd1/d51/da1/dba/fd4 [3537435,53478] 0 2024-04-05T03:02:53.894 INFO:tasks.workunit.client.0.smithi153.stdout:9/450: dread d1/f1f [0,4194304] 0 2024-04-05T03:02:53.896 INFO:tasks.workunit.client.0.smithi153.stdout:9/451: write d1/d10/f67 [188984,35945] 0 2024-04-05T03:02:53.899 INFO:tasks.workunit.client.0.smithi153.stdout:0/513: chown d9/df/d1d/d61/d8a/l94 199532 1 2024-04-05T03:02:53.900 INFO:tasks.workunit.client.0.smithi153.stdout:4/664: rmdir da/de 39 2024-04-05T03:02:53.903 INFO:tasks.workunit.client.0.smithi153.stdout:6/635: link d4/c27 d4/dc/df/d16/d17/cd5 0 2024-04-05T03:02:53.906 INFO:tasks.workunit.client.0.smithi153.stdout:1/499: write d5/d8/d20/d4b/d18/d16/d33/f3e [2216475,112587] 0 2024-04-05T03:02:53.908 INFO:tasks.workunit.client.0.smithi153.stdout:9/452: dwrite d1/d9/d1b/f30 [0,4194304] 0 2024-04-05T03:02:53.909 INFO:tasks.workunit.client.1.smithi203.stdout:9/949: write d13/d1b/d33/d64/fc6 [4651829,91144] 0 2024-04-05T03:02:53.913 INFO:tasks.workunit.client.1.smithi203.stdout:9/950: unlink d13/d1b/d6b/d7a/l127 0 2024-04-05T03:02:53.915 INFO:tasks.workunit.client.0.smithi153.stdout:7/683: dwrite d2/da/d3a/d41/d31/f45 [4194304,4194304] 0 2024-04-05T03:02:53.915 INFO:tasks.workunit.client.0.smithi153.stdout:1/500: unlink d5/dd/d1b/f56 0 2024-04-05T03:02:53.915 INFO:tasks.workunit.client.1.smithi203.stdout:9/951: fdatasync d13/d5c/dd1/d51/da1/dba/fee 0 2024-04-05T03:02:53.916 INFO:tasks.workunit.client.0.smithi153.stdout:6/636: write d4/d38/fbb [1622576,107468] 0 2024-04-05T03:02:53.918 INFO:tasks.workunit.client.0.smithi153.stdout:9/453: creat d1/d10/d16/d22/fa4 x:0 0 0 2024-04-05T03:02:53.918 INFO:tasks.workunit.client.0.smithi153.stdout:2/641: dwrite d9/da/fb [0,4194304] 0 2024-04-05T03:02:53.921 INFO:tasks.workunit.client.0.smithi153.stdout:6/637: rmdir d4/d38 39 2024-04-05T03:02:53.926 INFO:tasks.workunit.client.1.smithi203.stdout:9/952: getdents d13/d1b/d33/d64/dfa/d67 0 2024-04-05T03:02:53.927 INFO:tasks.workunit.client.0.smithi153.stdout:9/454: rename d1/d10/f92 to d1/d10/d36/d57/d97/fa5 0 2024-04-05T03:02:53.927 INFO:tasks.workunit.client.0.smithi153.stdout:7/684: creat d2/da/d47/d5d/d7c/d8d/d95/fe3 x:0 0 0 2024-04-05T03:02:53.928 INFO:tasks.workunit.client.0.smithi153.stdout:1/501: dwrite d5/d8/d20/d4b/fe [4194304,4194304] 0 2024-04-05T03:02:53.934 INFO:tasks.workunit.client.1.smithi203.stdout:9/953: dwrite d13/d5c/d9d/f115 [0,4194304] 0 2024-04-05T03:02:53.935 INFO:tasks.workunit.client.0.smithi153.stdout:6/638: symlink d4/dc/df/d16/d24/d5e/ld6 0 2024-04-05T03:02:53.936 INFO:tasks.workunit.client.0.smithi153.stdout:1/502: symlink d5/d8/d20/d4b/d18/d61/l9e 0 2024-04-05T03:02:53.937 INFO:tasks.workunit.client.0.smithi153.stdout:2/642: creat d9/da/d19/d2e/d7e/d7a/fda x:0 0 0 2024-04-05T03:02:53.937 INFO:tasks.workunit.client.1.smithi203.stdout:9/954: dread - d13/d1b/d33/d64/dfa/ff8 zero size 2024-04-05T03:02:53.940 INFO:tasks.workunit.client.1.smithi203.stdout:9/955: mkdir d13/d1b/d6b/d9e/d59/d9b/d129 0 2024-04-05T03:02:53.945 INFO:tasks.workunit.client.1.smithi203.stdout:9/956: dwrite d13/d5c/d9d/fde [0,4194304] 0 2024-04-05T03:02:53.949 INFO:tasks.workunit.client.1.smithi203.stdout:9/957: symlink d13/d5c/dd1/d51/dcb/l12a 0 2024-04-05T03:02:53.977 INFO:tasks.workunit.client.1.smithi203.stdout:9/958: dread d13/d1b/d6b/d9e/d77/fd3 [0,4194304] 0 2024-04-05T03:02:53.994 INFO:tasks.workunit.client.1.smithi203.stdout:0/879: sync 2024-04-05T03:02:53.994 INFO:tasks.workunit.client.1.smithi203.stdout:5/875: sync 2024-04-05T03:02:53.995 INFO:tasks.workunit.client.1.smithi203.stdout:6/870: sync 2024-04-05T03:02:53.996 INFO:tasks.workunit.client.1.smithi203.stdout:0/880: chown dd/d17/d1d/d26/d28/f2d 7 1 2024-04-05T03:02:53.997 INFO:tasks.workunit.client.0.smithi153.stdout:9/455: rmdir d1/d10/d16/d22 39 2024-04-05T03:02:53.997 INFO:tasks.workunit.client.0.smithi153.stdout:4/665: truncate da/db/d20/fad 2927854 0 2024-04-05T03:02:53.998 INFO:tasks.workunit.client.0.smithi153.stdout:2/643: write d9/d68/db9/dd8/f33 [3679722,54848] 0 2024-04-05T03:02:53.998 INFO:tasks.workunit.client.1.smithi203.stdout:9/959: creat d13/d1b/d6b/d9e/d59/dbc/f12b x:0 0 0 2024-04-05T03:02:54.000 INFO:tasks.workunit.client.1.smithi203.stdout:9/960: chown d13/d1b/d6b/d9e/d59/d9b/lef 31 1 2024-04-05T03:02:54.000 INFO:tasks.workunit.client.1.smithi203.stdout:5/876: readlink d3/d1f/d49/la5 0 2024-04-05T03:02:54.002 INFO:tasks.workunit.client.1.smithi203.stdout:0/881: getdents dd/d17/d1d/d22/d45/d51 0 2024-04-05T03:02:54.003 INFO:tasks.workunit.client.0.smithi153.stdout:6/639: creat d4/dc/df/fd7 x:0 0 0 2024-04-05T03:02:54.006 INFO:tasks.workunit.client.0.smithi153.stdout:4/666: unlink da/de/fb2 0 2024-04-05T03:02:54.007 INFO:tasks.workunit.client.0.smithi153.stdout:6/640: dread - d4/f89 zero size 2024-04-05T03:02:54.011 INFO:tasks.workunit.client.0.smithi153.stdout:4/667: creat da/de/d56/d78/d8f/fe8 x:0 0 0 2024-04-05T03:02:54.012 INFO:tasks.workunit.client.1.smithi203.stdout:6/871: dread d0/da/d54/d4c/da1/db7/fbf [4194304,4194304] 0 2024-04-05T03:02:54.066 INFO:tasks.workunit.client.0.smithi153.stdout:6/641: readlink d4/dc/df/d16/d33/lc4 0 2024-04-05T03:02:54.067 INFO:tasks.workunit.client.0.smithi153.stdout:2/644: creat d9/da/d19/d2e/d7e/d7a/fdb x:0 0 0 2024-04-05T03:02:54.067 INFO:tasks.workunit.client.0.smithi153.stdout:1/503: dread d5/d8/d20/d4b/d18/d16/d2c/f7d [0,4194304] 0 2024-04-05T03:02:54.070 INFO:tasks.workunit.client.1.smithi203.stdout:0/882: creat dd/d17/d1d/d22/d45/d51/f100 x:0 0 0 2024-04-05T03:02:54.070 INFO:tasks.workunit.client.0.smithi153.stdout:6/642: unlink d4/dc/d7a/d9c/fb3 0 2024-04-05T03:02:54.073 INFO:tasks.workunit.client.0.smithi153.stdout:1/504: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:54.074 INFO:tasks.workunit.client.1.smithi203.stdout:5/877: dwrite d3/d1f/d49/f84 [0,4194304] 0 2024-04-05T03:02:54.077 INFO:tasks.workunit.client.1.smithi203.stdout:9/961: write d13/d5c/dd1/d51/da1/dba/fee [4518766,54552] 0 2024-04-05T03:02:54.077 INFO:tasks.workunit.client.0.smithi153.stdout:1/505: fdatasync d5/f76 0 2024-04-05T03:02:54.079 INFO:tasks.workunit.client.0.smithi153.stdout:1/506: stat d5/d8/d20/d2f/d40/l46 0 2024-04-05T03:02:54.080 INFO:tasks.workunit.client.1.smithi203.stdout:0/883: creat dd/d17/d1d/d22/f101 x:0 0 0 2024-04-05T03:02:54.082 INFO:tasks.workunit.client.1.smithi203.stdout:5/878: mknod d3/d4/dd6/d110/c124 0 2024-04-05T03:02:54.084 INFO:tasks.workunit.client.1.smithi203.stdout:9/962: truncate d13/d1b/d6b/d9e/d59/d7b/f7f 4863188 0 2024-04-05T03:02:54.086 INFO:tasks.workunit.client.1.smithi203.stdout:0/884: creat dd/d17/d1d/d22/d53/dec/f102 x:0 0 0 2024-04-05T03:02:54.088 INFO:tasks.workunit.client.1.smithi203.stdout:9/963: write d13/d1b/d33/d64/dfa/d67/d82/f122 [252398,38144] 0 2024-04-05T03:02:54.088 INFO:tasks.workunit.client.1.smithi203.stdout:0/885: write dd/d17/ff8 [1306627,34985] 0 2024-04-05T03:02:54.089 INFO:tasks.workunit.client.1.smithi203.stdout:5/879: getdents d3/d4/d9e/dde 0 2024-04-05T03:02:54.090 INFO:tasks.workunit.client.1.smithi203.stdout:6/872: dread d0/da/d18/d38/d2a/fb1 [0,4194304] 0 2024-04-05T03:02:54.091 INFO:tasks.workunit.client.1.smithi203.stdout:9/964: read - d13/d1b/d6b/d9e/d59/d5f/d103/f10e zero size 2024-04-05T03:02:54.091 INFO:tasks.workunit.client.1.smithi203.stdout:5/880: mknod d3/c125 0 2024-04-05T03:02:54.092 INFO:tasks.workunit.client.1.smithi203.stdout:0/886: unlink dd/d17/d1d/d22/d80/lc8 0 2024-04-05T03:02:54.093 INFO:tasks.workunit.client.1.smithi203.stdout:6/873: write d0/da/d54/d4c/da1/db7/d71/f79 [8171316,71728] 0 2024-04-05T03:02:54.095 INFO:tasks.workunit.client.1.smithi203.stdout:0/887: read dd/d17/dd8/f47 [6361788,87334] 0 2024-04-05T03:02:54.096 INFO:tasks.workunit.client.1.smithi203.stdout:5/881: rmdir d3/d1f/d10e 39 2024-04-05T03:02:54.098 INFO:tasks.workunit.client.1.smithi203.stdout:6/874: mkdir d0/da/d18/d92/d104 0 2024-04-05T03:02:54.100 INFO:tasks.workunit.client.1.smithi203.stdout:0/888: mknod dd/d17/d1d/d22/d53/c103 0 2024-04-05T03:02:54.101 INFO:tasks.workunit.client.1.smithi203.stdout:9/965: write d13/d1b/d33/d64/dfa/fc3 [760407,36154] 0 2024-04-05T03:02:54.102 INFO:tasks.workunit.client.1.smithi203.stdout:6/875: chown d0/da/lf0 2625 1 2024-04-05T03:02:54.103 INFO:tasks.workunit.client.1.smithi203.stdout:0/889: mkdir dd/d17/d1d/d22/d53/d55/d104 0 2024-04-05T03:02:54.104 INFO:tasks.workunit.client.1.smithi203.stdout:9/966: mkdir d13/d5c/d9d/d12c 0 2024-04-05T03:02:54.104 INFO:tasks.workunit.client.1.smithi203.stdout:5/882: dwrite d3/d4/f8e [0,4194304] 0 2024-04-05T03:02:54.106 INFO:tasks.workunit.client.1.smithi203.stdout:0/890: dread - dd/d17/d1d/d26/d8c/ff0 zero size 2024-04-05T03:02:54.106 INFO:tasks.workunit.client.1.smithi203.stdout:5/883: mkdir d3/d126 0 2024-04-05T03:02:54.108 INFO:tasks.workunit.client.1.smithi203.stdout:9/967: truncate d13/d1b/d6b/f72 1565230 0 2024-04-05T03:02:54.109 INFO:tasks.workunit.client.1.smithi203.stdout:9/968: read d13/d1b/d6b/d9e/d77/f7d [2733365,48452] 0 2024-04-05T03:02:54.110 INFO:tasks.workunit.client.1.smithi203.stdout:5/884: creat d3/f127 x:0 0 0 2024-04-05T03:02:54.113 INFO:tasks.workunit.client.1.smithi203.stdout:9/969: symlink d13/d5c/l12d 0 2024-04-05T03:02:54.118 INFO:tasks.workunit.client.1.smithi203.stdout:9/970: creat d13/d1b/d6b/d9e/d59/d9b/f12e x:0 0 0 2024-04-05T03:02:54.119 INFO:tasks.workunit.client.1.smithi203.stdout:0/891: link dd/d17/d1d/d22/d53/f63 dd/d17/d1d/d95/f105 0 2024-04-05T03:02:54.122 INFO:tasks.workunit.client.1.smithi203.stdout:0/892: dread dd/d17/d1d/d26/fd1 [0,4194304] 0 2024-04-05T03:02:54.123 INFO:tasks.workunit.client.1.smithi203.stdout:6/876: truncate d0/da/d18/d38/d2a/d2c/f4e 7625880 0 2024-04-05T03:02:54.123 INFO:tasks.workunit.client.1.smithi203.stdout:5/885: link d3/d1f/d59/d64/d12/lce d3/d4/dc/d67/db5/dda/l128 0 2024-04-05T03:02:54.124 INFO:tasks.workunit.client.0.smithi153.stdout:4/668: unlink da/f48 0 2024-04-05T03:02:54.125 INFO:tasks.workunit.client.0.smithi153.stdout:6/643: creat d4/dc/df/d16/d24/d4e/fd8 x:0 0 0 2024-04-05T03:02:54.126 INFO:tasks.workunit.client.1.smithi203.stdout:9/971: rename d13/d5c/dd1/d31 to d13/d1b/d33/d64/dfa/d124/dd2/d12f 0 2024-04-05T03:02:54.126 INFO:tasks.workunit.client.0.smithi153.stdout:6/644: chown d4/dc/d7a/da8/fc5 0 1 2024-04-05T03:02:54.127 INFO:tasks.workunit.client.0.smithi153.stdout:4/669: dread da/de/d56/d78/d8f/f3d [0,4194304] 0 2024-04-05T03:02:54.127 INFO:tasks.workunit.client.1.smithi203.stdout:9/972: readlink d13/d1b/d33/l43 0 2024-04-05T03:02:54.127 INFO:tasks.workunit.client.1.smithi203.stdout:0/893: symlink dd/d17/d1d/d22/d53/d7f/l106 0 2024-04-05T03:02:54.128 INFO:tasks.workunit.client.0.smithi153.stdout:2/645: link d9/da/f5f d9/da/d19/d2e/d7e/fdc 0 2024-04-05T03:02:54.129 INFO:tasks.workunit.client.0.smithi153.stdout:2/646: write d9/da/d37/fb6 [684889,130575] 0 2024-04-05T03:02:54.131 INFO:tasks.workunit.client.1.smithi203.stdout:0/894: unlink dd/db1/ce9 0 2024-04-05T03:02:54.132 INFO:tasks.workunit.client.0.smithi153.stdout:4/670: creat da/d2a/d86/fe9 x:0 0 0 2024-04-05T03:02:54.134 INFO:tasks.workunit.client.0.smithi153.stdout:6/645: creat d4/dc/df/d16/d44/fd9 x:0 0 0 2024-04-05T03:02:54.136 INFO:tasks.workunit.client.0.smithi153.stdout:1/507: link d5/d8/d20/d4b/fe d5/dd/f9f 0 2024-04-05T03:02:54.139 INFO:tasks.workunit.client.0.smithi153.stdout:6/646: dread d4/dc/f97 [0,4194304] 0 2024-04-05T03:02:54.141 INFO:tasks.workunit.client.1.smithi203.stdout:9/973: dread d13/d1b/d6b/d9e/f5e [0,4194304] 0 2024-04-05T03:02:54.142 INFO:tasks.workunit.client.1.smithi203.stdout:9/974: chown d13/d5c/d9d 407 1 2024-04-05T03:02:54.143 INFO:tasks.workunit.client.1.smithi203.stdout:0/895: dwrite dd/d17/d1d/d22/d45/d51/f86 [0,4194304] 0 2024-04-05T03:02:54.144 INFO:tasks.workunit.client.0.smithi153.stdout:1/508: dwrite d5/d8/d20/d4b/d18/d16/d33/f70 [0,4194304] 0 2024-04-05T03:02:54.144 INFO:tasks.workunit.client.0.smithi153.stdout:6/647: read d4/dc/df/d16/d17/f3a [3460483,84675] 0 2024-04-05T03:02:54.147 INFO:tasks.workunit.client.1.smithi203.stdout:4/694: sync 2024-04-05T03:02:54.147 INFO:tasks.workunit.client.1.smithi203.stdout:0/896: read dd/d17/d1d/d26/f3d [688157,61569] 0 2024-04-05T03:02:54.148 INFO:tasks.workunit.client.0.smithi153.stdout:1/509: read d5/d8/f6e [2433980,14923] 0 2024-04-05T03:02:54.148 INFO:tasks.workunit.client.1.smithi203.stdout:6/877: dwrite d0/f3 [4194304,4194304] 0 2024-04-05T03:02:54.149 INFO:tasks.workunit.client.0.smithi153.stdout:6/648: stat d4/dc/df/d16/d24/d8d/c80 0 2024-04-05T03:02:54.150 INFO:tasks.workunit.client.1.smithi203.stdout:5/886: write d3/d1f/d59/d64/f4f [2799984,81978] 0 2024-04-05T03:02:54.152 INFO:tasks.workunit.client.1.smithi203.stdout:5/887: write d3/d4/dc/d67/dac/ff0 [892986,97863] 0 2024-04-05T03:02:54.153 INFO:tasks.workunit.client.0.smithi153.stdout:4/671: dwrite da/de/f6c [0,4194304] 0 2024-04-05T03:02:54.155 INFO:tasks.workunit.client.1.smithi203.stdout:5/888: chown d3/d4/c7b 1848 1 2024-04-05T03:02:54.157 INFO:tasks.workunit.client.0.smithi153.stdout:1/510: creat d5/d8/d20/d4b/d18/d16/d48/d96/fa0 x:0 0 0 2024-04-05T03:02:54.158 INFO:tasks.workunit.client.1.smithi203.stdout:5/889: creat d3/d4/dc/d67/dac/f129 x:0 0 0 2024-04-05T03:02:54.159 INFO:tasks.workunit.client.0.smithi153.stdout:6/649: creat d4/fda x:0 0 0 2024-04-05T03:02:54.161 INFO:tasks.workunit.client.0.smithi153.stdout:1/511: fsync d5/d8/d20/d4b/d18/d16/d33/f3c 0 2024-04-05T03:02:54.161 INFO:tasks.workunit.client.1.smithi203.stdout:0/897: rename dd/d17/d1d/d22/d80/d49/l4d to dd/d17/d1d/d22/l107 0 2024-04-05T03:02:54.163 INFO:tasks.workunit.client.0.smithi153.stdout:6/650: creat d4/dc/df/d16/d44/fdb x:0 0 0 2024-04-05T03:02:54.165 INFO:tasks.workunit.client.1.smithi203.stdout:9/975: write d13/d1b/d33/d64/dfa/d124/dd2/fe7 [1473429,17693] 0 2024-04-05T03:02:54.165 INFO:tasks.workunit.client.0.smithi153.stdout:1/512: truncate d5/dd/f89 1518911 0 2024-04-05T03:02:54.170 INFO:tasks.workunit.client.1.smithi203.stdout:0/898: link dd/d17/d1d/d22/d53/d7f/fe3 dd/d17/d1d/d26/d8c/d6d/dc4/f108 0 2024-04-05T03:02:54.171 INFO:tasks.workunit.client.0.smithi153.stdout:6/651: creat d4/dc/df/dcb/fdc x:0 0 0 2024-04-05T03:02:54.172 INFO:tasks.workunit.client.0.smithi153.stdout:1/513: unlink d5/d8/c52 0 2024-04-05T03:02:54.172 INFO:tasks.workunit.client.1.smithi203.stdout:6/878: write d0/da/f94 [119570,114498] 0 2024-04-05T03:02:54.173 INFO:tasks.workunit.client.1.smithi203.stdout:0/899: fsync dd/d17/d1d/d26/d8c/d6d/fa9 0 2024-04-05T03:02:54.174 INFO:tasks.workunit.client.0.smithi153.stdout:1/514: dread d5/d8/d20/f88 [0,4194304] 0 2024-04-05T03:02:54.177 INFO:tasks.workunit.client.0.smithi153.stdout:1/515: symlink d5/d8/d20/d4b/d18/d16/d48/d96/la1 0 2024-04-05T03:02:54.178 INFO:tasks.workunit.client.0.smithi153.stdout:6/652: dread d4/d38/fbb [0,4194304] 0 2024-04-05T03:02:54.178 INFO:tasks.workunit.client.1.smithi203.stdout:4/695: dwrite da/d3c/d4c/f79 [0,4194304] 0 2024-04-05T03:02:54.178 INFO:tasks.workunit.client.1.smithi203.stdout:6/879: truncate d0/da/d54/d53/f8d 2871842 0 2024-04-05T03:02:54.179 INFO:tasks.workunit.client.0.smithi153.stdout:1/516: write f3 [3508775,25710] 0 2024-04-05T03:02:54.179 INFO:tasks.workunit.client.1.smithi203.stdout:4/696: chown da/d29/d2e/d53/cda 354791113 1 2024-04-05T03:02:54.181 INFO:tasks.workunit.client.1.smithi203.stdout:0/900: rename dd/d17/ff8 to dd/d17/dd8/d56/f109 0 2024-04-05T03:02:54.181 INFO:tasks.workunit.client.0.smithi153.stdout:6/653: mknod d4/dc/df/d16/d24/cdd 0 2024-04-05T03:02:54.184 INFO:tasks.workunit.client.0.smithi153.stdout:6/654: chown d4/dc/df/d16/d24/d8d/lbf 812011925 1 2024-04-05T03:02:54.185 INFO:tasks.workunit.client.1.smithi203.stdout:4/697: write da/d3c/d4c/f73 [3902612,48407] 0 2024-04-05T03:02:54.186 INFO:tasks.workunit.client.0.smithi153.stdout:1/517: dwrite d5/dd/d1b/f66 [8388608,4194304] 0 2024-04-05T03:02:54.187 INFO:tasks.workunit.client.0.smithi153.stdout:1/518: chown f3 202241 1 2024-04-05T03:02:54.188 INFO:tasks.workunit.client.1.smithi203.stdout:0/901: readlink dd/d17/d1d/d22/d80/d49/l7c 0 2024-04-05T03:02:54.188 INFO:tasks.workunit.client.0.smithi153.stdout:6/655: truncate d4/dc/f3c 785234 0 2024-04-05T03:02:54.188 INFO:tasks.workunit.client.1.smithi203.stdout:6/880: dwrite d0/da/d54/f4b [0,4194304] 0 2024-04-05T03:02:54.189 INFO:tasks.workunit.client.0.smithi153.stdout:1/519: dread - d5/d8/f65 zero size 2024-04-05T03:02:54.190 INFO:tasks.workunit.client.1.smithi203.stdout:6/881: readlink d0/da/d18/d38/d2a/d2c/lef 0 2024-04-05T03:02:54.190 INFO:tasks.workunit.client.1.smithi203.stdout:4/698: mkdir da/d29/d2e/d57/d96/ddc/de0 0 2024-04-05T03:02:54.190 INFO:tasks.workunit.client.1.smithi203.stdout:0/902: write dd/d17/d1d/d26/d8c/ff0 [298190,17055] 0 2024-04-05T03:02:54.193 INFO:tasks.workunit.client.0.smithi153.stdout:6/656: mkdir d4/dc/df/d16/d36/d4d/dde 0 2024-04-05T03:02:54.194 INFO:tasks.workunit.client.1.smithi203.stdout:0/903: symlink dd/d17/d1d/d26/d8c/dcf/l10a 0 2024-04-05T03:02:54.196 INFO:tasks.workunit.client.0.smithi153.stdout:1/520: rename d5/d8/d20/d4b/d18/d16/d48/d82 to d5/d8/d20/d4b/d51/da2 0 2024-04-05T03:02:54.199 INFO:tasks.workunit.client.0.smithi153.stdout:6/657: unlink d4/dc/df/d16/d5f/f8c 0 2024-04-05T03:02:54.202 INFO:tasks.workunit.client.1.smithi203.stdout:4/699: dwrite da/d29/d2e/db0/dbe/dce/dd0/fb6 [0,4194304] 0 2024-04-05T03:02:54.203 INFO:tasks.workunit.client.1.smithi203.stdout:0/904: symlink dd/d17/d1d/d26/l10b 0 2024-04-05T03:02:54.204 INFO:tasks.workunit.client.1.smithi203.stdout:1/915: sync 2024-04-05T03:02:54.206 INFO:tasks.workunit.client.0.smithi153.stdout:6/658: truncate d4/dc/df/f56 409138 0 2024-04-05T03:02:54.210 INFO:tasks.workunit.client.1.smithi203.stdout:1/916: unlink d7/c22 0 2024-04-05T03:02:54.212 INFO:tasks.workunit.client.1.smithi203.stdout:0/905: rename dd/d17/d1d/d95/fb2 to dd/f10c 0 2024-04-05T03:02:54.213 INFO:tasks.workunit.client.0.smithi153.stdout:6/659: dwrite d4/dc/d7a/da8/fca [0,4194304] 0 2024-04-05T03:02:54.220 INFO:tasks.workunit.client.0.smithi153.stdout:6/660: mknod d4/d38/d4c/cdf 0 2024-04-05T03:02:54.220 INFO:tasks.workunit.client.1.smithi203.stdout:0/906: creat dd/d17/f10d x:0 0 0 2024-04-05T03:02:54.221 INFO:tasks.workunit.client.1.smithi203.stdout:0/907: read - dd/d17/f10d zero size 2024-04-05T03:02:54.222 INFO:tasks.workunit.client.0.smithi153.stdout:6/661: write d4/f6 [989623,117098] 0 2024-04-05T03:02:54.224 INFO:tasks.workunit.client.0.smithi153.stdout:6/662: truncate d4/dc/df/d16/d44/fdb 314745 0 2024-04-05T03:02:54.225 INFO:tasks.workunit.client.1.smithi203.stdout:0/908: getdents dd/d17/d1d/d22/d45/d51/d69/dfe 0 2024-04-05T03:02:54.226 INFO:tasks.workunit.client.1.smithi203.stdout:6/882: dread d0/da/d18/d38/d2a/ffc [0,4194304] 0 2024-04-05T03:02:54.228 INFO:tasks.workunit.client.1.smithi203.stdout:6/883: fsync d0/da/d54/d4c/da1/db7/fc4 0 2024-04-05T03:02:54.229 INFO:tasks.workunit.client.0.smithi153.stdout:6/663: dwrite d4/dc/df/d16/d44/f96 [0,4194304] 0 2024-04-05T03:02:54.231 INFO:tasks.workunit.client.1.smithi203.stdout:6/884: chown d0/da/d54/d53/f8d 1 1 2024-04-05T03:02:54.231 INFO:tasks.workunit.client.1.smithi203.stdout:5/890: dread d3/d4/dc/f4b [0,4194304] 0 2024-04-05T03:02:54.233 INFO:tasks.workunit.client.1.smithi203.stdout:6/885: write d0/da/d54/d53/f59 [5335132,123472] 0 2024-04-05T03:02:54.235 INFO:tasks.workunit.client.1.smithi203.stdout:5/891: symlink d3/d4/dc/d67/db5/dda/l12a 0 2024-04-05T03:02:54.236 INFO:tasks.workunit.client.1.smithi203.stdout:6/886: symlink d0/da/d54/d4c/l105 0 2024-04-05T03:02:54.238 INFO:tasks.workunit.client.1.smithi203.stdout:3/729: sync 2024-04-05T03:02:54.238 INFO:tasks.workunit.client.1.smithi203.stdout:5/892: chown d3/d1f/fc4 282 1 2024-04-05T03:02:54.239 INFO:tasks.workunit.client.0.smithi153.stdout:6/664: rename d4/dc/d7a/da8/cab to d4/dc/df/d16/d44/ce0 0 2024-04-05T03:02:54.244 INFO:tasks.workunit.client.1.smithi203.stdout:6/887: getdents d0/da/d18/d38/d2a/d44/d78 0 2024-04-05T03:02:54.245 INFO:tasks.workunit.client.1.smithi203.stdout:5/893: creat d3/d4/dc/d67/f12b x:0 0 0 2024-04-05T03:02:54.248 INFO:tasks.workunit.client.1.smithi203.stdout:9/976: dread d13/d5c/dd1/f4c [0,4194304] 0 2024-04-05T03:02:54.248 INFO:tasks.workunit.client.1.smithi203.stdout:5/894: creat d3/d1f/d49/d10c/d10d/f12c x:0 0 0 2024-04-05T03:02:54.250 INFO:tasks.workunit.client.1.smithi203.stdout:9/977: mkdir d13/d1b/d6b/d9e/df0/d130 0 2024-04-05T03:02:54.251 INFO:tasks.workunit.client.1.smithi203.stdout:5/895: symlink d3/d1f/d59/d60/l12d 0 2024-04-05T03:02:54.252 INFO:tasks.workunit.client.1.smithi203.stdout:9/978: creat d13/d1b/d33/d64/f131 x:0 0 0 2024-04-05T03:02:54.256 INFO:tasks.workunit.client.0.smithi153.stdout:4/672: dwrite da/de/da1/fc0 [4194304,4194304] 0 2024-04-05T03:02:54.260 INFO:tasks.workunit.client.0.smithi153.stdout:4/673: read da/db/f96 [2028620,7237] 0 2024-04-05T03:02:54.261 INFO:tasks.workunit.client.1.smithi203.stdout:6/888: dread d0/da/d54/d4c/da1/db7/fc4 [0,4194304] 0 2024-04-05T03:02:54.263 INFO:tasks.workunit.client.0.smithi153.stdout:4/674: write da/db/d8b/db7/fe0 [895357,27943] 0 2024-04-05T03:02:54.263 INFO:tasks.workunit.client.1.smithi203.stdout:6/889: readlink d0/da/d18/l6e 0 2024-04-05T03:02:54.265 INFO:tasks.workunit.client.1.smithi203.stdout:3/730: dread d0/dc/d1a/d76/d35/f3d [0,4194304] 0 2024-04-05T03:02:54.267 INFO:tasks.workunit.client.1.smithi203.stdout:6/890: creat d0/da/d18/d38/d2a/d44/d78/da7/f106 x:0 0 0 2024-04-05T03:02:54.269 INFO:tasks.workunit.client.1.smithi203.stdout:3/731: creat d0/dc/d38/dca/fd1 x:0 0 0 2024-04-05T03:02:54.270 INFO:tasks.workunit.client.0.smithi153.stdout:4/675: dread da/de/da1/fc0 [4194304,4194304] 0 2024-04-05T03:02:54.270 INFO:tasks.workunit.client.0.smithi153.stdout:4/676: stat da/db/d20/d3c/cd0 0 2024-04-05T03:02:54.270 INFO:tasks.workunit.client.0.smithi153.stdout:5/544: sync 2024-04-05T03:02:54.271 INFO:tasks.workunit.client.1.smithi203.stdout:6/891: rename d0/da/d18/d92/fe7 to d0/da/d18/d38/d2a/d2c/f107 0 2024-04-05T03:02:54.273 INFO:tasks.workunit.client.1.smithi203.stdout:6/892: write d0/da/d54/d4c/da1/db7/fcb [1074201,106005] 0 2024-04-05T03:02:54.273 INFO:tasks.workunit.client.1.smithi203.stdout:3/732: mknod d0/dc/d1a/cd2 0 2024-04-05T03:02:54.273 INFO:tasks.workunit.client.0.smithi153.stdout:5/545: mkdir d0/d25/d4d/d9c/dc4 0 2024-04-05T03:02:54.274 INFO:tasks.workunit.client.1.smithi203.stdout:6/893: chown d0/da/d18/d92/lda 1276701 1 2024-04-05T03:02:54.276 INFO:tasks.workunit.client.0.smithi153.stdout:4/677: dwrite f8 [4194304,4194304] 0 2024-04-05T03:02:54.277 INFO:tasks.workunit.client.0.smithi153.stdout:5/546: rename d0/c5d to d0/d2c/d3f/da8/cc5 0 2024-04-05T03:02:54.279 INFO:tasks.workunit.client.1.smithi203.stdout:6/894: dwrite d0/da/f61 [0,4194304] 0 2024-04-05T03:02:54.280 INFO:tasks.workunit.client.0.smithi153.stdout:4/678: creat da/db/d20/d35/fea x:0 0 0 2024-04-05T03:02:54.284 INFO:tasks.workunit.client.1.smithi203.stdout:6/895: readlink d0/da/d18/l1b 0 2024-04-05T03:02:54.284 INFO:tasks.workunit.client.0.smithi153.stdout:4/679: truncate da/db/d20/d35/d38/d76/d90/fca 594082 0 2024-04-05T03:02:54.288 INFO:tasks.workunit.client.1.smithi203.stdout:6/896: chown d0/da/d18/d38/d2a/la4 13059730 1 2024-04-05T03:02:54.289 INFO:tasks.workunit.client.0.smithi153.stdout:1/521: rmdir d5/d8 39 2024-04-05T03:02:54.292 INFO:tasks.workunit.client.1.smithi203.stdout:6/897: unlink d0/da/d54/d4c/da1/db7/f28 0 2024-04-05T03:02:54.293 INFO:tasks.workunit.client.0.smithi153.stdout:4/680: link da/db/d20/d35/l3a da/db/d20/d3c/d81/leb 0 2024-04-05T03:02:54.294 INFO:tasks.workunit.client.0.smithi153.stdout:4/681: chown da/db/d20/d3c/d81/db6/fe5 8 1 2024-04-05T03:02:54.296 INFO:tasks.workunit.client.0.smithi153.stdout:1/522: creat d5/d8/d20/d4b/d18/d16/d45/fa3 x:0 0 0 2024-04-05T03:02:54.297 INFO:tasks.workunit.client.0.smithi153.stdout:4/682: write da/db/d20/d3c/d81/db6/f71 [325553,124267] 0 2024-04-05T03:02:54.300 INFO:tasks.workunit.client.0.smithi153.stdout:6/665: truncate d4/d38/fbe 505274 0 2024-04-05T03:02:54.300 INFO:tasks.workunit.client.1.smithi203.stdout:4/700: write da/d29/d2e/d8b/fdb [1260579,55998] 0 2024-04-05T03:02:54.301 INFO:tasks.workunit.client.0.smithi153.stdout:6/666: fsync d4/dc/df/d16/d36/d4d/d64/f6c 0 2024-04-05T03:02:54.302 INFO:tasks.workunit.client.0.smithi153.stdout:4/683: rename da/db/d20/d35/c65 to da/de/d56/d78/d8f/cec 0 2024-04-05T03:02:54.302 INFO:tasks.workunit.client.0.smithi153.stdout:6/667: readlink d4/dc/df/l22 0 2024-04-05T03:02:54.305 INFO:tasks.workunit.client.0.smithi153.stdout:4/684: dread - da/db/d20/d3c/d91/fe7 zero size 2024-04-05T03:02:54.305 INFO:tasks.workunit.client.0.smithi153.stdout:4/685: dread - da/d2a/d86/fe2 zero size 2024-04-05T03:02:54.306 INFO:tasks.workunit.client.0.smithi153.stdout:4/686: chown da/db/d20/d45/f6b 40 1 2024-04-05T03:02:54.307 INFO:tasks.workunit.client.1.smithi203.stdout:1/917: dwrite d7/d24/d33/f5b [0,4194304] 0 2024-04-05T03:02:54.308 INFO:tasks.workunit.client.0.smithi153.stdout:4/687: read da/db/d20/d3c/d81/f8a [757718,49560] 0 2024-04-05T03:02:54.309 INFO:tasks.workunit.client.0.smithi153.stdout:4/688: stat da/db/d20/d3c/c3f 0 2024-04-05T03:02:54.309 INFO:tasks.workunit.client.1.smithi203.stdout:5/896: write d3/d1f/d10e/fd8 [489618,8112] 0 2024-04-05T03:02:54.310 INFO:tasks.workunit.client.1.smithi203.stdout:9/979: truncate d13/d1b/d6b/d9e/d59/fc1 4604943 0 2024-04-05T03:02:54.310 INFO:tasks.workunit.client.1.smithi203.stdout:1/918: write d7/d24/d33/f45 [4936121,122391] 0 2024-04-05T03:02:54.312 INFO:tasks.workunit.client.1.smithi203.stdout:5/897: readlink d3/d1f/d59/d64/d12/l52 0 2024-04-05T03:02:54.314 INFO:tasks.workunit.client.1.smithi203.stdout:3/733: truncate d0/dc/d38/f3b 69530 0 2024-04-05T03:02:54.315 INFO:tasks.workunit.client.1.smithi203.stdout:1/919: write d7/d24/f35 [4974705,47559] 0 2024-04-05T03:02:54.315 INFO:tasks.workunit.client.1.smithi203.stdout:5/898: chown d3/d4/d94/fed 0 1 2024-04-05T03:02:54.316 INFO:tasks.workunit.client.1.smithi203.stdout:6/898: write d0/da/d18/d38/d2a/d44/f4a [268180,117591] 0 2024-04-05T03:02:54.317 INFO:tasks.workunit.client.1.smithi203.stdout:5/899: stat d3/d5f 0 2024-04-05T03:02:54.317 INFO:tasks.workunit.client.0.smithi153.stdout:4/689: symlink da/db/d20/d3c/led 0 2024-04-05T03:02:54.318 INFO:tasks.workunit.client.1.smithi203.stdout:9/980: dread d13/d1b/d33/d64/dfa/d124/dd2/d12f/f99 [0,4194304] 0 2024-04-05T03:02:54.318 INFO:tasks.workunit.client.1.smithi203.stdout:6/899: stat d0/da/d54/f45 0 2024-04-05T03:02:54.319 INFO:tasks.workunit.client.1.smithi203.stdout:3/734: creat d0/dc/d1a/d8d/dc9/fd3 x:0 0 0 2024-04-05T03:02:54.320 INFO:tasks.workunit.client.1.smithi203.stdout:6/900: dread - d0/da/d18/d38/d2a/d2c/fee zero size 2024-04-05T03:02:54.323 INFO:tasks.workunit.client.1.smithi203.stdout:1/920: creat d7/d126/f12e x:0 0 0 2024-04-05T03:02:54.326 INFO:tasks.workunit.client.1.smithi203.stdout:5/900: creat d3/d4/dc/d67/db5/dda/f12e x:0 0 0 2024-04-05T03:02:54.326 INFO:tasks.workunit.client.1.smithi203.stdout:3/735: creat d0/d41/fd4 x:0 0 0 2024-04-05T03:02:54.327 INFO:tasks.workunit.client.0.smithi153.stdout:4/690: dwrite da/d2a/fd3 [0,4194304] 0 2024-04-05T03:02:54.328 INFO:tasks.workunit.client.0.smithi153.stdout:4/691: stat da/db/f96 0 2024-04-05T03:02:54.328 INFO:tasks.workunit.client.0.smithi153.stdout:4/692: chown da/de/d66 2260 1 2024-04-05T03:02:54.329 INFO:tasks.workunit.client.1.smithi203.stdout:1/921: truncate d7/d24/d33/d48/d4b/d51/f7f 1677071 0 2024-04-05T03:02:54.332 INFO:tasks.workunit.client.0.smithi153.stdout:6/668: read d4/d38/fbe [438994,20930] 0 2024-04-05T03:02:54.332 INFO:tasks.workunit.client.1.smithi203.stdout:1/922: write d7/d14/d2b/d58/d86/d91/da6/dee/ffe [559517,95187] 0 2024-04-05T03:02:54.335 INFO:tasks.workunit.client.0.smithi153.stdout:4/693: dwrite da/fb0 [0,4194304] 0 2024-04-05T03:02:54.336 INFO:tasks.workunit.client.0.smithi153.stdout:4/694: chown da/de/d56/d78/d8f/f99 3201105 1 2024-04-05T03:02:54.339 INFO:tasks.workunit.client.1.smithi203.stdout:6/901: dread - d0/da/d54/d4c/da1/db7/d71/fa6 zero size 2024-04-05T03:02:54.341 INFO:tasks.workunit.client.1.smithi203.stdout:4/701: dread da/d71/fc2 [0,4194304] 0 2024-04-05T03:02:54.342 INFO:tasks.workunit.client.1.smithi203.stdout:1/923: unlink d7/d14/d2b/d58/d86/d91/da6/df9/d106/f107 0 2024-04-05T03:02:54.342 INFO:tasks.workunit.client.1.smithi203.stdout:9/981: link d13/ffe d13/d1b/d33/d64/dfa/d124/f132 0 2024-04-05T03:02:54.343 INFO:tasks.workunit.client.1.smithi203.stdout:4/702: stat da/d29/d2e/c33 0 2024-04-05T03:02:54.344 INFO:tasks.workunit.client.0.smithi153.stdout:6/669: link d4/l95 d4/d38/le1 0 2024-04-05T03:02:54.345 INFO:tasks.workunit.client.1.smithi203.stdout:1/924: mkdir d7/d126/d12f 0 2024-04-05T03:02:54.345 INFO:tasks.workunit.client.0.smithi153.stdout:4/695: getdents da/db/d20/d35/d41/da8/dc1 0 2024-04-05T03:02:54.346 INFO:tasks.workunit.client.1.smithi203.stdout:4/703: write da/d29/d2e/db0/dcb/d80/fa6 [1358050,82694] 0 2024-04-05T03:02:54.346 INFO:tasks.workunit.client.0.smithi153.stdout:6/670: write d4/dc/df/d16/d36/d4d/f91 [2202251,47002] 0 2024-04-05T03:02:54.347 INFO:tasks.workunit.client.1.smithi203.stdout:9/982: write d13/d1b/d33/d64/dfa/d67/d82/f122 [38623,121657] 0 2024-04-05T03:02:54.348 INFO:tasks.workunit.client.0.smithi153.stdout:4/696: mknod da/de/d56/d6a/cee 0 2024-04-05T03:02:54.348 INFO:tasks.workunit.client.1.smithi203.stdout:6/902: dwrite d0/f9 [4194304,4194304] 0 2024-04-05T03:02:54.349 INFO:tasks.workunit.client.0.smithi153.stdout:6/671: creat d4/dc/df/d16/d24/d5e/fe2 x:0 0 0 2024-04-05T03:02:54.350 INFO:tasks.workunit.client.1.smithi203.stdout:1/925: write d7/d14/f10a [280478,130701] 0 2024-04-05T03:02:54.351 INFO:tasks.workunit.client.0.smithi153.stdout:4/697: creat da/db/d8b/fef x:0 0 0 2024-04-05T03:02:54.352 INFO:tasks.workunit.client.1.smithi203.stdout:4/704: chown da/d29/d2e/db0/dcb/d80 530 1 2024-04-05T03:02:54.354 INFO:tasks.workunit.client.0.smithi153.stdout:4/698: creat da/de/d66/ff0 x:0 0 0 2024-04-05T03:02:54.355 INFO:tasks.workunit.client.1.smithi203.stdout:4/705: dread - da/d29/d2e/db0/fd1 zero size 2024-04-05T03:02:54.356 INFO:tasks.workunit.client.0.smithi153.stdout:6/672: link d4/d38/d4c/fa5 d4/dc/d7a/da8/fe3 0 2024-04-05T03:02:54.359 INFO:tasks.workunit.client.1.smithi203.stdout:1/926: fsync d7/d24/f85 0 2024-04-05T03:02:54.360 INFO:tasks.workunit.client.1.smithi203.stdout:5/901: truncate d3/d4/f87 1762899 0 2024-04-05T03:02:54.362 INFO:tasks.workunit.client.1.smithi203.stdout:9/983: unlink d13/d1b/d6b/d9e/d59/dbc/f12b 0 2024-04-05T03:02:54.365 INFO:tasks.workunit.client.0.smithi153.stdout:6/673: dwrite d4/f51 [0,4194304] 0 2024-04-05T03:02:54.366 INFO:tasks.workunit.client.1.smithi203.stdout:1/927: creat d7/d14/d2b/d58/d86/f130 x:0 0 0 2024-04-05T03:02:54.367 INFO:tasks.workunit.client.0.smithi153.stdout:6/674: truncate d4/dc/df/d16/d36/d52/f55 286114 0 2024-04-05T03:02:54.367 INFO:tasks.workunit.client.1.smithi203.stdout:3/736: dread d0/dc/f14 [0,4194304] 0 2024-04-05T03:02:54.368 INFO:tasks.workunit.client.1.smithi203.stdout:5/902: creat d3/d1f/d59/d60/f12f x:0 0 0 2024-04-05T03:02:54.369 INFO:tasks.workunit.client.1.smithi203.stdout:9/984: creat d13/d1b/d6b/d9e/d59/d5f/f133 x:0 0 0 2024-04-05T03:02:54.370 INFO:tasks.workunit.client.1.smithi203.stdout:9/985: fsync d13/d1b/d33/d64/dfa/d67/d82/f8e 0 2024-04-05T03:02:54.371 INFO:tasks.workunit.client.0.smithi153.stdout:6/675: getdents d4/dc/df/d16/d5f/da2 0 2024-04-05T03:02:54.373 INFO:tasks.workunit.client.1.smithi203.stdout:1/928: creat d7/d14/d83/f131 x:0 0 0 2024-04-05T03:02:54.374 INFO:tasks.workunit.client.0.smithi153.stdout:6/676: mknod d4/d38/ce4 0 2024-04-05T03:02:54.375 INFO:tasks.workunit.client.1.smithi203.stdout:1/929: write d7/fe [6519968,128465] 0 2024-04-05T03:02:54.375 INFO:tasks.workunit.client.1.smithi203.stdout:6/903: link d0/da/d54/d4c/c52 d0/da/d54/d4c/d7f/c108 0 2024-04-05T03:02:54.377 INFO:tasks.workunit.client.1.smithi203.stdout:3/737: mknod d0/dc/d1a/d1f/da1/daf/cd5 0 2024-04-05T03:02:54.377 INFO:tasks.workunit.client.1.smithi203.stdout:6/904: chown d0/da/d18/d92/dfe 34900 1 2024-04-05T03:02:54.378 INFO:tasks.workunit.client.0.smithi153.stdout:4/699: fsync da/fb0 0 2024-04-05T03:02:54.379 INFO:tasks.workunit.client.1.smithi203.stdout:9/986: dwrite d13/d1b/d6b/d9e/d77/f120 [0,4194304] 0 2024-04-05T03:02:54.380 INFO:tasks.workunit.client.0.smithi153.stdout:6/677: truncate d4/dc/df/d16/d5f/da2/fa4 4089022 0 2024-04-05T03:02:54.386 INFO:tasks.workunit.client.0.smithi153.stdout:6/678: getdents d4/dc/d7a/da8 0 2024-04-05T03:02:54.386 INFO:tasks.workunit.client.0.smithi153.stdout:4/700: dwrite da/de/d56/f70 [4194304,4194304] 0 2024-04-05T03:02:54.387 INFO:tasks.workunit.client.1.smithi203.stdout:9/987: mkdir d13/d1b/d6b/d134 0 2024-04-05T03:02:54.389 INFO:tasks.workunit.client.0.smithi153.stdout:4/701: truncate da/d2a/d86/fe2 1004572 0 2024-04-05T03:02:54.390 INFO:tasks.workunit.client.0.smithi153.stdout:6/679: symlink d4/dc/d7a/le5 0 2024-04-05T03:02:54.391 INFO:tasks.workunit.client.0.smithi153.stdout:4/702: stat da/db/d20/d35/d38/d76/fc6 0 2024-04-05T03:02:54.393 INFO:tasks.workunit.client.0.smithi153.stdout:6/680: read d4/d38/d4c/fa0 [2815676,58899] 0 2024-04-05T03:02:54.393 INFO:tasks.workunit.client.1.smithi203.stdout:6/905: mknod d0/da/d18/d38/d2a/c109 0 2024-04-05T03:02:54.394 INFO:tasks.workunit.client.1.smithi203.stdout:6/906: chown d0/f9d 55576835 1 2024-04-05T03:02:54.395 INFO:tasks.workunit.client.1.smithi203.stdout:3/738: creat d0/dc/d1a/d1f/fd6 x:0 0 0 2024-04-05T03:02:54.397 INFO:tasks.workunit.client.1.smithi203.stdout:1/930: getdents d7/d24/d33/d48/d4b/dea/df2 0 2024-04-05T03:02:54.398 INFO:tasks.workunit.client.0.smithi153.stdout:6/681: getdents d4/dc/df/d16/d33 0 2024-04-05T03:02:54.399 INFO:tasks.workunit.client.1.smithi203.stdout:9/988: mknod d13/d1b/d6b/d7a/ddc/c135 0 2024-04-05T03:02:54.401 INFO:tasks.workunit.client.0.smithi153.stdout:6/682: symlink d4/dc/df/d16/d24/d5e/le6 0 2024-04-05T03:02:54.402 INFO:tasks.workunit.client.1.smithi203.stdout:3/739: fdatasync d0/dc/d1a/d1d/f2f 0 2024-04-05T03:02:54.403 INFO:tasks.workunit.client.1.smithi203.stdout:4/706: write da/d29/d2e/f35 [1393905,51244] 0 2024-04-05T03:02:54.404 INFO:tasks.workunit.client.1.smithi203.stdout:1/931: write d7/f4f [4316118,70778] 0 2024-04-05T03:02:54.404 INFO:tasks.workunit.client.0.smithi153.stdout:6/683: symlink d4/dc/df/d16/d17/le7 0 2024-04-05T03:02:54.406 INFO:tasks.workunit.client.1.smithi203.stdout:4/707: chown da/d29/d2e/d53/d78/fcf 2386440 1 2024-04-05T03:02:54.406 INFO:tasks.workunit.client.1.smithi203.stdout:1/932: fdatasync d7/d14/d9d/dbf/fcd 0 2024-04-05T03:02:54.409 INFO:tasks.workunit.client.1.smithi203.stdout:9/989: read d13/d1b/d6b/d9e/d59/d5f/fb9 [157085,14937] 0 2024-04-05T03:02:54.413 INFO:tasks.workunit.client.1.smithi203.stdout:3/740: dwrite d0/dc/d1a/d8d/dc9/fd3 [0,4194304] 0 2024-04-05T03:02:54.413 INFO:tasks.workunit.client.0.smithi153.stdout:4/703: write da/db/f79 [817603,20566] 0 2024-04-05T03:02:54.414 INFO:tasks.workunit.client.0.smithi153.stdout:6/684: rmdir d4 39 2024-04-05T03:02:54.415 INFO:tasks.workunit.client.0.smithi153.stdout:4/704: write da/db/d20/d35/d38/d76/d90/fd6 [390137,8841] 0 2024-04-05T03:02:54.416 INFO:tasks.workunit.client.0.smithi153.stdout:4/705: chown da/db/d20/d3c/led 11539 1 2024-04-05T03:02:54.419 INFO:tasks.workunit.client.0.smithi153.stdout:6/685: mkdir d4/dc/df/d16/d24/d5e/de8 0 2024-04-05T03:02:54.419 INFO:tasks.workunit.client.0.smithi153.stdout:3/582: sync 2024-04-05T03:02:54.419 INFO:tasks.workunit.client.0.smithi153.stdout:8/470: sync 2024-04-05T03:02:54.420 INFO:tasks.workunit.client.1.smithi203.stdout:5/903: dread d3/d5f/d92/fbc [0,4194304] 0 2024-04-05T03:02:54.420 INFO:tasks.workunit.client.0.smithi153.stdout:4/706: creat da/d2a/d86/ff1 x:0 0 0 2024-04-05T03:02:54.420 INFO:tasks.workunit.client.1.smithi203.stdout:4/708: mkdir da/d29/d2e/d57/d96/ddc/de1 0 2024-04-05T03:02:54.426 INFO:tasks.workunit.client.1.smithi203.stdout:3/741: creat d0/d21/d83/fd7 x:0 0 0 2024-04-05T03:02:54.426 INFO:tasks.workunit.client.1.smithi203.stdout:5/904: dwrite d3/d4/dd6/dfe/f101 [0,4194304] 0 2024-04-05T03:02:54.426 INFO:tasks.workunit.client.1.smithi203.stdout:6/907: write d0/da/d18/d38/d2a/fd8 [628726,50254] 0 2024-04-05T03:02:54.428 INFO:tasks.workunit.client.1.smithi203.stdout:3/742: chown d0/dc/d1a/d1f/da1 598434 1 2024-04-05T03:02:54.428 INFO:tasks.workunit.client.1.smithi203.stdout:4/709: creat da/d71/fe2 x:0 0 0 2024-04-05T03:02:54.430 INFO:tasks.workunit.client.0.smithi153.stdout:3/583: link d1d/f70 d1d/fcf 0 2024-04-05T03:02:54.430 INFO:tasks.workunit.client.0.smithi153.stdout:6/686: link d4/dc/df/l22 d4/dc/df/d16/d36/le9 0 2024-04-05T03:02:54.431 INFO:tasks.workunit.client.1.smithi203.stdout:1/933: link d7/d14/d2b/fd9 d7/d14/d9d/f132 0 2024-04-05T03:02:54.431 INFO:tasks.workunit.client.1.smithi203.stdout:9/990: creat d13/d5c/dd1/d51/dcb/d104/d113/f136 x:0 0 0 2024-04-05T03:02:54.432 INFO:tasks.workunit.client.0.smithi153.stdout:6/687: chown d4/dc/df/d16/d24/d5e/ld6 249093 1 2024-04-05T03:02:54.433 INFO:tasks.workunit.client.0.smithi153.stdout:3/584: write d1d/d32/d93/d3c/d3f/d90/da4/fb3 [9027625,111165] 0 2024-04-05T03:02:54.434 INFO:tasks.workunit.client.1.smithi203.stdout:9/991: readlink d13/d5c/dd1/d51/dcb/l12a 0 2024-04-05T03:02:54.436 INFO:tasks.workunit.client.0.smithi153.stdout:6/688: creat d4/dc/fea x:0 0 0 2024-04-05T03:02:54.437 INFO:tasks.workunit.client.0.smithi153.stdout:3/585: mkdir d1d/d32/d93/d3c/d3f/d90/db6/dd0 0 2024-04-05T03:02:54.439 INFO:tasks.workunit.client.0.smithi153.stdout:3/586: chown d1d/d32/d93/d3c/d3f/d7c/c8c 429003 1 2024-04-05T03:02:54.439 INFO:tasks.workunit.client.0.smithi153.stdout:6/689: rmdir d4/dc/df/d16/d36 39 2024-04-05T03:02:54.439 INFO:tasks.workunit.client.1.smithi203.stdout:3/743: creat d0/dc/d38/d5d/fd8 x:0 0 0 2024-04-05T03:02:54.441 INFO:tasks.workunit.client.0.smithi153.stdout:6/690: read - d4/d38/d4c/fa5 zero size 2024-04-05T03:02:54.442 INFO:tasks.workunit.client.1.smithi203.stdout:6/908: dwrite d0/da/d54/d53/f6a [0,4194304] 0 2024-04-05T03:02:54.443 INFO:tasks.workunit.client.1.smithi203.stdout:1/934: dwrite d7/f1c [0,4194304] 0 2024-04-05T03:02:54.444 INFO:tasks.workunit.client.1.smithi203.stdout:4/710: getdents da/d29/d2e/d53 0 2024-04-05T03:02:54.444 INFO:tasks.workunit.client.0.smithi153.stdout:4/707: write da/de/d56/d6a/d2d/f57 [4100583,120381] 0 2024-04-05T03:02:54.446 INFO:tasks.workunit.client.1.smithi203.stdout:6/909: write d0/da/d18/d38/d86/f5a [2798966,103928] 0 2024-04-05T03:02:54.447 INFO:tasks.workunit.client.0.smithi153.stdout:6/691: mknod d4/dc/df/d16/d33/ceb 0 2024-04-05T03:02:54.449 INFO:tasks.workunit.client.0.smithi153.stdout:6/692: rename d4/d38 to d4/d38/d4c/dec 22 2024-04-05T03:02:54.451 INFO:tasks.workunit.client.0.smithi153.stdout:4/708: unlink da/de/d56/d6a/d2d/f7a 0 2024-04-05T03:02:54.454 INFO:tasks.workunit.client.1.smithi203.stdout:3/744: dwrite d0/dc/d1a/d1d/f5c [0,4194304] 0 2024-04-05T03:02:54.455 INFO:tasks.workunit.client.0.smithi153.stdout:8/471: dwrite d7/d13/d19/d1b/d33/f3a [0,4194304] 0 2024-04-05T03:02:54.455 INFO:tasks.workunit.client.0.smithi153.stdout:6/693: mkdir d4/dc/df/d16/d44/ded 0 2024-04-05T03:02:54.456 INFO:tasks.workunit.client.1.smithi203.stdout:1/935: rename d7/f80 to d7/d14/d83/deb/dff/f133 0 2024-04-05T03:02:54.456 INFO:tasks.workunit.client.1.smithi203.stdout:6/910: dread d0/da/f10 [4194304,4194304] 0 2024-04-05T03:02:54.458 INFO:tasks.workunit.client.1.smithi203.stdout:3/745: dread - d0/dc/d1a/d1f/da1/daf/fc1 zero size 2024-04-05T03:02:54.459 INFO:tasks.workunit.client.1.smithi203.stdout:4/711: unlink da/d3c/l4e 0 2024-04-05T03:02:54.460 INFO:tasks.workunit.client.1.smithi203.stdout:1/936: creat d7/d79/f134 x:0 0 0 2024-04-05T03:02:54.461 INFO:tasks.workunit.client.0.smithi153.stdout:4/709: unlink da/de/d56/d6a/d2d/ddb/fdd 0 2024-04-05T03:02:54.461 INFO:tasks.workunit.client.0.smithi153.stdout:8/472: fsync d7/d13/d19/d1b/d6c/f90 0 2024-04-05T03:02:54.461 INFO:tasks.workunit.client.1.smithi203.stdout:1/937: fdatasync d7/fe 0 2024-04-05T03:02:54.463 INFO:tasks.workunit.client.0.smithi153.stdout:8/473: fdatasync f0 0 2024-04-05T03:02:54.463 INFO:tasks.workunit.client.1.smithi203.stdout:1/938: fsync d7/d14/f10a 0 2024-04-05T03:02:54.466 INFO:tasks.workunit.client.0.smithi153.stdout:8/474: fsync d7/d13/d19/d1b/d21/f69 0 2024-04-05T03:02:54.467 INFO:tasks.workunit.client.1.smithi203.stdout:9/992: truncate d13/d1b/fbd 615200 0 2024-04-05T03:02:54.467 INFO:tasks.workunit.client.0.smithi153.stdout:4/710: creat da/de/d56/d78/d8f/d68/ff2 x:0 0 0 2024-04-05T03:02:54.468 INFO:tasks.workunit.client.1.smithi203.stdout:4/712: write da/d29/d2e/fb1 [1563048,89679] 0 2024-04-05T03:02:54.469 INFO:tasks.workunit.client.0.smithi153.stdout:4/711: fdatasync da/de/d56/d6a/d2d/fa3 0 2024-04-05T03:02:54.470 INFO:tasks.workunit.client.1.smithi203.stdout:3/746: dwrite d0/dc/d38/dca/fd1 [0,4194304] 0 2024-04-05T03:02:54.471 INFO:tasks.workunit.client.0.smithi153.stdout:8/475: creat d7/d2a/f98 x:0 0 0 2024-04-05T03:02:54.471 INFO:tasks.workunit.client.1.smithi203.stdout:4/713: write da/d3c/fd8 [1031357,64291] 0 2024-04-05T03:02:54.474 INFO:tasks.workunit.client.1.smithi203.stdout:9/993: mkdir d13/d1b/d33/d64/dd6/d137 0 2024-04-05T03:02:54.476 INFO:tasks.workunit.client.0.smithi153.stdout:3/587: dread d1d/d27/f96 [0,4194304] 0 2024-04-05T03:02:54.476 INFO:tasks.workunit.client.1.smithi203.stdout:4/714: unlink da/d71/fc2 0 2024-04-05T03:02:54.479 INFO:tasks.workunit.client.1.smithi203.stdout:9/994: creat d13/d5c/dd1/d51/da1/f138 x:0 0 0 2024-04-05T03:02:54.479 INFO:tasks.workunit.client.0.smithi153.stdout:8/476: creat d7/d13/d19/d1b/d4f/d72/f99 x:0 0 0 2024-04-05T03:02:54.480 INFO:tasks.workunit.client.1.smithi203.stdout:9/995: write d13/d1b/d6b/d9e/d77/fed [2424187,117248] 0 2024-04-05T03:02:54.482 INFO:tasks.workunit.client.0.smithi153.stdout:8/477: mkdir d7/d9a 0 2024-04-05T03:02:54.485 INFO:tasks.workunit.client.1.smithi203.stdout:9/996: symlink d13/d1b/d33/d64/dfa/d67/d82/l139 0 2024-04-05T03:02:54.486 INFO:tasks.workunit.client.1.smithi203.stdout:5/905: dread d3/d4/d9e/f9f [0,4194304] 0 2024-04-05T03:02:54.486 INFO:tasks.workunit.client.0.smithi153.stdout:8/478: symlink d7/d13/d19/d1b/d4f/d5f/d54/l9b 0 2024-04-05T03:02:54.486 INFO:tasks.workunit.client.1.smithi203.stdout:9/997: fsync d13/d1b/d33/d64/f131 0 2024-04-05T03:02:54.487 INFO:tasks.workunit.client.1.smithi203.stdout:5/906: chown d3/d5f/d92/fc9 14 1 2024-04-05T03:02:54.488 INFO:tasks.workunit.client.0.smithi153.stdout:8/479: mknod d7/d13/c9c 0 2024-04-05T03:02:54.489 INFO:tasks.workunit.client.0.smithi153.stdout:8/480: write d7/f74 [497751,111048] 0 2024-04-05T03:02:54.489 INFO:tasks.workunit.client.1.smithi203.stdout:9/998: symlink d13/d5c/dd1/d51/dcb/d104/d113/l13a 0 2024-04-05T03:02:54.490 INFO:tasks.workunit.client.1.smithi203.stdout:0/909: sync 2024-04-05T03:02:54.491 INFO:tasks.workunit.client.1.smithi203.stdout:5/907: rmdir d3/d4/d9e/d118 0 2024-04-05T03:02:54.492 INFO:tasks.workunit.client.1.smithi203.stdout:9/999: truncate d13/d1b/d6b/d9e/d59/d5f/d103/f126 581769 0 2024-04-05T03:02:54.507 INFO:tasks.workunit.client.0.smithi153.stdout:3/588: dread d1d/d32/d93/d3c/f7b [0,4194304] 0 2024-04-05T03:02:54.509 INFO:tasks.workunit.client.0.smithi153.stdout:3/589: creat d1d/d32/d93/d3c/d94/fd1 x:0 0 0 2024-04-05T03:02:54.511 INFO:tasks.workunit.client.0.smithi153.stdout:3/590: mknod d1d/d27/cd2 0 2024-04-05T03:02:54.512 INFO:tasks.workunit.client.0.smithi153.stdout:3/591: symlink d1d/d32/d93/d3c/dc5/ld3 0 2024-04-05T03:02:54.522 INFO:tasks.workunit.client.1.smithi203.stdout:6/911: write d0/da/d18/d38/d2a/ffc [419840,78380] 0 2024-04-05T03:02:54.524 INFO:tasks.workunit.client.0.smithi153.stdout:4/712: write da/db/d20/d45/fa6 [201872,104716] 0 2024-04-05T03:02:54.525 INFO:tasks.workunit.client.1.smithi203.stdout:1/939: write d7/d24/d33/d48/d4b/d6b/f75 [3762422,84728] 0 2024-04-05T03:02:54.527 INFO:tasks.workunit.client.1.smithi203.stdout:4/715: write da/d3c/f49 [294643,40378] 0 2024-04-05T03:02:54.529 INFO:tasks.workunit.client.0.smithi153.stdout:8/481: truncate d7/d13/d19/d1b/d4f/d68/f75 4355022 0 2024-04-05T03:02:54.530 INFO:tasks.workunit.client.0.smithi153.stdout:6/694: dwrite d4/fb [4194304,4194304] 0 2024-04-05T03:02:54.530 INFO:tasks.workunit.client.1.smithi203.stdout:4/716: rmdir da/d29/d2e/dc9 39 2024-04-05T03:02:54.530 INFO:tasks.workunit.client.0.smithi153.stdout:3/592: write d1d/d27/d2b/f2d [1879773,109137] 0 2024-04-05T03:02:54.531 INFO:tasks.workunit.client.1.smithi203.stdout:6/912: dread d0/da/d54/d53/f63 [0,4194304] 0 2024-04-05T03:02:54.535 INFO:tasks.workunit.client.1.smithi203.stdout:1/940: rename d7/c46 to d7/d24/d33/d48/d4b/d6b/db9/c135 0 2024-04-05T03:02:54.536 INFO:tasks.workunit.client.1.smithi203.stdout:0/910: truncate f7 1414326 0 2024-04-05T03:02:54.538 INFO:tasks.workunit.client.1.smithi203.stdout:5/908: truncate d3/d1f/f93 10902259 0 2024-04-05T03:02:54.539 INFO:tasks.workunit.client.0.smithi153.stdout:8/482: write d7/d2a/f98 [18655,7720] 0 2024-04-05T03:02:54.540 INFO:tasks.workunit.client.1.smithi203.stdout:6/913: rename d0/da/d18/ff6 to d0/da/d54/d4c/f10a 0 2024-04-05T03:02:54.543 INFO:tasks.workunit.client.1.smithi203.stdout:1/941: creat d7/d14/d2b/d58/d86/d91/da6/df9/d106/d122/f136 x:0 0 0 2024-04-05T03:02:54.543 INFO:tasks.workunit.client.0.smithi153.stdout:4/713: dwrite da/db/d20/d35/fea [0,4194304] 0 2024-04-05T03:02:54.545 INFO:tasks.workunit.client.1.smithi203.stdout:5/909: mknod d3/d1f/d49/d10c/d10d/c130 0 2024-04-05T03:02:54.546 INFO:tasks.workunit.client.1.smithi203.stdout:1/942: write d7/d24/d33/d48/d4b/d6b/fb3 [4179833,318] 0 2024-04-05T03:02:54.547 INFO:tasks.workunit.client.1.smithi203.stdout:0/911: dwrite dd/d17/d1d/d22/d80/f72 [0,4194304] 0 2024-04-05T03:02:54.548 INFO:tasks.workunit.client.1.smithi203.stdout:4/717: dwrite f9 [4194304,4194304] 0 2024-04-05T03:02:54.548 INFO:tasks.workunit.client.1.smithi203.stdout:6/914: creat d0/da/d54/f10b x:0 0 0 2024-04-05T03:02:54.549 INFO:tasks.workunit.client.1.smithi203.stdout:1/943: chown d7/d14/d83/dfb/c119 0 1 2024-04-05T03:02:54.550 INFO:tasks.workunit.client.1.smithi203.stdout:4/718: chown da/f93 107 1 2024-04-05T03:02:54.551 INFO:tasks.workunit.client.1.smithi203.stdout:5/910: mknod d3/d4/dc/d67/db5/dda/df9/c131 0 2024-04-05T03:02:54.554 INFO:tasks.workunit.client.1.smithi203.stdout:0/912: fdatasync dd/d17/d1d/d22/d45/d51/feb 0 2024-04-05T03:02:54.557 INFO:tasks.workunit.client.1.smithi203.stdout:1/944: mkdir d7/d14/d2b/d58/d137 0 2024-04-05T03:02:54.560 INFO:tasks.workunit.client.1.smithi203.stdout:1/945: read d7/d24/f35 [1653789,17997] 0 2024-04-05T03:02:54.570 INFO:tasks.workunit.client.0.smithi153.stdout:3/593: rename d1d/d32/d93/d57 to d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4 0 2024-04-05T03:02:54.570 INFO:tasks.workunit.client.1.smithi203.stdout:4/719: mknod da/d29/d2e/db0/dbe/dce/ce3 0 2024-04-05T03:02:54.570 INFO:tasks.workunit.client.1.smithi203.stdout:4/720: fsync da/f27 0 2024-04-05T03:02:54.571 INFO:tasks.workunit.client.0.smithi153.stdout:8/483: symlink d7/d13/d19/l9d 0 2024-04-05T03:02:54.574 INFO:tasks.workunit.client.0.smithi153.stdout:3/594: dread d1d/d27/d2b/f71 [0,4194304] 0 2024-04-05T03:02:54.574 INFO:tasks.workunit.client.1.smithi203.stdout:0/913: symlink dd/db7/l10e 0 2024-04-05T03:02:54.574 INFO:tasks.workunit.client.0.smithi153.stdout:6/695: link d4/dc/df/d16/d44/fa1 d4/dc/df/d16/d36/d4d/dad/fee 0 2024-04-05T03:02:54.575 INFO:tasks.workunit.client.0.smithi153.stdout:3/595: chown d1d/d32/d93/d3c/f7b 0 1 2024-04-05T03:02:54.577 INFO:tasks.workunit.client.0.smithi153.stdout:8/484: symlink d7/d13/d19/d1b/d4f/d72/d73/l9e 0 2024-04-05T03:02:54.578 INFO:tasks.workunit.client.0.smithi153.stdout:4/714: getdents da/db/d20/d45 0 2024-04-05T03:02:54.578 INFO:tasks.workunit.client.0.smithi153.stdout:4/715: stat da/f1d 0 2024-04-05T03:02:54.579 INFO:tasks.workunit.client.0.smithi153.stdout:6/696: write d4/f15 [1815072,126758] 0 2024-04-05T03:02:54.579 INFO:tasks.workunit.client.1.smithi203.stdout:1/946: mknod d7/d126/c138 0 2024-04-05T03:02:54.582 INFO:tasks.workunit.client.0.smithi153.stdout:8/485: unlink d7/d13/d19/d1b/f97 0 2024-04-05T03:02:54.582 INFO:tasks.workunit.client.0.smithi153.stdout:4/716: rename da/de/d56/d78 to da/de/d56/d78/df3 22 2024-04-05T03:02:54.583 INFO:tasks.workunit.client.0.smithi153.stdout:3/596: dwrite d1d/d27/f64 [0,4194304] 0 2024-04-05T03:02:54.584 INFO:tasks.workunit.client.0.smithi153.stdout:6/697: symlink d4/lef 0 2024-04-05T03:02:54.586 INFO:tasks.workunit.client.0.smithi153.stdout:3/597: stat d1d/d32/d93/d3c/d3f/d90/d92/d99/l9d 0 2024-04-05T03:02:54.586 INFO:tasks.workunit.client.1.smithi203.stdout:5/911: getdents d3/d4/dc 0 2024-04-05T03:02:54.587 INFO:tasks.workunit.client.1.smithi203.stdout:1/947: creat d7/d14/d2b/d58/d137/f139 x:0 0 0 2024-04-05T03:02:54.587 INFO:tasks.workunit.client.1.smithi203.stdout:5/912: chown d3/d1f/d49/d10c/d10d 50 1 2024-04-05T03:02:54.588 INFO:tasks.workunit.client.1.smithi203.stdout:1/948: write d7/f1c [1728041,127935] 0 2024-04-05T03:02:54.589 INFO:tasks.workunit.client.0.smithi153.stdout:6/698: unlink d4/dc/df/d16/d24/d4e/f5a 0 2024-04-05T03:02:54.590 INFO:tasks.workunit.client.1.smithi203.stdout:1/949: fdatasync d7/d14/f29 0 2024-04-05T03:02:54.590 INFO:tasks.workunit.client.1.smithi203.stdout:4/721: creat da/d29/d2e/fe4 x:0 0 0 2024-04-05T03:02:54.591 INFO:tasks.workunit.client.0.smithi153.stdout:6/699: chown d4/dc/df/d16/d36/d4d/d64/fbd 934315962 1 2024-04-05T03:02:54.591 INFO:tasks.workunit.client.1.smithi203.stdout:0/914: dread dd/d17/d1d/d22/d53/d7f/fc7 [0,4194304] 0 2024-04-05T03:02:54.592 INFO:tasks.workunit.client.0.smithi153.stdout:4/717: rename da/de/d56/d78/d8f/cec to da/db/de6/cf4 0 2024-04-05T03:02:54.593 INFO:tasks.workunit.client.0.smithi153.stdout:4/718: chown da/db/fe4 110636 1 2024-04-05T03:02:54.595 INFO:tasks.workunit.client.1.smithi203.stdout:5/913: dread d3/d1f/d59/d60/f76 [0,4194304] 0 2024-04-05T03:02:54.595 INFO:tasks.workunit.client.0.smithi153.stdout:3/598: creat d1d/d32/d93/d3c/d3f/d7c/fd5 x:0 0 0 2024-04-05T03:02:54.596 INFO:tasks.workunit.client.1.smithi203.stdout:4/722: symlink da/d29/d2e/d57/le5 0 2024-04-05T03:02:54.596 INFO:tasks.workunit.client.1.smithi203.stdout:5/914: write d3/d1f/d59/d64/f4f [302586,89975] 0 2024-04-05T03:02:54.597 INFO:tasks.workunit.client.1.smithi203.stdout:4/723: chown da/d29/d2e/l2f 151624238 1 2024-04-05T03:02:54.598 INFO:tasks.workunit.client.1.smithi203.stdout:0/915: fdatasync dd/d17/d1d/d22/d53/d7f/fe3 0 2024-04-05T03:02:54.599 INFO:tasks.workunit.client.0.smithi153.stdout:6/700: dwrite d4/dc/df/d16/d5f/fac [0,4194304] 0 2024-04-05T03:02:54.599 INFO:tasks.workunit.client.1.smithi203.stdout:1/950: getdents d7/d24/d33/d48/d4b/dea/df2 0 2024-04-05T03:02:54.602 INFO:tasks.workunit.client.1.smithi203.stdout:4/724: chown da/d29/d2e/d57/lca 60249 1 2024-04-05T03:02:54.602 INFO:tasks.workunit.client.0.smithi153.stdout:4/719: creat da/db/d20/ff5 x:0 0 0 2024-04-05T03:02:54.605 INFO:tasks.workunit.client.0.smithi153.stdout:6/701: dread - d4/dc/df/d16/d17/d54/faa zero size 2024-04-05T03:02:54.606 INFO:tasks.workunit.client.1.smithi203.stdout:5/915: rename d3/d1f to d3/d4/dd6/dfe/d132 0 2024-04-05T03:02:54.606 INFO:tasks.workunit.client.0.smithi153.stdout:4/720: creat da/db/d20/d35/d41/d67/ff6 x:0 0 0 2024-04-05T03:02:54.607 INFO:tasks.workunit.client.1.smithi203.stdout:6/915: fsync d0/da/d54/d53/fbd 0 2024-04-05T03:02:54.608 INFO:tasks.workunit.client.0.smithi153.stdout:6/702: rmdir d4/dc/df/d16/d5f/da2 39 2024-04-05T03:02:54.610 INFO:tasks.workunit.client.1.smithi203.stdout:1/951: fdatasync d7/d24/d33/d48/d4b/d51/f87 0 2024-04-05T03:02:54.612 INFO:tasks.workunit.client.1.smithi203.stdout:5/916: symlink d3/d4/dd6/dfe/l133 0 2024-04-05T03:02:54.613 INFO:tasks.workunit.client.0.smithi153.stdout:6/703: rename d4/dc/df/d16/d5f/da6 to d4/dc/df/d16/d24/df0 0 2024-04-05T03:02:54.614 INFO:tasks.workunit.client.0.smithi153.stdout:8/486: dread f5 [0,4194304] 0 2024-04-05T03:02:54.614 INFO:tasks.workunit.client.1.smithi203.stdout:6/916: symlink d0/da/d18/d38/d2a/d44/d78/l10c 0 2024-04-05T03:02:54.615 INFO:tasks.workunit.client.0.smithi153.stdout:6/704: write d4/dc/df/d16/d24/fb9 [825412,62561] 0 2024-04-05T03:02:54.616 INFO:tasks.workunit.client.1.smithi203.stdout:1/952: readlink d7/d24/d33/l5d 0 2024-04-05T03:02:54.617 INFO:tasks.workunit.client.0.smithi153.stdout:6/705: write d4/dc/df/d16/d36/d4d/d64/fbd [696848,120603] 0 2024-04-05T03:02:54.619 INFO:tasks.workunit.client.1.smithi203.stdout:5/917: creat d3/d4/d9e/dde/f134 x:0 0 0 2024-04-05T03:02:54.620 INFO:tasks.workunit.client.0.smithi153.stdout:6/706: rmdir d4/dc/df/d16/d36/d52 39 2024-04-05T03:02:54.620 INFO:tasks.workunit.client.1.smithi203.stdout:6/917: mknod d0/da/d18/d38/d2a/d44/d78/da7/c10d 0 2024-04-05T03:02:54.621 INFO:tasks.workunit.client.0.smithi153.stdout:8/487: link d7/d13/d19/d1b/d4f/d72/d73/l82 d7/d46/l9f 0 2024-04-05T03:02:54.621 INFO:tasks.workunit.client.0.smithi153.stdout:6/707: dread - d4/dc/df/d16/d17/d54/faa zero size 2024-04-05T03:02:54.622 INFO:tasks.workunit.client.1.smithi203.stdout:0/916: getdents dd/d17/d1d/d26/d28 0 2024-04-05T03:02:54.623 INFO:tasks.workunit.client.1.smithi203.stdout:5/918: rmdir d3/d4/dd6/dfe/d132/d10e 39 2024-04-05T03:02:54.625 INFO:tasks.workunit.client.1.smithi203.stdout:6/918: symlink d0/da/d18/d92/de4/l10e 0 2024-04-05T03:02:54.625 INFO:tasks.workunit.client.1.smithi203.stdout:5/919: chown d3/d4/dd6/dfe/d132/la4 688377 1 2024-04-05T03:02:54.626 INFO:tasks.workunit.client.1.smithi203.stdout:0/917: mknod dd/d17/d1d/d22/d80/d49/c10f 0 2024-04-05T03:02:54.627 INFO:tasks.workunit.client.1.smithi203.stdout:1/953: getdents d7/d24/d33/d48/d4b/dea/def 0 2024-04-05T03:02:54.628 INFO:tasks.workunit.client.1.smithi203.stdout:5/920: chown d3/d4/dc/f37 1 1 2024-04-05T03:02:54.628 INFO:tasks.workunit.client.1.smithi203.stdout:1/954: fdatasync d7/d126/f12d 0 2024-04-05T03:02:54.630 INFO:tasks.workunit.client.1.smithi203.stdout:0/918: fsync dd/d17/d1d/d22/d53/d7f/fc7 0 2024-04-05T03:02:54.631 INFO:tasks.workunit.client.1.smithi203.stdout:5/921: fdatasync d3/d4/dd6/dfe/d132/d59/d64/f3a 0 2024-04-05T03:02:54.634 INFO:tasks.workunit.client.0.smithi153.stdout:3/599: write d1d/d32/f49 [857098,40010] 0 2024-04-05T03:02:54.634 INFO:tasks.workunit.client.1.smithi203.stdout:4/725: dread da/f13 [0,4194304] 0 2024-04-05T03:02:54.635 INFO:tasks.workunit.client.0.smithi153.stdout:3/600: stat d1d/d32/caf 0 2024-04-05T03:02:54.635 INFO:tasks.workunit.client.1.smithi203.stdout:1/955: dread d7/d14/d9d/fc4 [0,4194304] 0 2024-04-05T03:02:54.637 INFO:tasks.workunit.client.1.smithi203.stdout:5/922: rename d3/l81 to d3/d4/dc/d67/dac/l135 0 2024-04-05T03:02:54.637 INFO:tasks.workunit.client.0.smithi153.stdout:8/488: write d7/d2a/f2d [157128,8426] 0 2024-04-05T03:02:54.637 INFO:tasks.workunit.client.0.smithi153.stdout:3/601: write d1d/d27/f96 [3933161,104702] 0 2024-04-05T03:02:54.638 INFO:tasks.workunit.client.1.smithi203.stdout:5/923: fsync d3/d4/dd6/dfe/d132/d59/d64/f44 0 2024-04-05T03:02:54.639 INFO:tasks.workunit.client.0.smithi153.stdout:8/489: creat d7/d9a/fa0 x:0 0 0 2024-04-05T03:02:54.642 INFO:tasks.workunit.client.1.smithi203.stdout:0/919: rename dd/d17/d1d/d26/f67 to dd/d17/d1d/d26/d8c/f110 0 2024-04-05T03:02:54.642 INFO:tasks.workunit.client.0.smithi153.stdout:8/490: write d7/d85/f4a [928874,44125] 0 2024-04-05T03:02:54.642 INFO:tasks.workunit.client.0.smithi153.stdout:8/491: read - d7/d13/d19/d1b/f89 zero size 2024-04-05T03:02:54.643 INFO:tasks.workunit.client.1.smithi203.stdout:5/924: rmdir d3/d4/dd6/dfe/d132/d49/d10c 39 2024-04-05T03:02:54.644 INFO:tasks.workunit.client.1.smithi203.stdout:6/919: dread d0/da/d18/f9f [0,4194304] 0 2024-04-05T03:02:54.645 INFO:tasks.workunit.client.1.smithi203.stdout:4/726: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:54.645 INFO:tasks.workunit.client.0.smithi153.stdout:8/492: rename f5 to d7/d2a/fa1 0 2024-04-05T03:02:54.646 INFO:tasks.workunit.client.0.smithi153.stdout:8/493: chown d7/d13/d19/f5c 586189 1 2024-04-05T03:02:54.646 INFO:tasks.workunit.client.0.smithi153.stdout:8/494: fdatasync d7/d2a/f44 0 2024-04-05T03:02:54.647 INFO:tasks.workunit.client.0.smithi153.stdout:8/495: chown d7/d13/d19/d1b/d21/l4b 27254 1 2024-04-05T03:02:54.647 INFO:tasks.workunit.client.1.smithi203.stdout:6/920: creat d0/da/d54/d53/f10f x:0 0 0 2024-04-05T03:02:54.649 INFO:tasks.workunit.client.1.smithi203.stdout:4/727: dread da/d29/d2e/f7a [0,4194304] 0 2024-04-05T03:02:54.649 INFO:tasks.workunit.client.1.smithi203.stdout:6/921: chown d0/da/d18/d92/de4 193 1 2024-04-05T03:02:54.651 INFO:tasks.workunit.client.1.smithi203.stdout:0/920: dwrite dd/d17/d1d/d26/d28/ff7 [0,4194304] 0 2024-04-05T03:02:54.653 INFO:tasks.workunit.client.1.smithi203.stdout:6/922: symlink d0/da/l110 0 2024-04-05T03:02:54.655 INFO:tasks.workunit.client.1.smithi203.stdout:0/921: rmdir dd/d17/d1d/d26/d8c/d6d 39 2024-04-05T03:02:54.656 INFO:tasks.workunit.client.1.smithi203.stdout:6/923: symlink d0/db4/l111 0 2024-04-05T03:02:54.657 INFO:tasks.workunit.client.1.smithi203.stdout:4/728: mknod da/d29/d2e/d57/d96/ddc/de0/ce6 0 2024-04-05T03:02:54.663 INFO:tasks.workunit.client.1.smithi203.stdout:0/922: dwrite dd/d17/d1d/d22/d53/d7f/fea [0,4194304] 0 2024-04-05T03:02:54.664 INFO:tasks.workunit.client.1.smithi203.stdout:4/729: getdents da/d29 0 2024-04-05T03:02:54.664 INFO:tasks.workunit.client.1.smithi203.stdout:6/924: write d0/da/d18/d38/d2a/d2c/f46 [1401191,74138] 0 2024-04-05T03:02:54.666 INFO:tasks.workunit.client.1.smithi203.stdout:6/925: chown d0/l29 179 1 2024-04-05T03:02:54.666 INFO:tasks.workunit.client.1.smithi203.stdout:4/730: chown da/l1d 797620 1 2024-04-05T03:02:54.668 INFO:tasks.workunit.client.1.smithi203.stdout:6/926: write d0/da/d54/d4c/f10a [877044,33134] 0 2024-04-05T03:02:54.674 INFO:tasks.workunit.client.1.smithi203.stdout:0/923: rename dd/d17/lb4 to dd/d17/d1d/d22/l111 0 2024-04-05T03:02:54.674 INFO:tasks.workunit.client.1.smithi203.stdout:4/731: write da/d29/f32 [262581,64478] 0 2024-04-05T03:02:54.675 INFO:tasks.workunit.client.1.smithi203.stdout:6/927: write d0/da/d18/d38/d86/f2d [482137,129040] 0 2024-04-05T03:02:54.676 INFO:tasks.workunit.client.1.smithi203.stdout:0/924: write dd/d17/d1d/d22/d53/dec/f102 [464765,62215] 0 2024-04-05T03:02:54.676 INFO:tasks.workunit.client.0.smithi153.stdout:8/496: dread d7/d13/d19/f4e [0,4194304] 0 2024-04-05T03:02:54.677 INFO:tasks.workunit.client.1.smithi203.stdout:6/928: rmdir d0/da/d18/d92/de4 39 2024-04-05T03:02:54.678 INFO:tasks.workunit.client.0.smithi153.stdout:8/497: write d7/d2a/f7b [742659,28128] 0 2024-04-05T03:02:54.680 INFO:tasks.workunit.client.1.smithi203.stdout:4/732: rename da/d3c/d4c/f5c to da/d29/d2e/db0/dbe/dce/dd0/fe7 0 2024-04-05T03:02:54.681 INFO:tasks.workunit.client.1.smithi203.stdout:0/925: getdents dd/d17/d1d/d22/d53/dec 0 2024-04-05T03:02:54.682 INFO:tasks.workunit.client.1.smithi203.stdout:4/733: mknod da/d29/d2e/d8b/ce8 0 2024-04-05T03:02:54.682 INFO:tasks.workunit.client.1.smithi203.stdout:6/929: dread d0/da/d54/d53/fb3 [0,4194304] 0 2024-04-05T03:02:54.683 INFO:tasks.workunit.client.1.smithi203.stdout:6/930: chown d0/da/d54/d4c/da1/db7 6526489 1 2024-04-05T03:02:54.685 INFO:tasks.workunit.client.1.smithi203.stdout:6/931: symlink d0/da/d54/d53/l112 0 2024-04-05T03:02:54.690 INFO:tasks.workunit.client.1.smithi203.stdout:6/932: rename d0/da/d18/d38/d2a/d2c/l66 to d0/da/d54/d4c/d7f/l113 0 2024-04-05T03:02:54.692 INFO:tasks.workunit.client.1.smithi203.stdout:6/933: rmdir d0/da/d18/d38/d2a 39 2024-04-05T03:02:54.694 INFO:tasks.workunit.client.1.smithi203.stdout:6/934: fsync d0/da/d18/d38/d2a/d2c/f46 0 2024-04-05T03:02:54.695 INFO:tasks.workunit.client.1.smithi203.stdout:0/926: dwrite dd/d17/d1d/d22/d53/db0/fda [4194304,4194304] 0 2024-04-05T03:02:54.696 INFO:tasks.workunit.client.1.smithi203.stdout:6/935: write d0/da/d54/fb0 [660205,9462] 0 2024-04-05T03:02:54.787 INFO:tasks.workunit.client.1.smithi203.stdout:0/927: mkdir dd/d17/d112 0 2024-04-05T03:02:54.788 INFO:tasks.workunit.client.1.smithi203.stdout:0/928: read dd/d17/d1d/d22/d53/d61/fe5 [845603,99043] 0 2024-04-05T03:02:54.801 INFO:tasks.workunit.client.1.smithi203.stdout:6/936: truncate d0/da/d54/d4c/da1/db7/fc4 510059 0 2024-04-05T03:02:54.803 INFO:tasks.workunit.client.1.smithi203.stdout:6/937: chown d0/da/d18/d38/d86/c62 141024 1 2024-04-05T03:02:54.816 INFO:tasks.workunit.client.1.smithi203.stdout:0/929: write dd/d17/f79 [774985,64657] 0 2024-04-05T03:02:54.822 INFO:tasks.workunit.client.1.smithi203.stdout:6/938: dwrite d0/da/d54/d82/fd3 [0,4194304] 0 2024-04-05T03:02:54.824 INFO:tasks.workunit.client.1.smithi203.stdout:0/930: dwrite dd/d17/d1d/d22/d53/db0/fda [4194304,4194304] 0 2024-04-05T03:02:54.826 INFO:tasks.workunit.client.1.smithi203.stdout:6/939: creat d0/da/f114 x:0 0 0 2024-04-05T03:02:54.829 INFO:tasks.workunit.client.1.smithi203.stdout:0/931: dread - dd/d17/d1d/d22/d45/dbe/fd2 zero size 2024-04-05T03:02:54.830 INFO:tasks.workunit.client.1.smithi203.stdout:6/940: truncate d0/da/d54/d4c/f10a 1922863 0 2024-04-05T03:02:54.831 INFO:tasks.workunit.client.1.smithi203.stdout:0/932: dread - dd/d17/d1d/d22/d53/d61/f92 zero size 2024-04-05T03:02:54.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/941: dwrite d0/da/f114 [0,4194304] 0 2024-04-05T03:02:54.839 INFO:tasks.workunit.client.1.smithi203.stdout:0/933: dwrite dd/d17/d1d/d22/f3b [0,4194304] 0 2024-04-05T03:02:54.846 INFO:tasks.workunit.client.1.smithi203.stdout:0/934: link dd/d17/d1d/d22/d53/d7f/fc7 dd/d17/d1d/d26/f113 0 2024-04-05T03:02:54.849 INFO:tasks.workunit.client.1.smithi203.stdout:0/935: symlink dd/d17/d1d/d22/d53/dec/df1/l114 0 2024-04-05T03:02:54.850 INFO:tasks.workunit.client.1.smithi203.stdout:0/936: write dd/d17/f48 [1599091,14376] 0 2024-04-05T03:02:54.859 INFO:tasks.workunit.client.1.smithi203.stdout:3/747: sync 2024-04-05T03:02:54.862 INFO:tasks.workunit.client.1.smithi203.stdout:3/748: mkdir d0/d21/d81/dd9 0 2024-04-05T03:02:54.867 INFO:tasks.workunit.client.1.smithi203.stdout:3/749: dwrite d0/dc/d34/f61 [0,4194304] 0 2024-04-05T03:02:54.872 INFO:tasks.workunit.client.1.smithi203.stdout:3/750: link d0/d41/f8e d0/dc/d38/d5d/d66/d80/fda 0 2024-04-05T03:02:54.872 INFO:tasks.workunit.client.1.smithi203.stdout:3/751: rmdir d0/d21 39 2024-04-05T03:02:54.874 INFO:tasks.workunit.client.1.smithi203.stdout:3/752: chown d0/d21/lb4 8321081 1 2024-04-05T03:02:54.875 INFO:tasks.workunit.client.1.smithi203.stdout:3/753: write d0/d69/f88 [4397306,118854] 0 2024-04-05T03:02:54.939 INFO:tasks.workunit.client.0.smithi153.stdout:7/685: sync 2024-04-05T03:02:54.939 INFO:tasks.workunit.client.0.smithi153.stdout:0/514: sync 2024-04-05T03:02:54.942 INFO:tasks.workunit.client.0.smithi153.stdout:7/686: stat d2/d51/d93/l64 0 2024-04-05T03:02:54.943 INFO:tasks.workunit.client.0.smithi153.stdout:0/515: mknod d9/df/c97 0 2024-04-05T03:02:54.945 INFO:tasks.workunit.client.0.smithi153.stdout:0/516: creat d9/db/d42/d43/f98 x:0 0 0 2024-04-05T03:02:54.949 INFO:tasks.workunit.client.0.smithi153.stdout:7/687: rename d2/da/d3a/d41/d7a/f56 to d2/d51/d93/fe4 0 2024-04-05T03:02:54.950 INFO:tasks.workunit.client.0.smithi153.stdout:0/517: mknod d9/df/d1d/d61/c99 0 2024-04-05T03:02:54.955 INFO:tasks.workunit.client.0.smithi153.stdout:0/518: link d9/d14/f23 d9/db/d42/d7e/f9a 0 2024-04-05T03:02:54.957 INFO:tasks.workunit.client.0.smithi153.stdout:7/688: dread d2/da/d11/d39/f63 [0,4194304] 0 2024-04-05T03:02:54.957 INFO:tasks.workunit.client.0.smithi153.stdout:0/519: rmdir d9/db 39 2024-04-05T03:02:54.958 INFO:tasks.workunit.client.0.smithi153.stdout:0/520: stat d9/df/d1d/d61/d8a 0 2024-04-05T03:02:54.958 INFO:tasks.workunit.client.0.smithi153.stdout:7/689: chown d2/d51/f79 62 1 2024-04-05T03:02:54.961 INFO:tasks.workunit.client.0.smithi153.stdout:7/690: mknod d2/d51/d77/ce5 0 2024-04-05T03:02:54.963 INFO:tasks.workunit.client.0.smithi153.stdout:7/691: creat d2/da/d47/d5d/d7c/fe6 x:0 0 0 2024-04-05T03:02:54.966 INFO:tasks.workunit.client.0.smithi153.stdout:7/692: creat d2/d51/d93/fe7 x:0 0 0 2024-04-05T03:02:54.966 INFO:tasks.workunit.client.0.smithi153.stdout:0/521: rmdir d9/db/d42/d43 39 2024-04-05T03:02:54.968 INFO:tasks.workunit.client.0.smithi153.stdout:0/522: fsync d9/d14/f19 0 2024-04-05T03:02:54.971 INFO:tasks.workunit.client.0.smithi153.stdout:0/523: creat d9/d68/f9b x:0 0 0 2024-04-05T03:02:54.973 INFO:tasks.workunit.client.0.smithi153.stdout:7/693: link d2/f32 d2/da/d11/d39/fe8 0 2024-04-05T03:02:54.973 INFO:tasks.workunit.client.1.smithi203.stdout:1/956: sync 2024-04-05T03:02:54.974 INFO:tasks.workunit.client.0.smithi153.stdout:0/524: symlink d9/db/l9c 0 2024-04-05T03:02:54.975 INFO:tasks.workunit.client.0.smithi153.stdout:7/694: symlink d2/da/d3a/d41/ddd/de1/le9 0 2024-04-05T03:02:54.975 INFO:tasks.workunit.client.1.smithi203.stdout:1/957: rmdir d7/d90 39 2024-04-05T03:02:54.976 INFO:tasks.workunit.client.0.smithi153.stdout:7/695: write d2/da/d47/d5d/d7c/f81 [1808738,12868] 0 2024-04-05T03:02:54.978 INFO:tasks.workunit.client.0.smithi153.stdout:7/696: truncate d2/da/d11/d88/da6/fc2 147649 0 2024-04-05T03:02:54.979 INFO:tasks.workunit.client.0.smithi153.stdout:0/525: link d9/db/f4b d9/d14/d67/d6f/f9d 0 2024-04-05T03:02:54.981 INFO:tasks.workunit.client.0.smithi153.stdout:0/526: mkdir d9/d68/d9e 0 2024-04-05T03:02:54.981 INFO:tasks.workunit.client.0.smithi153.stdout:0/527: chown d9/df/d1e/c41 31 1 2024-04-05T03:02:54.982 INFO:tasks.workunit.client.0.smithi153.stdout:7/697: getdents d2/da/d47/d5d/d7c/d8d/d95/dce/dc8 0 2024-04-05T03:02:54.983 INFO:tasks.workunit.client.0.smithi153.stdout:0/528: mknod d9/df/d1e/d45/d62/c9f 0 2024-04-05T03:02:54.984 INFO:tasks.workunit.client.0.smithi153.stdout:0/529: chown d9/d68/f9b 583264 1 2024-04-05T03:02:54.986 INFO:tasks.workunit.client.0.smithi153.stdout:7/698: rename d2/d9b/l91 to d2/da/d11/lea 0 2024-04-05T03:02:54.986 INFO:tasks.workunit.client.0.smithi153.stdout:0/530: creat d9/df/d1e/d45/d62/fa0 x:0 0 0 2024-04-05T03:02:54.988 INFO:tasks.workunit.client.1.smithi203.stdout:1/958: dread d7/d24/d33/d48/d4b/dea/df2/ff5 [4194304,4194304] 0 2024-04-05T03:02:54.990 INFO:tasks.workunit.client.0.smithi153.stdout:0/531: creat d9/db/d42/d43/fa1 x:0 0 0 2024-04-05T03:02:54.990 INFO:tasks.workunit.client.0.smithi153.stdout:7/699: getdents d2/da/d11/d88/da6 0 2024-04-05T03:02:54.993 INFO:tasks.workunit.client.0.smithi153.stdout:0/532: truncate d9/db/d42/d43/f85 1378603 0 2024-04-05T03:02:54.996 INFO:tasks.workunit.client.0.smithi153.stdout:0/533: creat d9/db/d42/d43/fa2 x:0 0 0 2024-04-05T03:02:54.996 INFO:tasks.workunit.client.0.smithi153.stdout:7/700: dwrite d2/da/d3a/d41/f8a [0,4194304] 0 2024-04-05T03:02:54.998 INFO:tasks.workunit.client.0.smithi153.stdout:7/701: readlink d2/da/d11/lea 0 2024-04-05T03:02:55.000 INFO:tasks.workunit.client.0.smithi153.stdout:7/702: write d2/da/d47/d5d/f69 [4808879,35976] 0 2024-04-05T03:02:55.003 INFO:tasks.workunit.client.0.smithi153.stdout:0/534: rmdir d9/db/d42/d43 39 2024-04-05T03:02:55.005 INFO:tasks.workunit.client.0.smithi153.stdout:0/535: dread - d9/db/f44 zero size 2024-04-05T03:02:55.007 INFO:tasks.workunit.client.0.smithi153.stdout:7/703: rename d2/da/d11/d88/la4 to d2/da/d47/d5d/leb 0 2024-04-05T03:02:55.007 INFO:tasks.workunit.client.0.smithi153.stdout:0/536: mknod d9/d14/d67/ca3 0 2024-04-05T03:02:55.008 INFO:tasks.workunit.client.0.smithi153.stdout:7/704: read d2/da/d3a/d41/d7a/f82 [618518,91280] 0 2024-04-05T03:02:55.008 INFO:tasks.workunit.client.1.smithi203.stdout:1/959: dread d7/d24/d33/d48/d4b/d51/fb2 [0,4194304] 0 2024-04-05T03:02:55.008 INFO:tasks.workunit.client.0.smithi153.stdout:7/705: chown d2/da/d3a/d41/d31/f37 3485350 1 2024-04-05T03:02:55.013 INFO:tasks.workunit.client.1.smithi203.stdout:1/960: link d7/d14/d2b/fd9 d7/d14/d83/deb/dff/f13a 0 2024-04-05T03:02:55.013 INFO:tasks.workunit.client.0.smithi153.stdout:2/647: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:8/498: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:9/456: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:4/721: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:1/523: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:5/547: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:3/602: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:6/708: sync 2024-04-05T03:02:55.014 INFO:tasks.workunit.client.0.smithi153.stdout:0/537: dwrite d9/df/d1e/f80 [0,4194304] 0 2024-04-05T03:02:55.015 INFO:tasks.workunit.client.0.smithi153.stdout:7/706: rmdir d2/da/d47/d5d 39 2024-04-05T03:02:55.017 INFO:tasks.workunit.client.0.smithi153.stdout:9/457: creat d1/d3a/fa6 x:0 0 0 2024-04-05T03:02:55.018 INFO:tasks.workunit.client.0.smithi153.stdout:2/648: stat d9/da/d19/d2e/d4f/ld7 0 2024-04-05T03:02:55.018 INFO:tasks.workunit.client.0.smithi153.stdout:8/499: creat d7/d2a/fa2 x:0 0 0 2024-04-05T03:02:55.018 INFO:tasks.workunit.client.0.smithi153.stdout:5/548: dread - d0/d25/d4d/d62/d5f/db2/fb4 zero size 2024-04-05T03:02:55.019 INFO:tasks.workunit.client.0.smithi153.stdout:6/709: chown d4/dc/d7a/cc7 27238 1 2024-04-05T03:02:55.020 INFO:tasks.workunit.client.0.smithi153.stdout:4/722: readlink da/d2a/ld7 0 2024-04-05T03:02:55.021 INFO:tasks.workunit.client.0.smithi153.stdout:0/538: creat d9/d68/fa4 x:0 0 0 2024-04-05T03:02:55.021 INFO:tasks.workunit.client.0.smithi153.stdout:7/707: write d2/da/d3a/fd2 [796438,41579] 0 2024-04-05T03:02:55.022 INFO:tasks.workunit.client.1.smithi203.stdout:1/961: truncate d7/d24/d33/f5b 2062432 0 2024-04-05T03:02:55.022 INFO:tasks.workunit.client.0.smithi153.stdout:3/603: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fa5 10 1 2024-04-05T03:02:55.024 INFO:tasks.workunit.client.0.smithi153.stdout:5/549: mknod d0/d25/d4d/d62/d5f/d86/cc6 0 2024-04-05T03:02:55.025 INFO:tasks.workunit.client.1.smithi203.stdout:1/962: read d7/d24/d33/d48/fa1 [2993890,13537] 0 2024-04-05T03:02:55.025 INFO:tasks.workunit.client.1.smithi203.stdout:4/734: sync 2024-04-05T03:02:55.025 INFO:tasks.workunit.client.1.smithi203.stdout:5/925: sync 2024-04-05T03:02:55.025 INFO:tasks.workunit.client.1.smithi203.stdout:6/942: sync 2024-04-05T03:02:55.027 INFO:tasks.workunit.client.0.smithi153.stdout:8/500: rmdir d7/d13/d19/d1b/d4f/d5f 39 2024-04-05T03:02:55.027 INFO:tasks.workunit.client.0.smithi153.stdout:4/723: dread da/db/d20/d45/f6b [0,4194304] 0 2024-04-05T03:02:55.027 INFO:tasks.workunit.client.1.smithi203.stdout:4/735: write da/d3c/fd8 [13529,105424] 0 2024-04-05T03:02:55.029 INFO:tasks.workunit.client.0.smithi153.stdout:0/539: creat d9/df/d1d/d61/d8a/fa5 x:0 0 0 2024-04-05T03:02:55.029 INFO:tasks.workunit.client.0.smithi153.stdout:2/649: symlink d9/da/d19/d2e/d7e/d53/ldd 0 2024-04-05T03:02:55.031 INFO:tasks.workunit.client.1.smithi203.stdout:1/963: mknod d7/d24/d11f/c13b 0 2024-04-05T03:02:55.031 INFO:tasks.workunit.client.0.smithi153.stdout:6/710: dread d4/dc/df/d16/d24/d4e/fa9 [0,4194304] 0 2024-04-05T03:02:55.033 INFO:tasks.workunit.client.1.smithi203.stdout:6/943: dread d0/da/d54/f45 [4194304,4194304] 0 2024-04-05T03:02:55.033 INFO:tasks.workunit.client.1.smithi203.stdout:5/926: rename d3/d4/dd6/dfe/d132/d59/d64/d12/leb to d3/l136 0 2024-04-05T03:02:55.034 INFO:tasks.workunit.client.0.smithi153.stdout:9/458: dread d1/d9/d1b/f6a [0,4194304] 0 2024-04-05T03:02:55.034 INFO:tasks.workunit.client.0.smithi153.stdout:3/604: write d1d/f70 [133210,82488] 0 2024-04-05T03:02:55.035 INFO:tasks.workunit.client.1.smithi203.stdout:4/736: mknod da/d29/d2e/d57/ce9 0 2024-04-05T03:02:55.035 INFO:tasks.workunit.client.0.smithi153.stdout:8/501: creat d7/d13/fa3 x:0 0 0 2024-04-05T03:02:55.035 INFO:tasks.workunit.client.1.smithi203.stdout:1/964: truncate d7/d79/f88 654506 0 2024-04-05T03:02:55.037 INFO:tasks.workunit.client.0.smithi153.stdout:2/650: write d9/d68/db9/dd8/f33 [5801618,78905] 0 2024-04-05T03:02:55.039 INFO:tasks.workunit.client.0.smithi153.stdout:9/459: fsync d1/d10/f39 0 2024-04-05T03:02:55.040 INFO:tasks.workunit.client.0.smithi153.stdout:0/540: creat d9/db/d42/d7e/fa6 x:0 0 0 2024-04-05T03:02:55.042 INFO:tasks.workunit.client.0.smithi153.stdout:1/524: dwrite d5/d8/d20/d4b/d18/d16/d48/d96/fa0 [0,4194304] 0 2024-04-05T03:02:55.043 INFO:tasks.workunit.client.1.smithi203.stdout:4/737: readlink da/l16 0 2024-04-05T03:02:55.043 INFO:tasks.workunit.client.0.smithi153.stdout:8/502: unlink f0 0 2024-04-05T03:02:55.045 INFO:tasks.workunit.client.0.smithi153.stdout:4/724: mkdir da/db/d20/d35/d38/df7 0 2024-04-05T03:02:55.045 INFO:tasks.workunit.client.1.smithi203.stdout:1/965: mkdir d7/d14/d2b/d58/d7a/d13c 0 2024-04-05T03:02:55.046 INFO:tasks.workunit.client.0.smithi153.stdout:4/725: dread - da/d2a/fcd zero size 2024-04-05T03:02:55.047 INFO:tasks.workunit.client.1.smithi203.stdout:1/966: stat d7/d14/d2b/d58/d7a/f8a 0 2024-04-05T03:02:55.048 INFO:tasks.workunit.client.1.smithi203.stdout:6/944: unlink d0/da/d18/d38/d2a/lab 0 2024-04-05T03:02:55.048 INFO:tasks.workunit.client.0.smithi153.stdout:2/651: chown d9/da/d94/l95 85 1 2024-04-05T03:02:55.049 INFO:tasks.workunit.client.1.smithi203.stdout:5/927: rmdir d3/d4 39 2024-04-05T03:02:55.051 INFO:tasks.workunit.client.0.smithi153.stdout:8/503: creat d7/d13/d19/d1b/d21/fa4 x:0 0 0 2024-04-05T03:02:55.053 INFO:tasks.workunit.client.0.smithi153.stdout:8/504: write d7/d13/d19/f8f [513870,130723] 0 2024-04-05T03:02:55.053 INFO:tasks.workunit.client.1.smithi203.stdout:4/738: dwrite da/d29/d2e/db0/dcb/d80/fa6 [0,4194304] 0 2024-04-05T03:02:55.054 INFO:tasks.workunit.client.1.smithi203.stdout:4/739: chown da/d29/f9e 14020 1 2024-04-05T03:02:55.056 INFO:tasks.workunit.client.1.smithi203.stdout:6/945: unlink d0/da/d18/d38/d86/fe6 0 2024-04-05T03:02:55.057 INFO:tasks.workunit.client.0.smithi153.stdout:1/525: mkdir d5/d8/d20/d4b/d18/d16/d45/d68/da4 0 2024-04-05T03:02:55.058 INFO:tasks.workunit.client.0.smithi153.stdout:2/652: dwrite d9/da/d36/f81 [0,4194304] 0 2024-04-05T03:02:55.059 INFO:tasks.workunit.client.0.smithi153.stdout:5/550: dread d0/f3c [0,4194304] 0 2024-04-05T03:02:55.061 INFO:tasks.workunit.client.0.smithi153.stdout:0/541: fsync d9/df/d1e/d45/d62/f7d 0 2024-04-05T03:02:55.062 INFO:tasks.workunit.client.1.smithi203.stdout:5/928: unlink d3/d4/dd6/dfe/d132/le5 0 2024-04-05T03:02:55.065 INFO:tasks.workunit.client.0.smithi153.stdout:3/605: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fd6 x:0 0 0 2024-04-05T03:02:55.066 INFO:tasks.workunit.client.0.smithi153.stdout:1/526: mknod d5/d8/d20/d4b/d18/d16/d33/d83/ca5 0 2024-04-05T03:02:55.067 INFO:tasks.workunit.client.1.smithi203.stdout:5/929: mkdir d3/d4/dd6/dfe/d132/d137 0 2024-04-05T03:02:55.068 INFO:tasks.workunit.client.1.smithi203.stdout:6/946: dwrite d0/da/d54/d53/f59 [0,4194304] 0 2024-04-05T03:02:55.069 INFO:tasks.workunit.client.0.smithi153.stdout:6/711: dwrite d4/d38/d4c/fa5 [0,4194304] 0 2024-04-05T03:02:55.070 INFO:tasks.workunit.client.0.smithi153.stdout:7/708: dread d2/d51/d93/f4b [0,4194304] 0 2024-04-05T03:02:55.071 INFO:tasks.workunit.client.0.smithi153.stdout:1/527: chown d5/d8/d20/d4b/d18/d16/d48/f99 6 1 2024-04-05T03:02:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:54 smithi153 ceph-mon[29381]: pgmap v12: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 122 MiB/s rd, 357 MiB/s wr, 675 op/s 2024-04-05T03:02:55.072 INFO:tasks.workunit.client.0.smithi153.stdout:0/542: creat d9/df/d1e/fa7 x:0 0 0 2024-04-05T03:02:55.073 INFO:tasks.workunit.client.0.smithi153.stdout:0/543: read d9/f30 [1488013,109997] 0 2024-04-05T03:02:55.075 INFO:tasks.workunit.client.1.smithi203.stdout:6/947: dread d0/da/d54/d4c/d7f/dcc/ffb [0,4194304] 0 2024-04-05T03:02:55.081 INFO:tasks.workunit.client.1.smithi203.stdout:5/930: dwrite d3/d4/dc/d67/db5/dda/fef [0,4194304] 0 2024-04-05T03:02:55.084 INFO:tasks.workunit.client.1.smithi203.stdout:1/967: dread d7/d24/f66 [0,4194304] 0 2024-04-05T03:02:55.086 INFO:tasks.workunit.client.1.smithi203.stdout:5/931: fsync d3/f32 0 2024-04-05T03:02:55.090 INFO:tasks.workunit.client.0.smithi153.stdout:7/709: creat d2/da/d47/d5d/d7c/fec x:0 0 0 2024-04-05T03:02:55.090 INFO:tasks.workunit.client.0.smithi153.stdout:3/606: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fd7 x:0 0 0 2024-04-05T03:02:55.093 INFO:tasks.workunit.client.1.smithi203.stdout:1/968: rename d7/d24/d33/c4a to d7/d24/d11f/c13d 0 2024-04-05T03:02:55.094 INFO:tasks.workunit.client.1.smithi203.stdout:0/937: sync 2024-04-05T03:02:55.094 INFO:tasks.workunit.client.1.smithi203.stdout:3/754: sync 2024-04-05T03:02:55.095 INFO:tasks.workunit.client.1.smithi203.stdout:6/948: dwrite d0/da/d54/d4c/da1/db7/fa2 [8388608,4194304] 0 2024-04-05T03:02:55.095 INFO:tasks.workunit.client.0.smithi153.stdout:7/710: symlink d2/led 0 2024-04-05T03:02:55.098 INFO:tasks.workunit.client.0.smithi153.stdout:3/607: dwrite d1d/d27/f64 [0,4194304] 0 2024-04-05T03:02:55.099 INFO:tasks.workunit.client.1.smithi203.stdout:6/949: dread - d0/da/d18/d38/d2a/d44/fc0 zero size 2024-04-05T03:02:55.101 INFO:tasks.workunit.client.0.smithi153.stdout:7/711: mknod d2/d9b/cee 0 2024-04-05T03:02:55.102 INFO:tasks.workunit.client.1.smithi203.stdout:3/755: readlink d0/dc/d1a/d1f/lad 0 2024-04-05T03:02:55.104 INFO:tasks.workunit.client.1.smithi203.stdout:4/740: dread da/d29/f5b [0,4194304] 0 2024-04-05T03:02:55.105 INFO:tasks.workunit.client.0.smithi153.stdout:3/608: creat d1d/d32/d93/d3c/d94/fd8 x:0 0 0 2024-04-05T03:02:55.107 INFO:tasks.workunit.client.0.smithi153.stdout:3/609: write d1d/f21 [5357584,52836] 0 2024-04-05T03:02:55.107 INFO:tasks.workunit.client.0.smithi153.stdout:7/712: creat d2/da/d3a/d41/ddd/fef x:0 0 0 2024-04-05T03:02:55.108 INFO:tasks.workunit.client.1.smithi203.stdout:0/938: rename dd/d17/d1d/d26 to dd/d17/d115 0 2024-04-05T03:02:55.110 INFO:tasks.workunit.client.1.smithi203.stdout:6/950: dwrite d0/da/d54/f101 [0,4194304] 0 2024-04-05T03:02:55.111 INFO:tasks.workunit.client.1.smithi203.stdout:5/932: dread d3/d4/dd6/dfe/d132/d59/d64/f3a [0,4194304] 0 2024-04-05T03:02:55.112 INFO:tasks.workunit.client.0.smithi153.stdout:5/551: dread d0/d25/d4d/f66 [0,4194304] 0 2024-04-05T03:02:55.116 INFO:tasks.workunit.client.1.smithi203.stdout:3/756: rename d0/dc/d1a/d1f/c46 to d0/dc/d1a/d1f/da1/daf/cdb 0 2024-04-05T03:02:55.117 INFO:tasks.workunit.client.1.smithi203.stdout:3/757: chown d0/dc/d1a/d1f/da1 325334356 1 2024-04-05T03:02:55.118 INFO:tasks.workunit.client.0.smithi153.stdout:9/460: truncate d1/d10/d16/d22/f3e 1009399 0 2024-04-05T03:02:55.118 INFO:tasks.workunit.client.1.smithi203.stdout:6/951: creat d0/f115 x:0 0 0 2024-04-05T03:02:55.121 INFO:tasks.workunit.client.0.smithi153.stdout:0/544: write d9/d14/f19 [217445,31581] 0 2024-04-05T03:02:55.121 INFO:tasks.workunit.client.0.smithi153.stdout:5/552: dwrite d0/d9/f1e [0,4194304] 0 2024-04-05T03:02:55.123 INFO:tasks.workunit.client.1.smithi203.stdout:4/741: rename da/d3c/d4c/f73 to da/d3c/d4c/d99/fea 0 2024-04-05T03:02:55.123 INFO:tasks.workunit.client.0.smithi153.stdout:5/553: write d0/d4e/fa5 [399559,89309] 0 2024-04-05T03:02:55.124 INFO:tasks.workunit.client.0.smithi153.stdout:3/610: creat d1d/d32/d93/d3c/fd9 x:0 0 0 2024-04-05T03:02:55.124 INFO:tasks.workunit.client.0.smithi153.stdout:1/528: write d5/d8/d20/d4b/d18/d16/f5b [698809,88721] 0 2024-04-05T03:02:55.126 INFO:tasks.workunit.client.1.smithi203.stdout:3/758: dwrite d0/d21/d81/f59 [0,4194304] 0 2024-04-05T03:02:55.127 INFO:tasks.workunit.client.0.smithi153.stdout:9/461: creat d1/d10/d16/d22/d2c/d68/fa7 x:0 0 0 2024-04-05T03:02:55.128 INFO:tasks.workunit.client.1.smithi203.stdout:6/952: creat d0/da/d54/d4c/f116 x:0 0 0 2024-04-05T03:02:55.129 INFO:tasks.workunit.client.0.smithi153.stdout:6/712: truncate d4/dc/df/d16/d5f/f7c 3097394 0 2024-04-05T03:02:55.130 INFO:tasks.workunit.client.0.smithi153.stdout:5/554: readlink d0/d25/d4d/d62/dbb/lbd 0 2024-04-05T03:02:55.130 INFO:tasks.workunit.client.1.smithi203.stdout:3/759: write d0/dc/d38/dca/fd1 [263439,50488] 0 2024-04-05T03:02:55.130 INFO:tasks.workunit.client.0.smithi153.stdout:9/462: write d1/d3a/d91/f9f [861295,110903] 0 2024-04-05T03:02:55.130 INFO:tasks.workunit.client.0.smithi153.stdout:0/545: read d9/df/d1e/f54 [2018924,89711] 0 2024-04-05T03:02:55.132 INFO:tasks.workunit.client.1.smithi203.stdout:6/953: write d0/da/d18/d38/d2a/ffc [215171,32870] 0 2024-04-05T03:02:55.132 INFO:tasks.workunit.client.0.smithi153.stdout:5/555: readlink d0/d25/d4d/d62/d37/l57 0 2024-04-05T03:02:55.134 INFO:tasks.workunit.client.1.smithi203.stdout:4/742: dwrite da/d29/d2e/f35 [0,4194304] 0 2024-04-05T03:02:55.138 INFO:tasks.workunit.client.1.smithi203.stdout:4/743: write da/d29/d2e/d53/f97 [1065244,55860] 0 2024-04-05T03:02:55.138 INFO:tasks.workunit.client.1.smithi203.stdout:5/933: rename d3/d4/dc/d67/dac/l135 to d3/d4/dd6/d110/l138 0 2024-04-05T03:02:55.140 INFO:tasks.workunit.client.0.smithi153.stdout:6/713: dwrite d4/dc/d7a/da8/fca [0,4194304] 0 2024-04-05T03:02:55.140 INFO:tasks.workunit.client.0.smithi153.stdout:1/529: mknod d5/ca6 0 2024-04-05T03:02:55.141 INFO:tasks.workunit.client.1.smithi203.stdout:3/760: mkdir d0/dc/d1a/d1f/da1/daf/ddc 0 2024-04-05T03:02:55.144 INFO:tasks.workunit.client.0.smithi153.stdout:3/611: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/fda x:0 0 0 2024-04-05T03:02:55.144 INFO:tasks.workunit.client.1.smithi203.stdout:6/954: mknod d0/da/d18/d38/d86/c117 0 2024-04-05T03:02:55.145 INFO:tasks.workunit.client.1.smithi203.stdout:1/969: write d7/d14/d83/fb4 [610890,39763] 0 2024-04-05T03:02:55.145 INFO:tasks.workunit.client.0.smithi153.stdout:7/713: dwrite d2/da/d47/fdb [0,4194304] 0 2024-04-05T03:02:55.146 INFO:tasks.workunit.client.1.smithi203.stdout:6/955: chown d0/da/d54/d4c/da1/db7/d71/le1 365644 1 2024-04-05T03:02:55.148 INFO:tasks.workunit.client.0.smithi153.stdout:5/556: mkdir d0/d25/d4d/d9c/dc7 0 2024-04-05T03:02:55.153 INFO:tasks.workunit.client.0.smithi153.stdout:0/546: symlink d9/df/d1e/d84/la8 0 2024-04-05T03:02:55.155 INFO:tasks.workunit.client.1.smithi203.stdout:6/956: rmdir d0/da/d18/d38/d2a 39 2024-04-05T03:02:55.158 INFO:tasks.workunit.client.0.smithi153.stdout:9/463: dwrite d1/d9/d1b/f75 [0,4194304] 0 2024-04-05T03:02:55.158 INFO:tasks.workunit.client.0.smithi153.stdout:6/714: creat d4/dc/df/d16/d36/d4d/dad/ff1 x:0 0 0 2024-04-05T03:02:55.161 INFO:tasks.workunit.client.0.smithi153.stdout:3/612: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f97 [78731,108061] 0 2024-04-05T03:02:55.163 INFO:tasks.workunit.client.0.smithi153.stdout:0/547: creat d9/fa9 x:0 0 0 2024-04-05T03:02:55.164 INFO:tasks.workunit.client.0.smithi153.stdout:3/613: stat d1d/d32/d93/d3c/fd9 0 2024-04-05T03:02:55.165 INFO:tasks.workunit.client.1.smithi203.stdout:0/939: readlink dd/d17/d1d/d22/d80/d49/l7c 0 2024-04-05T03:02:55.166 INFO:tasks.workunit.client.1.smithi203.stdout:3/761: rename d0/f5 to d0/fdd 0 2024-04-05T03:02:55.168 INFO:tasks.workunit.client.0.smithi153.stdout:5/557: dwrite d0/d25/d4d/d62/d37/fa6 [0,4194304] 0 2024-04-05T03:02:55.169 INFO:tasks.workunit.client.0.smithi153.stdout:7/714: link d2/da/d11/f2b d2/da/d3a/d41/d7a/d54/ff0 0 2024-04-05T03:02:55.172 INFO:tasks.workunit.client.1.smithi203.stdout:1/970: link d7/d14/f29 d7/d24/d33/d48/d4b/d6b/db9/f13e 0 2024-04-05T03:02:55.172 INFO:tasks.workunit.client.1.smithi203.stdout:6/957: dwrite d0/da/d18/d38/d2a/fd8 [0,4194304] 0 2024-04-05T03:02:55.174 INFO:tasks.workunit.client.1.smithi203.stdout:1/971: chown d7/d24/d33/d48/d4b/d6b/db9/lfa 45739 1 2024-04-05T03:02:55.174 INFO:tasks.workunit.client.1.smithi203.stdout:1/972: chown d7/d79/caa 7481374 1 2024-04-05T03:02:55.175 INFO:tasks.workunit.client.1.smithi203.stdout:1/973: chown d7/d24/d33/d48/d4b/d51/fe2 3569901 1 2024-04-05T03:02:55.177 INFO:tasks.workunit.client.1.smithi203.stdout:1/974: truncate d7/d14/d9d/dbf/f10d 1043477 0 2024-04-05T03:02:55.178 INFO:tasks.workunit.client.1.smithi203.stdout:0/940: mkdir dd/d17/d1d/d22/d45/dbe/de1/d116 0 2024-04-05T03:02:55.179 INFO:tasks.workunit.client.1.smithi203.stdout:5/934: write d3/d4/dd6/dfe/d132/d59/d60/f74 [1648156,16389] 0 2024-04-05T03:02:55.179 INFO:tasks.workunit.client.0.smithi153.stdout:5/558: write d0/f27 [2048087,129683] 0 2024-04-05T03:02:55.179 INFO:tasks.workunit.client.0.smithi153.stdout:7/715: unlink d2/da/l21 0 2024-04-05T03:02:55.182 INFO:tasks.workunit.client.0.smithi153.stdout:6/715: getdents d4/dc/df/d16/d17 0 2024-04-05T03:02:55.184 INFO:tasks.workunit.client.1.smithi203.stdout:6/958: mknod d0/da/d54/d4c/da1/db7/c118 0 2024-04-05T03:02:55.184 INFO:tasks.workunit.client.1.smithi203.stdout:4/744: dwrite da/d29/d2e/d53/f68 [0,4194304] 0 2024-04-05T03:02:55.186 INFO:tasks.workunit.client.0.smithi153.stdout:3/614: dwrite d1d/d32/d93/d3c/d3f/d9c/fc4 [0,4194304] 0 2024-04-05T03:02:55.188 INFO:tasks.workunit.client.0.smithi153.stdout:6/716: fsync d4/dc/df/f26 0 2024-04-05T03:02:55.189 INFO:tasks.workunit.client.0.smithi153.stdout:6/717: write d4/dc/df/dcb/fdc [1042976,111219] 0 2024-04-05T03:02:55.189 INFO:tasks.workunit.client.1.smithi203.stdout:3/762: rename d0/dc/d1a/d8d/d96/f51 to d0/fde 0 2024-04-05T03:02:55.190 INFO:tasks.workunit.client.0.smithi153.stdout:7/716: creat d2/da/d3a/d41/ff1 x:0 0 0 2024-04-05T03:02:55.195 INFO:tasks.workunit.client.0.smithi153.stdout:3/615: mkdir d1d/d32/d93/d3c/d3f/d90/da4/ddb 0 2024-04-05T03:02:55.196 INFO:tasks.workunit.client.1.smithi203.stdout:4/745: write da/d3c/f43 [3189183,96629] 0 2024-04-05T03:02:55.198 INFO:tasks.workunit.client.1.smithi203.stdout:5/935: rename d3/d4/dd6/dfe/d132/d59/d64/f3a to d3/d4/dd6/d110/f139 0 2024-04-05T03:02:55.198 INFO:tasks.workunit.client.1.smithi203.stdout:4/746: dread - da/d29/d2e/d53/fa4 zero size 2024-04-05T03:02:55.201 INFO:tasks.workunit.client.0.smithi153.stdout:7/717: mkdir d2/da/d47/d5d/d7c/d8d/df2 0 2024-04-05T03:02:55.204 INFO:tasks.workunit.client.0.smithi153.stdout:6/718: mkdir d4/dc/d7a/dd2/df2 0 2024-04-05T03:02:55.204 INFO:tasks.workunit.client.1.smithi203.stdout:1/975: getdents d7/d14/d2b/d58 0 2024-04-05T03:02:55.205 INFO:tasks.workunit.client.1.smithi203.stdout:0/941: link dd/d17/d1d/d22/d53/d61/f92 dd/db7/f117 0 2024-04-05T03:02:55.205 INFO:tasks.workunit.client.1.smithi203.stdout:3/763: mkdir d0/dc/d1a/ddf 0 2024-04-05T03:02:55.208 INFO:tasks.workunit.client.1.smithi203.stdout:3/764: readlink d0/d21/lb4 0 2024-04-05T03:02:55.210 INFO:tasks.workunit.client.1.smithi203.stdout:1/976: write d7/d24/d33/d48/d4b/d6b/fb3 [2908625,64682] 0 2024-04-05T03:02:55.210 INFO:tasks.workunit.client.0.smithi153.stdout:7/718: dwrite d2/da/d47/fbc [0,4194304] 0 2024-04-05T03:02:55.212 INFO:tasks.workunit.client.0.smithi153.stdout:3/616: rename d1d/d32/d93/d3c/c88 to d1d/cdc 0 2024-04-05T03:02:55.213 INFO:tasks.workunit.client.1.smithi203.stdout:6/959: dwrite d0/da/d18/f30 [0,4194304] 0 2024-04-05T03:02:55.214 INFO:tasks.workunit.client.1.smithi203.stdout:6/960: chown d0/db4/lc3 1 1 2024-04-05T03:02:55.214 INFO:tasks.workunit.client.0.smithi153.stdout:0/548: dwrite d9/db/f66 [0,4194304] 0 2024-04-05T03:02:55.214 INFO:tasks.workunit.client.0.smithi153.stdout:9/464: dread d1/d10/d36/f60 [0,4194304] 0 2024-04-05T03:02:55.216 INFO:tasks.workunit.client.1.smithi203.stdout:6/961: write d0/da/d18/d38/d2a/d44/ffa [114386,55416] 0 2024-04-05T03:02:55.219 INFO:tasks.workunit.client.1.smithi203.stdout:3/765: readlink d0/dc/d1a/d76/d35/l4e 0 2024-04-05T03:02:55.220 INFO:tasks.workunit.client.1.smithi203.stdout:5/936: creat d3/d4/dd6/dfe/d132/d49/f13a x:0 0 0 2024-04-05T03:02:55.224 INFO:tasks.workunit.client.1.smithi203.stdout:1/977: mkdir d7/d14/d13f 0 2024-04-05T03:02:55.224 INFO:tasks.workunit.client.0.smithi153.stdout:3/617: mknod d1d/d32/d93/d3c/dc5/cdd 0 2024-04-05T03:02:55.224 INFO:tasks.workunit.client.0.smithi153.stdout:0/549: creat d9/db/d42/d7e/faa x:0 0 0 2024-04-05T03:02:55.226 INFO:tasks.workunit.client.0.smithi153.stdout:9/465: chown d1/d10/l99 75 1 2024-04-05T03:02:55.228 INFO:tasks.workunit.client.1.smithi203.stdout:5/937: chown d3/d4/d9e/f9f 892353085 1 2024-04-05T03:02:55.230 INFO:tasks.workunit.client.0.smithi153.stdout:7/719: dwrite d2/d9b/fc5 [0,4194304] 0 2024-04-05T03:02:55.232 INFO:tasks.workunit.client.1.smithi203.stdout:6/962: creat d0/da/d18/d38/f119 x:0 0 0 2024-04-05T03:02:55.233 INFO:tasks.workunit.client.0.smithi153.stdout:1/530: dread d5/d8/d20/d4b/d18/d16/d45/f4a [0,4194304] 0 2024-04-05T03:02:55.237 INFO:tasks.workunit.client.0.smithi153.stdout:5/559: truncate d0/d9/d22/f41 2649693 0 2024-04-05T03:02:55.238 INFO:tasks.workunit.client.1.smithi203.stdout:6/963: rename d0/da/d18/d38/d2a to d0/da/d54/d4c/daf/d11a 0 2024-04-05T03:02:55.238 INFO:tasks.workunit.client.0.smithi153.stdout:0/550: truncate d9/df/d1e/f32 1985684 0 2024-04-05T03:02:55.240 INFO:tasks.workunit.client.0.smithi153.stdout:9/466: truncate d1/d10/f5f 148299 0 2024-04-05T03:02:55.240 INFO:tasks.workunit.client.0.smithi153.stdout:5/560: chown d0/d25/d4d/d62/d5f/db2 1195 1 2024-04-05T03:02:55.241 INFO:tasks.workunit.client.0.smithi153.stdout:7/720: symlink d2/da/d3a/d41/ddd/de1/lf3 0 2024-04-05T03:02:55.242 INFO:tasks.workunit.client.0.smithi153.stdout:3/618: dwrite d1d/d32/d93/d3c/d3f/d7c/fd5 [0,4194304] 0 2024-04-05T03:02:55.244 INFO:tasks.workunit.client.0.smithi153.stdout:7/721: chown d2/da/d3a/d41/d7a/d54/la0 1805 1 2024-04-05T03:02:55.246 INFO:tasks.workunit.client.0.smithi153.stdout:5/561: fdatasync d0/d2c/f2d 0 2024-04-05T03:02:55.247 INFO:tasks.workunit.client.1.smithi203.stdout:6/964: rename d0/da/d18/d38/d86/cf7 to d0/da/d18/d92/de4/c11b 0 2024-04-05T03:02:55.248 INFO:tasks.workunit.client.0.smithi153.stdout:9/467: chown d1/d10/d16/f23 14114094 1 2024-04-05T03:02:55.250 INFO:tasks.workunit.client.0.smithi153.stdout:3/619: rename d1d/f70 to d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/fde 0 2024-04-05T03:02:55.251 INFO:tasks.workunit.client.0.smithi153.stdout:0/551: dread d9/db/d42/d43/f82 [0,4194304] 0 2024-04-05T03:02:55.255 INFO:tasks.workunit.client.0.smithi153.stdout:5/562: unlink d0/d25/d4d/d62/f9d 0 2024-04-05T03:02:55.257 INFO:tasks.workunit.client.0.smithi153.stdout:7/722: creat d2/da/d47/ff4 x:0 0 0 2024-04-05T03:02:55.258 INFO:tasks.workunit.client.0.smithi153.stdout:9/468: dwrite d1/d10/d16/d22/d2c/d68/d81/f76 [0,4194304] 0 2024-04-05T03:02:55.258 INFO:tasks.workunit.client.1.smithi203.stdout:4/747: dread da/d29/d2e/f3e [0,4194304] 0 2024-04-05T03:02:55.259 INFO:tasks.workunit.client.0.smithi153.stdout:7/723: chown d2/da/d3a 645287 1 2024-04-05T03:02:55.259 INFO:tasks.workunit.client.0.smithi153.stdout:0/552: rmdir d9/df/d1d/d61 39 2024-04-05T03:02:55.262 INFO:tasks.workunit.client.1.smithi203.stdout:4/748: rename da/d3c/d4c/l67 to da/d3c/d4c/d99/leb 0 2024-04-05T03:02:55.264 INFO:tasks.workunit.client.0.smithi153.stdout:3/620: mkdir d1d/ddf 0 2024-04-05T03:02:55.265 INFO:tasks.workunit.client.0.smithi153.stdout:5/563: mknod d0/d25/d4d/d62/d5f/d86/cc8 0 2024-04-05T03:02:55.265 INFO:tasks.workunit.client.0.smithi153.stdout:1/531: dread d5/d8/d20/d4b/d18/d16/f36 [0,4194304] 0 2024-04-05T03:02:55.266 INFO:tasks.workunit.client.1.smithi203.stdout:4/749: rmdir da/d29/d2e/d57/d96 39 2024-04-05T03:02:55.268 INFO:tasks.workunit.client.1.smithi203.stdout:4/750: stat da/d29/d2e/d8b/c9d 0 2024-04-05T03:02:55.268 INFO:tasks.workunit.client.0.smithi153.stdout:7/724: chown d2/da/d3a/d41/f8c 4007 1 2024-04-05T03:02:55.268 INFO:tasks.workunit.client.1.smithi203.stdout:0/942: fsync dd/db7/f117 0 2024-04-05T03:02:55.269 INFO:tasks.workunit.client.0.smithi153.stdout:3/621: creat d1d/d75/fe0 x:0 0 0 2024-04-05T03:02:55.269 INFO:tasks.workunit.client.1.smithi203.stdout:5/938: dread d3/d4/dc/d67/db5/fcb [0,4194304] 0 2024-04-05T03:02:55.270 INFO:tasks.workunit.client.1.smithi203.stdout:5/939: dread - d3/d4/dd6/dfe/d132/d59/d60/f12f zero size 2024-04-05T03:02:55.273 INFO:tasks.workunit.client.0.smithi153.stdout:7/725: write d2/d51/d93/fe7 [775427,40905] 0 2024-04-05T03:02:55.273 INFO:tasks.workunit.client.1.smithi203.stdout:0/943: mknod dd/d17/d1d/d22/d80/c118 0 2024-04-05T03:02:55.274 INFO:tasks.workunit.client.0.smithi153.stdout:6/719: dread d4/f15 [0,4194304] 0 2024-04-05T03:02:55.275 INFO:tasks.workunit.client.0.smithi153.stdout:5/564: dread - d0/d25/d68/f9b zero size 2024-04-05T03:02:55.275 INFO:tasks.workunit.client.1.smithi203.stdout:3/766: write d0/dc/d1a/d1d/f72 [220002,7354] 0 2024-04-05T03:02:55.278 INFO:tasks.workunit.client.0.smithi153.stdout:7/726: fsync d2/da/d47/fda 0 2024-04-05T03:02:55.278 INFO:tasks.workunit.client.0.smithi153.stdout:0/553: dwrite d9/d14/f19 [0,4194304] 0 2024-04-05T03:02:55.281 INFO:tasks.workunit.client.0.smithi153.stdout:9/469: dwrite d1/d9/d1b/f75 [0,4194304] 0 2024-04-05T03:02:55.284 INFO:tasks.workunit.client.1.smithi203.stdout:0/944: getdents dd/d17/d1d/d22/d45/dbe 0 2024-04-05T03:02:55.289 INFO:tasks.workunit.client.1.smithi203.stdout:3/767: dwrite d0/dc/d1a/d76/fc4 [0,4194304] 0 2024-04-05T03:02:55.290 INFO:tasks.workunit.client.0.smithi153.stdout:0/554: fsync d9/db/f37 0 2024-04-05T03:02:55.291 INFO:tasks.workunit.client.1.smithi203.stdout:1/978: dwrite d7/d24/d33/d48/d4b/d6b/f10f [0,4194304] 0 2024-04-05T03:02:55.293 INFO:tasks.workunit.client.0.smithi153.stdout:5/565: mkdir d0/d25/d4d/d9c/dc7/dc9 0 2024-04-05T03:02:55.294 INFO:tasks.workunit.client.0.smithi153.stdout:3/622: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/le1 0 2024-04-05T03:02:55.295 INFO:tasks.workunit.client.0.smithi153.stdout:6/720: dwrite d4/dc/df/d16/d5f/d73/fb1 [4194304,4194304] 0 2024-04-05T03:02:55.295 INFO:tasks.workunit.client.1.smithi203.stdout:5/940: rename d3/d4/dd6/dfe/d132/d49/d10c/d10d/c130 to d3/d4/dd6/dfe/d132/c13b 0 2024-04-05T03:02:55.295 INFO:tasks.workunit.client.1.smithi203.stdout:6/965: dwrite d0/da/d54/d4c/d7f/dcc/ffb [0,4194304] 0 2024-04-05T03:02:55.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:54 smithi203 ceph-mon[39911]: pgmap v12: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 122 MiB/s rd, 357 MiB/s wr, 675 op/s 2024-04-05T03:02:55.298 INFO:tasks.workunit.client.1.smithi203.stdout:0/945: unlink dd/d17/d115/c8d 0 2024-04-05T03:02:55.298 INFO:tasks.workunit.client.1.smithi203.stdout:6/966: dread - d0/da/d54/d4c/daf/d11a/d44/d78/ff8 zero size 2024-04-05T03:02:55.300 INFO:tasks.workunit.client.0.smithi153.stdout:9/470: rename d1/d1d/l4b to d1/d10/d16/d22/d2c/la8 0 2024-04-05T03:02:55.300 INFO:tasks.workunit.client.1.smithi203.stdout:3/768: symlink d0/d21/d81/le0 0 2024-04-05T03:02:55.301 INFO:tasks.workunit.client.0.smithi153.stdout:7/727: dwrite d2/da/d47/f9e [0,4194304] 0 2024-04-05T03:02:55.302 INFO:tasks.workunit.client.1.smithi203.stdout:0/946: truncate dd/d17/d115/d8c/ff0 354893 0 2024-04-05T03:02:55.310 INFO:tasks.workunit.client.0.smithi153.stdout:7/728: chown d2/l7 5993867 1 2024-04-05T03:02:55.311 INFO:tasks.workunit.client.1.smithi203.stdout:6/967: dwrite d0/da/d54/d4c/daf/d11a/d44/f4a [0,4194304] 0 2024-04-05T03:02:55.316 INFO:tasks.workunit.client.0.smithi153.stdout:9/471: unlink d1/d10/d16/d22/d2c/d68/d81/f4a 0 2024-04-05T03:02:55.317 INFO:tasks.workunit.client.1.smithi203.stdout:5/941: symlink d3/d5f/d92/dff/d103/l13c 0 2024-04-05T03:02:55.318 INFO:tasks.workunit.client.0.smithi153.stdout:3/623: dwrite d1d/d32/d93/d3c/d3f/d90/d92/fac [0,4194304] 0 2024-04-05T03:02:55.321 INFO:tasks.workunit.client.0.smithi153.stdout:1/532: dwrite d5/d8/d20/d4b/fe [0,4194304] 0 2024-04-05T03:02:55.323 INFO:tasks.workunit.client.0.smithi153.stdout:6/721: dwrite d4/dc/df/d16/d5f/fb7 [4194304,4194304] 0 2024-04-05T03:02:55.323 INFO:tasks.workunit.client.0.smithi153.stdout:7/729: chown d2/da/d3a/fd4 210 1 2024-04-05T03:02:55.325 INFO:tasks.workunit.client.1.smithi203.stdout:0/947: symlink dd/d17/d1d/d22/d80/d49/l119 0 2024-04-05T03:02:55.326 INFO:tasks.workunit.client.0.smithi153.stdout:7/730: read d2/da/d11/f13 [6799738,28038] 0 2024-04-05T03:02:55.326 INFO:tasks.workunit.client.1.smithi203.stdout:5/942: dread - d3/d4/d9e/dde/f111 zero size 2024-04-05T03:02:55.329 INFO:tasks.workunit.client.0.smithi153.stdout:6/722: write d4/fda [466570,118728] 0 2024-04-05T03:02:55.329 INFO:tasks.workunit.client.0.smithi153.stdout:6/723: chown d4/dc/df/d16/d24/d8d/c8f 1515 1 2024-04-05T03:02:55.331 INFO:tasks.workunit.client.0.smithi153.stdout:6/724: truncate d4/dc/df/d16/d24/fb9 1310444 0 2024-04-05T03:02:55.333 INFO:tasks.workunit.client.1.smithi203.stdout:3/769: dwrite d0/dc/d1a/d1d/d53/f73 [0,4194304] 0 2024-04-05T03:02:55.335 INFO:tasks.workunit.client.1.smithi203.stdout:5/943: getdents d3/d4/dc/d67/db5 0 2024-04-05T03:02:55.337 INFO:tasks.workunit.client.1.smithi203.stdout:0/948: dread dd/d17/d1d/d22/d53/d55/fa1 [0,4194304] 0 2024-04-05T03:02:55.341 INFO:tasks.workunit.client.0.smithi153.stdout:7/731: dread d2/da/d3a/d41/da8/fad [0,4194304] 0 2024-04-05T03:02:55.342 INFO:tasks.workunit.client.1.smithi203.stdout:3/770: creat d0/dc/d1a/d8d/fe1 x:0 0 0 2024-04-05T03:02:55.345 INFO:tasks.workunit.client.1.smithi203.stdout:0/949: creat dd/d17/d1d/d22/d53/db0/f11a x:0 0 0 2024-04-05T03:02:55.348 INFO:tasks.workunit.client.1.smithi203.stdout:0/950: symlink dd/d17/d1d/d22/d53/d55/l11b 0 2024-04-05T03:02:55.350 INFO:tasks.workunit.client.1.smithi203.stdout:0/951: symlink dd/db7/l11c 0 2024-04-05T03:02:55.373 INFO:tasks.workunit.client.1.smithi203.stdout:5/944: dread d3/d4/dc/d67/db5/fe8 [0,4194304] 0 2024-04-05T03:02:55.374 INFO:tasks.workunit.client.1.smithi203.stdout:5/945: write d3/d4/dc/d67/dac/f129 [502864,59227] 0 2024-04-05T03:02:55.378 INFO:tasks.workunit.client.1.smithi203.stdout:5/946: symlink d3/d4/dd6/dfe/d132/d59/d64/l13d 0 2024-04-05T03:02:55.381 INFO:tasks.workunit.client.1.smithi203.stdout:5/947: chown d3/d4/dd6/dfe/d132/d49/l65 0 1 2024-04-05T03:02:55.384 INFO:tasks.workunit.client.1.smithi203.stdout:5/948: mkdir d3/d4/d13e 0 2024-04-05T03:02:55.385 INFO:tasks.workunit.client.1.smithi203.stdout:0/952: write dd/d17/d115/d28/f4e [1605357,62695] 0 2024-04-05T03:02:55.386 INFO:tasks.workunit.client.0.smithi153.stdout:6/725: write d4/dc/f46 [872343,60734] 0 2024-04-05T03:02:55.387 INFO:tasks.workunit.client.0.smithi153.stdout:7/732: write d2/f40 [844724,117658] 0 2024-04-05T03:02:55.390 INFO:tasks.workunit.client.1.smithi203.stdout:1/979: dwrite d7/d24/f35 [0,4194304] 0 2024-04-05T03:02:55.391 INFO:tasks.workunit.client.0.smithi153.stdout:0/555: dwrite d9/f69 [0,4194304] 0 2024-04-05T03:02:55.392 INFO:tasks.workunit.client.1.smithi203.stdout:5/949: truncate d3/f7a 2692500 0 2024-04-05T03:02:55.393 INFO:tasks.workunit.client.0.smithi153.stdout:7/733: write d2/da/d3a/fd2 [1453303,55138] 0 2024-04-05T03:02:55.394 INFO:tasks.workunit.client.1.smithi203.stdout:3/771: dwrite d0/dc/d1a/d76/d35/f47 [0,4194304] 0 2024-04-05T03:02:55.394 INFO:tasks.workunit.client.0.smithi153.stdout:5/566: dread d0/f44 [0,4194304] 0 2024-04-05T03:02:55.396 INFO:tasks.workunit.client.1.smithi203.stdout:0/953: creat dd/d17/d115/d8c/d6d/dc4/f11d x:0 0 0 2024-04-05T03:02:55.396 INFO:tasks.workunit.client.0.smithi153.stdout:1/533: dwrite d5/d8/d20/f88 [0,4194304] 0 2024-04-05T03:02:55.399 INFO:tasks.workunit.client.0.smithi153.stdout:0/556: creat d9/d14/d67/d6f/fab x:0 0 0 2024-04-05T03:02:55.404 INFO:tasks.workunit.client.0.smithi153.stdout:1/534: truncate d5/d8/d20/d4b/d18/d16/d45/d93/f9a 748355 0 2024-04-05T03:02:55.404 INFO:tasks.workunit.client.1.smithi203.stdout:0/954: mkdir dd/db1/d11e 0 2024-04-05T03:02:55.405 INFO:tasks.workunit.client.0.smithi153.stdout:6/726: dwrite d4/dc/f3c [0,4194304] 0 2024-04-05T03:02:55.408 INFO:tasks.workunit.client.1.smithi203.stdout:1/980: rename d7/d14/d83/dfb/c102 to d7/d14/d2b/d58/d7a/d13c/c140 0 2024-04-05T03:02:55.409 INFO:tasks.workunit.client.1.smithi203.stdout:1/981: dread - d7/d14/d2b/d58/d137/f139 zero size 2024-04-05T03:02:55.410 INFO:tasks.workunit.client.0.smithi153.stdout:1/535: mkdir d5/d8/d20/d3f/da7 0 2024-04-05T03:02:55.410 INFO:tasks.workunit.client.1.smithi203.stdout:5/950: symlink d3/d4/dd6/dfe/d132/d137/l13f 0 2024-04-05T03:02:55.413 INFO:tasks.workunit.client.0.smithi153.stdout:6/727: symlink d4/dc/df/d16/d24/d5e/lf3 0 2024-04-05T03:02:55.414 INFO:tasks.workunit.client.0.smithi153.stdout:6/728: chown d4/dc/l30 39805 1 2024-04-05T03:02:55.416 INFO:tasks.workunit.client.0.smithi153.stdout:0/557: dwrite d9/db/f49 [0,4194304] 0 2024-04-05T03:02:55.417 INFO:tasks.workunit.client.1.smithi203.stdout:1/982: write d7/d24/d33/d48/d4b/d6b/f75 [3257995,36537] 0 2024-04-05T03:02:55.417 INFO:tasks.workunit.client.0.smithi153.stdout:1/536: fdatasync d5/d8/d20/d4b/d18/d16/d45/d93/f9a 0 2024-04-05T03:02:55.417 INFO:tasks.workunit.client.1.smithi203.stdout:3/772: dwrite d0/dc/d38/d92/fb1 [4194304,4194304] 0 2024-04-05T03:02:55.419 INFO:tasks.workunit.client.1.smithi203.stdout:1/983: fdatasync d7/d14/d9d/dbf/fcd 0 2024-04-05T03:02:55.419 INFO:tasks.workunit.client.1.smithi203.stdout:0/955: rename dd/d17/d115/d8c to dd/d17/d115/d11f 0 2024-04-05T03:02:55.420 INFO:tasks.workunit.client.0.smithi153.stdout:1/537: truncate d5/d8/d20/d4b/d18/d16/d33/d83/f8e 1033401 0 2024-04-05T03:02:55.424 INFO:tasks.workunit.client.1.smithi203.stdout:5/951: symlink d3/d4/dc/d67/db5/l140 0 2024-04-05T03:02:55.424 INFO:tasks.workunit.client.0.smithi153.stdout:6/729: symlink d4/dc/df/d16/d5f/da2/lf4 0 2024-04-05T03:02:55.425 INFO:tasks.workunit.client.1.smithi203.stdout:1/984: truncate d7/f3d 10579444 0 2024-04-05T03:02:55.427 INFO:tasks.workunit.client.1.smithi203.stdout:1/985: stat d7/d14/d83/d117 0 2024-04-05T03:02:55.427 INFO:tasks.workunit.client.0.smithi153.stdout:1/538: mkdir d5/d8/d20/d4b/d18/d16/d45/d68/da4/da8 0 2024-04-05T03:02:55.429 INFO:tasks.workunit.client.1.smithi203.stdout:3/773: creat d0/dc/d38/d5d/fe2 x:0 0 0 2024-04-05T03:02:55.429 INFO:tasks.workunit.client.0.smithi153.stdout:1/539: fdatasync d5/d8/d20/d4b/d18/d16/f22 0 2024-04-05T03:02:55.431 INFO:tasks.workunit.client.0.smithi153.stdout:6/730: rename d4/dc/df/d16/d5f/da2/fa4 to d4/dc/df/dcb/ff5 0 2024-04-05T03:02:55.431 INFO:tasks.workunit.client.0.smithi153.stdout:0/558: link d9/df/d1e/c41 d9/db/cac 0 2024-04-05T03:02:55.433 INFO:tasks.workunit.client.0.smithi153.stdout:6/731: stat d4/dc/df/d16/d5f/d73/fb1 0 2024-04-05T03:02:55.433 INFO:tasks.workunit.client.1.smithi203.stdout:3/774: symlink d0/dc/d38/d5d/d66/d80/le3 0 2024-04-05T03:02:55.433 INFO:tasks.workunit.client.1.smithi203.stdout:1/986: getdents d7/d14/d2b/d58/d86/d91/da6/df9/d106 0 2024-04-05T03:02:55.434 INFO:tasks.workunit.client.0.smithi153.stdout:0/559: fdatasync d9/df/d1d/d61/f1b 0 2024-04-05T03:02:55.436 INFO:tasks.workunit.client.0.smithi153.stdout:0/560: chown d9/db/d42/d7e/c96 12115614 1 2024-04-05T03:02:55.440 INFO:tasks.workunit.client.0.smithi153.stdout:6/732: rmdir d4/dc/df/d16/d36/d4d/dde 0 2024-04-05T03:02:55.440 INFO:tasks.workunit.client.1.smithi203.stdout:1/987: dwrite d7/d24/d33/d48/d4b/f5c [0,4194304] 0 2024-04-05T03:02:55.442 INFO:tasks.workunit.client.0.smithi153.stdout:2/653: sync 2024-04-05T03:02:55.442 INFO:tasks.workunit.client.0.smithi153.stdout:8/505: sync 2024-04-05T03:02:55.442 INFO:tasks.workunit.client.0.smithi153.stdout:4/726: sync 2024-04-05T03:02:55.442 INFO:tasks.workunit.client.0.smithi153.stdout:0/561: rename d9/df/d1d/d61/l70 to d9/df/lad 0 2024-04-05T03:02:55.443 INFO:tasks.workunit.client.1.smithi203.stdout:1/988: mknod d7/d14/d83/deb/dff/c141 0 2024-04-05T03:02:55.444 INFO:tasks.workunit.client.0.smithi153.stdout:0/562: chown d9/df/d1e/f80 3 1 2024-04-05T03:02:55.445 INFO:tasks.workunit.client.1.smithi203.stdout:1/989: dread - d7/d14/d83/f131 zero size 2024-04-05T03:02:55.447 INFO:tasks.workunit.client.0.smithi153.stdout:2/654: mknod d9/da/d19/d2e/d7e/cde 0 2024-04-05T03:02:55.449 INFO:tasks.workunit.client.0.smithi153.stdout:6/733: creat d4/dc/df/d16/d24/d4e/ff6 x:0 0 0 2024-04-05T03:02:55.449 INFO:tasks.workunit.client.0.smithi153.stdout:1/540: read d5/d8/d20/d4b/d18/f1a [3057691,97648] 0 2024-04-05T03:02:55.449 INFO:tasks.workunit.client.1.smithi203.stdout:1/990: mknod d7/d14/d2b/d58/d86/d91/da6/df9/d106/c142 0 2024-04-05T03:02:55.450 INFO:tasks.workunit.client.0.smithi153.stdout:0/563: mkdir d9/df/d1e/d45/dae 0 2024-04-05T03:02:55.451 INFO:tasks.workunit.client.0.smithi153.stdout:4/727: read da/de/f19 [2486670,70679] 0 2024-04-05T03:02:55.451 INFO:tasks.workunit.client.0.smithi153.stdout:2/655: write d9/da/d19/d2e/d4f/fa6 [1461786,50658] 0 2024-04-05T03:02:55.451 INFO:tasks.workunit.client.0.smithi153.stdout:0/564: read d9/d14/f19 [630399,73555] 0 2024-04-05T03:02:55.452 INFO:tasks.workunit.client.1.smithi203.stdout:1/991: mknod d7/d24/d33/d48/d4b/dea/def/c143 0 2024-04-05T03:02:55.454 INFO:tasks.workunit.client.0.smithi153.stdout:2/656: fdatasync d9/da/d36/f81 0 2024-04-05T03:02:55.454 INFO:tasks.workunit.client.0.smithi153.stdout:8/506: rename d7/d13/d19/c5e to d7/ca5 0 2024-04-05T03:02:55.457 INFO:tasks.workunit.client.0.smithi153.stdout:6/734: write d4/dc/df/d16/d5f/d73/fb1 [3207734,108014] 0 2024-04-05T03:02:55.460 INFO:tasks.workunit.client.1.smithi203.stdout:1/992: dwrite d7/d14/d2b/fa3 [0,4194304] 0 2024-04-05T03:02:55.461 INFO:tasks.workunit.client.1.smithi203.stdout:5/952: dread d3/d4/dc/d67/f70 [0,4194304] 0 2024-04-05T03:02:55.464 INFO:tasks.workunit.client.0.smithi153.stdout:0/565: rmdir d9/db/d42/d7e/d8b 39 2024-04-05T03:02:55.466 INFO:tasks.workunit.client.1.smithi203.stdout:0/956: write dd/d17/dd8/fb6 [542503,55102] 0 2024-04-05T03:02:55.468 INFO:tasks.workunit.client.1.smithi203.stdout:1/993: creat d7/d14/d83/deb/df8/f144 x:0 0 0 2024-04-05T03:02:55.468 INFO:tasks.workunit.client.0.smithi153.stdout:4/728: symlink da/db/d20/d3c/d81/db6/lf8 0 2024-04-05T03:02:55.468 INFO:tasks.workunit.client.1.smithi203.stdout:0/957: fdatasync dd/d17/d1d/d22/d53/db0/f11a 0 2024-04-05T03:02:55.469 INFO:tasks.workunit.client.0.smithi153.stdout:6/735: creat d4/dc/df/d16/d5f/d73/ff7 x:0 0 0 2024-04-05T03:02:55.470 INFO:tasks.workunit.client.1.smithi203.stdout:0/958: write dd/d17/d115/d11f/ff0 [318170,112280] 0 2024-04-05T03:02:55.471 INFO:tasks.workunit.client.1.smithi203.stdout:0/959: chown dd/d17/dd8 10 1 2024-04-05T03:02:55.472 INFO:tasks.workunit.client.0.smithi153.stdout:7/734: dwrite d2/da/d3a/d41/d31/f73 [0,4194304] 0 2024-04-05T03:02:55.472 INFO:tasks.workunit.client.0.smithi153.stdout:6/736: readlink d4/d38/d4c/lc8 0 2024-04-05T03:02:55.473 INFO:tasks.workunit.client.0.smithi153.stdout:5/567: dwrite d0/d25/d4d/d62/d5f/d86/da4/fa1 [0,4194304] 0 2024-04-05T03:02:55.478 INFO:tasks.workunit.client.0.smithi153.stdout:4/729: symlink da/db/d20/d3c/d81/lf9 0 2024-04-05T03:02:55.478 INFO:tasks.workunit.client.0.smithi153.stdout:8/507: creat d7/d2a/d8b/fa6 x:0 0 0 2024-04-05T03:02:55.478 INFO:tasks.workunit.client.1.smithi203.stdout:0/960: dread dd/d17/d1d/d22/d80/f72 [0,4194304] 0 2024-04-05T03:02:55.479 INFO:tasks.workunit.client.1.smithi203.stdout:5/953: creat d3/d4/dd6/dfe/d132/f141 x:0 0 0 2024-04-05T03:02:55.481 INFO:tasks.workunit.client.0.smithi153.stdout:8/508: write d7/d13/d19/d1b/f35 [1255613,121544] 0 2024-04-05T03:02:55.481 INFO:tasks.workunit.client.0.smithi153.stdout:0/566: mkdir d9/df/daf 0 2024-04-05T03:02:55.482 INFO:tasks.workunit.client.1.smithi203.stdout:5/954: stat d3/d4/dd6/dfe/d132/d59/d60/f74 0 2024-04-05T03:02:55.486 INFO:tasks.workunit.client.1.smithi203.stdout:1/994: creat d7/d14/d2b/f145 x:0 0 0 2024-04-05T03:02:55.487 INFO:tasks.workunit.client.1.smithi203.stdout:0/961: mknod dd/d17/d1d/d22/d45/c120 0 2024-04-05T03:02:55.488 INFO:tasks.workunit.client.0.smithi153.stdout:4/730: creat da/d2a/ffa x:0 0 0 2024-04-05T03:02:55.490 INFO:tasks.workunit.client.1.smithi203.stdout:0/962: rmdir dd/d17/d1d/d22/d45/dbe/de1 39 2024-04-05T03:02:55.493 INFO:tasks.workunit.client.0.smithi153.stdout:1/541: truncate d5/dd/f13 1467557 0 2024-04-05T03:02:55.494 INFO:tasks.workunit.client.0.smithi153.stdout:4/731: mknod da/db/d20/d35/d41/da8/dc1/cfb 0 2024-04-05T03:02:55.496 INFO:tasks.workunit.client.0.smithi153.stdout:7/735: creat d2/da/ff5 x:0 0 0 2024-04-05T03:02:55.497 INFO:tasks.workunit.client.0.smithi153.stdout:1/542: creat d5/d8/d20/d4b/d51/fa9 x:0 0 0 2024-04-05T03:02:55.498 INFO:tasks.workunit.client.0.smithi153.stdout:5/568: getdents d0/d4e/db6 0 2024-04-05T03:02:55.503 INFO:tasks.workunit.client.0.smithi153.stdout:5/569: chown d0/d2c/f3a 5 1 2024-04-05T03:02:55.504 INFO:tasks.workunit.client.0.smithi153.stdout:6/737: truncate d4/dc/df/d16/d44/fdb 207714 0 2024-04-05T03:02:55.505 INFO:tasks.workunit.client.0.smithi153.stdout:1/543: read d5/dd/f9f [7895430,128655] 0 2024-04-05T03:02:55.507 INFO:tasks.workunit.client.0.smithi153.stdout:4/732: dread da/db/d20/d3c/d81/f8a [0,4194304] 0 2024-04-05T03:02:55.508 INFO:tasks.workunit.client.1.smithi203.stdout:5/955: write d3/d4/dd6/dfe/d132/f33 [4660183,98929] 0 2024-04-05T03:02:55.508 INFO:tasks.workunit.client.0.smithi153.stdout:2/657: dwrite d9/d77/f58 [0,4194304] 0 2024-04-05T03:02:55.512 INFO:tasks.workunit.client.0.smithi153.stdout:4/733: chown da/db/de6 5791 1 2024-04-05T03:02:55.512 INFO:tasks.workunit.client.0.smithi153.stdout:0/567: dread d9/df/d1d/d61/f1b [0,4194304] 0 2024-04-05T03:02:55.513 INFO:tasks.workunit.client.1.smithi203.stdout:5/956: mknod d3/d4/dd6/dfe/d132/d59/d64/d12/c142 0 2024-04-05T03:02:55.513 INFO:tasks.workunit.client.0.smithi153.stdout:6/738: mknod d4/dc/df/d16/d36/cf8 0 2024-04-05T03:02:55.514 INFO:tasks.workunit.client.0.smithi153.stdout:0/568: chown d9/df/d1e/d45/d62/f7d 432 1 2024-04-05T03:02:55.517 INFO:tasks.workunit.client.0.smithi153.stdout:2/658: dread d9/d68/db9/dd8/f5b [0,4194304] 0 2024-04-05T03:02:55.517 INFO:tasks.workunit.client.1.smithi203.stdout:0/963: dwrite dd/db7/f117 [0,4194304] 0 2024-04-05T03:02:55.518 INFO:tasks.workunit.client.0.smithi153.stdout:1/544: chown d5/dd/l85 8657 1 2024-04-05T03:02:55.518 INFO:tasks.workunit.client.0.smithi153.stdout:0/569: stat d9/df/d1d/d61/f1b 0 2024-04-05T03:02:55.520 INFO:tasks.workunit.client.1.smithi203.stdout:0/964: rmdir dd/d17/d115/d11f/d6d/dc4 39 2024-04-05T03:02:55.523 INFO:tasks.workunit.client.0.smithi153.stdout:7/736: dwrite d2/da/d11/f24 [4194304,4194304] 0 2024-04-05T03:02:55.528 INFO:tasks.workunit.client.0.smithi153.stdout:6/739: dwrite d4/dc/df/d16/d36/fb2 [0,4194304] 0 2024-04-05T03:02:55.528 INFO:tasks.workunit.client.1.smithi203.stdout:0/965: creat dd/d17/d115/d11f/dcf/f121 x:0 0 0 2024-04-05T03:02:55.528 INFO:tasks.workunit.client.1.smithi203.stdout:5/957: dwrite d3/d4/dd6/dfe/d132/d59/d64/f91 [0,4194304] 0 2024-04-05T03:02:55.532 INFO:tasks.workunit.client.0.smithi153.stdout:7/737: write d2/d9b/fc5 [1948654,113463] 0 2024-04-05T03:02:55.532 INFO:tasks.workunit.client.0.smithi153.stdout:5/570: link d0/d9/lb0 d0/d25/d4d/d62/d37/d38/lca 0 2024-04-05T03:02:55.532 INFO:tasks.workunit.client.0.smithi153.stdout:2/659: chown d9/da/d37/fc2 1864274277 1 2024-04-05T03:02:55.532 INFO:tasks.workunit.client.0.smithi153.stdout:7/738: fsync d2/da/d3a/d41/d31/f73 0 2024-04-05T03:02:55.534 INFO:tasks.workunit.client.1.smithi203.stdout:5/958: rename d3/d4/dc/lba to d3/d4/d13e/l143 0 2024-04-05T03:02:55.535 INFO:tasks.workunit.client.1.smithi203.stdout:1/995: dread d7/f3d [4194304,4194304] 0 2024-04-05T03:02:55.535 INFO:tasks.workunit.client.0.smithi153.stdout:1/545: fsync d5/d8/d20/d4b/d18/f1a 0 2024-04-05T03:02:55.539 INFO:tasks.workunit.client.0.smithi153.stdout:2/660: mknod d9/da/d19/cdf 0 2024-04-05T03:02:55.539 INFO:tasks.workunit.client.1.smithi203.stdout:0/966: creat dd/d17/d1d/d22/d53/d55/d104/f122 x:0 0 0 2024-04-05T03:02:55.543 INFO:tasks.workunit.client.1.smithi203.stdout:0/967: dread dd/d17/d115/f98 [4194304,4194304] 0 2024-04-05T03:02:55.546 INFO:tasks.workunit.client.0.smithi153.stdout:7/739: unlink d2/da/d47/d5d/d7c/d8d/d95/fe3 0 2024-04-05T03:02:55.547 INFO:tasks.workunit.client.1.smithi203.stdout:1/996: creat d7/d14/d2b/d58/d86/d120/f146 x:0 0 0 2024-04-05T03:02:55.550 INFO:tasks.workunit.client.0.smithi153.stdout:0/570: read d9/db/f11 [906797,34373] 0 2024-04-05T03:02:55.552 INFO:tasks.workunit.client.1.smithi203.stdout:0/968: creat dd/d17/d1d/d22/d53/db0/f123 x:0 0 0 2024-04-05T03:02:55.553 INFO:tasks.workunit.client.0.smithi153.stdout:4/734: write da/d2a/f47 [1667748,99903] 0 2024-04-05T03:02:55.555 INFO:tasks.workunit.client.0.smithi153.stdout:6/740: dwrite d4/dc/df/d16/d36/d52/f55 [0,4194304] 0 2024-04-05T03:02:55.555 INFO:tasks.workunit.client.1.smithi203.stdout:5/959: truncate d3/d4/dd6/dfe/f101 2398786 0 2024-04-05T03:02:55.558 INFO:tasks.workunit.client.0.smithi153.stdout:7/740: dread d2/da/d3a/d41/d31/f9a [0,4194304] 0 2024-04-05T03:02:55.559 INFO:tasks.workunit.client.1.smithi203.stdout:1/997: getdents d7/d14/d9d 0 2024-04-05T03:02:55.562 INFO:tasks.workunit.client.0.smithi153.stdout:5/571: write d0/d25/f6e [331668,107774] 0 2024-04-05T03:02:55.563 INFO:tasks.workunit.client.1.smithi203.stdout:5/960: rename d3/d4/dc/c6d to d3/d126/c144 0 2024-04-05T03:02:55.566 INFO:tasks.workunit.client.1.smithi203.stdout:0/969: link dd/d17/d1d/d22/d53/d55/l11b dd/d17/d115/d11f/l124 0 2024-04-05T03:02:55.567 INFO:tasks.workunit.client.1.smithi203.stdout:1/998: symlink d7/d24/d33/d48/d4b/d6b/l147 0 2024-04-05T03:02:55.567 INFO:tasks.workunit.client.1.smithi203.stdout:5/961: read d3/f4d [1398502,122978] 0 2024-04-05T03:02:55.567 INFO:tasks.workunit.client.1.smithi203.stdout:1/999: readlink d7/le9 0 2024-04-05T03:02:55.569 INFO:tasks.workunit.client.1.smithi203.stdout:0/970: creat dd/d17/d1d/d22/d53/d55/d104/f125 x:0 0 0 2024-04-05T03:02:55.571 INFO:tasks.workunit.client.1.smithi203.stdout:0/971: truncate dd/d17/d1d/d22/d53/fff 661027 0 2024-04-05T03:02:55.575 INFO:tasks.workunit.client.0.smithi153.stdout:1/546: getdents d5/d8/d20/d4b/d18 0 2024-04-05T03:02:55.576 INFO:tasks.workunit.client.1.smithi203.stdout:5/962: link d3/d4/dd6/dfe/d132/f3b d3/d4/d9e/dde/f145 0 2024-04-05T03:02:55.576 INFO:tasks.workunit.client.0.smithi153.stdout:2/661: link d9/da/d19/d2e/d7e/d53/da1/ccf d9/da/d19/d2e/dc7/ce0 0 2024-04-05T03:02:55.577 INFO:tasks.workunit.client.1.smithi203.stdout:5/963: write d3/d5f/d92/dff/f107 [818837,58500] 0 2024-04-05T03:02:55.577 INFO:tasks.workunit.client.0.smithi153.stdout:1/547: mknod d5/d8/d20/d4b/d18/d16/d33/caa 0 2024-04-05T03:02:55.579 INFO:tasks.workunit.client.0.smithi153.stdout:7/741: getdents d2/da/d11 0 2024-04-05T03:02:55.579 INFO:tasks.workunit.client.0.smithi153.stdout:2/662: mknod d9/d68/d9b/ce1 0 2024-04-05T03:02:55.579 INFO:tasks.workunit.client.1.smithi203.stdout:5/964: mknod d3/d4/dc/c146 0 2024-04-05T03:02:55.580 INFO:tasks.workunit.client.0.smithi153.stdout:7/742: fdatasync d2/da/d3a/d41/f1e 0 2024-04-05T03:02:55.580 INFO:tasks.workunit.client.0.smithi153.stdout:1/548: stat d5/d8/d20/d4b/d18/d16 0 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 -- 172.21.15.153:0/661525287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fcf90 msgr2=0x7ff0600fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 --2- 172.21.15.153:0/661525287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fcf90 0x7ff0600fd400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff050004940 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 -- 172.21.15.153:0/661525287 shutdown_connections 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 --2- 172.21.15.153:0/661525287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fcf90 0x7ff0600fd400 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 --2- 172.21.15.153:0/661525287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fbce0 0x7ff0600fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:55.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.581+0000 7ff065589700 1 -- 172.21.15.153:0/661525287 >> 172.21.15.153:0/661525287 conn(0x7ff0600f7440 msgr2=0x7ff0600f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:55.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.582+0000 7ff065589700 1 -- 172.21.15.153:0/661525287 shutdown_connections 2024-04-05T03:02:55.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.582+0000 7ff065589700 1 -- 172.21.15.153:0/661525287 wait complete. 2024-04-05T03:02:55.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.583+0000 7ff065589700 1 Processor -- start 2024-04-05T03:02:55.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.583+0000 7ff065589700 1 -- start start 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.583+0000 7ff065589700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.583+0000 7ff065589700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fcf90 0x7ff0601144d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.583+0000 7ff065589700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff060114ad0 con 0x7ff0600fbce0 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff065589700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff060115320 con 0x7ff0600fcf90 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05e7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fcf90 0x7ff0601144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39758/0 (socket says 172.21.15.153:39758) 2024-04-05T03:02:55.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 -- 172.21.15.153:0/1768128357 learned_addr learned my addr 172.21.15.153:0/1768128357 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:55.584 INFO:tasks.workunit.client.0.smithi153.stdout:2/663: truncate d9/da/d19/d2e/d7e/d7a/f57 1038883 0 2024-04-05T03:02:55.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 -- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fcf90 msgr2=0x7ff0601144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:55.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fcf90 0x7ff0601144d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:55.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 -- 172.21.15.153:0/1768128357 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050005040 con 0x7ff0600fbce0 2024-04-05T03:02:55.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.584+0000 7ff05effd700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff05400eb40 tx=0x7ff05400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:55.585 INFO:tasks.workunit.client.0.smithi153.stdout:2/664: chown d9/d68/d98 1872663 1 2024-04-05T03:02:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.585+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff054007cc0 con 0x7ff0600fbce0 2024-04-05T03:02:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.585+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff05400f040 con 0x7ff0600fbce0 2024-04-05T03:02:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.585+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0601155a0 con 0x7ff0600fbce0 2024-04-05T03:02:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.585+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff054009620 con 0x7ff0600fbce0 2024-04-05T03:02:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.585+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff060115ae0 con 0x7ff0600fbce0 2024-04-05T03:02:55.586 INFO:tasks.workunit.client.0.smithi153.stdout:2/665: chown d9/da/d19/d2e/d4f/ld7 55919717 1 2024-04-05T03:02:55.586 INFO:tasks.workunit.client.0.smithi153.stdout:1/549: unlink d5/d8/d20/d4b/fe 0 2024-04-05T03:02:55.587 INFO:tasks.workunit.client.1.smithi203.stdout:5/965: fdatasync d3/d4/dd6/dfe/d132/d59/d64/f3d 0 2024-04-05T03:02:55.587 INFO:tasks.workunit.client.1.smithi203.stdout:4/751: sync 2024-04-05T03:02:55.587 INFO:tasks.workunit.client.0.smithi153.stdout:2/666: truncate d9/d68/fce 836354 0 2024-04-05T03:02:55.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.587+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff060100920 con 0x7ff0600fbce0 2024-04-05T03:02:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.588+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7ff054009780 con 0x7ff0600fbce0 2024-04-05T03:02:55.588 INFO:tasks.workunit.client.1.smithi203.stdout:5/966: symlink d3/d126/l147 0 2024-04-05T03:02:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.589+0000 7ff0477fe700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 0x7ff04806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:55.589 INFO:tasks.workunit.client.0.smithi153.stdout:7/743: rename d2/d51 to d2/da/d3a/d41/d7a/d54/d85/df6 0 2024-04-05T03:02:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.589+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ff054089d30 con 0x7ff0600fbce0 2024-04-05T03:02:55.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.589+0000 7ff05e7fc700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 0x7ff04806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:55.590 INFO:tasks.workunit.client.0.smithi153.stdout:1/550: fsync d5/d8/d20/d3f/f5c 0 2024-04-05T03:02:55.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.591+0000 7ff05e7fc700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 0x7ff04806d9f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff05000b0d0 tx=0x7ff050009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:55.592 INFO:tasks.workunit.client.1.smithi203.stdout:4/752: dwrite da/fc [4194304,4194304] 0 2024-04-05T03:02:55.592 INFO:tasks.workunit.client.0.smithi153.stdout:2/667: mkdir d9/da/d19/de2 0 2024-04-05T03:02:55.594 INFO:tasks.workunit.client.1.smithi203.stdout:5/967: mknod d3/d4/dd6/dfe/d132/d10e/c148 0 2024-04-05T03:02:55.596 INFO:tasks.workunit.client.0.smithi153.stdout:0/571: dread d9/df/d1e/f54 [0,4194304] 0 2024-04-05T03:02:55.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:55.597+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7ff05404fb80 con 0x7ff0600fbce0 2024-04-05T03:02:55.600 INFO:tasks.workunit.client.0.smithi153.stdout:1/551: link d5/d8/d20/d2f/c55 d5/d8/d20/d4b/cab 0 2024-04-05T03:02:55.600 INFO:tasks.workunit.client.0.smithi153.stdout:7/744: dwrite d2/da/d3a/d41/d31/f73 [0,4194304] 0 2024-04-05T03:02:55.602 INFO:tasks.workunit.client.1.smithi203.stdout:5/968: mkdir d3/d4/dd6/dfe/d132/d49/d149 0 2024-04-05T03:02:55.604 INFO:tasks.workunit.client.1.smithi203.stdout:4/753: creat da/d29/d2e/d57/d96/ddc/de1/fec x:0 0 0 2024-04-05T03:02:55.604 INFO:tasks.workunit.client.1.smithi203.stdout:0/972: dwrite dd/d17/d1d/d22/d53/f63 [0,4194304] 0 2024-04-05T03:02:55.608 INFO:tasks.workunit.client.1.smithi203.stdout:0/973: chown dd/d17/d115/l10b 1432822 1 2024-04-05T03:02:55.608 INFO:tasks.workunit.client.0.smithi153.stdout:0/572: dread d9/f1a [0,4194304] 0 2024-04-05T03:02:55.609 INFO:tasks.workunit.client.0.smithi153.stdout:1/552: creat d5/d8/d20/d4b/d18/d16/d45/d68/da4/fac x:0 0 0 2024-04-05T03:02:55.609 INFO:tasks.workunit.client.0.smithi153.stdout:4/735: dwrite da/de/d56/d6a/d2d/f4a [0,4194304] 0 2024-04-05T03:02:55.610 INFO:tasks.workunit.client.1.smithi203.stdout:4/754: fsync da/d29/d2e/fb1 0 2024-04-05T03:02:55.611 INFO:tasks.workunit.client.1.smithi203.stdout:5/969: getdents d3/d4/dd6/dfe/d132/d137 0 2024-04-05T03:02:55.615 INFO:tasks.workunit.client.1.smithi203.stdout:5/970: read d3/d4/dd6/dfe/d132/d59/d60/f99 [4177192,12852] 0 2024-04-05T03:02:55.617 INFO:tasks.workunit.client.0.smithi153.stdout:7/745: link d2/da/d3a/d41/d7a/d54/d85/df6/d77/fdf d2/da/d3a/d41/d7a/d54/d85/ff7 0 2024-04-05T03:02:55.618 INFO:tasks.workunit.client.1.smithi203.stdout:4/755: creat da/fed x:0 0 0 2024-04-05T03:02:55.619 INFO:tasks.workunit.client.0.smithi153.stdout:4/736: rename da/db/f23 to da/db/dbd/ffc 0 2024-04-05T03:02:55.621 INFO:tasks.workunit.client.0.smithi153.stdout:0/573: dwrite d9/db/d42/d43/f98 [0,4194304] 0 2024-04-05T03:02:55.621 INFO:tasks.workunit.client.0.smithi153.stdout:1/553: symlink d5/lad 0 2024-04-05T03:02:55.624 INFO:tasks.workunit.client.0.smithi153.stdout:4/737: creat da/de/d66/ffd x:0 0 0 2024-04-05T03:02:55.630 INFO:tasks.workunit.client.1.smithi203.stdout:4/756: dwrite da/f5a [0,4194304] 0 2024-04-05T03:02:55.633 INFO:tasks.workunit.client.0.smithi153.stdout:4/738: mknod da/db/d20/cfe 0 2024-04-05T03:02:55.633 INFO:tasks.workunit.client.0.smithi153.stdout:0/574: dwrite d9/df/f18 [0,4194304] 0 2024-04-05T03:02:55.634 INFO:tasks.workunit.client.0.smithi153.stdout:2/668: dread d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:02:55.636 INFO:tasks.workunit.client.0.smithi153.stdout:4/739: write da/d2a/d86/fe2 [1274900,97107] 0 2024-04-05T03:02:55.636 INFO:tasks.workunit.client.1.smithi203.stdout:3/775: sync 2024-04-05T03:02:55.638 INFO:tasks.workunit.client.1.smithi203.stdout:3/776: stat d0/dc/d1a/d8d/d96/f90 0 2024-04-05T03:02:55.639 INFO:tasks.workunit.client.0.smithi153.stdout:7/746: dwrite d2/da/d3a/d41/d31/fd5 [0,4194304] 0 2024-04-05T03:02:55.640 INFO:tasks.workunit.client.1.smithi203.stdout:6/968: sync 2024-04-05T03:02:55.645 INFO:tasks.workunit.client.1.smithi203.stdout:3/777: link d0/dc/d38/dca/fd1 d0/d21/fe4 0 2024-04-05T03:02:55.646 INFO:tasks.workunit.client.0.smithi153.stdout:2/669: write d9/da/d19/d2e/dc7/fc0 [1438907,86274] 0 2024-04-05T03:02:55.648 INFO:tasks.workunit.client.1.smithi203.stdout:0/974: truncate dd/d17/d115/d28/ff7 2795555 0 2024-04-05T03:02:55.650 INFO:tasks.workunit.client.0.smithi153.stdout:0/575: dwrite d9/f30 [0,4194304] 0 2024-04-05T03:02:55.652 INFO:tasks.workunit.client.0.smithi153.stdout:0/576: readlink d9/db/d42/d43/l73 0 2024-04-05T03:02:55.655 INFO:tasks.workunit.client.1.smithi203.stdout:6/969: symlink d0/da/d18/l11c 0 2024-04-05T03:02:55.657 INFO:tasks.workunit.client.0.smithi153.stdout:2/670: rename d9/da/d19/f2f to d9/da/d19/d2e/d7e/d53/fe3 0 2024-04-05T03:02:55.657 INFO:tasks.workunit.client.1.smithi203.stdout:4/757: write da/d29/d2e/d8b/dc8/fd6 [3692508,39226] 0 2024-04-05T03:02:55.660 INFO:tasks.workunit.client.1.smithi203.stdout:0/975: dwrite dd/d17/d1d/d22/d53/db0/f123 [0,4194304] 0 2024-04-05T03:02:55.660 INFO:tasks.workunit.client.0.smithi153.stdout:7/747: getdents d2/da/d47 0 2024-04-05T03:02:55.664 INFO:tasks.workunit.client.0.smithi153.stdout:0/577: fdatasync d9/db/f11 0 2024-04-05T03:02:55.664 INFO:tasks.workunit.client.1.smithi203.stdout:6/970: dread - d0/da/d54/d4c/daf/d11a/ff5 zero size 2024-04-05T03:02:55.665 INFO:tasks.workunit.client.1.smithi203.stdout:5/971: dwrite d3/d4/dd6/dfe/d132/fc4 [0,4194304] 0 2024-04-05T03:02:55.665 INFO:tasks.workunit.client.0.smithi153.stdout:0/578: chown d9/db/d42/d7e/f9a 216591418 1 2024-04-05T03:02:55.667 INFO:tasks.workunit.client.0.smithi153.stdout:0/579: write d9/db/d42/f74 [2284868,113744] 0 2024-04-05T03:02:55.671 INFO:tasks.workunit.client.0.smithi153.stdout:1/554: truncate d5/d8/d20/d4b/d18/d16/d48/d96/fa0 2954856 0 2024-04-05T03:02:55.671 INFO:tasks.workunit.client.1.smithi203.stdout:0/976: creat dd/d17/d1d/d22/d53/db0/f126 x:0 0 0 2024-04-05T03:02:55.674 INFO:tasks.workunit.client.0.smithi153.stdout:2/671: mkdir d9/d62/de4 0 2024-04-05T03:02:55.674 INFO:tasks.workunit.client.0.smithi153.stdout:2/672: fdatasync d9/d68/f7f 0 2024-04-05T03:02:55.676 INFO:tasks.workunit.client.0.smithi153.stdout:1/555: unlink d5/d8/d20/d2f/l35 0 2024-04-05T03:02:55.677 INFO:tasks.workunit.client.0.smithi153.stdout:0/580: mknod d9/df/d1e/d45/dae/cb0 0 2024-04-05T03:02:55.678 INFO:tasks.workunit.client.0.smithi153.stdout:1/556: write d5/d8/d20/f5f [3960520,16985] 0 2024-04-05T03:02:55.679 INFO:tasks.workunit.client.1.smithi203.stdout:4/758: dwrite da/d29/d2e/db0/dcb/d80/fb9 [4194304,4194304] 0 2024-04-05T03:02:55.679 INFO:tasks.workunit.client.0.smithi153.stdout:0/581: stat d9/df/d1e/f54 0 2024-04-05T03:02:55.687 INFO:tasks.workunit.client.0.smithi153.stdout:2/673: mkdir d9/da/d37/d8a/da8/de5 0 2024-04-05T03:02:55.687 INFO:tasks.workunit.client.1.smithi203.stdout:0/977: dwrite dd/d17/d115/d11f/ff0 [0,4194304] 0 2024-04-05T03:02:55.689 INFO:tasks.workunit.client.1.smithi203.stdout:0/978: readlink dd/db7/l10e 0 2024-04-05T03:02:55.689 INFO:tasks.workunit.client.1.smithi203.stdout:3/778: dread d0/dc/d38/f8f [0,4194304] 0 2024-04-05T03:02:55.690 INFO:tasks.workunit.client.0.smithi153.stdout:7/748: getdents d2/da/d3a/d41/d7a/d54/d85 0 2024-04-05T03:02:55.692 INFO:tasks.workunit.client.1.smithi203.stdout:4/759: link da/l1f da/d29/d2e/db0/lee 0 2024-04-05T03:02:55.693 INFO:tasks.workunit.client.0.smithi153.stdout:4/740: dwrite da/db/d20/d45/f6b [0,4194304] 0 2024-04-05T03:02:55.696 INFO:tasks.workunit.client.0.smithi153.stdout:0/582: unlink d9/db/c77 0 2024-04-05T03:02:55.698 INFO:tasks.workunit.client.0.smithi153.stdout:2/674: rename d9/d68/db9/dd8 to d9/da/d19/d2e/d7e/de6 0 2024-04-05T03:02:55.700 INFO:tasks.workunit.client.0.smithi153.stdout:2/675: truncate d9/da/d19/d2e/d4f/fc4 578702 0 2024-04-05T03:02:55.703 INFO:tasks.workunit.client.0.smithi153.stdout:7/749: chown d2/da/c1c 130692 1 2024-04-05T03:02:55.705 INFO:tasks.workunit.client.1.smithi203.stdout:5/972: dwrite d3/f8c [0,4194304] 0 2024-04-05T03:02:55.707 INFO:tasks.workunit.client.0.smithi153.stdout:0/583: creat d9/df/d1e/d45/d62/d79/fb1 x:0 0 0 2024-04-05T03:02:55.708 INFO:tasks.workunit.client.1.smithi203.stdout:3/779: dwrite d0/dc/d1a/d1d/f9b [0,4194304] 0 2024-04-05T03:02:55.710 INFO:tasks.workunit.client.1.smithi203.stdout:6/971: dwrite d0/da/d54/d4c/daf/d11a/d44/faa [0,4194304] 0 2024-04-05T03:02:55.712 INFO:tasks.workunit.client.0.smithi153.stdout:1/557: creat d5/dd/d1b/fae x:0 0 0 2024-04-05T03:02:55.714 INFO:tasks.workunit.client.1.smithi203.stdout:5/973: mkdir d3/d4/dd6/dfe/d132/d59/d60/dca/d14a 0 2024-04-05T03:02:55.714 INFO:tasks.workunit.client.0.smithi153.stdout:4/741: mkdir da/db/d20/d35/d41/dff 0 2024-04-05T03:02:55.714 INFO:tasks.workunit.client.0.smithi153.stdout:1/558: chown d5/d8/d20/d4b/d18/d16/d33/l95 14 1 2024-04-05T03:02:55.714 INFO:tasks.workunit.client.1.smithi203.stdout:3/780: creat d0/dc/d1a/d1f/da1/fe5 x:0 0 0 2024-04-05T03:02:55.716 INFO:tasks.workunit.client.0.smithi153.stdout:7/750: unlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/f44 0 2024-04-05T03:02:55.716 INFO:tasks.workunit.client.1.smithi203.stdout:5/974: chown d3/d4/dc/f95 2074 1 2024-04-05T03:02:55.717 INFO:tasks.workunit.client.0.smithi153.stdout:7/751: chown d2/da/d11/d88 2 1 2024-04-05T03:02:55.718 INFO:tasks.workunit.client.1.smithi203.stdout:5/975: write d3/d4/d9e/dde/f134 [1035543,62633] 0 2024-04-05T03:02:55.719 INFO:tasks.workunit.client.1.smithi203.stdout:6/972: creat d0/da/d54/d4c/da1/db7/d71/f11d x:0 0 0 2024-04-05T03:02:55.721 INFO:tasks.workunit.client.1.smithi203.stdout:3/781: rmdir d0/d21/d81 39 2024-04-05T03:02:55.726 INFO:tasks.workunit.client.0.smithi153.stdout:1/559: mkdir d5/d8/d20/d4b/d18/d16/d48/d96/daf 0 2024-04-05T03:02:55.727 INFO:tasks.workunit.client.0.smithi153.stdout:4/742: read da/de/d56/f61 [80560,80413] 0 2024-04-05T03:02:55.728 INFO:tasks.workunit.client.1.smithi203.stdout:3/782: mknod d0/dc/d1a/d1f/da1/daf/ddc/ce6 0 2024-04-05T03:02:55.728 INFO:tasks.workunit.client.0.smithi153.stdout:1/560: fsync d5/d8/d20/d4b/d18/d16/d45/d93/f9a 0 2024-04-05T03:02:55.728 INFO:tasks.workunit.client.0.smithi153.stdout:4/743: chown da/d2a/ffa 16009 1 2024-04-05T03:02:55.729 INFO:tasks.workunit.client.0.smithi153.stdout:1/561: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:55.729 INFO:tasks.workunit.client.1.smithi203.stdout:5/976: dread d3/d4/d9e/dde/f134 [0,4194304] 0 2024-04-05T03:02:55.730 INFO:tasks.workunit.client.1.smithi203.stdout:3/783: truncate d0/d41/faa 716601 0 2024-04-05T03:02:55.732 INFO:tasks.workunit.client.1.smithi203.stdout:3/784: chown d0/d21/d83/fd7 314 1 2024-04-05T03:02:55.732 INFO:tasks.workunit.client.0.smithi153.stdout:7/752: link d2/da/d3a/d41/d7a/d54/fa2 d2/da/d3a/ff8 0 2024-04-05T03:02:55.733 INFO:tasks.workunit.client.1.smithi203.stdout:4/760: dread da/fc [0,4194304] 0 2024-04-05T03:02:55.733 INFO:tasks.workunit.client.1.smithi203.stdout:5/977: write d3/d4/dd6/dfe/d132/d59/d64/d12/f46 [4407198,36623] 0 2024-04-05T03:02:55.734 INFO:tasks.workunit.client.0.smithi153.stdout:2/676: dwrite d9/da/d19/d2e/d7e/de6/f5b [0,4194304] 0 2024-04-05T03:02:55.735 INFO:tasks.workunit.client.1.smithi203.stdout:3/785: fdatasync d0/dc/d38/fc2 0 2024-04-05T03:02:55.735 INFO:tasks.workunit.client.0.smithi153.stdout:7/753: write d2/da/d3a/d41/d7a/f82 [590808,59733] 0 2024-04-05T03:02:55.737 INFO:tasks.workunit.client.0.smithi153.stdout:4/744: symlink da/db/d20/d35/d38/d76/d90/da7/ddc/l100 0 2024-04-05T03:02:55.738 INFO:tasks.workunit.client.0.smithi153.stdout:7/754: chown d2/da/d3a/d41/d7a/d54/d85/df6/d77/ce5 58471592 1 2024-04-05T03:02:55.739 INFO:tasks.workunit.client.1.smithi203.stdout:5/978: creat d3/d4/d13e/f14b x:0 0 0 2024-04-05T03:02:55.739 INFO:tasks.workunit.client.0.smithi153.stdout:4/745: fsync da/de/d56/d6a/d2d/fa3 0 2024-04-05T03:02:55.739 INFO:tasks.workunit.client.1.smithi203.stdout:6/973: write d0/da/d18/f26 [4170544,76498] 0 2024-04-05T03:02:55.740 INFO:tasks.workunit.client.0.smithi153.stdout:2/677: symlink d9/da/d94/le7 0 2024-04-05T03:02:55.743 INFO:tasks.workunit.client.0.smithi153.stdout:2/678: read d9/da/d19/d2e/dc7/fc0 [1582214,47803] 0 2024-04-05T03:02:55.743 INFO:tasks.workunit.client.1.smithi203.stdout:3/786: unlink d0/dc/d38/d5d/d66/lc7 0 2024-04-05T03:02:55.743 INFO:tasks.workunit.client.1.smithi203.stdout:4/761: rename da/d29/d2e/d8b/fdb to da/d29/d2e/db0/dbe/dce/fef 0 2024-04-05T03:02:55.744 INFO:tasks.workunit.client.0.smithi153.stdout:7/755: mknod d2/da/d47/d5d/d7c/d8d/d95/dce/dc8/cf9 0 2024-04-05T03:02:55.744 INFO:tasks.workunit.client.1.smithi203.stdout:3/787: chown d0/dc/d1a/d1d/f9b 141 1 2024-04-05T03:02:55.745 INFO:tasks.workunit.client.1.smithi203.stdout:4/762: write da/d29/d2e/f35 [3999496,80967] 0 2024-04-05T03:02:55.749 INFO:tasks.workunit.client.0.smithi153.stdout:4/746: link da/d2a/fcd da/de/f101 0 2024-04-05T03:02:55.752 INFO:tasks.workunit.client.0.smithi153.stdout:2/679: fsync d9/da/d19/d2e/d7e/d53/f83 0 2024-04-05T03:02:55.752 INFO:tasks.workunit.client.1.smithi203.stdout:6/974: link d0/da/f43 d0/da/d54/d53/f11e 0 2024-04-05T03:02:55.754 INFO:tasks.workunit.client.1.smithi203.stdout:5/979: creat d3/d4/dd6/dfe/d132/f14c x:0 0 0 2024-04-05T03:02:55.754 INFO:tasks.workunit.client.0.smithi153.stdout:7/756: rename d2/d9b/c80 to d2/da/d3a/d41/d7a/d54/d85/df6/d59/cfa 0 2024-04-05T03:02:55.755 INFO:tasks.workunit.client.1.smithi203.stdout:4/763: dwrite da/d3c/f49 [0,4194304] 0 2024-04-05T03:02:55.756 INFO:tasks.workunit.client.0.smithi153.stdout:4/747: symlink da/db/d20/d45/l102 0 2024-04-05T03:02:55.756 INFO:tasks.workunit.client.0.smithi153.stdout:7/757: truncate d2/da/d11/d39/f76 890384 0 2024-04-05T03:02:55.757 INFO:tasks.workunit.client.1.smithi203.stdout:5/980: creat d3/d5f/d92/f14d x:0 0 0 2024-04-05T03:02:55.758 INFO:tasks.workunit.client.0.smithi153.stdout:0/584: truncate d9/df/d1e/f80 3122863 0 2024-04-05T03:02:55.759 INFO:tasks.workunit.client.1.smithi203.stdout:4/764: symlink da/d29/d2e/lf0 0 2024-04-05T03:02:55.759 INFO:tasks.workunit.client.1.smithi203.stdout:5/981: chown d3/d4/dd6/dfe/d132/d10e/c148 12007828 1 2024-04-05T03:02:55.760 INFO:tasks.workunit.client.1.smithi203.stdout:3/788: dwrite d0/d21/f6c [4194304,4194304] 0 2024-04-05T03:02:55.761 INFO:tasks.workunit.client.1.smithi203.stdout:6/975: link d0/da/d18/cf1 d0/da/d18/d38/d86/c11f 0 2024-04-05T03:02:55.761 INFO:tasks.workunit.client.0.smithi153.stdout:7/758: creat d2/da/d3a/ffb x:0 0 0 2024-04-05T03:02:55.762 INFO:tasks.workunit.client.1.smithi203.stdout:3/789: fdatasync d0/dc/d1a/d76/fc4 0 2024-04-05T03:02:55.763 INFO:tasks.workunit.client.0.smithi153.stdout:2/680: truncate d9/da/f5f 2369443 0 2024-04-05T03:02:55.763 INFO:tasks.workunit.client.1.smithi203.stdout:5/982: mkdir d3/d4/d94/d14e 0 2024-04-05T03:02:55.769 INFO:tasks.workunit.client.1.smithi203.stdout:5/983: mknod d3/d4/dc/d67/db5/dda/df9/d117/c14f 0 2024-04-05T03:02:55.769 INFO:tasks.workunit.client.0.smithi153.stdout:0/585: mknod d9/d68/d6d/cb2 0 2024-04-05T03:02:55.770 INFO:tasks.workunit.client.1.smithi203.stdout:5/984: chown d3/d4/d94/d14e 227 1 2024-04-05T03:02:55.773 INFO:tasks.workunit.client.0.smithi153.stdout:2/681: mkdir d9/da/d19/d2e/d7e/d7a/d6d/de8 0 2024-04-05T03:02:55.774 INFO:tasks.workunit.client.1.smithi203.stdout:5/985: symlink d3/d4/dd6/dfe/l150 0 2024-04-05T03:02:55.774 INFO:tasks.workunit.client.1.smithi203.stdout:6/976: dwrite d0/da/d54/f101 [0,4194304] 0 2024-04-05T03:02:55.774 INFO:tasks.workunit.client.0.smithi153.stdout:1/562: truncate d5/d8/d20/d3f/f91 3499134 0 2024-04-05T03:02:55.775 INFO:tasks.workunit.client.0.smithi153.stdout:4/748: creat da/f103 x:0 0 0 2024-04-05T03:02:55.776 INFO:tasks.workunit.client.1.smithi203.stdout:5/986: write d3/d4/dd6/dfe/d132/d59/d60/f74 [4604508,50348] 0 2024-04-05T03:02:55.777 INFO:tasks.workunit.client.0.smithi153.stdout:7/759: creat d2/da/d47/d5d/d7c/d8d/d95/dce/dc8/dde/ffc x:0 0 0 2024-04-05T03:02:55.777 INFO:tasks.workunit.client.1.smithi203.stdout:5/987: dread - d3/d4/dc/d67/dac/ff1 zero size 2024-04-05T03:02:55.778 INFO:tasks.workunit.client.1.smithi203.stdout:5/988: dread - d3/d4/dd6/dfe/d132/d59/d60/dca/f121 zero size 2024-04-05T03:02:55.779 INFO:tasks.workunit.client.0.smithi153.stdout:0/586: unlink d9/d14/f19 0 2024-04-05T03:02:55.779 INFO:tasks.workunit.client.1.smithi203.stdout:6/977: creat d0/da/d54/d4c/da1/f120 x:0 0 0 2024-04-05T03:02:55.780 INFO:tasks.workunit.client.0.smithi153.stdout:7/760: dread - d2/da/d3a/ff8 zero size 2024-04-05T03:02:55.783 INFO:tasks.workunit.client.0.smithi153.stdout:4/749: creat da/db/d20/d3c/d81/f104 x:0 0 0 2024-04-05T03:02:55.783 INFO:tasks.workunit.client.0.smithi153.stdout:2/682: dwrite d9/da/d19/f6a [0,4194304] 0 2024-04-05T03:02:55.786 INFO:tasks.workunit.client.1.smithi203.stdout:5/989: mknod d3/d4/dd6/dfe/d132/d137/c151 0 2024-04-05T03:02:55.791 INFO:tasks.workunit.client.1.smithi203.stdout:6/978: creat d0/da/d18/d92/dfe/f121 x:0 0 0 2024-04-05T03:02:55.791 INFO:tasks.workunit.client.0.smithi153.stdout:4/750: write da/db/d20/d3c/d81/db6/f50 [1336648,93041] 0 2024-04-05T03:02:55.792 INFO:tasks.workunit.client.1.smithi203.stdout:4/765: truncate da/d29/d2e/db0/dbe/dce/dd0/fb6 381064 0 2024-04-05T03:02:55.793 INFO:tasks.workunit.client.1.smithi203.stdout:5/990: read - d3/d5f/d92/fee zero size 2024-04-05T03:02:55.795 INFO:tasks.workunit.client.1.smithi203.stdout:3/790: dread d0/d41/f9a [0,4194304] 0 2024-04-05T03:02:55.795 INFO:tasks.workunit.client.0.smithi153.stdout:2/683: dwrite d9/d68/fba [0,4194304] 0 2024-04-05T03:02:55.796 INFO:tasks.workunit.client.1.smithi203.stdout:4/766: write da/d29/d2e/fb1 [869736,101440] 0 2024-04-05T03:02:55.799 INFO:tasks.workunit.client.0.smithi153.stdout:1/563: getdents d5 0 2024-04-05T03:02:55.799 INFO:tasks.workunit.client.0.smithi153.stdout:2/684: chown d9/da/d19/d2e/d7e/d7a/fda 101556784 1 2024-04-05T03:02:55.800 INFO:tasks.workunit.client.0.smithi153.stdout:4/751: mknod da/db/c105 0 2024-04-05T03:02:55.802 INFO:tasks.workunit.client.0.smithi153.stdout:2/685: write d9/da/d19/d2e/d7e/f5e [1582748,40421] 0 2024-04-05T03:02:55.805 INFO:tasks.workunit.client.0.smithi153.stdout:4/752: fdatasync da/f1d 0 2024-04-05T03:02:55.805 INFO:tasks.workunit.client.0.smithi153.stdout:3/624: sync 2024-04-05T03:02:55.806 INFO:tasks.workunit.client.0.smithi153.stdout:9/472: sync 2024-04-05T03:02:55.808 INFO:tasks.workunit.client.0.smithi153.stdout:2/686: mkdir d9/d68/d98/de9 0 2024-04-05T03:02:55.811 INFO:tasks.workunit.client.0.smithi153.stdout:9/473: chown d1/d6c/la2 11 1 2024-04-05T03:02:55.812 INFO:tasks.workunit.client.0.smithi153.stdout:1/564: dwrite d5/d8/d20/d4b/d18/d16/f5b [0,4194304] 0 2024-04-05T03:02:55.814 INFO:tasks.workunit.client.0.smithi153.stdout:3/625: unlink d1d/d32/d93/l6e 0 2024-04-05T03:02:55.817 INFO:tasks.workunit.client.0.smithi153.stdout:7/761: dread d2/da/d3a/d41/d7a/f4e [4194304,4194304] 0 2024-04-05T03:02:55.818 INFO:tasks.workunit.client.0.smithi153.stdout:1/565: mknod d5/d8/d20/d4b/d18/d16/d48/cb0 0 2024-04-05T03:02:55.819 INFO:tasks.workunit.client.1.smithi203.stdout:5/991: mknod d3/d4/d9e/dde/c152 0 2024-04-05T03:02:55.820 INFO:tasks.workunit.client.1.smithi203.stdout:3/791: creat d0/dc/d38/d5d/d66/d80/fe7 x:0 0 0 2024-04-05T03:02:55.823 INFO:tasks.workunit.client.0.smithi153.stdout:0/587: dwrite d9/db/d42/d43/f76 [0,4194304] 0 2024-04-05T03:02:55.824 INFO:tasks.workunit.client.1.smithi203.stdout:6/979: mkdir d0/d122 0 2024-04-05T03:02:55.824 INFO:tasks.workunit.client.1.smithi203.stdout:3/792: read d0/dc/d34/f6e [3780720,102575] 0 2024-04-05T03:02:55.826 INFO:tasks.workunit.client.1.smithi203.stdout:5/992: unlink d3/d4/dc/d67/db5/dda/fef 0 2024-04-05T03:02:55.826 INFO:tasks.workunit.client.0.smithi153.stdout:3/626: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/lc7 to d1d/d27/le2 0 2024-04-05T03:02:55.829 INFO:tasks.workunit.client.1.smithi203.stdout:6/980: truncate d0/da/d54/d4c/daf/d11a/d44/d78/f89 955183 0 2024-04-05T03:02:55.831 INFO:tasks.workunit.client.1.smithi203.stdout:4/767: link da/d29/l8a da/d29/lf1 0 2024-04-05T03:02:55.831 INFO:tasks.workunit.client.1.smithi203.stdout:3/793: rename d0/d21/lb5 to d0/dc/d1a/d8d/dc9/le8 0 2024-04-05T03:02:55.833 INFO:tasks.workunit.client.0.smithi153.stdout:1/566: dread - d5/d8/d20/d4b/d51/f7f zero size 2024-04-05T03:02:55.833 INFO:tasks.workunit.client.0.smithi153.stdout:0/588: symlink d9/df/d1e/d84/lb3 0 2024-04-05T03:02:55.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/567: dread - d5/d8/d20/d4b/d51/f7f zero size 2024-04-05T03:02:55.836 INFO:tasks.workunit.client.1.smithi203.stdout:6/981: symlink d0/da/d54/l123 0 2024-04-05T03:02:55.837 INFO:tasks.workunit.client.1.smithi203.stdout:6/982: readlink d0/db4/l111 0 2024-04-05T03:02:55.839 INFO:tasks.workunit.client.0.smithi153.stdout:4/753: link da/db/d20/d35/d38/ccc da/db/d20/d45/dc3/c106 0 2024-04-05T03:02:55.841 INFO:tasks.workunit.client.1.smithi203.stdout:5/993: getdents d3/d4/dd6/dfe/d132/d59/d60/dca 0 2024-04-05T03:02:55.841 INFO:tasks.workunit.client.0.smithi153.stdout:4/754: dread - da/f103 zero size 2024-04-05T03:02:55.842 INFO:tasks.workunit.client.0.smithi153.stdout:2/687: truncate d9/da/d37/f3d 3217464 0 2024-04-05T03:02:55.842 INFO:tasks.workunit.client.0.smithi153.stdout:3/627: dread d1d/d27/f8f [0,4194304] 0 2024-04-05T03:02:55.844 INFO:tasks.workunit.client.0.smithi153.stdout:0/589: read d9/f4e [3594189,116575] 0 2024-04-05T03:02:55.844 INFO:tasks.workunit.client.0.smithi153.stdout:1/568: dwrite d5/d8/d20/d4b/d18/d16/d33/f3c [0,4194304] 0 2024-04-05T03:02:55.844 INFO:tasks.workunit.client.0.smithi153.stdout:7/762: creat d2/ffd x:0 0 0 2024-04-05T03:02:55.845 INFO:tasks.workunit.client.1.smithi203.stdout:5/994: creat d3/d4/dc/d67/dac/f153 x:0 0 0 2024-04-05T03:02:55.846 INFO:tasks.workunit.client.0.smithi153.stdout:9/474: dwrite d1/d1d/f5a [0,4194304] 0 2024-04-05T03:02:55.846 INFO:tasks.workunit.client.0.smithi153.stdout:4/755: chown da/de/da1/fc0 3 1 2024-04-05T03:02:55.848 INFO:tasks.workunit.client.1.smithi203.stdout:6/983: dwrite d0/da/d54/d4c/da1/db7/d95/fe8 [4194304,4194304] 0 2024-04-05T03:02:55.848 INFO:tasks.workunit.client.0.smithi153.stdout:3/628: chown d1d/d32/d93/d3c/d3f/d7c 0 1 2024-04-05T03:02:55.849 INFO:tasks.workunit.client.0.smithi153.stdout:7/763: fdatasync d2/da/d11/f72 0 2024-04-05T03:02:55.850 INFO:tasks.workunit.client.0.smithi153.stdout:2/688: chown d9/da/d19/d2e/d7e/de6/c32 9244554 1 2024-04-05T03:02:55.854 INFO:tasks.workunit.client.0.smithi153.stdout:2/689: readlink l2 0 2024-04-05T03:02:55.855 INFO:tasks.workunit.client.0.smithi153.stdout:1/569: creat d5/d8/d20/d4b/fb1 x:0 0 0 2024-04-05T03:02:55.857 INFO:tasks.workunit.client.0.smithi153.stdout:8/509: sync 2024-04-05T03:02:55.857 INFO:tasks.workunit.client.0.smithi153.stdout:6/741: sync 2024-04-05T03:02:55.857 INFO:tasks.workunit.client.0.smithi153.stdout:5/572: sync 2024-04-05T03:02:55.860 INFO:tasks.workunit.client.0.smithi153.stdout:0/590: symlink d9/df/d1d/d61/d8a/lb4 0 2024-04-05T03:02:55.861 INFO:tasks.workunit.client.1.smithi203.stdout:5/995: unlink d3/d4/dc/d67/db5/dda/l128 0 2024-04-05T03:02:55.862 INFO:tasks.workunit.client.0.smithi153.stdout:8/510: write d7/d2a/d50/f6b [914594,107229] 0 2024-04-05T03:02:55.863 INFO:tasks.workunit.client.1.smithi203.stdout:5/996: write d3/d4/dd6/dfe/d132/f33 [5350274,106937] 0 2024-04-05T03:02:55.865 INFO:tasks.workunit.client.1.smithi203.stdout:5/997: truncate d3/d4/dd6/dfe/d132/d49/f13a 958929 0 2024-04-05T03:02:55.866 INFO:tasks.workunit.client.0.smithi153.stdout:3/629: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/fda zero size 2024-04-05T03:02:55.866 INFO:tasks.workunit.client.0.smithi153.stdout:7/764: fdatasync d2/da/d3a/d41/d7a/d54/d85/df6/fa5 0 2024-04-05T03:02:55.868 INFO:tasks.workunit.client.0.smithi153.stdout:9/475: mkdir d1/d10/d16/d22/d2c/d68/da9 0 2024-04-05T03:02:55.870 INFO:tasks.workunit.client.1.smithi203.stdout:5/998: creat d3/d4/dc/d67/db5/dda/df9/d117/f154 x:0 0 0 2024-04-05T03:02:55.871 INFO:tasks.workunit.client.0.smithi153.stdout:4/756: dread - da/db/d20/d35/d38/f6d zero size 2024-04-05T03:02:55.871 INFO:tasks.workunit.client.0.smithi153.stdout:9/476: readlink d1/d6c/la2 0 2024-04-05T03:02:55.872 INFO:tasks.workunit.client.0.smithi153.stdout:7/765: fdatasync d2/da/d11/f52 0 2024-04-05T03:02:55.872 INFO:tasks.workunit.client.0.smithi153.stdout:2/690: creat d9/da/d19/de2/fea x:0 0 0 2024-04-05T03:02:55.873 INFO:tasks.workunit.client.1.smithi203.stdout:5/999: read d3/d4/d9e/fab [4162406,99512] 0 2024-04-05T03:02:55.874 INFO:tasks.workunit.client.0.smithi153.stdout:1/570: dwrite d5/d8/d20/f88 [4194304,4194304] 0 2024-04-05T03:02:55.874 INFO:tasks.workunit.client.0.smithi153.stdout:6/742: mkdir d4/dc/df/d16/d36/df9 0 2024-04-05T03:02:55.875 INFO:tasks.workunit.client.0.smithi153.stdout:8/511: dread f3 [4194304,4194304] 0 2024-04-05T03:02:55.878 INFO:tasks.workunit.client.1.smithi203.stdout:4/768: mknod da/d29/d2e/d57/d96/ddc/de0/cf2 0 2024-04-05T03:02:55.878 INFO:tasks.workunit.client.0.smithi153.stdout:2/691: mknod d9/d68/ceb 0 2024-04-05T03:02:55.878 INFO:tasks.workunit.client.0.smithi153.stdout:7/766: fsync d2/da/d3a/d41/ff1 0 2024-04-05T03:02:55.879 INFO:tasks.workunit.client.1.smithi203.stdout:3/794: truncate d0/dc/d1a/d8d/d96/f90 2563248 0 2024-04-05T03:02:55.880 INFO:tasks.workunit.client.0.smithi153.stdout:9/477: write d1/d10/d36/d57/d5c/f89 [949195,76502] 0 2024-04-05T03:02:55.880 INFO:tasks.workunit.client.0.smithi153.stdout:3/630: rmdir d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f 39 2024-04-05T03:02:55.885 INFO:tasks.workunit.client.0.smithi153.stdout:2/692: fsync d9/da/d19/d2e/d7e/d7a/fda 0 2024-04-05T03:02:55.885 INFO:tasks.workunit.client.0.smithi153.stdout:9/478: read d1/d10/f67 [194007,17667] 0 2024-04-05T03:02:55.885 INFO:tasks.workunit.client.1.smithi203.stdout:4/769: rename da/d29/f2b to da/d29/d2e/d57/d96/ddc/ff3 0 2024-04-05T03:02:55.887 INFO:tasks.workunit.client.1.smithi203.stdout:0/979: sync 2024-04-05T03:02:55.887 INFO:tasks.workunit.client.0.smithi153.stdout:7/767: mkdir d2/da/d11/d39/dfe 0 2024-04-05T03:02:55.889 INFO:tasks.workunit.client.1.smithi203.stdout:3/795: dwrite d0/d21/d81/f7b [0,4194304] 0 2024-04-05T03:02:55.894 INFO:tasks.workunit.client.1.smithi203.stdout:3/796: chown d0/dc/d38/d5d/d66/d80/le3 3 1 2024-04-05T03:02:55.894 INFO:tasks.workunit.client.1.smithi203.stdout:0/980: rmdir dd/d17/d1d/d95 39 2024-04-05T03:02:55.896 INFO:tasks.workunit.client.0.smithi153.stdout:9/479: dwrite d1/d10/f39 [0,4194304] 0 2024-04-05T03:02:55.897 INFO:tasks.workunit.client.1.smithi203.stdout:3/797: unlink d0/dc/d1a/d76/la4 0 2024-04-05T03:02:55.897 INFO:tasks.workunit.client.0.smithi153.stdout:8/512: link d7/d2a/fa1 d7/d46/fa7 0 2024-04-05T03:02:55.904 INFO:tasks.workunit.client.0.smithi153.stdout:3/631: link d1d/fcf d1d/d27/fe3 0 2024-04-05T03:02:55.904 INFO:tasks.workunit.client.1.smithi203.stdout:3/798: rename d0/d21/d83/l8a to d0/dc/d1a/d1f/da1/daf/le9 0 2024-04-05T03:02:55.909 INFO:tasks.workunit.client.1.smithi203.stdout:6/984: dread d0/f3 [0,4194304] 0 2024-04-05T03:02:55.911 INFO:tasks.workunit.client.0.smithi153.stdout:2/693: dwrite d9/d68/d98/fac [0,4194304] 0 2024-04-05T03:02:55.913 INFO:tasks.workunit.client.0.smithi153.stdout:0/591: dwrite d9/df/d1e/f54 [0,4194304] 0 2024-04-05T03:02:55.914 INFO:tasks.workunit.client.1.smithi203.stdout:3/799: mknod d0/dc/d38/cea 0 2024-04-05T03:02:55.918 INFO:tasks.workunit.client.0.smithi153.stdout:9/480: mknod d1/d10/d16/d22/caa 0 2024-04-05T03:02:55.920 INFO:tasks.workunit.client.0.smithi153.stdout:3/632: write d1d/d32/d93/d3c/d94/fd1 [926776,130585] 0 2024-04-05T03:02:55.920 INFO:tasks.workunit.client.0.smithi153.stdout:7/768: creat d2/da/d3a/d41/d7a/d54/d85/df6/d93/fff x:0 0 0 2024-04-05T03:02:55.921 INFO:tasks.workunit.client.0.smithi153.stdout:1/571: dwrite d5/dd/f89 [0,4194304] 0 2024-04-05T03:02:55.922 INFO:tasks.workunit.client.0.smithi153.stdout:2/694: write d9/da/d19/d2e/d7e/d53/f9c [343340,102611] 0 2024-04-05T03:02:55.926 INFO:tasks.workunit.client.0.smithi153.stdout:9/481: readlink d1/d1d/l42 0 2024-04-05T03:02:55.926 INFO:tasks.workunit.client.0.smithi153.stdout:7/769: write d2/da/d3a/ffb [146721,60264] 0 2024-04-05T03:02:55.928 INFO:tasks.workunit.client.0.smithi153.stdout:9/482: dread - d1/d3a/d91/f9b zero size 2024-04-05T03:02:55.929 INFO:tasks.workunit.client.0.smithi153.stdout:7/770: write d2/da/d47/d5d/d7c/f81 [1003609,79481] 0 2024-04-05T03:02:55.929 INFO:tasks.workunit.client.0.smithi153.stdout:9/483: readlink d1/d1d/l90 0 2024-04-05T03:02:55.929 INFO:tasks.workunit.client.0.smithi153.stdout:9/484: fdatasync d1/f2e 0 2024-04-05T03:02:55.931 INFO:tasks.workunit.client.0.smithi153.stdout:6/743: dread d4/dc/df/d16/d36/d4d/f91 [0,4194304] 0 2024-04-05T03:02:55.934 INFO:tasks.workunit.client.0.smithi153.stdout:4/757: dread da/de/d56/d78/fce [0,4194304] 0 2024-04-05T03:02:55.935 INFO:tasks.workunit.client.0.smithi153.stdout:3/633: symlink d1d/d32/d93/le4 0 2024-04-05T03:02:55.936 INFO:tasks.workunit.client.0.smithi153.stdout:2/695: mknod d9/da/d94/dbf/cec 0 2024-04-05T03:02:55.941 INFO:tasks.workunit.client.0.smithi153.stdout:9/485: mknod d1/d10/d16/d22/cab 0 2024-04-05T03:02:55.942 INFO:tasks.workunit.client.0.smithi153.stdout:7/771: symlink d2/da/d11/d88/da6/l100 0 2024-04-05T03:02:55.944 INFO:tasks.workunit.client.0.smithi153.stdout:9/486: write d1/d3a/d91/f9b [801209,32448] 0 2024-04-05T03:02:55.945 INFO:tasks.workunit.client.0.smithi153.stdout:6/744: fdatasync d4/f20 0 2024-04-05T03:02:55.947 INFO:tasks.workunit.client.0.smithi153.stdout:3/634: dwrite d1d/d27/d2b/f62 [0,4194304] 0 2024-04-05T03:02:55.947 INFO:tasks.workunit.client.0.smithi153.stdout:7/772: chown d2/da/d3a/d41/ff1 20678292 1 2024-04-05T03:02:55.947 INFO:tasks.workunit.client.0.smithi153.stdout:4/758: creat da/db/dcf/f107 x:0 0 0 2024-04-05T03:02:55.948 INFO:tasks.workunit.client.0.smithi153.stdout:3/635: chown d1d/f28 1 1 2024-04-05T03:02:55.949 INFO:tasks.workunit.client.0.smithi153.stdout:4/759: chown da/l4c 26958 1 2024-04-05T03:02:55.950 INFO:tasks.workunit.client.0.smithi153.stdout:9/487: symlink d1/d10/d36/d57/d97/lac 0 2024-04-05T03:02:55.951 INFO:tasks.workunit.client.0.smithi153.stdout:2/696: link d9/da/d37/l5d d9/da/d37/d8a/da8/led 0 2024-04-05T03:02:55.951 INFO:tasks.workunit.client.0.smithi153.stdout:4/760: chown da/db/d20/f32 297477 1 2024-04-05T03:02:55.954 INFO:tasks.workunit.client.0.smithi153.stdout:6/745: mkdir d4/dc/df/d16/d24/d5e/de8/dfa 0 2024-04-05T03:02:55.954 INFO:tasks.workunit.client.0.smithi153.stdout:8/513: write d7/d13/f14 [3351094,83111] 0 2024-04-05T03:02:55.956 INFO:tasks.workunit.client.0.smithi153.stdout:2/697: chown d9/da/d19/d2e/d7e/d53/cc8 158015294 1 2024-04-05T03:02:55.956 INFO:tasks.workunit.client.0.smithi153.stdout:9/488: creat d1/d10/d16/d22/d2c/d68/fad x:0 0 0 2024-04-05T03:02:55.957 INFO:tasks.workunit.client.0.smithi153.stdout:7/773: creat d2/da/d3a/d41/d7a/d54/d85/df6/d77/f101 x:0 0 0 2024-04-05T03:02:55.958 INFO:tasks.workunit.client.0.smithi153.stdout:2/698: chown d9/da/d19/f1f 428294 1 2024-04-05T03:02:55.959 INFO:tasks.workunit.client.0.smithi153.stdout:4/761: fsync da/f26 0 2024-04-05T03:02:55.960 INFO:tasks.workunit.client.0.smithi153.stdout:8/514: dread d7/d13/d19/f8f [0,4194304] 0 2024-04-05T03:02:55.961 INFO:tasks.workunit.client.0.smithi153.stdout:2/699: chown d9/c9f 441 1 2024-04-05T03:02:55.962 INFO:tasks.workunit.client.0.smithi153.stdout:7/774: mknod d2/d9b/d6f/c102 0 2024-04-05T03:02:55.963 INFO:tasks.workunit.client.0.smithi153.stdout:1/572: write d5/dd/d1b/f74 [600349,57675] 0 2024-04-05T03:02:55.967 INFO:tasks.workunit.client.0.smithi153.stdout:4/762: write da/de/d56/d78/fce [473008,35111] 0 2024-04-05T03:02:55.967 INFO:tasks.workunit.client.0.smithi153.stdout:8/515: truncate d7/d13/f18 1709444 0 2024-04-05T03:02:55.968 INFO:tasks.workunit.client.1.smithi203.stdout:4/770: fdatasync da/d29/d2e/d53/d78/f7d 0 2024-04-05T03:02:55.970 INFO:tasks.workunit.client.1.smithi203.stdout:3/800: dwrite d0/dc/d1a/d76/fc4 [0,4194304] 0 2024-04-05T03:02:55.971 INFO:tasks.workunit.client.0.smithi153.stdout:4/763: read da/db/d20/d35/d38/d76/d90/fd6 [349881,15267] 0 2024-04-05T03:02:55.972 INFO:tasks.workunit.client.0.smithi153.stdout:0/592: dwrite d9/df/d1e/f50 [0,4194304] 0 2024-04-05T03:02:55.972 INFO:tasks.workunit.client.1.smithi203.stdout:6/985: dwrite d0/da/d54/d4c/da1/db7/f5f [0,4194304] 0 2024-04-05T03:02:55.973 INFO:tasks.workunit.client.0.smithi153.stdout:9/489: creat d1/d10/d16/fae x:0 0 0 2024-04-05T03:02:55.975 INFO:tasks.workunit.client.1.smithi203.stdout:4/771: rename da/d3c/d4c/d99/fea to da/d29/d2e/d8b/ff4 0 2024-04-05T03:02:55.976 INFO:tasks.workunit.client.1.smithi203.stdout:3/801: creat d0/dc/d38/d5d/d66/d80/feb x:0 0 0 2024-04-05T03:02:55.978 INFO:tasks.workunit.client.0.smithi153.stdout:1/573: dread - d5/d8/d20/d3f/f7b zero size 2024-04-05T03:02:55.978 INFO:tasks.workunit.client.1.smithi203.stdout:3/802: stat d0/dc/d1a/d8d/d96 0 2024-04-05T03:02:55.982 INFO:tasks.workunit.client.1.smithi203.stdout:6/986: rename d0/da/d54/d4c/d7f/dcc/cb5 to d0/da/d54/d4c/da1/db7/d71/c124 0 2024-04-05T03:02:55.983 INFO:tasks.workunit.client.0.smithi153.stdout:6/746: link d4/dc/df/l35 d4/dc/lfb 0 2024-04-05T03:02:55.983 INFO:tasks.workunit.client.1.smithi203.stdout:4/772: mkdir da/d29/d2e/d57/d96/dbc/df5 0 2024-04-05T03:02:55.984 INFO:tasks.workunit.client.1.smithi203.stdout:3/803: dread d0/dc/d1a/d1d/d53/f73 [0,4194304] 0 2024-04-05T03:02:55.987 INFO:tasks.workunit.client.1.smithi203.stdout:6/987: mkdir d0/da/d54/d4c/daf/d11a/d44/d78/d125 0 2024-04-05T03:02:55.988 INFO:tasks.workunit.client.1.smithi203.stdout:4/773: symlink da/d29/d2e/d57/d96/dbc/df5/lf6 0 2024-04-05T03:02:55.989 INFO:tasks.workunit.client.0.smithi153.stdout:9/490: symlink d1/d10/d16/d22/laf 0 2024-04-05T03:02:55.990 INFO:tasks.workunit.client.1.smithi203.stdout:6/988: write d0/da/d54/d4c/da1/db7/fcb [1160658,2608] 0 2024-04-05T03:02:55.991 INFO:tasks.workunit.client.0.smithi153.stdout:7/775: creat d2/da/d47/d5d/f103 x:0 0 0 2024-04-05T03:02:55.991 INFO:tasks.workunit.client.1.smithi203.stdout:6/989: chown d0/da/d54/d4c/da1/db7/d71 225709692 1 2024-04-05T03:02:55.991 INFO:tasks.workunit.client.1.smithi203.stdout:4/774: stat da/d29/d2e/dc9/cde 0 2024-04-05T03:02:55.993 INFO:tasks.workunit.client.1.smithi203.stdout:6/990: stat d0/da/d54/d4c/daf/d11a/d44/d78/cf3 0 2024-04-05T03:02:55.993 INFO:tasks.workunit.client.1.smithi203.stdout:4/775: truncate da/d29/d2e/d53/f68 4319383 0 2024-04-05T03:02:55.994 INFO:tasks.workunit.client.0.smithi153.stdout:7/776: chown d2/da/d11/d88/da6/fd9 3800 1 2024-04-05T03:02:55.994 INFO:tasks.workunit.client.0.smithi153.stdout:1/574: creat d5/d8/d20/d4b/d18/fb2 x:0 0 0 2024-04-05T03:02:55.995 INFO:tasks.workunit.client.0.smithi153.stdout:3/636: truncate d1d/fa7 5583533 0 2024-04-05T03:02:55.995 INFO:tasks.workunit.client.0.smithi153.stdout:7/777: fsync d2/da/d11/d39/fb6 0 2024-04-05T03:02:55.996 INFO:tasks.workunit.client.1.smithi203.stdout:6/991: symlink d0/da/d54/d4c/daf/d11a/d44/d78/d125/l126 0 2024-04-05T03:02:55.998 INFO:tasks.workunit.client.0.smithi153.stdout:2/700: rename d9/da/d37/d8a/da8/l8b to d9/da/d19/d2e/d7e/d7a/lee 0 2024-04-05T03:02:55.999 INFO:tasks.workunit.client.0.smithi153.stdout:9/491: dwrite d1/d10/d16/d22/d2c/f58 [0,4194304] 0 2024-04-05T03:02:56.001 INFO:tasks.workunit.client.0.smithi153.stdout:1/575: chown d5/d8/d20/l42 747023 1 2024-04-05T03:02:56.002 INFO:tasks.workunit.client.1.smithi203.stdout:4/776: dread da/d29/d2e/f63 [0,4194304] 0 2024-04-05T03:02:56.004 INFO:tasks.workunit.client.0.smithi153.stdout:8/516: link d7/d13/d19/d1b/d21/c25 d7/d13/ca8 0 2024-04-05T03:02:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.007+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff060115d50 con 0x7ff04806b540 2024-04-05T03:02:56.009 INFO:tasks.workunit.client.0.smithi153.stdout:7/778: rmdir d2/da/d3a/d41/d7a/d54/d85/df6/d93 39 2024-04-05T03:02:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.009+0000 7ff0477fe700 1 -- 172.21.15.153:0/1768128357 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7ff060115d50 con 0x7ff04806b540 2024-04-05T03:02:56.010 INFO:tasks.workunit.client.1.smithi203.stdout:4/777: link da/d29/d2e/d53/f97 da/d29/d2e/db0/dbe/dce/dd0/ff7 0 2024-04-05T03:02:56.011 INFO:tasks.workunit.client.0.smithi153.stdout:9/492: truncate d1/d10/d16/f17 2908977 0 2024-04-05T03:02:56.011 INFO:tasks.workunit.client.1.smithi203.stdout:4/778: truncate da/f93 698219 0 2024-04-05T03:02:56.012 INFO:tasks.workunit.client.0.smithi153.stdout:2/701: dwrite d9/da/d19/d2e/d7e/de6/f33 [4194304,4194304] 0 2024-04-05T03:02:56.012 INFO:tasks.workunit.client.1.smithi203.stdout:4/779: readlink da/d29/d2e/d57/da5/lb8 0 2024-04-05T03:02:56.012 INFO:tasks.workunit.client.1.smithi203.stdout:3/804: truncate d0/dc/d1a/d1d/d53/d5a/fab 2072610 0 2024-04-05T03:02:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 msgr2=0x7ff04806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 0x7ff04806d9f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff05000b0d0 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 msgr2=0x7ff060113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.015 INFO:tasks.workunit.client.1.smithi203.stdout:4/780: creat da/d29/d2e/d57/d96/ddc/de1/ff8 x:0 0 0 2024-04-05T03:02:56.015 INFO:tasks.workunit.client.1.smithi203.stdout:6/992: write d0/f9d [486488,33441] 0 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff05400eb40 tx=0x7ff05400b040 comp rx=0 tx=0).stop 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 shutdown_connections 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0600fbce0 0x7ff060113f90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff04806b540 0x7ff04806d9f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 --2- 172.21.15.153:0/1768128357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0600fcf90 0x7ff0601144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 >> 172.21.15.153:0/1768128357 conn(0x7ff0600f7440 msgr2=0x7ff0600f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 shutdown_connections 2024-04-05T03:02:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.014+0000 7ff065589700 1 -- 172.21.15.153:0/1768128357 wait complete. 2024-04-05T03:02:56.017 INFO:tasks.workunit.client.0.smithi153.stdout:0/593: getdents d9/db/d42/d7e 0 2024-04-05T03:02:56.018 INFO:tasks.workunit.client.1.smithi203.stdout:4/781: mkdir da/d29/d2e/d53/df9 0 2024-04-05T03:02:56.019 INFO:tasks.workunit.client.0.smithi153.stdout:6/747: getdents d4/dc/df/d16/d36 0 2024-04-05T03:02:56.020 INFO:tasks.workunit.client.1.smithi203.stdout:6/993: creat d0/da/d54/d4c/daf/d11a/d44/d78/f127 x:0 0 0 2024-04-05T03:02:56.022 INFO:tasks.workunit.client.1.smithi203.stdout:4/782: creat da/d3c/d4c/d99/ffa x:0 0 0 2024-04-05T03:02:56.024 INFO:tasks.workunit.client.0.smithi153.stdout:8/517: dwrite d7/d13/f91 [0,4194304] 0 2024-04-05T03:02:56.026 INFO:tasks.workunit.client.0.smithi153.stdout:1/576: dwrite d5/d8/d20/d4b/d18/d16/d2c/f7d [0,4194304] 0 2024-04-05T03:02:56.026 INFO:tasks.workunit.client.1.smithi203.stdout:4/783: dread da/d29/d2e/db0/dcb/d80/fc7 [0,4194304] 0 2024-04-05T03:02:56.032 INFO:tasks.workunit.client.0.smithi153.stdout:3/637: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f67 [0,4194304] 0 2024-04-05T03:02:56.032 INFO:tasks.workunit.client.1.smithi203.stdout:6/994: rename d0/da/d54/d4c/daf/d11a/d44/d78/f127 to d0/da/d54/d4c/da1/db7/f128 0 2024-04-05T03:02:56.033 INFO:tasks.workunit.client.0.smithi153.stdout:8/518: write d7/d13/d19/f22 [443493,104835] 0 2024-04-05T03:02:56.035 INFO:tasks.workunit.client.1.smithi203.stdout:3/805: dread d0/dc/d1a/d8d/d96/fcc [0,4194304] 0 2024-04-05T03:02:56.036 INFO:tasks.workunit.client.0.smithi153.stdout:7/779: symlink d2/da/d47/d5d/d7c/d8d/df2/l104 0 2024-04-05T03:02:56.037 INFO:tasks.workunit.client.0.smithi153.stdout:3/638: stat d1d/d32/d93/d3c/d3f/d7c/f8e 0 2024-04-05T03:02:56.039 INFO:tasks.workunit.client.0.smithi153.stdout:8/519: chown d7/d85/f6a 54 1 2024-04-05T03:02:56.046 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:02:56.047 INFO:tasks.workunit.client.0.smithi153.stdout:9/493: write d1/d9/d1b/f26 [866155,84880] 0 2024-04-05T03:02:56.050 INFO:tasks.workunit.client.0.smithi153.stdout:6/748: dread d4/d38/f8b [0,4194304] 0 2024-04-05T03:02:56.053 INFO:tasks.workunit.client.0.smithi153.stdout:9/494: write d1/d10/d16/d22/d2c/d68/d81/f76 [1514580,96058] 0 2024-04-05T03:02:56.053 INFO:tasks.workunit.client.1.smithi203.stdout:4/784: dwrite da/d71/f81 [0,4194304] 0 2024-04-05T03:02:56.054 INFO:tasks.workunit.client.0.smithi153.stdout:8/520: creat d7/fa9 x:0 0 0 2024-04-05T03:02:56.056 INFO:tasks.workunit.client.1.smithi203.stdout:4/785: stat da/d3c/c8f 0 2024-04-05T03:02:56.056 INFO:tasks.workunit.client.0.smithi153.stdout:0/594: dwrite d9/d14/f23 [4194304,4194304] 0 2024-04-05T03:02:56.057 INFO:tasks.workunit.client.0.smithi153.stdout:8/521: creat d7/d46/faa x:0 0 0 2024-04-05T03:02:56.060 INFO:tasks.workunit.client.1.smithi203.stdout:4/786: unlink da/d71/cd2 0 2024-04-05T03:02:56.061 INFO:tasks.workunit.client.0.smithi153.stdout:7/780: link d2/da/d11/d39/fe8 d2/da/d47/d5d/d7c/d8d/d95/dce/f105 0 2024-04-05T03:02:56.068 INFO:tasks.workunit.client.0.smithi153.stdout:6/749: dwrite d4/dc/df/d16/d5f/fac [0,4194304] 0 2024-04-05T03:02:56.070 INFO:tasks.workunit.client.1.smithi203.stdout:4/787: link da/l25 da/d29/d2e/d53/df9/lfb 0 2024-04-05T03:02:56.070 INFO:tasks.workunit.client.0.smithi153.stdout:9/495: dwrite d1/d3a/d91/f9f [0,4194304] 0 2024-04-05T03:02:56.074 INFO:tasks.workunit.client.1.smithi203.stdout:4/788: creat da/d29/d2e/d57/d96/ddc/ffc x:0 0 0 2024-04-05T03:02:56.076 INFO:tasks.workunit.client.0.smithi153.stdout:6/750: rename d4/d38/d4c/lc8 to d4/dc/df/d16/d36/d4d/dad/lfc 0 2024-04-05T03:02:56.078 INFO:tasks.workunit.client.0.smithi153.stdout:9/496: creat d1/d9/d1b/fb0 x:0 0 0 2024-04-05T03:02:56.081 INFO:tasks.workunit.client.1.smithi203.stdout:4/789: creat da/d29/ffd x:0 0 0 2024-04-05T03:02:56.081 INFO:tasks.workunit.client.0.smithi153.stdout:6/751: creat d4/dc/ffd x:0 0 0 2024-04-05T03:02:56.082 INFO:tasks.workunit.client.0.smithi153.stdout:9/497: chown d1/d9/d1b/f7b 914 1 2024-04-05T03:02:56.082 INFO:tasks.workunit.client.0.smithi153.stdout:7/781: dwrite d2/da/d3a/d41/d7a/d54/d85/df6/d93/fff [0,4194304] 0 2024-04-05T03:02:56.084 INFO:tasks.workunit.client.0.smithi153.stdout:6/752: fsync d4/dc/d7a/da8/fc5 0 2024-04-05T03:02:56.085 INFO:tasks.workunit.client.0.smithi153.stdout:2/702: truncate d9/da/fb 3849497 0 2024-04-05T03:02:56.085 INFO:tasks.workunit.client.1.smithi203.stdout:4/790: creat da/d29/d2e/d53/ffe x:0 0 0 2024-04-05T03:02:56.087 INFO:tasks.workunit.client.0.smithi153.stdout:9/498: mknod d1/d10/d36/d57/d97/cb1 0 2024-04-05T03:02:56.087 INFO:tasks.workunit.client.1.smithi203.stdout:4/791: creat da/d3c/d4c/fff x:0 0 0 2024-04-05T03:02:56.088 INFO:tasks.workunit.client.0.smithi153.stdout:2/703: dread - d9/da/fd5 zero size 2024-04-05T03:02:56.091 INFO:tasks.workunit.client.0.smithi153.stdout:7/782: mkdir d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106 0 2024-04-05T03:02:56.092 INFO:tasks.workunit.client.1.smithi203.stdout:4/792: rename da/d29/d2e/d53/c6e to da/d29/d2e/db0/c100 0 2024-04-05T03:02:56.094 INFO:tasks.workunit.client.1.smithi203.stdout:4/793: creat da/d71/f101 x:0 0 0 2024-04-05T03:02:56.095 INFO:tasks.workunit.client.0.smithi153.stdout:2/704: mkdir d9/da/d36/def 0 2024-04-05T03:02:56.095 INFO:tasks.workunit.client.0.smithi153.stdout:1/577: truncate d5/d8/d20/d4b/d18/d16/d45/f4a 86202 0 2024-04-05T03:02:56.095 INFO:tasks.workunit.client.0.smithi153.stdout:3/639: dwrite d1d/d27/f8f [0,4194304] 0 2024-04-05T03:02:56.096 INFO:tasks.workunit.client.1.smithi203.stdout:4/794: creat da/d71/f102 x:0 0 0 2024-04-05T03:02:56.097 INFO:tasks.workunit.client.0.smithi153.stdout:1/578: readlink d5/d8/d20/d3f/l73 0 2024-04-05T03:02:56.099 INFO:tasks.workunit.client.0.smithi153.stdout:2/705: mkdir d9/d68/d9b/df0 0 2024-04-05T03:02:56.099 INFO:tasks.workunit.client.1.smithi203.stdout:4/795: rmdir da/d29/d2e/db0 39 2024-04-05T03:02:56.102 INFO:tasks.workunit.client.0.smithi153.stdout:7/783: link d2/da/d3a/d41/f8a d2/da/d3a/d41/ddd/f107 0 2024-04-05T03:02:56.104 INFO:tasks.workunit.client.1.smithi203.stdout:4/796: symlink da/d29/d2e/db0/dbe/l103 0 2024-04-05T03:02:56.106 INFO:tasks.workunit.client.0.smithi153.stdout:0/595: dread d9/db/d42/f7f [0,4194304] 0 2024-04-05T03:02:56.107 INFO:tasks.workunit.client.0.smithi153.stdout:2/706: rename d9/da/d19/f1f to d9/da/d37/d8a/ff1 0 2024-04-05T03:02:56.108 INFO:tasks.workunit.client.0.smithi153.stdout:1/579: dwrite d5/d8/d20/d4b/d18/d16/d45/d93/f9a [0,4194304] 0 2024-04-05T03:02:56.110 INFO:tasks.workunit.client.0.smithi153.stdout:2/707: readlink d9/da/d19/d2e/d7e/d7a/l89 0 2024-04-05T03:02:56.110 INFO:tasks.workunit.client.0.smithi153.stdout:7/784: readlink d2/da/d3a/d41/d7a/l2f 0 2024-04-05T03:02:56.111 INFO:tasks.workunit.client.1.smithi203.stdout:4/797: read da/d3c/f44 [4010105,99456] 0 2024-04-05T03:02:56.112 INFO:tasks.workunit.client.0.smithi153.stdout:0/596: rename d9/df/c97 to d9/df/d1e/d45/dae/cb5 0 2024-04-05T03:02:56.116 INFO:tasks.workunit.client.0.smithi153.stdout:1/580: dread d5/d8/d20/d4b/d18/d16/d45/d93/f9a [0,4194304] 0 2024-04-05T03:02:56.118 INFO:tasks.workunit.client.0.smithi153.stdout:7/785: write d2/da/d11/d88/da6/fd9 [767041,51062] 0 2024-04-05T03:02:56.122 INFO:tasks.workunit.client.0.smithi153.stdout:1/581: mknod d5/d8/d20/d4b/d18/d16/d2c/cb3 0 2024-04-05T03:02:56.125 INFO:tasks.workunit.client.0.smithi153.stdout:9/499: truncate d1/f18 2034840 0 2024-04-05T03:02:56.125 INFO:tasks.workunit.client.0.smithi153.stdout:1/582: chown d5/lad 3386772 1 2024-04-05T03:02:56.126 INFO:tasks.workunit.client.0.smithi153.stdout:6/753: dwrite d4/dc/df/f56 [0,4194304] 0 2024-04-05T03:02:56.127 INFO:tasks.workunit.client.0.smithi153.stdout:3/640: write d1d/d27/f54 [433365,10198] 0 2024-04-05T03:02:56.129 INFO:tasks.workunit.client.0.smithi153.stdout:0/597: getdents d9/d68/d6d 0 2024-04-05T03:02:56.130 INFO:tasks.workunit.client.1.smithi203.stdout:4/798: dwrite da/d29/f9e [0,4194304] 0 2024-04-05T03:02:56.131 INFO:tasks.workunit.client.1.smithi203.stdout:4/799: fsync da/f10 0 2024-04-05T03:02:56.133 INFO:tasks.workunit.client.0.smithi153.stdout:1/583: chown d5/d8/d20/d4b/d18/d16/d33/d83/f8e 15015 1 2024-04-05T03:02:56.134 INFO:tasks.workunit.client.0.smithi153.stdout:9/500: mkdir d1/d1d/db2 0 2024-04-05T03:02:56.135 INFO:tasks.workunit.client.0.smithi153.stdout:7/786: dwrite d2/da/d3a/d41/d7a/f82 [0,4194304] 0 2024-04-05T03:02:56.137 INFO:tasks.workunit.client.0.smithi153.stdout:3/641: dread - d1d/d27/f74 zero size 2024-04-05T03:02:56.138 INFO:tasks.workunit.client.1.smithi203.stdout:4/800: creat da/d29/d2e/db0/dbe/f104 x:0 0 0 2024-04-05T03:02:56.138 INFO:tasks.workunit.client.0.smithi153.stdout:1/584: stat d5/d8/d20/d4b/d18/d16/d45/d93/f9a 0 2024-04-05T03:02:56.139 INFO:tasks.workunit.client.0.smithi153.stdout:6/754: mkdir d4/dc/df/dcb/dfe 0 2024-04-05T03:02:56.139 INFO:tasks.workunit.client.0.smithi153.stdout:7/787: dread - d2/da/d3a/d41/d7a/d54/d85/ff7 zero size 2024-04-05T03:02:56.141 INFO:tasks.workunit.client.0.smithi153.stdout:3/642: fdatasync d1d/d27/d2b/f2f 0 2024-04-05T03:02:56.141 INFO:tasks.workunit.client.0.smithi153.stdout:0/598: rename d9/df to d9/db/d42/d7e/db6 0 2024-04-05T03:02:56.144 INFO:tasks.workunit.client.0.smithi153.stdout:9/501: rmdir d1/d10 39 2024-04-05T03:02:56.144 INFO:tasks.workunit.client.1.smithi203.stdout:4/801: link da/d3c/d4c/fb7 da/d29/d2e/d57/d96/ddc/de1/f105 0 2024-04-05T03:02:56.148 INFO:tasks.workunit.client.0.smithi153.stdout:7/788: rename d2/da/d3a/d41/d7a/d54/d85/df6/d59 to d2/da/d3a/d41/d7a/d54/d85/df6/d59/d108 22 2024-04-05T03:02:56.148 INFO:tasks.workunit.client.0.smithi153.stdout:2/708: dwrite d9/da/f64 [0,4194304] 0 2024-04-05T03:02:56.152 INFO:tasks.workunit.client.0.smithi153.stdout:1/585: symlink d5/d8/d20/d4b/d18/d16/lb4 0 2024-04-05T03:02:56.158 INFO:tasks.workunit.client.0.smithi153.stdout:7/789: creat d2/da/d11/d39/f109 x:0 0 0 2024-04-05T03:02:56.159 INFO:tasks.workunit.client.0.smithi153.stdout:7/790: fdatasync d2/da/d47/d5d/d7c/fec 0 2024-04-05T03:02:56.159 INFO:tasks.workunit.client.0.smithi153.stdout:2/709: rename d9/da/d19/d2e/d7e/de6/f5b to d9/d68/d98/de9/ff2 0 2024-04-05T03:02:56.160 INFO:tasks.workunit.client.0.smithi153.stdout:2/710: chown d9/da/d94/l95 15 1 2024-04-05T03:02:56.163 INFO:tasks.workunit.client.0.smithi153.stdout:9/502: mknod d1/d9/cb3 0 2024-04-05T03:02:56.164 INFO:tasks.workunit.client.0.smithi153.stdout:3/643: getdents d1d/d32/d93/d3c/d3f/d90/db6/dd0 0 2024-04-05T03:02:56.166 INFO:tasks.workunit.client.0.smithi153.stdout:7/791: mkdir d2/d9b/d6f/d10a 0 2024-04-05T03:02:56.169 INFO:tasks.workunit.client.1.smithi203.stdout:4/802: dwrite da/d29/d2e/d53/d78/f7d [0,4194304] 0 2024-04-05T03:02:56.171 INFO:tasks.workunit.client.0.smithi153.stdout:7/792: creat d2/da/d47/da9/f10b x:0 0 0 2024-04-05T03:02:56.172 INFO:tasks.workunit.client.0.smithi153.stdout:9/503: fdatasync d1/d10/d16/d22/d2c/d68/fa7 0 2024-04-05T03:02:56.172 INFO:tasks.workunit.client.0.smithi153.stdout:3/644: mknod d1d/ce5 0 2024-04-05T03:02:56.174 INFO:tasks.workunit.client.1.smithi203.stdout:4/803: read da/f27 [245006,37973] 0 2024-04-05T03:02:56.174 INFO:tasks.workunit.client.0.smithi153.stdout:0/599: mknod d9/db/d42/d7e/db6/daf/cb7 0 2024-04-05T03:02:56.179 INFO:tasks.workunit.client.0.smithi153.stdout:6/755: dwrite d4/dc/df/d16/d33/fcd [0,4194304] 0 2024-04-05T03:02:56.179 INFO:tasks.workunit.client.0.smithi153.stdout:7/793: truncate d2/da/d11/f52 5365447 0 2024-04-05T03:02:56.179 INFO:tasks.workunit.client.0.smithi153.stdout:1/586: dwrite d5/dd/f9f [4194304,4194304] 0 2024-04-05T03:02:56.179 INFO:tasks.workunit.client.1.smithi203.stdout:4/804: link da/d29/d2e/d57/d96/ddc/de1/fec da/d29/d2e/d53/df9/f106 0 2024-04-05T03:02:56.182 INFO:tasks.workunit.client.0.smithi153.stdout:3/645: mknod d1d/d32/d93/d3c/d3f/d90/db6/ce6 0 2024-04-05T03:02:56.182 INFO:tasks.workunit.client.1.smithi203.stdout:4/805: mknod da/d3c/c107 0 2024-04-05T03:02:56.185 INFO:tasks.workunit.client.0.smithi153.stdout:9/504: mknod d1/d10/d36/d57/d5c/cb4 0 2024-04-05T03:02:56.185 INFO:tasks.workunit.client.1.smithi203.stdout:4/806: creat da/d3c/d4c/d99/f108 x:0 0 0 2024-04-05T03:02:56.186 INFO:tasks.workunit.client.1.smithi203.stdout:4/807: stat da/d3c/d4c/fff 0 2024-04-05T03:02:56.186 INFO:tasks.workunit.client.0.smithi153.stdout:0/600: mknod d9/db/d42/d7e/db6/d1e/d45/d62/cb8 0 2024-04-05T03:02:56.186 INFO:tasks.workunit.client.0.smithi153.stdout:7/794: creat d2/da/d3a/f10c x:0 0 0 2024-04-05T03:02:56.191 INFO:tasks.workunit.client.0.smithi153.stdout:3/646: mkdir d1d/d32/d93/d3c/d3f/d7c/de7 0 2024-04-05T03:02:56.191 INFO:tasks.workunit.client.0.smithi153.stdout:9/505: creat d1/d3a/fb5 x:0 0 0 2024-04-05T03:02:56.191 INFO:tasks.workunit.client.0.smithi153.stdout:1/587: mkdir d5/d8/d20/d4b/d51/db5 0 2024-04-05T03:02:56.193 INFO:tasks.workunit.client.0.smithi153.stdout:1/588: chown d5/d8/d20/d4b/d18/d16/d48/f50 0 1 2024-04-05T03:02:56.197 INFO:tasks.workunit.client.0.smithi153.stdout:7/795: rename d2/da/d3a/d41/d7a/d54/d85/df6/d93/fb0 to d2/da/d47/d5d/f10d 0 2024-04-05T03:02:56.198 INFO:tasks.workunit.client.0.smithi153.stdout:1/589: readlink d5/d8/d20/l63 0 2024-04-05T03:02:56.199 INFO:tasks.workunit.client.0.smithi153.stdout:1/590: write d5/dd/d1b/fae [501780,97294] 0 2024-04-05T03:02:56.200 INFO:tasks.workunit.client.0.smithi153.stdout:1/591: fsync d5/d8/d20/d3f/f5c 0 2024-04-05T03:02:56.201 INFO:tasks.workunit.client.0.smithi153.stdout:7/796: mkdir d2/da/d10e 0 2024-04-05T03:02:56.202 INFO:tasks.workunit.client.0.smithi153.stdout:1/592: write d5/d8/ff [1777983,27708] 0 2024-04-05T03:02:56.209 INFO:tasks.workunit.client.0.smithi153.stdout:1/593: write d5/d8/d20/d4b/d18/d16/d45/d68/f77 [3284847,108392] 0 2024-04-05T03:02:56.213 INFO:tasks.workunit.client.0.smithi153.stdout:7/797: read d2/da/d3a/d41/f19 [2365538,76950] 0 2024-04-05T03:02:56.213 INFO:tasks.workunit.client.0.smithi153.stdout:0/601: getdents d9/db/d42/d7e/db6/daf 0 2024-04-05T03:02:56.215 INFO:tasks.workunit.client.0.smithi153.stdout:0/602: write f0 [2246426,74270] 0 2024-04-05T03:02:56.217 INFO:tasks.workunit.client.0.smithi153.stdout:9/506: dread d1/d10/d36/f62 [0,4194304] 0 2024-04-05T03:02:56.219 INFO:tasks.workunit.client.0.smithi153.stdout:9/507: readlink d1/d3a/d91/d96/l9e 0 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.220+0000 7f7f6895c700 1 -- 172.21.15.153:0/3266006403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fce40 msgr2=0x7f7f640fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.220+0000 7f7f6895c700 1 --2- 172.21.15.153:0/3266006403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fce40 0x7f7f640fd250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004880 tx=0x7f7f58009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 -- 172.21.15.153:0/3266006403 shutdown_connections 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 --2- 172.21.15.153:0/3266006403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fe040 0x7f7f640fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 --2- 172.21.15.153:0/3266006403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fce40 0x7f7f640fd250 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 -- 172.21.15.153:0/3266006403 >> 172.21.15.153:0/3266006403 conn(0x7f7f640f8550 msgr2=0x7f7f640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 -- 172.21.15.153:0/3266006403 shutdown_connections 2024-04-05T03:02:56.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 -- 172.21.15.153:0/3266006403 wait complete. 2024-04-05T03:02:56.222 INFO:tasks.workunit.client.0.smithi153.stdout:7/798: symlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/l10f 0 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 Processor -- start 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.221+0000 7f7f6895c700 1 -- start start 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6895c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6895c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fe040 0x7f7f64070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6895c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640708c0 con 0x7f7f640fce40 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6895c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64070a30 con 0x7f7f640fe040 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39766/0 (socket says 172.21.15.153:39766) 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 -- 172.21.15.153:0/205678907 learned_addr learned my addr 172.21.15.153:0/205678907 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 -- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fe040 msgr2=0x7f7f64070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fe040 0x7f7f64070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.222+0000 7f7f6259c700 1 -- 172.21.15.153:0/205678907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f58005040 con 0x7f7f640fce40 2024-04-05T03:02:56.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f6259c700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7f580138f0 tx=0x7f7f58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:56.224 INFO:tasks.workunit.client.0.smithi153.stdout:1/594: getdents d5/d8/d20/d2f/d40 0 2024-04-05T03:02:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f5800e030 con 0x7f7f640fce40 2024-04-05T03:02:56.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f58013e60 con 0x7f7f640fce40 2024-04-05T03:02:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f6895c700 1 -- 172.21.15.153:0/205678907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f64070cb0 con 0x7f7f640fce40 2024-04-05T03:02:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f580123c0 con 0x7f7f640fce40 2024-04-05T03:02:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.223+0000 7f7f6895c700 1 -- 172.21.15.153:0/205678907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f64115a80 con 0x7f7f640fce40 2024-04-05T03:02:56.225 INFO:tasks.workunit.client.0.smithi153.stdout:9/508: rmdir d1/d10/d36/d57 39 2024-04-05T03:02:56.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.225+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f480052b0 con 0x7f7f640fce40 2024-04-05T03:02:56.226 INFO:tasks.workunit.client.0.smithi153.stdout:7/799: chown d2/da/d3a/d41/ddd/f107 364 1 2024-04-05T03:02:56.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.225+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f7f5801c400 con 0x7f7f640fce40 2024-04-05T03:02:56.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.226+0000 7f7f52ffd700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 0x7f7f4c06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.226+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f7f5808b8a0 con 0x7f7f640fce40 2024-04-05T03:02:56.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.227+0000 7f7f61d9b700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 0x7f7f4c06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:56.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.227+0000 7f7f61d9b700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 0x7f7f4c06dab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009fd0 tx=0x7f7f5400b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:56.227 INFO:tasks.workunit.client.1.smithi203.stdout:4/808: dread da/d29/d2e/d53/f97 [0,4194304] 0 2024-04-05T03:02:56.228 INFO:tasks.workunit.client.0.smithi153.stdout:6/756: dwrite d4/dc/df/d16/d17/f2a [0,4194304] 0 2024-04-05T03:02:56.229 INFO:tasks.workunit.client.0.smithi153.stdout:3/647: dread d1d/d32/d93/d3c/d3f/d90/d92/fac [0,4194304] 0 2024-04-05T03:02:56.229 INFO:tasks.workunit.client.1.smithi203.stdout:4/809: chown da/d29/d2e/d57/d96/dbc/df5/lf6 734442709 1 2024-04-05T03:02:56.229 INFO:tasks.workunit.client.1.smithi203.stdout:4/810: dread - da/fed zero size 2024-04-05T03:02:56.230 INFO:tasks.workunit.client.1.smithi203.stdout:4/811: chown da/d29/d2e/db0/dbe/dce/dd0/ff7 1296995 1 2024-04-05T03:02:56.231 INFO:tasks.workunit.client.0.smithi153.stdout:1/595: write d5/d8/d20/d4b/d1e/f4f [5515122,126943] 0 2024-04-05T03:02:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.232+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f7f58055820 con 0x7f7f640fce40 2024-04-05T03:02:56.232 INFO:tasks.workunit.client.0.smithi153.stdout:3/648: readlink l13 0 2024-04-05T03:02:56.232 INFO:tasks.workunit.client.1.smithi203.stdout:4/812: mkdir da/d29/d2e/d8b/d109 0 2024-04-05T03:02:56.237 INFO:tasks.workunit.client.1.smithi203.stdout:4/813: creat da/d3c/f10a x:0 0 0 2024-04-05T03:02:56.237 INFO:tasks.workunit.client.1.smithi203.stdout:3/806: sync 2024-04-05T03:02:56.237 INFO:tasks.workunit.client.1.smithi203.stdout:6/995: sync 2024-04-05T03:02:56.238 INFO:tasks.workunit.client.1.smithi203.stdout:0/981: sync 2024-04-05T03:02:56.239 INFO:tasks.workunit.client.0.smithi153.stdout:3/649: mknod d1d/d32/d93/d3c/d3f/d90/db6/ce8 0 2024-04-05T03:02:56.240 INFO:tasks.workunit.client.0.smithi153.stdout:1/596: truncate d5/d8/d20/f38 1693812 0 2024-04-05T03:02:56.241 INFO:tasks.workunit.client.0.smithi153.stdout:7/800: dwrite d2/da/d47/d5d/d7c/d8d/d95/fe2 [0,4194304] 0 2024-04-05T03:02:56.242 INFO:tasks.workunit.client.0.smithi153.stdout:9/509: dread d1/d10/d36/d57/d5c/f89 [0,4194304] 0 2024-04-05T03:02:56.245 INFO:tasks.workunit.client.1.smithi203.stdout:0/982: chown dd/d17/d1d/d22/dc1 111312299 1 2024-04-05T03:02:56.245 INFO:tasks.workunit.client.1.smithi203.stdout:6/996: dread d0/da/d54/f3f [0,4194304] 0 2024-04-05T03:02:56.246 INFO:tasks.workunit.client.0.smithi153.stdout:3/650: write d1d/d32/f45 [826226,22015] 0 2024-04-05T03:02:56.247 INFO:tasks.workunit.client.1.smithi203.stdout:3/807: dwrite d0/dc/d38/d5d/d66/d80/feb [0,4194304] 0 2024-04-05T03:02:56.247 INFO:tasks.workunit.client.1.smithi203.stdout:4/814: getdents da/d29/d2e/db0/dcb 0 2024-04-05T03:02:56.248 INFO:tasks.workunit.client.0.smithi153.stdout:1/597: write d5/d8/d20/d4b/d18/d16/d33/f3c [1993475,54967] 0 2024-04-05T03:02:56.250 INFO:tasks.workunit.client.0.smithi153.stdout:3/651: write d1d/d32/d93/d3c/d3f/d7c/fad [2899389,6944] 0 2024-04-05T03:02:56.250 INFO:tasks.workunit.client.0.smithi153.stdout:1/598: chown f4 2904772 1 2024-04-05T03:02:56.253 INFO:tasks.workunit.client.1.smithi203.stdout:0/983: rmdir dd/d17 39 2024-04-05T03:02:56.255 INFO:tasks.workunit.client.0.smithi153.stdout:3/652: symlink d1d/d27/d2b/le9 0 2024-04-05T03:02:56.259 INFO:tasks.workunit.client.0.smithi153.stdout:7/801: dread d2/da/d11/f24 [4194304,4194304] 0 2024-04-05T03:02:56.261 INFO:tasks.workunit.client.0.smithi153.stdout:0/603: dread d9/db/d42/f60 [0,4194304] 0 2024-04-05T03:02:56.262 INFO:tasks.workunit.client.1.smithi203.stdout:6/997: link d0/da/d54/d53/c85 d0/da/d18/d92/dfe/c129 0 2024-04-05T03:02:56.263 INFO:tasks.workunit.client.0.smithi153.stdout:9/510: write d1/d3a/d91/f9f [4713333,79045] 0 2024-04-05T03:02:56.264 INFO:tasks.workunit.client.1.smithi203.stdout:4/815: creat da/d29/d2e/d57/f10b x:0 0 0 2024-04-05T03:02:56.265 INFO:tasks.workunit.client.1.smithi203.stdout:3/808: getdents d0/dc/d1a/d1f/da2 0 2024-04-05T03:02:56.265 INFO:tasks.workunit.client.0.smithi153.stdout:6/757: dread d4/d38/fbb [0,4194304] 0 2024-04-05T03:02:56.266 INFO:tasks.workunit.client.1.smithi203.stdout:0/984: getdents dd/db7 0 2024-04-05T03:02:56.266 INFO:tasks.workunit.client.0.smithi153.stdout:1/599: dwrite f4 [4194304,4194304] 0 2024-04-05T03:02:56.269 INFO:tasks.workunit.client.0.smithi153.stdout:1/600: rename d5/dd to d5/dd/d1b/db6 22 2024-04-05T03:02:56.272 INFO:tasks.workunit.client.1.smithi203.stdout:4/816: creat da/d29/d2e/d57/d96/ddc/f10c x:0 0 0 2024-04-05T03:02:56.272 INFO:tasks.workunit.client.0.smithi153.stdout:1/601: fsync d5/d8/d20/d3f/f5c 0 2024-04-05T03:02:56.274 INFO:tasks.workunit.client.1.smithi203.stdout:3/809: rename d0/dc/d1a/d1d/cbe to d0/dc/d38/dca/cec 0 2024-04-05T03:02:56.274 INFO:tasks.workunit.client.0.smithi153.stdout:0/604: mknod d9/db/d42/d7e/cb9 0 2024-04-05T03:02:56.274 INFO:tasks.workunit.client.0.smithi153.stdout:6/758: write d4/dc/df/dcb/fdc [1908855,45905] 0 2024-04-05T03:02:56.275 INFO:tasks.workunit.client.0.smithi153.stdout:7/802: mkdir d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106/d110 0 2024-04-05T03:02:56.276 INFO:tasks.workunit.client.1.smithi203.stdout:0/985: symlink dd/d17/d115/d28/l127 0 2024-04-05T03:02:56.277 INFO:tasks.workunit.client.1.smithi203.stdout:6/998: truncate d0/da/d54/d4c/d7f/dcc/ffb 1111825 0 2024-04-05T03:02:56.277 INFO:tasks.workunit.client.0.smithi153.stdout:1/602: mknod d5/d8/d20/d4b/d18/cb7 0 2024-04-05T03:02:56.279 INFO:tasks.workunit.client.1.smithi203.stdout:3/810: symlink d0/dc/d1a/d1d/led 0 2024-04-05T03:02:56.279 INFO:tasks.workunit.client.0.smithi153.stdout:0/605: truncate d9/db/f13 316425 0 2024-04-05T03:02:56.281 INFO:tasks.workunit.client.1.smithi203.stdout:6/999: unlink d0/da/d18/l87 0 2024-04-05T03:02:56.282 INFO:tasks.workunit.client.0.smithi153.stdout:1/603: write d5/d8/d20/d4b/d18/d16/f36 [1693125,19810] 0 2024-04-05T03:02:56.283 INFO:tasks.workunit.client.1.smithi203.stdout:4/817: link da/d29/c4b da/d29/d2e/d57/da7/c10d 0 2024-04-05T03:02:56.286 INFO:tasks.workunit.client.0.smithi153.stdout:1/604: truncate d5/d8/d20/d4b/d18/d16/d48/f99 62400 0 2024-04-05T03:02:56.287 INFO:tasks.workunit.client.1.smithi203.stdout:3/811: link d0/d21/lb4 d0/d21/d81/dd9/lee 0 2024-04-05T03:02:56.288 INFO:tasks.workunit.client.0.smithi153.stdout:1/605: chown d5/d8/d20/d3f/f86 3418 1 2024-04-05T03:02:56.289 INFO:tasks.workunit.client.0.smithi153.stdout:1/606: readlink d5/d8/d20/l63 0 2024-04-05T03:02:56.292 INFO:tasks.workunit.client.0.smithi153.stdout:0/606: write d9/d14/f40 [1106211,100411] 0 2024-04-05T03:02:56.293 INFO:tasks.workunit.client.0.smithi153.stdout:3/653: dwrite d1d/d32/d93/d3c/d3f/f4f [0,4194304] 0 2024-04-05T03:02:56.293 INFO:tasks.workunit.client.1.smithi203.stdout:3/812: symlink d0/d41/lef 0 2024-04-05T03:02:56.296 INFO:tasks.workunit.client.1.smithi203.stdout:3/813: mknod d0/dc/d34/cf0 0 2024-04-05T03:02:56.297 INFO:tasks.workunit.client.0.smithi153.stdout:9/511: dread d1/d9/d1b/f7d [0,4194304] 0 2024-04-05T03:02:56.299 INFO:tasks.workunit.client.0.smithi153.stdout:1/607: mkdir d5/d8/d20/d2f/db8 0 2024-04-05T03:02:56.301 INFO:tasks.workunit.client.0.smithi153.stdout:3/654: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fbf 1 1 2024-04-05T03:02:56.301 INFO:tasks.workunit.client.0.smithi153.stdout:0/607: chown d9/db/d42/d7e/db6/l2e 4092606 1 2024-04-05T03:02:56.302 INFO:tasks.workunit.client.0.smithi153.stdout:9/512: symlink d1/d10/d36/lb6 0 2024-04-05T03:02:56.303 INFO:tasks.workunit.client.0.smithi153.stdout:3/655: write d1d/d27/d2b/f7e [1331848,112305] 0 2024-04-05T03:02:56.304 INFO:tasks.workunit.client.0.smithi153.stdout:0/608: write d9/db/d42/d7e/db6/d1e/d45/d62/d79/fb1 [453164,112247] 0 2024-04-05T03:02:56.304 INFO:tasks.workunit.client.0.smithi153.stdout:1/608: creat d5/d8/d20/d4b/d18/d16/d48/fb9 x:0 0 0 2024-04-05T03:02:56.305 INFO:tasks.workunit.client.0.smithi153.stdout:9/513: chown d1/d10/d16/d22/d2c/d68/d81/c13 527 1 2024-04-05T03:02:56.306 INFO:tasks.workunit.client.0.smithi153.stdout:0/609: write d9/db/d42/d43/f98 [850287,21662] 0 2024-04-05T03:02:56.307 INFO:tasks.workunit.client.1.smithi203.stdout:4/818: dwrite da/f9a [4194304,4194304] 0 2024-04-05T03:02:56.309 INFO:tasks.workunit.client.0.smithi153.stdout:0/610: write d9/d14/f40 [1778957,106711] 0 2024-04-05T03:02:56.309 INFO:tasks.workunit.client.0.smithi153.stdout:9/514: write d1/d3a/d91/f9b [1390567,72348] 0 2024-04-05T03:02:56.311 INFO:tasks.workunit.client.1.smithi203.stdout:4/819: rmdir da/d29/d2e/dc9 39 2024-04-05T03:02:56.312 INFO:tasks.workunit.client.0.smithi153.stdout:1/609: creat d5/d8/d20/d4b/d18/fba x:0 0 0 2024-04-05T03:02:56.312 INFO:tasks.workunit.client.1.smithi203.stdout:4/820: stat da/d29/d2e/db0/dbe/f104 0 2024-04-05T03:02:56.318 INFO:tasks.workunit.client.0.smithi153.stdout:3/656: dwrite d1d/d32/f49 [0,4194304] 0 2024-04-05T03:02:56.320 INFO:tasks.workunit.client.0.smithi153.stdout:7/803: dread d2/da/d3a/d41/d7a/d54/d85/df6/d93/fe4 [0,4194304] 0 2024-04-05T03:02:56.322 INFO:tasks.workunit.client.1.smithi203.stdout:3/814: dread d0/dc/d1a/d1d/d53/d5a/fab [0,4194304] 0 2024-04-05T03:02:56.322 INFO:tasks.workunit.client.1.smithi203.stdout:4/821: getdents da/d29/d2e/d57/d96/dbc/df5 0 2024-04-05T03:02:56.323 INFO:tasks.workunit.client.0.smithi153.stdout:1/610: creat d5/fbb x:0 0 0 2024-04-05T03:02:56.323 INFO:tasks.workunit.client.0.smithi153.stdout:9/515: fsync d1/d10/f39 0 2024-04-05T03:02:56.324 INFO:tasks.workunit.client.1.smithi203.stdout:3/815: creat d0/dc/d1a/d1d/d53/ff1 x:0 0 0 2024-04-05T03:02:56.325 INFO:tasks.workunit.client.1.smithi203.stdout:4/822: stat da/d3c/l4d 0 2024-04-05T03:02:56.326 INFO:tasks.workunit.client.0.smithi153.stdout:0/611: dwrite d9/d14/f40 [0,4194304] 0 2024-04-05T03:02:56.329 INFO:tasks.workunit.client.1.smithi203.stdout:4/823: creat da/f10e x:0 0 0 2024-04-05T03:02:56.330 INFO:tasks.workunit.client.1.smithi203.stdout:3/816: getdents d0/dc 0 2024-04-05T03:02:56.333 INFO:tasks.workunit.client.0.smithi153.stdout:0/612: creat d9/db/d42/d7e/db6/d1e/d45/fba x:0 0 0 2024-04-05T03:02:56.334 INFO:tasks.workunit.client.1.smithi203.stdout:3/817: rmdir d0/dc/d1a/d8d/dc9 39 2024-04-05T03:02:56.335 INFO:tasks.workunit.client.0.smithi153.stdout:3/657: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fea x:0 0 0 2024-04-05T03:02:56.337 INFO:tasks.workunit.client.1.smithi203.stdout:3/818: creat d0/dc/d1a/d1f/da1/ff2 x:0 0 0 2024-04-05T03:02:56.337 INFO:tasks.workunit.client.0.smithi153.stdout:7/804: write d2/da/d3a/d41/da8/fad [756011,104303] 0 2024-04-05T03:02:56.338 INFO:tasks.workunit.client.1.smithi203.stdout:4/824: write da/d71/f92 [193215,95284] 0 2024-04-05T03:02:56.338 INFO:tasks.workunit.client.0.smithi153.stdout:3/658: read - d1d/d75/fe0 zero size 2024-04-05T03:02:56.340 INFO:tasks.workunit.client.0.smithi153.stdout:3/659: truncate d1d/d32/d93/d3c/d3f/d9c/faa 582423 0 2024-04-05T03:02:56.341 INFO:tasks.workunit.client.1.smithi203.stdout:3/819: creat d0/dc/ff3 x:0 0 0 2024-04-05T03:02:56.341 INFO:tasks.workunit.client.0.smithi153.stdout:1/611: symlink d5/d8/d20/d4b/d18/d16/d2c/lbc 0 2024-04-05T03:02:56.341 INFO:tasks.workunit.client.0.smithi153.stdout:0/613: link d9/db/d42/d7e/cb9 d9/db/d42/d7e/db6/daf/cbb 0 2024-04-05T03:02:56.342 INFO:tasks.workunit.client.1.smithi203.stdout:3/820: chown d0 1813593981 1 2024-04-05T03:02:56.343 INFO:tasks.workunit.client.1.smithi203.stdout:4/825: truncate da/d29/fbf 8084274 0 2024-04-05T03:02:56.343 INFO:tasks.workunit.client.0.smithi153.stdout:3/660: truncate d1d/f72 4541467 0 2024-04-05T03:02:56.344 INFO:tasks.workunit.client.0.smithi153.stdout:3/661: fsync d1d/d32/f45 0 2024-04-05T03:02:56.345 INFO:tasks.workunit.client.0.smithi153.stdout:1/612: creat d5/d8/d20/d4b/d18/d16/d48/d96/fbd x:0 0 0 2024-04-05T03:02:56.346 INFO:tasks.workunit.client.1.smithi203.stdout:4/826: unlink da/d29/d2e/db0/fb5 0 2024-04-05T03:02:56.347 INFO:tasks.workunit.client.0.smithi153.stdout:1/613: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:56.350 INFO:tasks.workunit.client.0.smithi153.stdout:1/614: creat d5/d8/d20/d4b/d18/d16/d48/d96/fbe x:0 0 0 2024-04-05T03:02:56.351 INFO:tasks.workunit.client.1.smithi203.stdout:4/827: link da/d29/d2e/d57/ce9 da/d29/d2e/db0/dbe/c10f 0 2024-04-05T03:02:56.354 INFO:tasks.workunit.client.1.smithi203.stdout:4/828: creat da/d29/d2e/d8b/dc8/f110 x:0 0 0 2024-04-05T03:02:56.355 INFO:tasks.workunit.client.1.smithi203.stdout:3/821: chown d0/dc/d34/cf0 2 1 2024-04-05T03:02:56.356 INFO:tasks.workunit.client.0.smithi153.stdout:3/662: write d1d/d32/f89 [430049,54350] 0 2024-04-05T03:02:56.357 INFO:tasks.workunit.client.1.smithi203.stdout:3/822: write d0/d41/fd4 [8282,52983] 0 2024-04-05T03:02:56.357 INFO:tasks.workunit.client.1.smithi203.stdout:4/829: read da/d29/d2e/db0/dcb/d80/fa0 [361216,103429] 0 2024-04-05T03:02:56.357 INFO:tasks.workunit.client.0.smithi153.stdout:9/516: dread d1/d9/d1b/f75 [0,4194304] 0 2024-04-05T03:02:56.357 INFO:tasks.workunit.client.0.smithi153.stdout:1/615: write d5/d8/d20/d4b/d18/d16/d45/f4a [692723,121149] 0 2024-04-05T03:02:56.360 INFO:tasks.workunit.client.0.smithi153.stdout:3/663: creat d1d/d27/feb x:0 0 0 2024-04-05T03:02:56.362 INFO:tasks.workunit.client.0.smithi153.stdout:1/616: getdents d5/d8/d20/d4b/d51/db5 0 2024-04-05T03:02:56.363 INFO:tasks.workunit.client.1.smithi203.stdout:4/830: rename da/d29/d2e/d8b/fd7 to da/d29/d2e/d53/d78/f111 0 2024-04-05T03:02:56.363 INFO:tasks.workunit.client.0.smithi153.stdout:1/617: read - d5/d8/d20/d4b/fb1 zero size 2024-04-05T03:02:56.363 INFO:tasks.workunit.client.0.smithi153.stdout:9/517: write d1/d10/d16/f23 [722033,40489] 0 2024-04-05T03:02:56.365 INFO:tasks.workunit.client.1.smithi203.stdout:3/823: unlink d0/d41/ca7 0 2024-04-05T03:02:56.366 INFO:tasks.workunit.client.0.smithi153.stdout:9/518: symlink d1/d10/d36/d57/d5c/lb7 0 2024-04-05T03:02:56.368 INFO:tasks.workunit.client.1.smithi203.stdout:3/824: chown d0/dc/f54 491 1 2024-04-05T03:02:56.369 INFO:tasks.workunit.client.1.smithi203.stdout:4/831: creat da/d29/d2e/d8b/d109/f112 x:0 0 0 2024-04-05T03:02:56.369 INFO:tasks.workunit.client.0.smithi153.stdout:9/519: truncate d1/d10/d16/d22/d2c/d68/f7c 2246034 0 2024-04-05T03:02:56.371 INFO:tasks.workunit.client.0.smithi153.stdout:1/618: getdents d5/d8/d20/d4b/d18/d16/d2c 0 2024-04-05T03:02:56.372 INFO:tasks.workunit.client.0.smithi153.stdout:3/664: dread d1d/d75/f84 [0,4194304] 0 2024-04-05T03:02:56.372 INFO:tasks.workunit.client.0.smithi153.stdout:9/520: symlink d1/d1d/lb8 0 2024-04-05T03:02:56.372 INFO:tasks.workunit.client.1.smithi203.stdout:4/832: truncate da/d29/d2e/f3e 1664214 0 2024-04-05T03:02:56.373 INFO:tasks.workunit.client.0.smithi153.stdout:0/614: dread d9/d14/f23 [0,4194304] 0 2024-04-05T03:02:56.374 INFO:tasks.workunit.client.0.smithi153.stdout:1/619: read - d5/d8/d20/f25 zero size 2024-04-05T03:02:56.374 INFO:tasks.workunit.client.1.smithi203.stdout:4/833: chown da/d29/lac 62854 1 2024-04-05T03:02:56.375 INFO:tasks.workunit.client.1.smithi203.stdout:4/834: chown da/d29/d2e/d53/df9 106148 1 2024-04-05T03:02:56.376 INFO:tasks.workunit.client.0.smithi153.stdout:9/521: symlink d1/d10/d16/lb9 0 2024-04-05T03:02:56.376 INFO:tasks.workunit.client.0.smithi153.stdout:1/620: write d5/dd/f89 [5199165,114811] 0 2024-04-05T03:02:56.377 INFO:tasks.workunit.client.0.smithi153.stdout:9/522: stat d1/d1d/l50 0 2024-04-05T03:02:56.378 INFO:tasks.workunit.client.0.smithi153.stdout:9/523: readlink d1/d10/d16/d22/laf 0 2024-04-05T03:02:56.379 INFO:tasks.workunit.client.0.smithi153.stdout:9/524: readlink d1/d10/d16/lb9 0 2024-04-05T03:02:56.380 INFO:tasks.workunit.client.0.smithi153.stdout:1/621: fdatasync d5/d8/d20/f25 0 2024-04-05T03:02:56.380 INFO:tasks.workunit.client.0.smithi153.stdout:9/525: readlink d1/d10/d16/l95 0 2024-04-05T03:02:56.380 INFO:tasks.workunit.client.0.smithi153.stdout:0/615: fsync d9/db/d42/d7e/db6/f17 0 2024-04-05T03:02:56.381 INFO:tasks.workunit.client.0.smithi153.stdout:9/526: chown d1/d3a/d91/f9b 42702 1 2024-04-05T03:02:56.382 INFO:tasks.workunit.client.0.smithi153.stdout:9/527: fdatasync d1/d9/f8e 0 2024-04-05T03:02:56.383 INFO:tasks.workunit.client.0.smithi153.stdout:3/665: truncate d1d/d27/d2b/f2d 3940429 0 2024-04-05T03:02:56.384 INFO:tasks.workunit.client.0.smithi153.stdout:9/528: chown d1/d10/d16/d22/f74 1340759 1 2024-04-05T03:02:56.384 INFO:tasks.workunit.client.1.smithi203.stdout:4/835: dread da/d29/d2e/f3e [0,4194304] 0 2024-04-05T03:02:56.386 INFO:tasks.workunit.client.1.smithi203.stdout:4/836: write da/d29/d2e/d8b/dc8/f110 [489157,4951] 0 2024-04-05T03:02:56.387 INFO:tasks.workunit.client.1.smithi203.stdout:3/825: dwrite d0/dc/d38/d5d/d66/d80/f97 [0,4194304] 0 2024-04-05T03:02:56.389 INFO:tasks.workunit.client.0.smithi153.stdout:9/529: rmdir d1/d10/d16/d22/d2c/d68 39 2024-04-05T03:02:56.391 INFO:tasks.workunit.client.0.smithi153.stdout:0/616: write d9/db/d42/d7e/db6/d1e/f80 [1228316,82925] 0 2024-04-05T03:02:56.392 INFO:tasks.workunit.client.0.smithi153.stdout:1/622: mknod d5/d8/d20/d4b/d51/db5/cbf 0 2024-04-05T03:02:56.393 INFO:tasks.workunit.client.1.smithi203.stdout:3/826: creat d0/dc/d1a/d1d/d53/ff4 x:0 0 0 2024-04-05T03:02:56.394 INFO:tasks.workunit.client.0.smithi153.stdout:1/623: fdatasync d5/d8/d20/f5f 0 2024-04-05T03:02:56.394 INFO:tasks.workunit.client.0.smithi153.stdout:9/530: rename d1/d10/d36/d57/d97/cb1 to d1/d10/d16/d22/cba 0 2024-04-05T03:02:56.394 INFO:tasks.workunit.client.0.smithi153.stdout:9/531: stat d1/f2e 0 2024-04-05T03:02:56.395 INFO:tasks.workunit.client.1.smithi203.stdout:4/837: link da/d29/f32 da/d29/d2e/d53/f113 0 2024-04-05T03:02:56.399 INFO:tasks.workunit.client.1.smithi203.stdout:3/827: mknod d0/dc/d38/d5d/cf5 0 2024-04-05T03:02:56.399 INFO:tasks.workunit.client.0.smithi153.stdout:9/532: write d1/d10/d36/f56 [556421,10134] 0 2024-04-05T03:02:56.400 INFO:tasks.workunit.client.0.smithi153.stdout:1/624: truncate d5/d8/d20/f38 87623 0 2024-04-05T03:02:56.401 INFO:tasks.workunit.client.0.smithi153.stdout:1/625: chown d5/d8/f6e 26247 1 2024-04-05T03:02:56.403 INFO:tasks.workunit.client.0.smithi153.stdout:1/626: chown d5/d8/d20/l42 186 1 2024-04-05T03:02:56.404 INFO:tasks.workunit.client.0.smithi153.stdout:0/617: rename d9/db/d42/d7e/db6/d1e/c5b to d9/db/d42/d7e/db6/d1e/d45/d62/cbc 0 2024-04-05T03:02:56.406 INFO:tasks.workunit.client.0.smithi153.stdout:3/666: dread d1d/d27/d2b/f7e [0,4194304] 0 2024-04-05T03:02:56.406 INFO:tasks.workunit.client.0.smithi153.stdout:0/618: symlink d9/d14/lbd 0 2024-04-05T03:02:56.409 INFO:tasks.workunit.client.1.smithi203.stdout:4/838: dread da/f10 [0,4194304] 0 2024-04-05T03:02:56.411 INFO:tasks.workunit.client.1.smithi203.stdout:4/839: chown da/l1f 1627465599 1 2024-04-05T03:02:56.411 INFO:tasks.workunit.client.0.smithi153.stdout:1/627: dwrite d5/d8/d20/d4b/d51/fa9 [0,4194304] 0 2024-04-05T03:02:56.412 INFO:tasks.workunit.client.0.smithi153.stdout:3/667: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/fec x:0 0 0 2024-04-05T03:02:56.414 INFO:tasks.workunit.client.0.smithi153.stdout:1/628: truncate d5/d8/d20/d4b/d18/d16/d33/f3c 5080133 0 2024-04-05T03:02:56.415 INFO:tasks.workunit.client.0.smithi153.stdout:9/533: write d1/d9/f9a [216361,60108] 0 2024-04-05T03:02:56.418 INFO:tasks.workunit.client.1.smithi203.stdout:4/840: truncate da/f27 6225196 0 2024-04-05T03:02:56.419 INFO:tasks.workunit.client.1.smithi203.stdout:3/828: write d0/dc/d38/f3b [526294,7878] 0 2024-04-05T03:02:56.419 INFO:tasks.workunit.client.0.smithi153.stdout:3/668: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/fda 134832 1 2024-04-05T03:02:56.420 INFO:tasks.workunit.client.1.smithi203.stdout:4/841: stat da/d29/d2e/d8b 0 2024-04-05T03:02:56.422 INFO:tasks.workunit.client.1.smithi203.stdout:4/842: mkdir da/d29/d2e/d57/d96/d114 0 2024-04-05T03:02:56.422 INFO:tasks.workunit.client.1.smithi203.stdout:3/829: mkdir d0/dc/d1a/ddf/df6 0 2024-04-05T03:02:56.424 INFO:tasks.workunit.client.0.smithi153.stdout:3/669: symlink d1d/d32/d93/d3c/d3f/d90/d92/led 0 2024-04-05T03:02:56.424 INFO:tasks.workunit.client.0.smithi153.stdout:9/534: getdents d1/d10/d36/d71 0 2024-04-05T03:02:56.425 INFO:tasks.workunit.client.0.smithi153.stdout:3/670: chown d1d/d27/d2b/fc9 11 1 2024-04-05T03:02:56.427 INFO:tasks.workunit.client.1.smithi203.stdout:4/843: rename da/d71/lae to da/d29/d2e/d8b/d109/l115 0 2024-04-05T03:02:56.428 INFO:tasks.workunit.client.1.smithi203.stdout:3/830: unlink d0/dc/d1a/d1d/f2f 0 2024-04-05T03:02:56.429 INFO:tasks.workunit.client.1.smithi203.stdout:3/831: dread - d0/dc/d1a/d76/d35/fc0 zero size 2024-04-05T03:02:56.429 INFO:tasks.workunit.client.0.smithi153.stdout:3/671: creat d1d/d32/d93/d3c/d3f/d90/db6/fee x:0 0 0 2024-04-05T03:02:56.431 INFO:tasks.workunit.client.1.smithi203.stdout:3/832: symlink d0/dc/d1a/d76/lf7 0 2024-04-05T03:02:56.431 INFO:tasks.workunit.client.0.smithi153.stdout:3/672: unlink d1d/d32/d93/d3c/d3f/d7c/c8c 0 2024-04-05T03:02:56.432 INFO:tasks.workunit.client.1.smithi203.stdout:3/833: write d0/dc/d38/d5d/fd8 [475764,86500] 0 2024-04-05T03:02:56.432 INFO:tasks.workunit.client.0.smithi153.stdout:3/673: chown d1d/d27/feb 7767616 1 2024-04-05T03:02:56.437 INFO:tasks.workunit.client.0.smithi153.stdout:1/629: truncate d5/d8/d20/d4b/d18/d16/f5b 803425 0 2024-04-05T03:02:56.439 INFO:tasks.workunit.client.1.smithi203.stdout:3/834: dwrite d0/d21/d81/f5b [0,4194304] 0 2024-04-05T03:02:56.440 INFO:tasks.workunit.client.0.smithi153.stdout:1/630: rmdir d5/d8/d20/d2f 39 2024-04-05T03:02:56.440 INFO:tasks.workunit.client.0.smithi153.stdout:0/619: dread d9/d14/d67/d6f/f7c [0,4194304] 0 2024-04-05T03:02:56.440 INFO:tasks.workunit.client.1.smithi203.stdout:3/835: fdatasync d0/dc/d1a/d8d/fe1 0 2024-04-05T03:02:56.441 INFO:tasks.workunit.client.1.smithi203.stdout:4/844: dread da/d29/d2e/f35 [0,4194304] 0 2024-04-05T03:02:56.443 INFO:tasks.workunit.client.0.smithi153.stdout:1/631: write d5/d8/d20/d4b/d1e/f4f [5911549,82470] 0 2024-04-05T03:02:56.446 INFO:tasks.workunit.client.0.smithi153.stdout:0/620: mknod d9/db/cbe 0 2024-04-05T03:02:56.446 INFO:tasks.workunit.client.0.smithi153.stdout:9/535: dwrite d1/d9/d1b/f6a [0,4194304] 0 2024-04-05T03:02:56.449 INFO:tasks.workunit.client.0.smithi153.stdout:1/632: readlink d5/d8/d20/d4b/d51/l53 0 2024-04-05T03:02:56.449 INFO:tasks.workunit.client.1.smithi203.stdout:4/845: symlink da/d29/d2e/db0/dbe/dce/dd0/l116 0 2024-04-05T03:02:56.450 INFO:tasks.workunit.client.1.smithi203.stdout:4/846: dread - da/d29/ffd zero size 2024-04-05T03:02:56.453 INFO:tasks.workunit.client.0.smithi153.stdout:0/621: creat d9/d14/d67/fbf x:0 0 0 2024-04-05T03:02:56.453 INFO:tasks.workunit.client.1.smithi203.stdout:4/847: creat da/d29/d2e/db0/f117 x:0 0 0 2024-04-05T03:02:56.455 INFO:tasks.workunit.client.0.smithi153.stdout:9/536: creat d1/d10/d16/d22/d2c/d68/d81/fbb x:0 0 0 2024-04-05T03:02:56.455 INFO:tasks.workunit.client.1.smithi203.stdout:4/848: stat da/d29/d2e/d57/d96/ddc/de1 0 2024-04-05T03:02:56.457 INFO:tasks.workunit.client.0.smithi153.stdout:0/622: creat d9/db/d42/d7e/db6/fc0 x:0 0 0 2024-04-05T03:02:56.458 INFO:tasks.workunit.client.0.smithi153.stdout:9/537: mkdir d1/d9/dbc 0 2024-04-05T03:02:56.459 INFO:tasks.workunit.client.1.smithi203.stdout:4/849: symlink da/d29/l118 0 2024-04-05T03:02:56.459 INFO:tasks.workunit.client.0.smithi153.stdout:0/623: dread - d9/d14/f57 zero size 2024-04-05T03:02:56.460 INFO:tasks.workunit.client.1.smithi203.stdout:4/850: write da/d29/d2e/d53/d78/f7d [299525,81513] 0 2024-04-05T03:02:56.462 INFO:tasks.workunit.client.0.smithi153.stdout:9/538: truncate d1/d9/d1b/f73 18595 0 2024-04-05T03:02:56.463 INFO:tasks.workunit.client.0.smithi153.stdout:0/624: mknod d9/db/d42/cc1 0 2024-04-05T03:02:56.463 INFO:tasks.workunit.client.1.smithi203.stdout:3/836: write d0/dc/d38/f9e [1020012,30041] 0 2024-04-05T03:02:56.464 INFO:tasks.workunit.client.0.smithi153.stdout:9/539: mkdir d1/d3a/d91/d96/dbd 0 2024-04-05T03:02:56.466 INFO:tasks.workunit.client.0.smithi153.stdout:0/625: creat d9/db/d42/d7e/db6/d1e/d45/fc2 x:0 0 0 2024-04-05T03:02:56.467 INFO:tasks.workunit.client.0.smithi153.stdout:9/540: mkdir d1/d3a/d91/d96/dbd/dbe 0 2024-04-05T03:02:56.470 INFO:tasks.workunit.client.0.smithi153.stdout:9/541: unlink d1/d10/d36/d57/l87 0 2024-04-05T03:02:56.472 INFO:tasks.workunit.client.0.smithi153.stdout:0/626: dwrite d9/d14/f40 [0,4194304] 0 2024-04-05T03:02:56.474 INFO:tasks.workunit.client.0.smithi153.stdout:9/542: rename d1/d10/l99 to d1/d10/d36/d71/lbf 0 2024-04-05T03:02:56.475 INFO:tasks.workunit.client.0.smithi153.stdout:0/627: creat d9/d14/d67/fc3 x:0 0 0 2024-04-05T03:02:56.477 INFO:tasks.workunit.client.0.smithi153.stdout:9/543: creat d1/d6c/fc0 x:0 0 0 2024-04-05T03:02:56.480 INFO:tasks.workunit.client.0.smithi153.stdout:0/628: creat d9/db/d42/d43/fc4 x:0 0 0 2024-04-05T03:02:56.483 INFO:tasks.workunit.client.0.smithi153.stdout:0/629: write d9/db/d42/d7e/db6/f18 [415675,115278] 0 2024-04-05T03:02:56.484 INFO:tasks.workunit.client.1.smithi203.stdout:4/851: dwrite da/d29/f32 [0,4194304] 0 2024-04-05T03:02:56.485 INFO:tasks.workunit.client.0.smithi153.stdout:0/630: stat d9/db/d42/f60 0 2024-04-05T03:02:56.485 INFO:tasks.workunit.client.0.smithi153.stdout:9/544: dwrite d1/d10/f64 [0,4194304] 0 2024-04-05T03:02:56.486 INFO:tasks.workunit.client.1.smithi203.stdout:3/837: dwrite d0/dc/f54 [4194304,4194304] 0 2024-04-05T03:02:56.486 INFO:tasks.workunit.client.0.smithi153.stdout:0/631: chown d9/d14/f23 73030268 1 2024-04-05T03:02:56.487 INFO:tasks.workunit.client.1.smithi203.stdout:4/852: fdatasync da/d29/d2e/d57/da7/fd9 0 2024-04-05T03:02:56.488 INFO:tasks.workunit.client.0.smithi153.stdout:9/545: chown d1/d9/d1b/f1c 890 1 2024-04-05T03:02:56.492 INFO:tasks.workunit.client.1.smithi203.stdout:3/838: rmdir d0/dc/d1a/d76/d35 39 2024-04-05T03:02:56.492 INFO:tasks.workunit.client.0.smithi153.stdout:0/632: write d9/db/d42/d7e/db6/d1e/d45/d62/fa0 [776539,121749] 0 2024-04-05T03:02:56.494 INFO:tasks.workunit.client.0.smithi153.stdout:5/573: sync 2024-04-05T03:02:56.494 INFO:tasks.workunit.client.0.smithi153.stdout:2/711: sync 2024-04-05T03:02:56.494 INFO:tasks.workunit.client.0.smithi153.stdout:8/522: sync 2024-04-05T03:02:56.494 INFO:tasks.workunit.client.0.smithi153.stdout:4/764: sync 2024-04-05T03:02:56.497 INFO:tasks.workunit.client.1.smithi203.stdout:4/853: creat da/d3c/d4c/d99/f119 x:0 0 0 2024-04-05T03:02:56.497 INFO:tasks.workunit.client.1.smithi203.stdout:3/839: dread d0/d21/f6c [4194304,4194304] 0 2024-04-05T03:02:56.498 INFO:tasks.workunit.client.0.smithi153.stdout:9/546: truncate d1/d9/d1b/f28 613829 0 2024-04-05T03:02:56.499 INFO:tasks.workunit.client.0.smithi153.stdout:2/712: write d9/da/d37/fcb [231169,83582] 0 2024-04-05T03:02:56.500 INFO:tasks.workunit.client.0.smithi153.stdout:1/633: dread d5/d8/d20/d3f/f91 [0,4194304] 0 2024-04-05T03:02:56.502 INFO:tasks.workunit.client.0.smithi153.stdout:5/574: unlink d0/d25/c8a 0 2024-04-05T03:02:56.503 INFO:tasks.workunit.client.0.smithi153.stdout:2/713: write d9/da/fd0 [499314,68868] 0 2024-04-05T03:02:56.503 INFO:tasks.workunit.client.1.smithi203.stdout:3/840: rename d0/dc/d38/f3b to d0/dc/d1a/d8d/d96/ff8 0 2024-04-05T03:02:56.505 INFO:tasks.workunit.client.0.smithi153.stdout:1/634: readlink d5/d8/d20/d4b/d1e/l87 0 2024-04-05T03:02:56.505 INFO:tasks.workunit.client.0.smithi153.stdout:4/765: unlink da/de/d56/d6a/c75 0 2024-04-05T03:02:56.505 INFO:tasks.workunit.client.1.smithi203.stdout:4/854: creat da/d29/d2e/d57/d96/d114/f11a x:0 0 0 2024-04-05T03:02:56.506 INFO:tasks.workunit.client.0.smithi153.stdout:8/523: mkdir d7/d13/dab 0 2024-04-05T03:02:56.508 INFO:tasks.workunit.client.0.smithi153.stdout:8/524: chown d7/d2a/f65 15954 1 2024-04-05T03:02:56.508 INFO:tasks.workunit.client.1.smithi203.stdout:4/855: write da/d29/d2e/db0/dbe/dce/fef [1853242,3776] 0 2024-04-05T03:02:56.509 INFO:tasks.workunit.client.0.smithi153.stdout:4/766: write da/d2a/d86/fe2 [1872776,67230] 0 2024-04-05T03:02:56.511 INFO:tasks.workunit.client.1.smithi203.stdout:3/841: getdents d0/d69 0 2024-04-05T03:02:56.512 INFO:tasks.workunit.client.1.smithi203.stdout:4/856: mkdir da/d29/d2e/db0/dbe/dce/d11b 0 2024-04-05T03:02:56.513 INFO:tasks.workunit.client.1.smithi203.stdout:3/842: read d0/f23 [732273,79348] 0 2024-04-05T03:02:56.513 INFO:tasks.workunit.client.0.smithi153.stdout:2/714: creat d9/ff3 x:0 0 0 2024-04-05T03:02:56.513 INFO:tasks.workunit.client.0.smithi153.stdout:1/635: dwrite d5/dd/f89 [0,4194304] 0 2024-04-05T03:02:56.515 INFO:tasks.workunit.client.1.smithi203.stdout:4/857: rmdir da/d29/d2e/d8b/d109 39 2024-04-05T03:02:56.519 INFO:tasks.workunit.client.0.smithi153.stdout:8/525: rename d7/d13/d19/d1b/d6c/f90 to d7/d13/d19/d1b/d6c/fac 0 2024-04-05T03:02:56.522 INFO:tasks.workunit.client.1.smithi203.stdout:3/843: rename d0/dc/d1a/d76/d35/l6d to d0/dc/d1a/d8d/dc9/lf9 0 2024-04-05T03:02:56.522 INFO:tasks.workunit.client.0.smithi153.stdout:8/526: symlink d7/d13/d19/d1b/d33/lad 0 2024-04-05T03:02:56.522 INFO:tasks.workunit.client.0.smithi153.stdout:1/636: link d5/d8/d20/f25 d5/d8/fc0 0 2024-04-05T03:02:56.523 INFO:tasks.workunit.client.1.smithi203.stdout:3/844: rename d0 to d0/dc/d1a/ddf/df6/dfa 22 2024-04-05T03:02:56.523 INFO:tasks.workunit.client.1.smithi203.stdout:3/845: chown d0/dc/d1a/d8d/d96/f6a 23 1 2024-04-05T03:02:56.525 INFO:tasks.workunit.client.0.smithi153.stdout:4/767: read da/db/d20/f21 [7918541,70119] 0 2024-04-05T03:02:56.526 INFO:tasks.workunit.client.1.smithi203.stdout:3/846: mknod d0/d41/cfb 0 2024-04-05T03:02:56.529 INFO:tasks.workunit.client.0.smithi153.stdout:8/527: link d7/d9a/fa0 d7/d13/d19/d1b/d6d/fae 0 2024-04-05T03:02:56.530 INFO:tasks.workunit.client.0.smithi153.stdout:1/637: dread d5/d8/d20/d4b/d18/d16/f36 [0,4194304] 0 2024-04-05T03:02:56.532 INFO:tasks.workunit.client.1.smithi203.stdout:3/847: dwrite d0/d21/d81/f5b [4194304,4194304] 0 2024-04-05T03:02:56.536 INFO:tasks.workunit.client.1.smithi203.stdout:3/848: rmdir d0/dc/d1a/ddf/df6 0 2024-04-05T03:02:56.537 INFO:tasks.workunit.client.1.smithi203.stdout:0/986: sync 2024-04-05T03:02:56.538 INFO:tasks.workunit.client.1.smithi203.stdout:3/849: chown d0/dc/d1a/d8d/d96/f90 1 1 2024-04-05T03:02:56.540 INFO:tasks.workunit.client.1.smithi203.stdout:3/850: write d0/dc/d1a/d1f/da1/fe5 [415793,63249] 0 2024-04-05T03:02:56.540 INFO:tasks.workunit.client.1.smithi203.stdout:0/987: creat dd/d17/dd8/d56/f128 x:0 0 0 2024-04-05T03:02:56.540 INFO:tasks.workunit.client.1.smithi203.stdout:0/988: readlink dd/lf 0 2024-04-05T03:02:56.541 INFO:tasks.workunit.client.1.smithi203.stdout:0/989: write dd/d17/d115/d28/f4e [1380768,60978] 0 2024-04-05T03:02:56.541 INFO:tasks.workunit.client.1.smithi203.stdout:3/851: read d0/dc/d1a/d8d/d96/fcc [2594731,11822] 0 2024-04-05T03:02:56.543 INFO:tasks.workunit.client.1.smithi203.stdout:3/852: creat d0/d21/d83/ffc x:0 0 0 2024-04-05T03:02:56.546 INFO:tasks.workunit.client.1.smithi203.stdout:3/853: unlink d0/l40 0 2024-04-05T03:02:56.547 INFO:tasks.workunit.client.1.smithi203.stdout:3/854: fdatasync d0/dc/d1a/fc3 0 2024-04-05T03:02:56.549 INFO:tasks.workunit.client.1.smithi203.stdout:3/855: mknod d0/dc/d1a/d1f/da1/daf/cfd 0 2024-04-05T03:02:56.551 INFO:tasks.workunit.client.1.smithi203.stdout:4/858: rmdir da/d3c/d4c 39 2024-04-05T03:02:56.553 INFO:tasks.workunit.client.1.smithi203.stdout:3/856: dread d0/dc/d1a/d76/d35/f47 [0,4194304] 0 2024-04-05T03:02:56.560 INFO:tasks.workunit.client.0.smithi153.stdout:5/575: dwrite d0/d2c/f2d [0,4194304] 0 2024-04-05T03:02:56.560 INFO:tasks.workunit.client.0.smithi153.stdout:4/768: write da/de/d56/d78/d8f/f3d [414166,64621] 0 2024-04-05T03:02:56.560 INFO:tasks.workunit.client.0.smithi153.stdout:9/547: dwrite d1/d10/d36/d57/f78 [0,4194304] 0 2024-04-05T03:02:56.563 INFO:tasks.workunit.client.0.smithi153.stdout:8/528: truncate d7/d13/d19/f4e 3173184 0 2024-04-05T03:02:56.564 INFO:tasks.workunit.client.0.smithi153.stdout:9/548: dread - d1/d3a/fb5 zero size 2024-04-05T03:02:56.566 INFO:tasks.workunit.client.0.smithi153.stdout:9/549: stat d1/d10/d16/d22/d2c/c80 0 2024-04-05T03:02:56.568 INFO:tasks.workunit.client.0.smithi153.stdout:9/550: fdatasync d1/f2e 0 2024-04-05T03:02:56.570 INFO:tasks.workunit.client.0.smithi153.stdout:0/633: dread d9/f69 [0,4194304] 0 2024-04-05T03:02:56.571 INFO:tasks.workunit.client.0.smithi153.stdout:4/769: mkdir da/de/d56/d78/d8f/d108 0 2024-04-05T03:02:56.572 INFO:tasks.workunit.client.0.smithi153.stdout:8/529: rename d7/d13/f64 to d7/d2a/d50/faf 0 2024-04-05T03:02:56.574 INFO:tasks.workunit.client.0.smithi153.stdout:5/576: rmdir d0/d25/d4d/d62/d5f/d86/da4 39 2024-04-05T03:02:56.574 INFO:tasks.workunit.client.0.smithi153.stdout:4/770: readlink da/db/d20/d35/d41/da8/dc1/lba 0 2024-04-05T03:02:56.576 INFO:tasks.workunit.client.0.smithi153.stdout:8/530: creat d7/d13/d19/d1b/d6c/fb0 x:0 0 0 2024-04-05T03:02:56.578 INFO:tasks.workunit.client.0.smithi153.stdout:4/771: creat da/db/f109 x:0 0 0 2024-04-05T03:02:56.579 INFO:tasks.workunit.client.0.smithi153.stdout:5/577: dread d0/d9/f1f [0,4194304] 0 2024-04-05T03:02:56.585 INFO:tasks.workunit.client.0.smithi153.stdout:8/531: dwrite d7/d13/f14 [0,4194304] 0 2024-04-05T03:02:56.587 INFO:tasks.workunit.client.0.smithi153.stdout:8/532: mkdir d7/d46/db1 0 2024-04-05T03:02:56.597 INFO:tasks.workunit.client.1.smithi203.stdout:4/859: symlink da/d29/d2e/d57/d96/d114/l11c 0 2024-04-05T03:02:56.598 INFO:tasks.workunit.client.1.smithi203.stdout:3/857: dread - d0/dc/d1a/d1f/da2/fae zero size 2024-04-05T03:02:56.599 INFO:tasks.workunit.client.1.smithi203.stdout:3/858: chown d0/dc/d38/d5d 979094 1 2024-04-05T03:02:56.604 INFO:tasks.workunit.client.1.smithi203.stdout:3/859: dwrite d0/dc/d1a/d1d/f5c [4194304,4194304] 0 2024-04-05T03:02:56.610 INFO:tasks.workunit.client.1.smithi203.stdout:3/860: dwrite d0/dc/d38/d5d/d66/fc8 [0,4194304] 0 2024-04-05T03:02:56.614 INFO:tasks.workunit.client.1.smithi203.stdout:3/861: creat d0/dc/d1a/d1d/d53/ffe x:0 0 0 2024-04-05T03:02:56.614 INFO:tasks.workunit.client.1.smithi203.stdout:3/862: stat d0/dc/d38/d5d/cf5 0 2024-04-05T03:02:56.618 INFO:tasks.workunit.client.1.smithi203.stdout:3/863: readlink d0/dc/d1a/d8d/dc9/lf9 0 2024-04-05T03:02:56.618 INFO:tasks.workunit.client.0.smithi153.stdout:1/638: dread d5/d8/d20/d4b/f11 [0,4194304] 0 2024-04-05T03:02:56.621 INFO:tasks.workunit.client.0.smithi153.stdout:1/639: symlink d5/d8/d20/d4b/lc1 0 2024-04-05T03:02:56.627 INFO:tasks.workunit.client.0.smithi153.stdout:1/640: truncate d5/d8/d20/f25 646141 0 2024-04-05T03:02:56.628 INFO:tasks.workunit.client.0.smithi153.stdout:1/641: write d5/d8/d20/d4b/fb1 [744041,29231] 0 2024-04-05T03:02:56.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.632+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f48000bc0 con 0x7f7f4c06b600 2024-04-05T03:02:56.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.634+0000 7f7f52ffd700 1 -- 172.21.15.153:0/205678907 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f7f48000bc0 con 0x7f7f4c06b600 2024-04-05T03:02:56.634 INFO:tasks.workunit.client.1.smithi203.stdout:4/860: write da/d29/d2e/d53/df9/f106 [1002269,127373] 0 2024-04-05T03:02:56.636 INFO:tasks.workunit.client.1.smithi203.stdout:4/861: write da/d29/d2e/d57/d96/ddc/ffc [376296,119780] 0 2024-04-05T03:02:56.636 INFO:tasks.workunit.client.1.smithi203.stdout:3/864: truncate d0/dc/d38/d92/fb1 889510 0 2024-04-05T03:02:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 msgr2=0x7f7f4c06dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 0x7f7f4c06dab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009fd0 tx=0x7f7f5400b040 comp rx=0 tx=0).stop 2024-04-05T03:02:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 msgr2=0x7f7f64071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7f580138f0 tx=0x7f7f58009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 shutdown_connections 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f640fce40 0x7f7f64071e40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f7f4c06b600 0x7f7f4c06dab0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 --2- 172.21.15.153:0/205678907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f640fe040 0x7f7f64070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.637+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 >> 172.21.15.153:0/205678907 conn(0x7f7f640f8550 msgr2=0x7f7f640f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.638+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 shutdown_connections 2024-04-05T03:02:56.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.638+0000 7f7f50ff9700 1 -- 172.21.15.153:0/205678907 wait complete. 2024-04-05T03:02:56.639 INFO:tasks.workunit.client.0.smithi153.stdout:1/642: write d5/dd/f5d [124521,92691] 0 2024-04-05T03:02:56.642 INFO:tasks.workunit.client.0.smithi153.stdout:8/533: dread d7/d2a/f65 [0,4194304] 0 2024-04-05T03:02:56.644 INFO:tasks.workunit.client.0.smithi153.stdout:5/578: dwrite d0/d25/d4d/d62/d5f/d86/f85 [0,4194304] 0 2024-04-05T03:02:56.644 INFO:tasks.workunit.client.1.smithi203.stdout:3/865: link d0/dc/d1a/d1f/da2/cb8 d0/dc/d1a/d1d/d53/cff 0 2024-04-05T03:02:56.645 INFO:tasks.workunit.client.0.smithi153.stdout:1/643: dread - d5/d8/d20/d4b/d18/d16/d45/fa3 zero size 2024-04-05T03:02:56.649 INFO:tasks.workunit.client.0.smithi153.stdout:5/579: mkdir d0/d25/d4d/d9c/dc4/dcb 0 2024-04-05T03:02:56.651 INFO:tasks.workunit.client.0.smithi153.stdout:8/534: getdents d7/d13/d19/d1b/d4f/d72 0 2024-04-05T03:02:56.651 INFO:tasks.workunit.client.1.smithi203.stdout:3/866: read d0/d41/f8e [767477,17918] 0 2024-04-05T03:02:56.653 INFO:tasks.workunit.client.0.smithi153.stdout:9/551: dread d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:02:56.657 INFO:tasks.workunit.client.0.smithi153.stdout:4/772: dread da/de/da1/fc0 [0,4194304] 0 2024-04-05T03:02:56.657 INFO:tasks.workunit.client.0.smithi153.stdout:5/580: creat d0/d25/d4d/d9c/dc7/dc9/fcc x:0 0 0 2024-04-05T03:02:56.660 INFO:tasks.workunit.client.1.smithi203.stdout:3/867: dwrite d0/dc/ff3 [0,4194304] 0 2024-04-05T03:02:56.660 INFO:tasks.workunit.client.1.smithi203.stdout:4/862: dwrite da/d29/d2e/d57/d96/ddc/ff3 [0,4194304] 0 2024-04-05T03:02:56.661 INFO:tasks.workunit.client.0.smithi153.stdout:9/552: rename d1/d10/d16/d22/f74 to d1/d9/fc1 0 2024-04-05T03:02:56.661 INFO:tasks.workunit.client.0.smithi153.stdout:1/644: write d5/d8/f75 [386096,15962] 0 2024-04-05T03:02:56.661 INFO:tasks.workunit.client.0.smithi153.stdout:8/535: dread d7/d2a/f7b [0,4194304] 0 2024-04-05T03:02:56.662 INFO:tasks.workunit.client.1.smithi203.stdout:4/863: fdatasync da/d29/d2e/d57/f86 0 2024-04-05T03:02:56.662 INFO:tasks.workunit.client.1.smithi203.stdout:3/868: read d0/dc/d38/f8f [3522777,117397] 0 2024-04-05T03:02:56.662 INFO:tasks.workunit.client.0.smithi153.stdout:5/581: fsync d0/d2c/f3d 0 2024-04-05T03:02:56.663 INFO:tasks.workunit.client.0.smithi153.stdout:8/536: dread - d7/d46/faa zero size 2024-04-05T03:02:56.664 INFO:tasks.workunit.client.0.smithi153.stdout:8/537: chown d7/d13/d19/d95 146561137 1 2024-04-05T03:02:56.666 INFO:tasks.workunit.client.0.smithi153.stdout:4/773: creat da/d2a/f10a x:0 0 0 2024-04-05T03:02:56.667 INFO:tasks.workunit.client.0.smithi153.stdout:5/582: mkdir d0/d25/d4d/d62/d37/d38/dcd 0 2024-04-05T03:02:56.668 INFO:tasks.workunit.client.1.smithi203.stdout:3/869: symlink d0/l100 0 2024-04-05T03:02:56.668 INFO:tasks.workunit.client.0.smithi153.stdout:1/645: mknod d5/d8/d20/d4b/d18/cc2 0 2024-04-05T03:02:56.672 INFO:tasks.workunit.client.0.smithi153.stdout:9/553: rename d1/d10/d16/d22/fa4 to d1/d3a/d91/d96/dbd/fc2 0 2024-04-05T03:02:56.673 INFO:tasks.workunit.client.1.smithi203.stdout:3/870: mknod d0/dc/d1a/d1d/d53/c101 0 2024-04-05T03:02:56.676 INFO:tasks.workunit.client.1.smithi203.stdout:4/864: dread da/d29/d2e/d53/d78/fdd [0,4194304] 0 2024-04-05T03:02:56.680 INFO:tasks.workunit.client.0.smithi153.stdout:9/554: fdatasync d1/d9/d1b/f7b 0 2024-04-05T03:02:56.681 INFO:tasks.workunit.client.1.smithi203.stdout:3/871: creat d0/dc/d1a/d1f/f102 x:0 0 0 2024-04-05T03:02:56.681 INFO:tasks.workunit.client.1.smithi203.stdout:4/865: getdents da/d29/d2e/d57/da5 0 2024-04-05T03:02:56.681 INFO:tasks.workunit.client.0.smithi153.stdout:9/555: chown d1/d10/d16/d22/d2c/f31 1 1 2024-04-05T03:02:56.682 INFO:tasks.workunit.client.0.smithi153.stdout:8/538: creat d7/d13/d19/d1b/d4f/d5f/fb2 x:0 0 0 2024-04-05T03:02:56.682 INFO:tasks.workunit.client.0.smithi153.stdout:4/774: getdents da/db/d20/d35/d38/df7 0 2024-04-05T03:02:56.683 INFO:tasks.workunit.client.0.smithi153.stdout:9/556: chown d1/d9/d1b/fb0 46961 1 2024-04-05T03:02:56.684 INFO:tasks.workunit.client.1.smithi203.stdout:3/872: symlink d0/dc/d1a/d1f/l103 0 2024-04-05T03:02:56.684 INFO:tasks.workunit.client.1.smithi203.stdout:3/873: chown d0/dc/d38/d5d/la0 338620091 1 2024-04-05T03:02:56.685 INFO:tasks.workunit.client.1.smithi203.stdout:4/866: mknod da/d29/d2e/d57/d96/ddc/de0/c11d 0 2024-04-05T03:02:56.685 INFO:tasks.workunit.client.0.smithi153.stdout:8/539: dread d7/f2b [0,4194304] 0 2024-04-05T03:02:56.685 INFO:tasks.workunit.client.0.smithi153.stdout:4/775: creat da/db/d20/d35/d38/d76/d90/f10b x:0 0 0 2024-04-05T03:02:56.687 INFO:tasks.workunit.client.0.smithi153.stdout:4/776: chown da/db/d20/d35/d41/da8/dc1/f9b 160735 1 2024-04-05T03:02:56.688 INFO:tasks.workunit.client.1.smithi203.stdout:3/874: rmdir d0/dc/d1a/d8d 39 2024-04-05T03:02:56.691 INFO:tasks.workunit.client.1.smithi203.stdout:4/867: dwrite da/d29/d2e/db0/dcb/d80/fb9 [0,4194304] 0 2024-04-05T03:02:56.699 INFO:tasks.workunit.client.1.smithi203.stdout:4/868: dread da/d29/d2e/d8b/dc8/f110 [0,4194304] 0 2024-04-05T03:02:56.705 INFO:tasks.workunit.client.0.smithi153.stdout:8/540: dread d7/f28 [0,4194304] 0 2024-04-05T03:02:56.705 INFO:tasks.workunit.client.1.smithi203.stdout:4/869: creat da/d3c/d4c/d99/f11e x:0 0 0 2024-04-05T03:02:56.705 INFO:tasks.workunit.client.0.smithi153.stdout:5/583: write d0/d9/d22/f28 [2144207,89719] 0 2024-04-05T03:02:56.707 INFO:tasks.workunit.client.0.smithi153.stdout:9/557: write d1/f4 [20209,103724] 0 2024-04-05T03:02:56.708 INFO:tasks.workunit.client.0.smithi153.stdout:4/777: write da/ff [3650226,77406] 0 2024-04-05T03:02:56.708 INFO:tasks.workunit.client.1.smithi203.stdout:4/870: creat da/d29/d2e/d57/d96/ddc/f11f x:0 0 0 2024-04-05T03:02:56.709 INFO:tasks.workunit.client.0.smithi153.stdout:8/541: write d7/d13/d19/d1b/d21/f69 [253860,76518] 0 2024-04-05T03:02:56.709 INFO:tasks.workunit.client.0.smithi153.stdout:5/584: creat d0/d25/d68/fce x:0 0 0 2024-04-05T03:02:56.710 INFO:tasks.workunit.client.0.smithi153.stdout:6/759: sync 2024-04-05T03:02:56.710 INFO:tasks.workunit.client.0.smithi153.stdout:7/805: sync 2024-04-05T03:02:56.713 INFO:tasks.workunit.client.1.smithi203.stdout:4/871: dread da/d29/d2e/db0/dbe/dce/dd0/ff7 [0,4194304] 0 2024-04-05T03:02:56.713 INFO:tasks.workunit.client.0.smithi153.stdout:8/542: fdatasync d7/d2a/d50/f6b 0 2024-04-05T03:02:56.716 INFO:tasks.workunit.client.1.smithi203.stdout:4/872: readlink da/d3c/d4c/d99/leb 0 2024-04-05T03:02:56.716 INFO:tasks.workunit.client.0.smithi153.stdout:5/585: mknod d0/d25/d4d/d9c/dc7/ccf 0 2024-04-05T03:02:56.716 INFO:tasks.workunit.client.0.smithi153.stdout:1/646: dwrite d5/d8/d20/d4b/d18/f1a [4194304,4194304] 0 2024-04-05T03:02:56.716 INFO:tasks.workunit.client.0.smithi153.stdout:4/778: dread da/de/d56/f61 [0,4194304] 0 2024-04-05T03:02:56.718 INFO:tasks.workunit.client.1.smithi203.stdout:4/873: truncate da/d29/fb2 1023230 0 2024-04-05T03:02:56.719 INFO:tasks.workunit.client.0.smithi153.stdout:9/558: dread d1/d10/d36/f56 [0,4194304] 0 2024-04-05T03:02:56.721 INFO:tasks.workunit.client.1.smithi203.stdout:4/874: creat da/d29/d2e/db0/dbe/dce/dd0/f120 x:0 0 0 2024-04-05T03:02:56.723 INFO:tasks.workunit.client.1.smithi203.stdout:4/875: write da/d29/f9e [4964286,67009] 0 2024-04-05T03:02:56.724 INFO:tasks.workunit.client.0.smithi153.stdout:7/806: dwrite d2/f1d [4194304,4194304] 0 2024-04-05T03:02:56.725 INFO:tasks.workunit.client.1.smithi203.stdout:4/876: symlink da/d29/d2e/d57/d96/l121 0 2024-04-05T03:02:56.727 INFO:tasks.workunit.client.1.smithi203.stdout:4/877: creat da/d29/d2e/db0/dbe/f122 x:0 0 0 2024-04-05T03:02:56.730 INFO:tasks.workunit.client.0.smithi153.stdout:7/807: dread d2/da/d3a/ffb [0,4194304] 0 2024-04-05T03:02:56.730 INFO:tasks.workunit.client.0.smithi153.stdout:7/808: readlink d2/d9b/d6f/l84 0 2024-04-05T03:02:56.737 INFO:tasks.workunit.client.0.smithi153.stdout:9/559: rename d1/d10/d16/fae to d1/d10/d16/d22/fc3 0 2024-04-05T03:02:56.738 INFO:tasks.workunit.client.1.smithi203.stdout:4/878: getdents da/d29/d2e/d8b 0 2024-04-05T03:02:56.741 INFO:tasks.workunit.client.1.smithi203.stdout:4/879: symlink da/d3c/l123 0 2024-04-05T03:02:56.743 INFO:tasks.workunit.client.1.smithi203.stdout:4/880: chown da/d29/d2e/dc9/cde 178924 1 2024-04-05T03:02:56.743 INFO:tasks.workunit.client.0.smithi153.stdout:9/560: chown d1/d10/d16/d22/d2c/d68/d81/f76 0 1 2024-04-05T03:02:56.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/543: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:56.746 INFO:tasks.workunit.client.1.smithi203.stdout:4/881: truncate da/d29/d2e/d53/fa4 417238 0 2024-04-05T03:02:56.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/586: dwrite d0/d25/f96 [0,4194304] 0 2024-04-05T03:02:56.750 INFO:tasks.workunit.client.1.smithi203.stdout:4/882: rename da/d29/d2e/d57/d96/ddc/f11f to da/d29/d2e/d57/d96/ddc/f124 0 2024-04-05T03:02:56.751 INFO:tasks.workunit.client.0.smithi153.stdout:8/544: truncate d7/d13/f8e 515466 0 2024-04-05T03:02:56.751 INFO:tasks.workunit.client.0.smithi153.stdout:5/587: creat d0/d25/d68/fd0 x:0 0 0 2024-04-05T03:02:56.752 INFO:tasks.workunit.client.1.smithi203.stdout:4/883: dread da/d29/d2e/f63 [0,4194304] 0 2024-04-05T03:02:56.757 INFO:tasks.workunit.client.0.smithi153.stdout:8/545: creat d7/d13/d19/d95/fb3 x:0 0 0 2024-04-05T03:02:56.759 INFO:tasks.workunit.client.0.smithi153.stdout:8/546: truncate d7/f74 806578 0 2024-04-05T03:02:56.762 INFO:tasks.workunit.client.0.smithi153.stdout:8/547: symlink d7/d13/d19/d1b/d4f/d68/lb4 0 2024-04-05T03:02:56.766 INFO:tasks.workunit.client.0.smithi153.stdout:6/760: dwrite d4/dc/df/d16/d44/fdb [0,4194304] 0 2024-04-05T03:02:56.770 INFO:tasks.workunit.client.0.smithi153.stdout:8/548: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:56.772 INFO:tasks.workunit.client.0.smithi153.stdout:6/761: creat d4/dc/df/d16/d44/ded/fff x:0 0 0 2024-04-05T03:02:56.780 INFO:tasks.workunit.client.0.smithi153.stdout:7/809: write d2/da/d3a/d41/ddd/f107 [6699608,109901] 0 2024-04-05T03:02:56.782 INFO:tasks.workunit.client.0.smithi153.stdout:1/647: write d5/d8/d20/d2f/f60 [1699251,62162] 0 2024-04-05T03:02:56.782 INFO:tasks.workunit.client.0.smithi153.stdout:7/810: dread - d2/fd6 zero size 2024-04-05T03:02:56.785 INFO:tasks.workunit.client.0.smithi153.stdout:5/588: write d0/d25/d4d/d62/d37/f4f [5474947,45794] 0 2024-04-05T03:02:56.786 INFO:tasks.workunit.client.1.smithi203.stdout:4/884: dwrite da/d29/d2e/f61 [0,4194304] 0 2024-04-05T03:02:56.787 INFO:tasks.workunit.client.0.smithi153.stdout:4/779: dwrite da/db/d20/d35/d38/f5c [0,4194304] 0 2024-04-05T03:02:56.787 INFO:tasks.workunit.client.0.smithi153.stdout:1/648: chown d5/d8/d20/d2f/d40 0 1 2024-04-05T03:02:56.789 INFO:tasks.workunit.client.0.smithi153.stdout:7/811: symlink d2/d9b/d6f/l111 0 2024-04-05T03:02:56.790 INFO:tasks.workunit.client.1.smithi203.stdout:4/885: symlink da/d29/d2e/d8b/l125 0 2024-04-05T03:02:56.794 INFO:tasks.workunit.client.0.smithi153.stdout:9/561: dwrite d1/d10/d36/d57/f6f [0,4194304] 0 2024-04-05T03:02:56.796 INFO:tasks.workunit.client.0.smithi153.stdout:1/649: write d5/d8/d20/d4b/f11 [2911147,25287] 0 2024-04-05T03:02:56.799 INFO:tasks.workunit.client.0.smithi153.stdout:1/650: write d5/dd/f89 [4122325,10352] 0 2024-04-05T03:02:56.799 INFO:tasks.workunit.client.0.smithi153.stdout:9/562: fdatasync d1/d10/d16/d22/d2c/f31 0 2024-04-05T03:02:56.803 INFO:tasks.workunit.client.0.smithi153.stdout:4/780: rename da/db/fe4 to da/de/d56/d78/d8f/d68/f10c 0 2024-04-05T03:02:56.808 INFO:tasks.workunit.client.0.smithi153.stdout:5/589: dread d0/f14 [4194304,4194304] 0 2024-04-05T03:02:56.810 INFO:tasks.workunit.client.0.smithi153.stdout:4/781: truncate da/db/d20/d35/d38/d76/d90/fd6 433839 0 2024-04-05T03:02:56.811 INFO:tasks.workunit.client.0.smithi153.stdout:9/563: creat d1/fc4 x:0 0 0 2024-04-05T03:02:56.815 INFO:tasks.workunit.client.0.smithi153.stdout:1/651: getdents d5/dd 0 2024-04-05T03:02:56.815 INFO:tasks.workunit.client.0.smithi153.stdout:8/549: dread d7/d13/f62 [0,4194304] 0 2024-04-05T03:02:56.819 INFO:tasks.workunit.client.0.smithi153.stdout:4/782: getdents da/db/d20/d35/d41/d67 0 2024-04-05T03:02:56.823 INFO:tasks.workunit.client.0.smithi153.stdout:1/652: rename d5/d8/d20/d4b/d18/d16/d48/d96/daf to d5/d8/d20/d4b/d18/d16/d45/d68/da4/dc3 0 2024-04-05T03:02:56.825 INFO:tasks.workunit.client.0.smithi153.stdout:9/564: dread d1/d10/d36/f60 [0,4194304] 0 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 -- 172.21.15.153:0/3838698961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 msgr2=0x7efd4c0fc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 --2- 172.21.15.153:0/3838698961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c0fc240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efd3c004940 tx=0x7efd3c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 -- 172.21.15.153:0/3838698961 shutdown_connections 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 --2- 172.21.15.153:0/3838698961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 0x7efd4c0fd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 --2- 172.21.15.153:0/3838698961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c0fc240 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.829+0000 7efd515ba700 1 -- 172.21.15.153:0/3838698961 >> 172.21.15.153:0/3838698961 conn(0x7efd4c0f7480 msgr2=0x7efd4c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.830+0000 7efd515ba700 1 -- 172.21.15.153:0/3838698961 shutdown_connections 2024-04-05T03:02:56.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.830+0000 7efd515ba700 1 -- 172.21.15.153:0/3838698961 wait complete. 2024-04-05T03:02:56.830 INFO:tasks.workunit.client.0.smithi153.stdout:4/783: dwrite da/d2a/d86/fe9 [0,4194304] 0 2024-04-05T03:02:56.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.831+0000 7efd515ba700 1 Processor -- start 2024-04-05T03:02:56.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.831+0000 7efd515ba700 1 -- start start 2024-04-05T03:02:56.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd515ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd515ba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 0x7efd4c10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd515ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c10e4d0 con 0x7efd4c0fbe30 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd515ba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c10ed20 con 0x7efd4c0fcfa0 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd4affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd4affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39782/0 (socket says 172.21.15.153:39782) 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd4affd700 1 -- 172.21.15.153:0/362667951 learned_addr learned my addr 172.21.15.153:0/362667951 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.832+0000 7efd4a7fc700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 0x7efd4c10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:56.832 INFO:tasks.workunit.client.0.smithi153.stdout:9/565: rename d1/f2e to d1/d3a/d91/fc5 0 2024-04-05T03:02:56.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.833+0000 7efd4affd700 1 -- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 msgr2=0x7efd4c10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:56.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.833+0000 7efd4affd700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 0x7efd4c10ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:56.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.833+0000 7efd4affd700 1 -- 172.21.15.153:0/362667951 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c005040 con 0x7efd4c0fbe30 2024-04-05T03:02:56.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.833+0000 7efd4affd700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b0d0 tx=0x7efd3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:56.834 INFO:tasks.workunit.client.0.smithi153.stdout:4/784: mkdir da/db/d20/d3c/d91/dd5/d10d 0 2024-04-05T03:02:56.840 INFO:tasks.workunit.client.0.smithi153.stdout:4/785: read da/db/f7b [606603,118801] 0 2024-04-05T03:02:56.842 INFO:tasks.workunit.client.0.smithi153.stdout:4/786: symlink da/db/d8b/db7/l10e 0 2024-04-05T03:02:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.844+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c00e070 con 0x7efd4c0fbe30 2024-04-05T03:02:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.844+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4c10efa0 con 0x7efd4c0fbe30 2024-04-05T03:02:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.844+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd3c00b410 con 0x7efd4c0fbe30 2024-04-05T03:02:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.844+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c0125e0 con 0x7efd4c0fbe30 2024-04-05T03:02:56.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.844+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4c10f4e0 con 0x7efd4c0fbe30 2024-04-05T03:02:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.845+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4c101b30 con 0x7efd4c0fbe30 2024-04-05T03:02:56.845 INFO:tasks.workunit.client.0.smithi153.stdout:6/762: dread d4/dc/d7a/da8/fca [0,4194304] 0 2024-04-05T03:02:56.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.846+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7efd3c00b580 con 0x7efd4c0fbe30 2024-04-05T03:02:56.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.847+0000 7efd337fe700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 0x7efd3406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:56.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.847+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7efd3c08ad80 con 0x7efd4c0fbe30 2024-04-05T03:02:56.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.847+0000 7efd4a7fc700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 0x7efd3406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:56.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.848+0000 7efd4a7fc700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 0x7efd3406d9f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd40009b80 tx=0x7efd4000b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:56.847 INFO:tasks.workunit.client.0.smithi153.stdout:4/787: dread da/db/d20/d45/f82 [0,4194304] 0 2024-04-05T03:02:56.848 INFO:tasks.workunit.client.0.smithi153.stdout:6/763: symlink d4/dc/df/d16/d24/d5e/de8/l100 0 2024-04-05T03:02:56.849 INFO:tasks.workunit.client.0.smithi153.stdout:4/788: write da/fb0 [3056365,21805] 0 2024-04-05T03:02:56.849 INFO:tasks.workunit.client.0.smithi153.stdout:6/764: stat d4/dc/d7a/da8 0 2024-04-05T03:02:56.851 INFO:tasks.workunit.client.0.smithi153.stdout:6/765: mkdir d4/dc/d101 0 2024-04-05T03:02:56.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:56.852+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7efd3c058710 con 0x7efd4c0fbe30 2024-04-05T03:02:56.852 INFO:tasks.workunit.client.0.smithi153.stdout:4/789: truncate da/db/d20/d35/d41/da8/dc1/f9b 539047 0 2024-04-05T03:02:56.853 INFO:tasks.workunit.client.0.smithi153.stdout:6/766: mknod d4/dc/d101/c102 0 2024-04-05T03:02:56.855 INFO:tasks.workunit.client.0.smithi153.stdout:4/790: fdatasync da/db/d20/d45/f43 0 2024-04-05T03:02:56.856 INFO:tasks.workunit.client.0.smithi153.stdout:6/767: mknod d4/dc/df/d16/d44/ded/c103 0 2024-04-05T03:02:56.857 INFO:tasks.workunit.client.0.smithi153.stdout:4/791: mkdir da/db/d8b/d10f 0 2024-04-05T03:02:56.858 INFO:tasks.workunit.client.1.smithi203.stdout:4/886: mknod da/d29/d2e/d8b/d109/c126 0 2024-04-05T03:02:56.859 INFO:tasks.workunit.client.0.smithi153.stdout:1/653: write d5/d8/d20/d4b/f79 [725361,58025] 0 2024-04-05T03:02:56.859 INFO:tasks.workunit.client.0.smithi153.stdout:6/768: dread d4/d38/fbb [0,4194304] 0 2024-04-05T03:02:56.861 INFO:tasks.workunit.client.1.smithi203.stdout:4/887: write da/d29/d2e/db0/dcb/d80/f84 [1038772,49913] 0 2024-04-05T03:02:56.865 INFO:tasks.workunit.client.0.smithi153.stdout:4/792: rename f8 to da/db/d20/d45/db1/f110 0 2024-04-05T03:02:56.867 INFO:tasks.workunit.client.0.smithi153.stdout:8/550: dwrite d7/d13/f18 [0,4194304] 0 2024-04-05T03:02:56.868 INFO:tasks.workunit.client.0.smithi153.stdout:6/769: unlink d4/dc/df/d16/d36/d4d/d64/fbd 0 2024-04-05T03:02:56.872 INFO:tasks.workunit.client.0.smithi153.stdout:4/793: write da/db/d20/d45/f6b [2180629,118437] 0 2024-04-05T03:02:56.872 INFO:tasks.workunit.client.0.smithi153.stdout:1/654: dwrite d5/dd/d1b/f74 [0,4194304] 0 2024-04-05T03:02:56.875 INFO:tasks.workunit.client.0.smithi153.stdout:6/770: fdatasync d4/dc/df/d16/d44/fd9 0 2024-04-05T03:02:56.881 INFO:tasks.workunit.client.0.smithi153.stdout:4/794: rename da/d2a to da/de/d66/d111 0 2024-04-05T03:02:56.883 INFO:tasks.workunit.client.0.smithi153.stdout:4/795: dread - da/db/d20/d35/d41/d67/fae zero size 2024-04-05T03:02:56.884 INFO:tasks.workunit.client.1.smithi203.stdout:4/888: dwrite da/d29/d2e/db0/dcb/d80/fc7 [0,4194304] 0 2024-04-05T03:02:56.885 INFO:tasks.workunit.client.0.smithi153.stdout:6/771: mknod d4/dc/d101/c104 0 2024-04-05T03:02:56.889 INFO:tasks.workunit.client.0.smithi153.stdout:8/551: getdents d7/d13/d19/d1b/d4f/d72/d73 0 2024-04-05T03:02:56.890 INFO:tasks.workunit.client.1.smithi203.stdout:4/889: fdatasync da/d29/d2e/d53/f97 0 2024-04-05T03:02:56.891 INFO:tasks.workunit.client.0.smithi153.stdout:6/772: symlink d4/dc/df/d16/d36/d4d/l105 0 2024-04-05T03:02:56.892 INFO:tasks.workunit.client.1.smithi203.stdout:4/890: rmdir da/d29/d2e/d8b/d109 39 2024-04-05T03:02:56.894 INFO:tasks.workunit.client.0.smithi153.stdout:8/552: fsync d7/d13/d19/f8f 0 2024-04-05T03:02:56.895 INFO:tasks.workunit.client.0.smithi153.stdout:4/796: link da/db/f46 da/de/d56/d78/d8f/f112 0 2024-04-05T03:02:56.897 INFO:tasks.workunit.client.0.smithi153.stdout:6/773: rename d4/dc/df/d16/d33/fcd to d4/dc/df/d16/d36/d4d/d64/f106 0 2024-04-05T03:02:56.901 INFO:tasks.workunit.client.1.smithi203.stdout:4/891: rename da/d29/d2e/d8b/dc8 to da/d29/d2e/d57/da5/d127 0 2024-04-05T03:02:56.903 INFO:tasks.workunit.client.0.smithi153.stdout:6/774: mknod d4/dc/df/d16/d17/c107 0 2024-04-05T03:02:56.905 INFO:tasks.workunit.client.0.smithi153.stdout:8/553: dwrite d7/d85/f4a [4194304,4194304] 0 2024-04-05T03:02:56.907 INFO:tasks.workunit.client.0.smithi153.stdout:4/797: creat da/db/d20/d35/d38/d76/f113 x:0 0 0 2024-04-05T03:02:56.908 INFO:tasks.workunit.client.0.smithi153.stdout:1/655: dread d5/f7 [4194304,4194304] 0 2024-04-05T03:02:56.909 INFO:tasks.workunit.client.0.smithi153.stdout:1/656: fsync f4 0 2024-04-05T03:02:56.910 INFO:tasks.workunit.client.1.smithi203.stdout:4/892: dwrite da/d29/d2e/fb1 [4194304,4194304] 0 2024-04-05T03:02:56.911 INFO:tasks.workunit.client.0.smithi153.stdout:1/657: stat d5/d8/d20/d4b/d51/db5/cbf 0 2024-04-05T03:02:56.914 INFO:tasks.workunit.client.0.smithi153.stdout:6/775: rename d4/dc/df/d16/d36/d52/c5d to d4/dc/df/d16/d24/d5e/de8/dfa/c108 0 2024-04-05T03:02:56.914 INFO:tasks.workunit.client.0.smithi153.stdout:8/554: dwrite d7/d2a/fa2 [0,4194304] 0 2024-04-05T03:02:56.917 INFO:tasks.workunit.client.1.smithi203.stdout:4/893: creat da/d29/d2e/d8b/d109/f128 x:0 0 0 2024-04-05T03:02:56.924 INFO:tasks.workunit.client.1.smithi203.stdout:4/894: creat da/d3c/d4c/d99/f129 x:0 0 0 2024-04-05T03:02:56.925 INFO:tasks.workunit.client.0.smithi153.stdout:6/776: creat d4/dc/df/d16/d24/d4e/f109 x:0 0 0 2024-04-05T03:02:56.926 INFO:tasks.workunit.client.0.smithi153.stdout:8/555: creat d7/d13/d19/fb5 x:0 0 0 2024-04-05T03:02:56.927 INFO:tasks.workunit.client.0.smithi153.stdout:6/777: chown d4/d38/d4c/l77 41212236 1 2024-04-05T03:02:56.928 INFO:tasks.workunit.client.1.smithi203.stdout:4/895: fsync da/d29/d2e/d53/fa4 0 2024-04-05T03:02:56.932 INFO:tasks.workunit.client.0.smithi153.stdout:6/778: mknod d4/dc/df/d16/c10a 0 2024-04-05T03:02:56.939 INFO:tasks.workunit.client.0.smithi153.stdout:1/658: dwrite d5/d8/d20/d3f/f64 [0,4194304] 0 2024-04-05T03:02:56.947 INFO:tasks.workunit.client.1.smithi203.stdout:4/896: rmdir da/d3c/d4c/d99 39 2024-04-05T03:02:56.948 INFO:tasks.workunit.client.1.smithi203.stdout:4/897: write f9 [9240797,112039] 0 2024-04-05T03:02:56.950 INFO:tasks.workunit.client.1.smithi203.stdout:4/898: mkdir da/d3c/d12a 0 2024-04-05T03:02:56.955 INFO:tasks.workunit.client.0.smithi153.stdout:8/556: dread d7/d2a/f77 [0,4194304] 0 2024-04-05T03:02:56.956 INFO:tasks.workunit.client.0.smithi153.stdout:4/798: dread da/db/d20/d45/db1/f110 [0,4194304] 0 2024-04-05T03:02:56.959 INFO:tasks.workunit.client.0.smithi153.stdout:4/799: creat da/f114 x:0 0 0 2024-04-05T03:02:56.960 INFO:tasks.workunit.client.0.smithi153.stdout:8/557: read d7/d13/d19/d1b/d33/f59 [252490,87769] 0 2024-04-05T03:02:56.962 INFO:tasks.workunit.client.0.smithi153.stdout:8/558: symlink d7/d13/lb6 0 2024-04-05T03:02:56.963 INFO:tasks.workunit.client.0.smithi153.stdout:4/800: creat da/db/d20/d35/d38/d76/f115 x:0 0 0 2024-04-05T03:02:56.965 INFO:tasks.workunit.client.1.smithi203.stdout:4/899: dread da/fdf [4194304,4194304] 0 2024-04-05T03:02:56.966 INFO:tasks.workunit.client.0.smithi153.stdout:8/559: unlink d7/d2a/d50/faf 0 2024-04-05T03:02:56.968 INFO:tasks.workunit.client.0.smithi153.stdout:4/801: creat da/db/d8b/d10f/f116 x:0 0 0 2024-04-05T03:02:56.968 INFO:tasks.workunit.client.1.smithi203.stdout:4/900: mknod da/d29/d2e/db0/dbe/c12b 0 2024-04-05T03:02:56.968 INFO:tasks.workunit.client.1.smithi203.stdout:4/901: stat da/d29/d2e/db0/dbe/dce/ce3 0 2024-04-05T03:02:56.970 INFO:tasks.workunit.client.1.smithi203.stdout:4/902: read da/d29/d2e/db0/dcb/d80/fa6 [2253173,108004] 0 2024-04-05T03:02:56.972 INFO:tasks.workunit.client.0.smithi153.stdout:1/659: dread d5/d8/d20/d4b/d18/d16/d48/f50 [0,4194304] 0 2024-04-05T03:02:56.973 INFO:tasks.workunit.client.0.smithi153.stdout:8/560: rename d7/d13/d19/d1b/d21/fa4 to d7/d13/d19/d1b/d4f/d96/fb7 0 2024-04-05T03:02:56.974 INFO:tasks.workunit.client.1.smithi203.stdout:4/903: unlink da/c1c 0 2024-04-05T03:02:56.975 INFO:tasks.workunit.client.0.smithi153.stdout:1/660: fsync d5/d8/d20/d4b/d18/d16/d48/d96/fbe 0 2024-04-05T03:02:56.975 INFO:tasks.workunit.client.1.smithi203.stdout:4/904: read - da/d29/d2e/d8b/d109/f128 zero size 2024-04-05T03:02:56.975 INFO:tasks.workunit.client.0.smithi153.stdout:1/661: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:56.978 INFO:tasks.workunit.client.0.smithi153.stdout:6/779: dread d4/dc/df/d16/d36/d4d/d64/f106 [0,4194304] 0 2024-04-05T03:02:56.978 INFO:tasks.workunit.client.1.smithi203.stdout:4/905: creat da/d29/d2e/d57/f12c x:0 0 0 2024-04-05T03:02:56.978 INFO:tasks.workunit.client.0.smithi153.stdout:8/561: rename d7/d46/c83 to d7/d13/d19/d1b/d6c/cb8 0 2024-04-05T03:02:56.980 INFO:tasks.workunit.client.0.smithi153.stdout:1/662: creat d5/d8/d20/d2f/db8/fc4 x:0 0 0 2024-04-05T03:02:56.980 INFO:tasks.workunit.client.1.smithi203.stdout:4/906: rmdir da/d29/d2e/d57/d96/ddc/de1 39 2024-04-05T03:02:56.980 INFO:tasks.workunit.client.0.smithi153.stdout:6/780: mknod d4/dc/df/d16/c10b 0 2024-04-05T03:02:56.982 INFO:tasks.workunit.client.0.smithi153.stdout:8/562: mkdir d7/d13/d19/d1b/db9 0 2024-04-05T03:02:56.984 INFO:tasks.workunit.client.0.smithi153.stdout:8/563: rmdir d7/d9a 39 2024-04-05T03:02:56.985 INFO:tasks.workunit.client.0.smithi153.stdout:1/663: symlink d5/d8/d20/d3f/da7/lc5 0 2024-04-05T03:02:56.985 INFO:tasks.workunit.client.0.smithi153.stdout:8/564: chown d7/d13/d19/d1b/d4f/d96 51 1 2024-04-05T03:02:56.985 INFO:tasks.workunit.client.0.smithi153.stdout:6/781: write d4/dc/df/d16/d17/f84 [1965445,74030] 0 2024-04-05T03:02:56.986 INFO:tasks.workunit.client.1.smithi203.stdout:4/907: link da/d29/d2e/d53/c98 da/d29/d2e/db0/c12d 0 2024-04-05T03:02:56.987 INFO:tasks.workunit.client.0.smithi153.stdout:8/565: chown d7/d13/d19/d1b/d4f/d5f/f87 6 1 2024-04-05T03:02:56.988 INFO:tasks.workunit.client.1.smithi203.stdout:4/908: mknod da/d71/c12e 0 2024-04-05T03:02:56.989 INFO:tasks.workunit.client.1.smithi203.stdout:4/909: write da/d29/ffd [1030604,82578] 0 2024-04-05T03:02:56.992 INFO:tasks.workunit.client.0.smithi153.stdout:6/782: rename d4/dc/df/d16/d24/d4e/ff6 to d4/dc/d101/f10c 0 2024-04-05T03:02:56.993 INFO:tasks.workunit.client.1.smithi203.stdout:4/910: rename da/d29/d2e/d53/d78/f111 to da/d29/d2e/d8b/f12f 0 2024-04-05T03:02:56.994 INFO:tasks.workunit.client.0.smithi153.stdout:6/783: chown d4/dc/df/d16/d24/d5e/f60 5667443 1 2024-04-05T03:02:56.994 INFO:tasks.workunit.client.0.smithi153.stdout:8/566: mknod d7/d13/d19/d1b/d6d/cba 0 2024-04-05T03:02:56.995 INFO:tasks.workunit.client.0.smithi153.stdout:1/664: truncate d5/dd/d1b/fae 8729 0 2024-04-05T03:02:56.996 INFO:tasks.workunit.client.0.smithi153.stdout:6/784: write d4/f15 [647905,73152] 0 2024-04-05T03:02:56.997 INFO:tasks.workunit.client.0.smithi153.stdout:8/567: mknod d7/d13/d19/d1b/d33/cbb 0 2024-04-05T03:02:56.998 INFO:tasks.workunit.client.0.smithi153.stdout:8/568: fsync d7/d13/d19/d1b/d4f/d72/f99 0 2024-04-05T03:02:56.999 INFO:tasks.workunit.client.1.smithi203.stdout:4/911: dwrite da/d29/d2e/d57/d96/ddc/ff3 [0,4194304] 0 2024-04-05T03:02:57.000 INFO:tasks.workunit.client.0.smithi153.stdout:6/785: creat d4/f10d x:0 0 0 2024-04-05T03:02:57.000 INFO:tasks.workunit.client.1.smithi203.stdout:4/912: stat da/d29/d2e/db0/f117 0 2024-04-05T03:02:57.002 INFO:tasks.workunit.client.1.smithi203.stdout:4/913: dread - da/d29/d2e/d57/d96/dbc/fbd zero size 2024-04-05T03:02:57.002 INFO:tasks.workunit.client.0.smithi153.stdout:8/569: unlink d7/d13/f18 0 2024-04-05T03:02:57.003 INFO:tasks.workunit.client.0.smithi153.stdout:1/665: dwrite d5/d8/d20/d4b/d18/d16/d45/f4a [0,4194304] 0 2024-04-05T03:02:57.005 INFO:tasks.workunit.client.0.smithi153.stdout:1/666: write d5/d8/f9d [441773,113783] 0 2024-04-05T03:02:57.005 INFO:tasks.workunit.client.0.smithi153.stdout:8/570: mkdir d7/d13/d19/d1b/d4f/d68/dbc 0 2024-04-05T03:02:57.008 INFO:tasks.workunit.client.1.smithi203.stdout:4/914: dwrite da/d3c/d4c/d99/f108 [0,4194304] 0 2024-04-05T03:02:57.009 INFO:tasks.workunit.client.0.smithi153.stdout:8/571: read - d7/d85/f6a zero size 2024-04-05T03:02:57.009 INFO:tasks.workunit.client.1.smithi203.stdout:4/915: readlink da/d3c/d4c/d99/leb 0 2024-04-05T03:02:57.012 INFO:tasks.workunit.client.0.smithi153.stdout:8/572: write d7/f74 [532226,106219] 0 2024-04-05T03:02:57.013 INFO:tasks.workunit.client.0.smithi153.stdout:8/573: chown d7/d13/d19/f5c 6 1 2024-04-05T03:02:57.015 INFO:tasks.workunit.client.1.smithi203.stdout:4/916: mkdir da/d29/d2e/d57/d96/dbc/d130 0 2024-04-05T03:02:57.016 INFO:tasks.workunit.client.0.smithi153.stdout:1/667: rename d5/d8/d20/d2f/c55 to d5/d8/d20/d4b/d18/d16/d2c/cc6 0 2024-04-05T03:02:57.016 INFO:tasks.workunit.client.0.smithi153.stdout:8/574: fsync d7/d13/d19/fb5 0 2024-04-05T03:02:57.016 INFO:tasks.workunit.client.0.smithi153.stdout:6/786: dwrite d4/dc/df/d16/d5f/d73/ff7 [0,4194304] 0 2024-04-05T03:02:57.019 INFO:tasks.workunit.client.0.smithi153.stdout:8/575: readlink d7/d2a/l40 0 2024-04-05T03:02:57.023 INFO:tasks.workunit.client.0.smithi153.stdout:8/576: truncate d7/d13/d19/d1b/d4f/d5f/fb2 1004215 0 2024-04-05T03:02:57.023 INFO:tasks.workunit.client.0.smithi153.stdout:1/668: mknod d5/d8/d20/d4b/d18/d16/d2c/cc7 0 2024-04-05T03:02:57.025 INFO:tasks.workunit.client.0.smithi153.stdout:6/787: mkdir d4/dc/df/d16/d36/d10e 0 2024-04-05T03:02:57.025 INFO:tasks.workunit.client.1.smithi203.stdout:0/990: sync 2024-04-05T03:02:57.026 INFO:tasks.workunit.client.0.smithi153.stdout:8/577: fdatasync d7/d2a/f98 0 2024-04-05T03:02:57.026 INFO:tasks.workunit.client.1.smithi203.stdout:3/875: sync 2024-04-05T03:02:57.027 INFO:tasks.workunit.client.1.smithi203.stdout:0/991: rmdir dd/d17/d115/d11f 39 2024-04-05T03:02:57.027 INFO:tasks.workunit.client.1.smithi203.stdout:3/876: dread - d0/dc/d38/d5d/d66/d80/fe7 zero size 2024-04-05T03:02:57.030 INFO:tasks.workunit.client.1.smithi203.stdout:0/992: unlink dd/c10 0 2024-04-05T03:02:57.031 INFO:tasks.workunit.client.0.smithi153.stdout:6/788: mkdir d4/d38/d10f 0 2024-04-05T03:02:57.033 INFO:tasks.workunit.client.0.smithi153.stdout:8/578: creat d7/d9a/fbd x:0 0 0 2024-04-05T03:02:57.034 INFO:tasks.workunit.client.1.smithi203.stdout:3/877: mknod d0/d21/c104 0 2024-04-05T03:02:57.036 INFO:tasks.workunit.client.1.smithi203.stdout:0/993: mkdir dd/d17/d1d/d22/d45/d51/d129 0 2024-04-05T03:02:57.039 INFO:tasks.workunit.client.0.smithi153.stdout:8/579: creat d7/d13/d19/d1b/d4f/d68/dbc/fbe x:0 0 0 2024-04-05T03:02:57.039 INFO:tasks.workunit.client.1.smithi203.stdout:4/917: write da/f22 [1858789,79188] 0 2024-04-05T03:02:57.040 INFO:tasks.workunit.client.0.smithi153.stdout:8/580: read d7/d2a/fa2 [3402635,72702] 0 2024-04-05T03:02:57.042 INFO:tasks.workunit.client.1.smithi203.stdout:0/994: rename dd/d17/d1d/d22/l111 to dd/d17/d1d/d22/d45/d51/d129/l12a 0 2024-04-05T03:02:57.042 INFO:tasks.workunit.client.1.smithi203.stdout:0/995: chown dd/d17/dd8/d56 915153345 1 2024-04-05T03:02:57.043 INFO:tasks.workunit.client.1.smithi203.stdout:4/918: dread - da/d29/d2e/d57/d96/ddc/de1/ff8 zero size 2024-04-05T03:02:57.045 INFO:tasks.workunit.client.1.smithi203.stdout:0/996: mknod dd/db1/c12b 0 2024-04-05T03:02:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:56 smithi203 ceph-mon[39911]: pgmap v13: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 99 MiB/s rd, 276 MiB/s wr, 511 op/s 2024-04-05T03:02:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:56 smithi203 ceph-mon[39911]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:57.051 INFO:tasks.workunit.client.0.smithi153.stdout:6/789: rename d4/dc/d7a/dd2 to d4/dc/df/d16/d36/d4d/d64/d110 0 2024-04-05T03:02:57.052 INFO:tasks.workunit.client.0.smithi153.stdout:1/669: truncate d5/d8/d20/d4b/d18/d16/d33/f3e 2998841 0 2024-04-05T03:02:57.053 INFO:tasks.workunit.client.1.smithi203.stdout:3/878: dwrite d0/dc/d1a/d1f/f32 [4194304,4194304] 0 2024-04-05T03:02:57.057 INFO:tasks.workunit.client.0.smithi153.stdout:8/581: write d7/d13/d19/d1b/d6d/f7a [144737,17493] 0 2024-04-05T03:02:57.060 INFO:tasks.workunit.client.0.smithi153.stdout:6/790: dwrite d4/dc/f46 [0,4194304] 0 2024-04-05T03:02:57.061 INFO:tasks.workunit.client.1.smithi203.stdout:4/919: rename da/d29/d2e/db0/dbe/dce/dd0/f120 to da/d29/d2e/d57/d96/ddc/f131 0 2024-04-05T03:02:57.062 INFO:tasks.workunit.client.0.smithi153.stdout:1/670: getdents d5/d8/d20/d4b/d18/d16/d45/d68 0 2024-04-05T03:02:57.063 INFO:tasks.workunit.client.0.smithi153.stdout:1/671: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:57.064 INFO:tasks.workunit.client.1.smithi203.stdout:3/879: truncate d0/dc/d1a/d8d/d96/f90 2922371 0 2024-04-05T03:02:57.064 INFO:tasks.workunit.client.1.smithi203.stdout:3/880: chown d0/dc/d1a 5 1 2024-04-05T03:02:57.066 INFO:tasks.workunit.client.1.smithi203.stdout:0/997: creat dd/d17/d115/d11f/d6d/f12c x:0 0 0 2024-04-05T03:02:57.068 INFO:tasks.workunit.client.1.smithi203.stdout:3/881: mknod d0/dc/d1a/d1d/d53/d5a/c105 0 2024-04-05T03:02:57.069 INFO:tasks.workunit.client.0.smithi153.stdout:6/791: rename d4/dc/df/d16/d17/le7 to d4/dc/df/d16/d36/d4d/d64/d110/l111 0 2024-04-05T03:02:57.071 INFO:tasks.workunit.client.1.smithi203.stdout:0/998: mkdir dd/d17/d1d/d22/d53/db0/d12d 0 2024-04-05T03:02:57.072 INFO:tasks.workunit.client.1.smithi203.stdout:0/999: readlink dd/d17/d1d/d22/d53/d7f/ld9 0 2024-04-05T03:02:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:56 smithi153 ceph-mon[29381]: pgmap v13: 65 pgs: 65 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 99 MiB/s rd, 276 MiB/s wr, 511 op/s 2024-04-05T03:02:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:56 smithi153 ceph-mon[29381]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:57.072 INFO:tasks.workunit.client.0.smithi153.stdout:1/672: dwrite d5/d8/d20/d4b/d18/d16/d33/f3c [0,4194304] 0 2024-04-05T03:02:57.075 INFO:tasks.workunit.client.0.smithi153.stdout:6/792: rename d4/dc/df/c1c to d4/dc/df/d16/d36/d4d/d64/d110/c112 0 2024-04-05T03:02:57.078 INFO:tasks.workunit.client.0.smithi153.stdout:1/673: unlink d5/d8/d20/d2f/f60 0 2024-04-05T03:02:57.080 INFO:tasks.workunit.client.0.smithi153.stdout:1/674: write d5/d8/d20/d4b/f79 [503724,59011] 0 2024-04-05T03:02:57.080 INFO:tasks.workunit.client.1.smithi203.stdout:4/920: dwrite da/f27 [4194304,4194304] 0 2024-04-05T03:02:57.085 INFO:tasks.workunit.client.1.smithi203.stdout:3/882: dwrite d0/d21/fe4 [0,4194304] 0 2024-04-05T03:02:57.087 INFO:tasks.workunit.client.0.smithi153.stdout:1/675: link d5/dd/d1b/f74 d5/d8/d20/d4b/d18/d16/d33/d83/fc8 0 2024-04-05T03:02:57.087 INFO:tasks.workunit.client.1.smithi203.stdout:3/883: truncate d0/d21/d83/ffc 1024488 0 2024-04-05T03:02:57.097 INFO:tasks.workunit.client.1.smithi203.stdout:3/884: write d0/dc/d38/d5d/d66/f93 [803520,98905] 0 2024-04-05T03:02:57.099 INFO:tasks.workunit.client.1.smithi203.stdout:3/885: write d0/d41/f9a [1740978,86045] 0 2024-04-05T03:02:57.100 INFO:tasks.workunit.client.0.smithi153.stdout:6/793: write d4/dc/df/d16/f23 [2047777,38758] 0 2024-04-05T03:02:57.101 INFO:tasks.workunit.client.0.smithi153.stdout:6/794: read d4/dc/df/d16/d5f/d73/ff7 [3809968,77183] 0 2024-04-05T03:02:57.102 INFO:tasks.workunit.client.1.smithi203.stdout:4/921: dwrite da/d29/d2e/d53/d78/fdd [0,4194304] 0 2024-04-05T03:02:57.104 INFO:tasks.workunit.client.1.smithi203.stdout:3/886: getdents d0/dc/d1a/d1f/da2 0 2024-04-05T03:02:57.105 INFO:tasks.workunit.client.0.smithi153.stdout:6/795: rmdir d4/dc/df/d16/d5f/d73 39 2024-04-05T03:02:57.109 INFO:tasks.workunit.client.0.smithi153.stdout:6/796: fdatasync d4/dc/df/d16/d36/d4d/f66 0 2024-04-05T03:02:57.109 INFO:tasks.workunit.client.1.smithi203.stdout:4/922: creat da/d29/d2e/d8b/f132 x:0 0 0 2024-04-05T03:02:57.109 INFO:tasks.workunit.client.1.smithi203.stdout:4/923: fdatasync da/d29/d2e/db0/f117 0 2024-04-05T03:02:57.112 INFO:tasks.workunit.client.1.smithi203.stdout:3/887: link d0/dc/d1a/d8d/dc9/le8 d0/dc/d38/dca/l106 0 2024-04-05T03:02:57.113 INFO:tasks.workunit.client.0.smithi153.stdout:6/797: mkdir d4/dc/df/d16/d113 0 2024-04-05T03:02:57.116 INFO:tasks.workunit.client.1.smithi203.stdout:3/888: creat d0/dc/d1a/d76/d35/f107 x:0 0 0 2024-04-05T03:02:57.116 INFO:tasks.workunit.client.1.smithi203.stdout:4/924: dwrite da/d29/d2e/d57/f12c [0,4194304] 0 2024-04-05T03:02:57.117 INFO:tasks.workunit.client.0.smithi153.stdout:1/676: dread d5/dd/f9f [4194304,4194304] 0 2024-04-05T03:02:57.119 INFO:tasks.workunit.client.0.smithi153.stdout:1/677: mknod d5/d8/cc9 0 2024-04-05T03:02:57.120 INFO:tasks.workunit.client.1.smithi203.stdout:3/889: write d0/dc/d38/f8f [666319,35047] 0 2024-04-05T03:02:57.120 INFO:tasks.workunit.client.0.smithi153.stdout:6/798: dwrite d4/dc/ffd [0,4194304] 0 2024-04-05T03:02:57.122 INFO:tasks.workunit.client.0.smithi153.stdout:1/678: write d5/d8/f9d [204886,70488] 0 2024-04-05T03:02:57.124 INFO:tasks.workunit.client.0.smithi153.stdout:6/799: dread d4/dc/df/d16/d44/fa1 [0,4194304] 0 2024-04-05T03:02:57.126 INFO:tasks.workunit.client.0.smithi153.stdout:1/679: write d5/d8/d20/d4b/d18/d16/d33/d83/f8e [352164,108200] 0 2024-04-05T03:02:57.129 INFO:tasks.workunit.client.0.smithi153.stdout:1/680: fdatasync d5/d8/d20/d4b/d18/d16/f22 0 2024-04-05T03:02:57.129 INFO:tasks.workunit.client.0.smithi153.stdout:6/800: creat d4/dc/df/d16/d36/d4d/f114 x:0 0 0 2024-04-05T03:02:57.132 INFO:tasks.workunit.client.0.smithi153.stdout:6/801: fdatasync d4/dc/df/d16/d17/d54/faa 0 2024-04-05T03:02:57.132 INFO:tasks.workunit.client.0.smithi153.stdout:6/802: dread - d4/dc/df/d16/d44/ded/fff zero size 2024-04-05T03:02:57.135 INFO:tasks.workunit.client.0.smithi153.stdout:1/681: read d5/d8/f9d [394186,96324] 0 2024-04-05T03:02:57.136 INFO:tasks.workunit.client.0.smithi153.stdout:6/803: truncate d4/dc/df/f13 4150930 0 2024-04-05T03:02:57.136 INFO:tasks.workunit.client.0.smithi153.stdout:1/682: truncate d5/dd/f5d 1221422 0 2024-04-05T03:02:57.137 INFO:tasks.workunit.client.0.smithi153.stdout:1/683: readlink d5/dd/d1b/l29 0 2024-04-05T03:02:57.138 INFO:tasks.workunit.client.0.smithi153.stdout:6/804: symlink d4/dc/df/d16/d5f/da2/l115 0 2024-04-05T03:02:57.139 INFO:tasks.workunit.client.0.smithi153.stdout:1/684: symlink d5/d8/d20/d4b/d18/d16/d45/d68/da4/da8/lca 0 2024-04-05T03:02:57.141 INFO:tasks.workunit.client.0.smithi153.stdout:1/685: chown d5/d8/d20/f38 22 1 2024-04-05T03:02:57.145 INFO:tasks.workunit.client.1.smithi203.stdout:4/925: dread da/d3c/f43 [0,4194304] 0 2024-04-05T03:02:57.147 INFO:tasks.workunit.client.0.smithi153.stdout:6/805: dwrite d4/dc/df/f26 [0,4194304] 0 2024-04-05T03:02:57.147 INFO:tasks.workunit.client.0.smithi153.stdout:1/686: read d5/d8/d20/d4b/d18/d16/d48/d96/fa0 [668841,52522] 0 2024-04-05T03:02:57.147 INFO:tasks.workunit.client.0.smithi153.stdout:1/687: chown d5/d8/d20/d4b/d18/d16/d48 29159458 1 2024-04-05T03:02:57.147 INFO:tasks.workunit.client.0.smithi153.stdout:6/806: fdatasync d4/dc/df/d16/d44/fa1 0 2024-04-05T03:02:57.149 INFO:tasks.workunit.client.0.smithi153.stdout:6/807: chown d4/dc/df/d16/d24/d5e/de8 83584560 1 2024-04-05T03:02:57.150 INFO:tasks.workunit.client.1.smithi203.stdout:4/926: dread da/d3c/d4c/d99/fc1 [0,4194304] 0 2024-04-05T03:02:57.150 INFO:tasks.workunit.client.0.smithi153.stdout:6/808: chown d4/dc/df/d16/d24/d8d 457 1 2024-04-05T03:02:57.150 INFO:tasks.workunit.client.0.smithi153.stdout:6/809: stat d4/dc 0 2024-04-05T03:02:57.151 INFO:tasks.workunit.client.0.smithi153.stdout:6/810: fdatasync d4/dc/ffd 0 2024-04-05T03:02:57.152 INFO:tasks.workunit.client.0.smithi153.stdout:1/688: creat d5/d8/d20/d4b/d18/d16/d2c/d62/fcb x:0 0 0 2024-04-05T03:02:57.153 INFO:tasks.workunit.client.1.smithi203.stdout:4/927: mkdir da/d29/d133 0 2024-04-05T03:02:57.155 INFO:tasks.workunit.client.1.smithi203.stdout:4/928: creat da/d71/f134 x:0 0 0 2024-04-05T03:02:57.156 INFO:tasks.workunit.client.1.smithi203.stdout:4/929: dread - da/d3c/d4c/d99/f129 zero size 2024-04-05T03:02:57.157 INFO:tasks.workunit.client.0.smithi153.stdout:1/689: creat d5/d8/d20/d4b/d18/d16/d45/d93/fcc x:0 0 0 2024-04-05T03:02:57.158 INFO:tasks.workunit.client.0.smithi153.stdout:6/811: truncate d4/d38/f75 2292075 0 2024-04-05T03:02:57.159 INFO:tasks.workunit.client.0.smithi153.stdout:1/690: readlink d5/d8/d20/d2f/d40/l46 0 2024-04-05T03:02:57.160 INFO:tasks.workunit.client.0.smithi153.stdout:6/812: creat d4/dc/df/dcb/f116 x:0 0 0 2024-04-05T03:02:57.162 INFO:tasks.workunit.client.1.smithi203.stdout:4/930: dwrite da/f39 [4194304,4194304] 0 2024-04-05T03:02:57.164 INFO:tasks.workunit.client.0.smithi153.stdout:6/813: unlink d4/dc/df/d16/d36/d4d/dad/lbc 0 2024-04-05T03:02:57.166 INFO:tasks.workunit.client.0.smithi153.stdout:6/814: mkdir d4/dc/df/d117 0 2024-04-05T03:02:57.167 INFO:tasks.workunit.client.1.smithi203.stdout:4/931: getdents da/d29/d2e/d57/d96/ddc/de1 0 2024-04-05T03:02:57.173 INFO:tasks.workunit.client.0.smithi153.stdout:6/815: dread d4/d38/d4c/f6a [0,4194304] 0 2024-04-05T03:02:57.175 INFO:tasks.workunit.client.0.smithi153.stdout:1/691: dread d5/d8/d20/d4b/d18/d16/f22 [0,4194304] 0 2024-04-05T03:02:57.176 INFO:tasks.workunit.client.0.smithi153.stdout:1/692: fsync d5/d8/d20/d4b/d18/d16/d48/f99 0 2024-04-05T03:02:57.178 INFO:tasks.workunit.client.0.smithi153.stdout:1/693: creat d5/d8/d20/d4b/d18/d16/d45/d68/da4/dc3/fcd x:0 0 0 2024-04-05T03:02:57.180 INFO:tasks.workunit.client.0.smithi153.stdout:1/694: mknod d5/cce 0 2024-04-05T03:02:57.186 INFO:tasks.workunit.client.1.smithi203.stdout:4/932: dread da/d29/d2e/db0/dbe/dce/fef [0,4194304] 0 2024-04-05T03:02:57.188 INFO:tasks.workunit.client.1.smithi203.stdout:4/933: symlink da/d29/d2e/db0/dbe/dce/l135 0 2024-04-05T03:02:57.189 INFO:tasks.workunit.client.1.smithi203.stdout:4/934: stat da/d3c/d4c/d99 0 2024-04-05T03:02:57.191 INFO:tasks.workunit.client.1.smithi203.stdout:4/935: creat da/d29/d2e/db0/dbe/dce/f136 x:0 0 0 2024-04-05T03:02:57.193 INFO:tasks.workunit.client.1.smithi203.stdout:4/936: unlink da/d29/d2e/f3e 0 2024-04-05T03:02:57.194 INFO:tasks.workunit.client.0.smithi153.stdout:6/816: dread d4/dc/f97 [0,4194304] 0 2024-04-05T03:02:57.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.196+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efd4c10f7c0 con 0x7efd3406b540 2024-04-05T03:02:57.197 INFO:tasks.workunit.client.1.smithi203.stdout:4/937: rename da/d29/d2e/db0/lc3 to da/d29/d2e/db0/dcb/l137 0 2024-04-05T03:02:57.197 INFO:tasks.workunit.client.0.smithi153.stdout:6/817: mkdir d4/dc/df/d16/d36/d4d/d64/d110/df2/d118 0 2024-04-05T03:02:57.198 INFO:tasks.workunit.client.0.smithi153.stdout:1/695: dread d5/d8/d20/f88 [4194304,4194304] 0 2024-04-05T03:02:57.199 INFO:tasks.workunit.client.0.smithi153.stdout:6/818: mkdir d4/dc/df/dcb/d119 0 2024-04-05T03:02:57.199 INFO:tasks.workunit.client.1.smithi203.stdout:4/938: mkdir da/d29/d2e/d57/d96/dbc/df5/d138 0 2024-04-05T03:02:57.199 INFO:tasks.workunit.client.1.smithi203.stdout:4/939: fsync da/d3c/d4c/d99/f108 0 2024-04-05T03:02:57.200 INFO:tasks.workunit.client.0.smithi153.stdout:6/819: chown d4/dc/df/d16/c10b 3 1 2024-04-05T03:02:57.205 INFO:tasks.workunit.client.0.smithi153.stdout:1/696: dwrite d5/d8/d20/d4b/d1e/f8f [0,4194304] 0 2024-04-05T03:02:57.208 INFO:tasks.workunit.client.1.smithi203.stdout:4/940: write da/f74 [1369187,57898] 0 2024-04-05T03:02:57.208 INFO:tasks.workunit.client.0.smithi153.stdout:1/697: mknod d5/d8/d20/d2f/db8/ccf 0 2024-04-05T03:02:57.209 INFO:tasks.workunit.client.1.smithi203.stdout:4/941: read - da/fed zero size 2024-04-05T03:02:57.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.209+0000 7efd337fe700 1 -- 172.21.15.153:0/362667951 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7efd4c10f7c0 con 0x7efd3406b540 2024-04-05T03:02:57.209 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:02:57.209 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 2m ago 8m 26.6M - 0.25.0 c8568f914cd2 db2172977a37 2024-04-05T03:02:57.209 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (9m) 2m ago 9m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (7m) 37s ago 7m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 2m ago 8m 88.5M - 9.4.7 954c08fa6188 0fafa83f96b9 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (3m) 2m ago 3m 14.5M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (3m) 2m ago 3m 11.1M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (3m) 37s ago 3m 44.8M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (3m) 37s ago 3m 70.6M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (10m) 2m ago 10m 479M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (56s) 37s ago 7m 82.5M - 19.0.0-2013-g0028525c d8c35eaf3c25 f4c2fc4aeae0 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (11m) 2m ago 11m 43.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (7m) 37s ago 7m 39.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 2m ago 8m 15.8M - 1.5.0 0da6a335fe13 61d64f55ec69 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (7m) 37s ago 7m 16.0M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 2m ago 6m 46.7M 4629M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:02:57.210 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 2m ago 6m 46.5M 4629M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:02:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 2m ago 5m 44.9M 4629M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:02:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (5m) 37s ago 5m 94.2M 4970M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:02:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (5m) 37s ago 5m 93.6M 4970M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:02:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (4m) 37s ago 4m 86.5M 4970M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:02:57.211 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (7m) 37s ago 7m 58.9M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 msgr2=0x7efd3406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 0x7efd3406d9f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd40009b80 tx=0x7efd4000b040 comp rx=0 tx=0).stop 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 msgr2=0x7efd4c10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00b0d0 tx=0x7efd3c009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 shutdown_connections 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd4c0fbe30 0x7efd4c10d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efd3406b540 0x7efd3406d9f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 --2- 172.21.15.153:0/362667951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd4c0fcfa0 0x7efd4c10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 >> 172.21.15.153:0/362667951 conn(0x7efd4c0f7480 msgr2=0x7efd4c0f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 shutdown_connections 2024-04-05T03:02:57.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.212+0000 7efd515ba700 1 -- 172.21.15.153:0/362667951 wait complete. 2024-04-05T03:02:57.214 INFO:tasks.workunit.client.0.smithi153.stdout:1/698: getdents d5/d8/d20/d4b/d18/d16/d45 0 2024-04-05T03:02:57.214 INFO:tasks.workunit.client.1.smithi203.stdout:4/942: getdents da/d29/d2e/d8b/d109 0 2024-04-05T03:02:57.215 INFO:tasks.workunit.client.1.smithi203.stdout:4/943: chown da/d3c/f43 53127537 1 2024-04-05T03:02:57.215 INFO:tasks.workunit.client.0.smithi153.stdout:1/699: write d5/d8/d20/d2f/db8/fc4 [184339,119418] 0 2024-04-05T03:02:57.217 INFO:tasks.workunit.client.1.smithi203.stdout:4/944: rmdir da/d29/d2e/d53/df9 39 2024-04-05T03:02:57.220 INFO:tasks.workunit.client.1.smithi203.stdout:4/945: unlink da/d29/d2e/db0/dbe/dce/dd0/fe7 0 2024-04-05T03:02:57.223 INFO:tasks.workunit.client.1.smithi203.stdout:4/946: rename da/d29/f47 to da/d29/d2e/db0/f139 0 2024-04-05T03:02:57.225 INFO:tasks.workunit.client.1.smithi203.stdout:4/947: mkdir da/d29/d2e/d57/d13a 0 2024-04-05T03:02:57.226 INFO:tasks.workunit.client.1.smithi203.stdout:4/948: mknod da/d3c/d4c/d99/c13b 0 2024-04-05T03:02:57.229 INFO:tasks.workunit.client.1.smithi203.stdout:4/949: dread da/f10 [0,4194304] 0 2024-04-05T03:02:57.231 INFO:tasks.workunit.client.1.smithi203.stdout:4/950: creat da/d29/d2e/d57/da7/f13c x:0 0 0 2024-04-05T03:02:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.315+0000 7f05b1ba1700 1 -- 172.21.15.153:0/993643637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 msgr2=0x7f05ac0fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/993643637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac0fe450 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0594004940 tx=0x7f0594009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- 172.21.15.153:0/993643637 shutdown_connections 2024-04-05T03:02:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/993643637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac0fe450 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/993643637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 0x7f05ac0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- 172.21.15.153:0/993643637 >> 172.21.15.153:0/993643637 conn(0x7f05ac0f8550 msgr2=0x7f05ac0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- 172.21.15.153:0/993643637 shutdown_connections 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- 172.21.15.153:0/993643637 wait complete. 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 Processor -- start 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- start start 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 0x7f05ac10d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac10e3d0 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.316+0000 7f05b1ba1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac10e540 con 0x7f05ac0fce70 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39798/0 (socket says 172.21.15.153:39798) 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 -- 172.21.15.153:0/1512693990 learned_addr learned my addr 172.21.15.153:0/1512693990 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:57.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05ab7fe700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 0x7f05ac10d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 -- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 msgr2=0x7f05ac10d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 0x7f05ac10d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 -- 172.21.15.153:0/1512693990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594005040 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05aaffd700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0594013870 tx=0x7f0594009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0594021070 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0594009430 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac112f40 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059400e070 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.317+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac113450 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.318+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac04c710 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.319+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f059401c750 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.319+0000 7f05a3fff700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 0x7f059806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.319+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f059408b260 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.319+0000 7f05ab7fe700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 0x7f059806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.320+0000 7f05ab7fe700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 0x7f059806d990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f059c009b80 tx=0x7f059c00b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:57.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.324+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f0594058c50 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.350 INFO:tasks.workunit.client.0.smithi153.stdout:3/674: sync 2024-04-05T03:02:57.353 INFO:tasks.workunit.client.0.smithi153.stdout:3/675: unlink d1d/f1e 0 2024-04-05T03:02:57.353 INFO:tasks.workunit.client.0.smithi153.stdout:3/676: dread - d1d/d75/fe0 zero size 2024-04-05T03:02:57.357 INFO:tasks.workunit.client.0.smithi153.stdout:3/677: rename d1d/d27/d2b/f35 to d1d/d32/d93/d3c/d3f/d7c/fef 0 2024-04-05T03:02:57.358 INFO:tasks.workunit.client.0.smithi153.stdout:3/678: fdatasync d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fcb 0 2024-04-05T03:02:57.367 INFO:tasks.workunit.client.0.smithi153.stdout:3/679: dwrite d1d/fa7 [0,4194304] 0 2024-04-05T03:02:57.369 INFO:tasks.workunit.client.0.smithi153.stdout:3/680: chown d1d/d32/d93/d3c/d3f/f4f 64573383 1 2024-04-05T03:02:57.369 INFO:tasks.workunit.client.0.smithi153.stdout:3/681: readlink lf 0 2024-04-05T03:02:57.373 INFO:tasks.workunit.client.0.smithi153.stdout:3/682: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 x:0 0 0 2024-04-05T03:02:57.378 INFO:tasks.workunit.client.0.smithi153.stdout:3/683: rmdir d1d/d32/d93 39 2024-04-05T03:02:57.382 INFO:tasks.workunit.client.0.smithi153.stdout:3/684: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f81 [5359879,72697] 0 2024-04-05T03:02:57.385 INFO:tasks.workunit.client.0.smithi153.stdout:3/685: write d1d/d27/f74 [891003,84432] 0 2024-04-05T03:02:57.388 INFO:tasks.workunit.client.0.smithi153.stdout:3/686: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/ff1 x:0 0 0 2024-04-05T03:02:57.389 INFO:tasks.workunit.client.0.smithi153.stdout:3/687: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f67 [3860451,45173] 0 2024-04-05T03:02:57.391 INFO:tasks.workunit.client.0.smithi153.stdout:3/688: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fcb [59274,38274] 0 2024-04-05T03:02:57.392 INFO:tasks.workunit.client.0.smithi153.stdout:3/689: stat d1d/d32/d93/d3c/d3f/d90/db6/fb7 0 2024-04-05T03:02:57.394 INFO:tasks.workunit.client.0.smithi153.stdout:3/690: readlink d1d/d32/d93/d3c/d3f/d9c/la2 0 2024-04-05T03:02:57.399 INFO:tasks.workunit.client.0.smithi153.stdout:3/691: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fae [0,4194304] 0 2024-04-05T03:02:57.402 INFO:tasks.workunit.client.0.smithi153.stdout:3/692: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff2 x:0 0 0 2024-04-05T03:02:57.411 INFO:tasks.workunit.client.0.smithi153.stdout:3/693: dwrite d1d/d32/d93/d3c/d3f/d90/d92/fac [4194304,4194304] 0 2024-04-05T03:02:57.413 INFO:tasks.workunit.client.0.smithi153.stdout:3/694: symlink d1d/d27/lf3 0 2024-04-05T03:02:57.415 INFO:tasks.workunit.client.0.smithi153.stdout:3/695: write d1d/d32/d93/d3c/d3f/d90/da4/fb0 [1038726,79581] 0 2024-04-05T03:02:57.427 INFO:tasks.workunit.client.1.smithi203.stdout:3/890: sync 2024-04-05T03:02:57.427 INFO:tasks.workunit.client.1.smithi203.stdout:4/951: sync 2024-04-05T03:02:57.429 INFO:tasks.workunit.client.1.smithi203.stdout:3/891: chown d0/dc/d1a/d1f/da1/daf/le9 302 1 2024-04-05T03:02:57.429 INFO:tasks.workunit.client.1.smithi203.stdout:4/952: symlink da/d29/d2e/db0/dbe/dce/dd0/l13d 0 2024-04-05T03:02:57.432 INFO:tasks.workunit.client.1.smithi203.stdout:4/953: creat da/d29/d2e/d57/d96/dbc/df5/f13e x:0 0 0 2024-04-05T03:02:57.432 INFO:tasks.workunit.client.1.smithi203.stdout:4/954: write da/f4a [7145637,90749] 0 2024-04-05T03:02:57.434 INFO:tasks.workunit.client.1.smithi203.stdout:4/955: read - da/d29/d2e/d57/d96/dbc/df5/f13e zero size 2024-04-05T03:02:57.438 INFO:tasks.workunit.client.1.smithi203.stdout:4/956: dwrite da/f9a [0,4194304] 0 2024-04-05T03:02:57.439 INFO:tasks.workunit.client.1.smithi203.stdout:3/892: truncate d0/dc/d1a/d1d/d53/f73 1778660 0 2024-04-05T03:02:57.439 INFO:tasks.workunit.client.1.smithi203.stdout:4/957: chown da/d3c/d4c/d99 1748 1 2024-04-05T03:02:57.439 INFO:tasks.workunit.client.1.smithi203.stdout:3/893: chown d0/l1b 0 1 2024-04-05T03:02:57.440 INFO:tasks.workunit.client.1.smithi203.stdout:3/894: write d0/dc/ff3 [3755604,17360] 0 2024-04-05T03:02:57.442 INFO:tasks.workunit.client.1.smithi203.stdout:4/958: creat da/d29/d2e/db0/dbe/f13f x:0 0 0 2024-04-05T03:02:57.444 INFO:tasks.workunit.client.1.smithi203.stdout:3/895: dread d0/dc/d38/fc2 [0,4194304] 0 2024-04-05T03:02:57.445 INFO:tasks.workunit.client.1.smithi203.stdout:4/959: unlink da/f28 0 2024-04-05T03:02:57.448 INFO:tasks.workunit.client.1.smithi203.stdout:3/896: truncate d0/dc/d38/fc2 1394689 0 2024-04-05T03:02:57.452 INFO:tasks.workunit.client.1.smithi203.stdout:3/897: symlink d0/dc/d38/l108 0 2024-04-05T03:02:57.456 INFO:tasks.workunit.client.1.smithi203.stdout:3/898: mknod d0/dc/d1a/d76/c109 0 2024-04-05T03:02:57.458 INFO:tasks.workunit.client.1.smithi203.stdout:3/899: creat d0/d21/d81/f10a x:0 0 0 2024-04-05T03:02:57.459 INFO:tasks.workunit.client.1.smithi203.stdout:3/900: readlink d0/d21/d83/l95 0 2024-04-05T03:02:57.459 INFO:tasks.workunit.client.1.smithi203.stdout:4/960: truncate da/d29/d2e/db0/dcb/d80/fb9 1277954 0 2024-04-05T03:02:57.459 INFO:tasks.workunit.client.1.smithi203.stdout:3/901: readlink d0/dc/d38/d5d/la0 0 2024-04-05T03:02:57.460 INFO:tasks.workunit.client.1.smithi203.stdout:3/902: write d0/dc/d1a/d1f/f32 [5080129,45959] 0 2024-04-05T03:02:57.462 INFO:tasks.workunit.client.1.smithi203.stdout:3/903: symlink d0/d41/l10b 0 2024-04-05T03:02:57.462 INFO:tasks.workunit.client.1.smithi203.stdout:4/961: dread da/fdf [4194304,4194304] 0 2024-04-05T03:02:57.464 INFO:tasks.workunit.client.1.smithi203.stdout:3/904: mkdir d0/dc/d1a/d1f/da1/d10c 0 2024-04-05T03:02:57.466 INFO:tasks.workunit.client.1.smithi203.stdout:4/962: dread f9 [8388608,4194304] 0 2024-04-05T03:02:57.467 INFO:tasks.workunit.client.1.smithi203.stdout:3/905: rename d0/dc/d1a/d1f/da1/daf/ddc/ce6 to d0/dc/d1a/d1d/d53/d5a/c10d 0 2024-04-05T03:02:57.469 INFO:tasks.workunit.client.1.smithi203.stdout:4/963: getdents da/d3c/d4c/d99 0 2024-04-05T03:02:57.470 INFO:tasks.workunit.client.1.smithi203.stdout:4/964: read - da/d29/d2e/fe4 zero size 2024-04-05T03:02:57.470 INFO:tasks.workunit.client.1.smithi203.stdout:3/906: rename d0/dc/d1a/d1f/da2/fd0 to d0/dc/d1a/d1d/d53/d5a/f10e 0 2024-04-05T03:02:57.471 INFO:tasks.workunit.client.1.smithi203.stdout:4/965: mknod da/d29/d2e/d57/d96/ddc/de0/c140 0 2024-04-05T03:02:57.472 INFO:tasks.workunit.client.1.smithi203.stdout:3/907: mkdir d0/d69/d10f 0 2024-04-05T03:02:57.472 INFO:tasks.workunit.client.1.smithi203.stdout:4/966: readlink da/d29/d2e/d53/df9/lfb 0 2024-04-05T03:02:57.481 INFO:tasks.workunit.client.1.smithi203.stdout:3/908: write d0/dc/d38/d5d/f8b [1081986,43319] 0 2024-04-05T03:02:57.482 INFO:tasks.workunit.client.1.smithi203.stdout:3/909: stat d0/d69/f88 0 2024-04-05T03:02:57.485 INFO:tasks.workunit.client.1.smithi203.stdout:3/910: creat d0/d21/f110 x:0 0 0 2024-04-05T03:02:57.486 INFO:tasks.workunit.client.1.smithi203.stdout:3/911: readlink d0/dc/d1a/l87 0 2024-04-05T03:02:57.487 INFO:tasks.workunit.client.1.smithi203.stdout:4/967: dwrite da/d29/d2e/d53/f6f [0,4194304] 0 2024-04-05T03:02:57.488 INFO:tasks.workunit.client.1.smithi203.stdout:3/912: creat d0/dc/d34/f111 x:0 0 0 2024-04-05T03:02:57.490 INFO:tasks.workunit.client.1.smithi203.stdout:3/913: symlink d0/d21/l112 0 2024-04-05T03:02:57.492 INFO:tasks.workunit.client.0.smithi153.stdout:2/715: sync 2024-04-05T03:02:57.492 INFO:tasks.workunit.client.0.smithi153.stdout:0/634: sync 2024-04-05T03:02:57.493 INFO:tasks.workunit.client.1.smithi203.stdout:4/968: rename da/d29/d2e/d53/fa4 to da/d29/d2e/d53/f141 0 2024-04-05T03:02:57.493 INFO:tasks.workunit.client.1.smithi203.stdout:4/969: dread - da/d29/d2e/db0/dbe/fc5 zero size 2024-04-05T03:02:57.494 INFO:tasks.workunit.client.1.smithi203.stdout:3/914: rename d0/dc/d38/d5d/f62 to d0/d69/d10f/f113 0 2024-04-05T03:02:57.495 INFO:tasks.workunit.client.0.smithi153.stdout:0/635: mknod d9/db/d42/d43/cc5 0 2024-04-05T03:02:57.495 INFO:tasks.workunit.client.1.smithi203.stdout:4/970: symlink da/d29/d2e/d57/da5/l142 0 2024-04-05T03:02:57.496 INFO:tasks.workunit.client.1.smithi203.stdout:4/971: stat da/f37 0 2024-04-05T03:02:57.496 INFO:tasks.workunit.client.0.smithi153.stdout:0/636: chown d9/d14/d67/ca3 19344 1 2024-04-05T03:02:57.496 INFO:tasks.workunit.client.0.smithi153.stdout:2/716: mknod d9/da/d19/d2e/d7e/d7a/d6d/cf4 0 2024-04-05T03:02:57.497 INFO:tasks.workunit.client.1.smithi203.stdout:3/915: mkdir d0/dc/d1a/d1f/dac/d114 0 2024-04-05T03:02:57.498 INFO:tasks.workunit.client.0.smithi153.stdout:0/637: readlink d9/db/d42/d7e/db6/lad 0 2024-04-05T03:02:57.499 INFO:tasks.workunit.client.1.smithi203.stdout:4/972: creat da/d29/d2e/d57/d96/ddc/de0/f143 x:0 0 0 2024-04-05T03:02:57.499 INFO:tasks.workunit.client.1.smithi203.stdout:3/916: fsync d0/dc/d1a/d76/d35/f3d 0 2024-04-05T03:02:57.500 INFO:tasks.workunit.client.0.smithi153.stdout:2/717: getdents d9/d68/d98 0 2024-04-05T03:02:57.502 INFO:tasks.workunit.client.0.smithi153.stdout:0/638: creat d9/db/d42/d7e/db6/d1e/d45/d62/d79/fc6 x:0 0 0 2024-04-05T03:02:57.503 INFO:tasks.workunit.client.0.smithi153.stdout:0/639: fdatasync d9/db/d42/d43/fa1 0 2024-04-05T03:02:57.503 INFO:tasks.workunit.client.0.smithi153.stdout:2/718: creat d9/da/d19/de2/ff5 x:0 0 0 2024-04-05T03:02:57.505 INFO:tasks.workunit.client.0.smithi153.stdout:2/719: mkdir d9/da/d19/df6 0 2024-04-05T03:02:57.506 INFO:tasks.workunit.client.1.smithi203.stdout:4/973: dwrite da/d71/f102 [0,4194304] 0 2024-04-05T03:02:57.508 INFO:tasks.workunit.client.1.smithi203.stdout:3/917: dwrite d0/dc/d1a/d1f/f9d [0,4194304] 0 2024-04-05T03:02:57.510 INFO:tasks.workunit.client.1.smithi203.stdout:3/918: dread - d0/dc/d34/f111 zero size 2024-04-05T03:02:57.515 INFO:tasks.workunit.client.1.smithi203.stdout:3/919: mkdir d0/dc/d1a/d1f/da1/d115 0 2024-04-05T03:02:57.516 INFO:tasks.workunit.client.1.smithi203.stdout:4/974: dwrite da/d3c/d4c/f87 [0,4194304] 0 2024-04-05T03:02:57.517 INFO:tasks.workunit.client.1.smithi203.stdout:3/920: stat d0/dc/d1a/d1d/cb3 0 2024-04-05T03:02:57.523 INFO:tasks.workunit.client.1.smithi203.stdout:4/975: dread da/d71/f102 [0,4194304] 0 2024-04-05T03:02:57.526 INFO:tasks.workunit.client.0.smithi153.stdout:2/720: dwrite d9/da/fb [0,4194304] 0 2024-04-05T03:02:57.527 INFO:tasks.workunit.client.1.smithi203.stdout:4/976: unlink da/d3c/d4c/d99/c13b 0 2024-04-05T03:02:57.528 INFO:tasks.workunit.client.1.smithi203.stdout:4/977: chown da/d29/d2e/d53/f113 5141371 1 2024-04-05T03:02:57.528 INFO:tasks.workunit.client.1.smithi203.stdout:4/978: fdatasync da/d29/f9e 0 2024-04-05T03:02:57.529 INFO:tasks.workunit.client.1.smithi203.stdout:4/979: chown da/f74 61720 1 2024-04-05T03:02:57.530 INFO:tasks.workunit.client.1.smithi203.stdout:4/980: read - da/d29/d2e/d57/d96/dbc/fbd zero size 2024-04-05T03:02:57.530 INFO:tasks.workunit.client.0.smithi153.stdout:0/640: dread d9/db/d42/d7e/db6/d1e/f50 [0,4194304] 0 2024-04-05T03:02:57.530 INFO:tasks.workunit.client.0.smithi153.stdout:2/721: truncate d9/da/d19/d2e/d7e/de6/f6b 719075 0 2024-04-05T03:02:57.532 INFO:tasks.workunit.client.0.smithi153.stdout:0/641: rmdir d9/d68 39 2024-04-05T03:02:57.534 INFO:tasks.workunit.client.0.smithi153.stdout:0/642: fsync d9/db/d42/d7e/db6/d1e/fa7 0 2024-04-05T03:02:57.535 INFO:tasks.workunit.client.0.smithi153.stdout:2/722: mkdir d9/da/d94/dca/df7 0 2024-04-05T03:02:57.535 INFO:tasks.workunit.client.0.smithi153.stdout:0/643: fdatasync d9/d14/d67/d6f/fab 0 2024-04-05T03:02:57.538 INFO:tasks.workunit.client.0.smithi153.stdout:0/644: write d9/db/d42/d7e/db6/d1d/d61/f2b [2813358,101817] 0 2024-04-05T03:02:57.541 INFO:tasks.workunit.client.0.smithi153.stdout:0/645: rmdir d9/db/d42/d7e/db6/d1e/d45/d62 39 2024-04-05T03:02:57.541 INFO:tasks.workunit.client.0.smithi153.stdout:2/723: dwrite d9/da/d19/d2e/d4f/fa6 [0,4194304] 0 2024-04-05T03:02:57.542 INFO:tasks.workunit.client.0.smithi153.stdout:2/724: write d9/da/d19/d2e/d7e/d7a/fdb [488879,1194] 0 2024-04-05T03:02:57.545 INFO:tasks.workunit.client.0.smithi153.stdout:0/646: creat d9/fc7 x:0 0 0 2024-04-05T03:02:57.546 INFO:tasks.workunit.client.0.smithi153.stdout:0/647: fdatasync f0 0 2024-04-05T03:02:57.547 INFO:tasks.workunit.client.0.smithi153.stdout:2/725: rename d9/da/d19/d2e/d4f/fa6 to d9/da/d94/dca/df7/ff8 0 2024-04-05T03:02:57.551 INFO:tasks.workunit.client.0.smithi153.stdout:5/590: sync 2024-04-05T03:02:57.551 INFO:tasks.workunit.client.0.smithi153.stdout:7/812: sync 2024-04-05T03:02:57.554 INFO:tasks.workunit.client.1.smithi203.stdout:3/921: write d0/dc/d1a/d1f/dac/fba [471606,79758] 0 2024-04-05T03:02:57.556 INFO:tasks.workunit.client.1.smithi203.stdout:3/922: dread - d0/dc/d1a/d1f/f102 zero size 2024-04-05T03:02:57.558 INFO:tasks.workunit.client.1.smithi203.stdout:3/923: mknod d0/d21/c116 0 2024-04-05T03:02:57.559 INFO:tasks.workunit.client.0.smithi153.stdout:5/591: readlink d0/d25/d4d/l7f 0 2024-04-05T03:02:57.560 INFO:tasks.workunit.client.1.smithi203.stdout:3/924: fsync d0/d41/f8e 0 2024-04-05T03:02:57.562 INFO:tasks.workunit.client.0.smithi153.stdout:0/648: write d9/db/fd [1260205,38573] 0 2024-04-05T03:02:57.562 INFO:tasks.workunit.client.0.smithi153.stdout:0/649: dread - d9/d14/d67/fc3 zero size 2024-04-05T03:02:57.563 INFO:tasks.workunit.client.0.smithi153.stdout:0/650: chown d9/db 31 1 2024-04-05T03:02:57.566 INFO:tasks.workunit.client.1.smithi203.stdout:3/925: dwrite d0/d41/f9a [0,4194304] 0 2024-04-05T03:02:57.572 INFO:tasks.workunit.client.1.smithi203.stdout:3/926: write d0/dc/d1a/d1d/f29 [1286433,104437] 0 2024-04-05T03:02:57.574 INFO:tasks.workunit.client.1.smithi203.stdout:3/927: truncate d0/dc/d1a/d1f/fd6 487728 0 2024-04-05T03:02:57.575 INFO:tasks.workunit.client.1.smithi203.stdout:3/928: write d0/dc/d38/dca/fd1 [1602956,104251] 0 2024-04-05T03:02:57.576 INFO:tasks.workunit.client.1.smithi203.stdout:3/929: truncate d0/fde 5983266 0 2024-04-05T03:02:57.582 INFO:tasks.workunit.client.0.smithi153.stdout:5/592: mknod d0/d25/d4d/d9c/dc7/cd1 0 2024-04-05T03:02:57.582 INFO:tasks.workunit.client.1.smithi203.stdout:3/930: dwrite d0/d21/d81/f10a [0,4194304] 0 2024-04-05T03:02:57.583 INFO:tasks.workunit.client.0.smithi153.stdout:0/651: write d9/db/d42/d7e/db6/d1e/d45/d62/f7d [2305538,129106] 0 2024-04-05T03:02:57.586 INFO:tasks.workunit.client.1.smithi203.stdout:3/931: mknod d0/dc/d1a/d1f/da1/c117 0 2024-04-05T03:02:57.587 INFO:tasks.workunit.client.0.smithi153.stdout:0/652: mknod d9/db/d42/d7e/db6/d1e/d84/cc8 0 2024-04-05T03:02:57.590 INFO:tasks.workunit.client.1.smithi203.stdout:3/932: dread d0/dc/d1a/d1d/f72 [0,4194304] 0 2024-04-05T03:02:57.590 INFO:tasks.workunit.client.0.smithi153.stdout:0/653: creat d9/d68/fc9 x:0 0 0 2024-04-05T03:02:57.592 INFO:tasks.workunit.client.1.smithi203.stdout:3/933: mkdir d0/dc/d38/d92/d118 0 2024-04-05T03:02:57.593 INFO:tasks.workunit.client.0.smithi153.stdout:0/654: symlink d9/db/d42/d7e/db6/d1e/d45/d62/d79/lca 0 2024-04-05T03:02:57.593 INFO:tasks.workunit.client.0.smithi153.stdout:0/655: chown d9/f1a 4552483 1 2024-04-05T03:02:57.594 INFO:tasks.workunit.client.1.smithi203.stdout:3/934: mkdir d0/d21/d119 0 2024-04-05T03:02:57.595 INFO:tasks.workunit.client.1.smithi203.stdout:3/935: truncate d0/d69/f88 4981168 0 2024-04-05T03:02:57.597 INFO:tasks.workunit.client.1.smithi203.stdout:3/936: symlink d0/dc/d38/d5d/d66/d80/l11a 0 2024-04-05T03:02:57.599 INFO:tasks.workunit.client.1.smithi203.stdout:3/937: creat d0/d69/d10f/f11b x:0 0 0 2024-04-05T03:02:57.600 INFO:tasks.workunit.client.1.smithi203.stdout:3/938: chown d0/dc/d1a/d1d/d53/f98 5727943 1 2024-04-05T03:02:57.601 INFO:tasks.workunit.client.0.smithi153.stdout:5/593: dread d0/d9/d22/fc2 [0,4194304] 0 2024-04-05T03:02:57.603 INFO:tasks.workunit.client.0.smithi153.stdout:5/594: mknod d0/d25/d4d/d9c/dc7/cd2 0 2024-04-05T03:02:57.608 INFO:tasks.workunit.client.0.smithi153.stdout:7/813: write d2/da/d11/f2b [5039236,70046] 0 2024-04-05T03:02:57.608 INFO:tasks.workunit.client.0.smithi153.stdout:5/595: dwrite d0/d25/d68/f9b [0,4194304] 0 2024-04-05T03:02:57.610 INFO:tasks.workunit.client.0.smithi153.stdout:7/814: readlink d2/da/d11/d88/da6/ld0 0 2024-04-05T03:02:57.610 INFO:tasks.workunit.client.0.smithi153.stdout:0/656: fsync d9/d68/fc9 0 2024-04-05T03:02:57.611 INFO:tasks.workunit.client.1.smithi203.stdout:3/939: dwrite d0/d21/f6c [8388608,4194304] 0 2024-04-05T03:02:57.615 INFO:tasks.workunit.client.0.smithi153.stdout:0/657: write d9/db/d42/d7e/faa [950272,88174] 0 2024-04-05T03:02:57.616 INFO:tasks.workunit.client.0.smithi153.stdout:7/815: creat d2/da/d47/d5d/d7c/d8d/df2/f112 x:0 0 0 2024-04-05T03:02:57.617 INFO:tasks.workunit.client.1.smithi203.stdout:3/940: getdents d0 0 2024-04-05T03:02:57.618 INFO:tasks.workunit.client.0.smithi153.stdout:5/596: creat d0/d25/d4d/d62/d37/fd3 x:0 0 0 2024-04-05T03:02:57.619 INFO:tasks.workunit.client.0.smithi153.stdout:5/597: chown d0/d9/f1f 1014767 1 2024-04-05T03:02:57.619 INFO:tasks.workunit.client.0.smithi153.stdout:7/816: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d59/c113 0 2024-04-05T03:02:57.620 INFO:tasks.workunit.client.1.smithi203.stdout:3/941: symlink d0/dc/d1a/d1f/dac/l11c 0 2024-04-05T03:02:57.625 INFO:tasks.workunit.client.0.smithi153.stdout:5/598: getdents d0/d2c 0 2024-04-05T03:02:57.625 INFO:tasks.workunit.client.0.smithi153.stdout:7/817: getdents d2/da/d11/d88/da6 0 2024-04-05T03:02:57.633 INFO:tasks.workunit.client.0.smithi153.stdout:5/599: dwrite d0/d25/f6e [0,4194304] 0 2024-04-05T03:02:57.636 INFO:tasks.workunit.client.0.smithi153.stdout:5/600: mknod d0/d25/d4d/d9c/dc7/dc9/cd4 0 2024-04-05T03:02:57.640 INFO:tasks.workunit.client.0.smithi153.stdout:5/601: creat d0/d25/d4d/d9c/db7/fd5 x:0 0 0 2024-04-05T03:02:57.642 INFO:tasks.workunit.client.1.smithi203.stdout:3/942: dwrite d0/fbd [0,4194304] 0 2024-04-05T03:02:57.642 INFO:tasks.workunit.client.0.smithi153.stdout:0/658: dread d9/f30 [0,4194304] 0 2024-04-05T03:02:57.642 INFO:tasks.workunit.client.0.smithi153.stdout:7/818: write d2/da/d3a/d41/d7a/d54/d85/ff7 [37392,102316] 0 2024-04-05T03:02:57.643 INFO:tasks.workunit.client.0.smithi153.stdout:5/602: creat d0/d25/d4d/d9c/db7/fd6 x:0 0 0 2024-04-05T03:02:57.643 INFO:tasks.workunit.client.1.smithi203.stdout:3/943: write d0/dc/ff3 [4913882,60218] 0 2024-04-05T03:02:57.644 INFO:tasks.workunit.client.0.smithi153.stdout:7/819: chown d2/da/d3a/f94 8 1 2024-04-05T03:02:57.645 INFO:tasks.workunit.client.0.smithi153.stdout:0/659: chown d9/d14/d67/d6f/f9d 1 1 2024-04-05T03:02:57.648 INFO:tasks.workunit.client.0.smithi153.stdout:0/660: dread - d9/db/d42/d7e/f8c zero size 2024-04-05T03:02:57.649 INFO:tasks.workunit.client.0.smithi153.stdout:5/603: mkdir d0/d25/d4d/d62/d5f/d86/dd7 0 2024-04-05T03:02:57.649 INFO:tasks.workunit.client.1.smithi203.stdout:3/944: dwrite d0/dc/d1a/d76/f94 [0,4194304] 0 2024-04-05T03:02:57.651 INFO:tasks.workunit.client.1.smithi203.stdout:3/945: write d0/dc/d1a/d1f/f32 [4368100,29402] 0 2024-04-05T03:02:57.654 INFO:tasks.workunit.client.0.smithi153.stdout:0/661: mknod d9/db/d42/d7e/db6/d1e/d45/ccb 0 2024-04-05T03:02:57.657 INFO:tasks.workunit.client.1.smithi203.stdout:3/946: symlink d0/dc/d1a/d1d/l11d 0 2024-04-05T03:02:57.658 INFO:tasks.workunit.client.1.smithi203.stdout:3/947: read - d0/dc/d1a/d1d/d53/ff4 zero size 2024-04-05T03:02:57.660 INFO:tasks.workunit.client.1.smithi203.stdout:3/948: stat d0/fdd 0 2024-04-05T03:02:57.660 INFO:tasks.workunit.client.0.smithi153.stdout:5/604: dwrite d0/d25/d4d/d62/d5f/d86/f74 [0,4194304] 0 2024-04-05T03:02:57.664 INFO:tasks.workunit.client.1.smithi203.stdout:3/949: rmdir d0/d21/d119 0 2024-04-05T03:02:57.668 INFO:tasks.workunit.client.0.smithi153.stdout:6/820: sync 2024-04-05T03:02:57.668 INFO:tasks.workunit.client.0.smithi153.stdout:9/566: sync 2024-04-05T03:02:57.669 INFO:tasks.workunit.client.0.smithi153.stdout:3/696: sync 2024-04-05T03:02:57.669 INFO:tasks.workunit.client.0.smithi153.stdout:4/802: sync 2024-04-05T03:02:57.669 INFO:tasks.workunit.client.0.smithi153.stdout:1/700: sync 2024-04-05T03:02:57.669 INFO:tasks.workunit.client.0.smithi153.stdout:8/582: sync 2024-04-05T03:02:57.672 INFO:tasks.workunit.client.0.smithi153.stdout:9/567: write d1/fc4 [77737,31457] 0 2024-04-05T03:02:57.672 INFO:tasks.workunit.client.0.smithi153.stdout:3/697: mkdir d1d/d27/df4 0 2024-04-05T03:02:57.674 INFO:tasks.workunit.client.0.smithi153.stdout:1/701: creat d5/d8/d20/d4b/d51/da2/fd0 x:0 0 0 2024-04-05T03:02:57.675 INFO:tasks.workunit.client.0.smithi153.stdout:6/821: chown d4/dc/df/f13 74718408 1 2024-04-05T03:02:57.676 INFO:tasks.workunit.client.0.smithi153.stdout:7/820: dread d2/da/d3a/d41/ddd/f107 [0,4194304] 0 2024-04-05T03:02:57.678 INFO:tasks.workunit.client.0.smithi153.stdout:4/803: rename da/de/d56/d6a/f7f to da/db/d8b/d10f/f117 0 2024-04-05T03:02:57.680 INFO:tasks.workunit.client.0.smithi153.stdout:1/702: chown d5/d8/d20/d4b/d51/da2/c8c 56457369 1 2024-04-05T03:02:57.681 INFO:tasks.workunit.client.0.smithi153.stdout:7/821: chown d2/da/d47/d5d/d7c/d8d/d95 117 1 2024-04-05T03:02:57.681 INFO:tasks.workunit.client.1.smithi203.stdout:3/950: write d0/d41/f8e [1455631,60715] 0 2024-04-05T03:02:57.682 INFO:tasks.workunit.client.1.smithi203.stdout:3/951: write d0/dc/d1a/d8d/fe1 [104859,64792] 0 2024-04-05T03:02:57.683 INFO:tasks.workunit.client.1.smithi203.stdout:3/952: chown d0/d21/fe4 5 1 2024-04-05T03:02:57.683 INFO:tasks.workunit.client.0.smithi153.stdout:6/822: creat d4/dc/df/d16/d17/d54/f11a x:0 0 0 2024-04-05T03:02:57.683 INFO:tasks.workunit.client.0.smithi153.stdout:5/605: dread d0/d25/d4d/d62/f56 [0,4194304] 0 2024-04-05T03:02:57.684 INFO:tasks.workunit.client.0.smithi153.stdout:4/804: mknod da/db/c118 0 2024-04-05T03:02:57.684 INFO:tasks.workunit.client.1.smithi203.stdout:3/953: read d0/dc/d38/d5d/d66/fc8 [2457399,76031] 0 2024-04-05T03:02:57.685 INFO:tasks.workunit.client.1.smithi203.stdout:3/954: read - d0/d21/f110 zero size 2024-04-05T03:02:57.686 INFO:tasks.workunit.client.0.smithi153.stdout:8/583: dwrite d7/fa [0,4194304] 0 2024-04-05T03:02:57.686 INFO:tasks.workunit.client.0.smithi153.stdout:1/703: stat d5/d8/d20/d4b/d1e/c59 0 2024-04-05T03:02:57.687 INFO:tasks.workunit.client.1.smithi203.stdout:3/955: creat d0/d41/f11e x:0 0 0 2024-04-05T03:02:57.688 INFO:tasks.workunit.client.0.smithi153.stdout:5/606: readlink d0/d2c/d3f/da8/lac 0 2024-04-05T03:02:57.688 INFO:tasks.workunit.client.0.smithi153.stdout:9/568: dwrite d1/d1d/f5a [4194304,4194304] 0 2024-04-05T03:02:57.689 INFO:tasks.workunit.client.0.smithi153.stdout:3/698: rename d1d/d27/c5a to d1d/d32/d93/d3c/cf5 0 2024-04-05T03:02:57.691 INFO:tasks.workunit.client.0.smithi153.stdout:7/822: rmdir d2/da/d3a/d41/d7a 39 2024-04-05T03:02:57.692 INFO:tasks.workunit.client.0.smithi153.stdout:4/805: chown da/db/de6/cf4 5 1 2024-04-05T03:02:57.695 INFO:tasks.workunit.client.0.smithi153.stdout:6/823: fsync d4/dc/df/d16/d36/d52/f55 0 2024-04-05T03:02:57.695 INFO:tasks.workunit.client.1.smithi203.stdout:3/956: dwrite d0/dc/d1a/d76/d35/f107 [0,4194304] 0 2024-04-05T03:02:57.697 INFO:tasks.workunit.client.0.smithi153.stdout:3/699: creat d1d/d75/ff6 x:0 0 0 2024-04-05T03:02:57.703 INFO:tasks.workunit.client.1.smithi203.stdout:3/957: getdents d0/dc/d1a/d1f/da1 0 2024-04-05T03:02:57.703 INFO:tasks.workunit.client.0.smithi153.stdout:5/607: dwrite d0/f44 [0,4194304] 0 2024-04-05T03:02:57.706 INFO:tasks.workunit.client.1.smithi203.stdout:3/958: mkdir d0/dc/d34/d11f 0 2024-04-05T03:02:57.707 INFO:tasks.workunit.client.1.smithi203.stdout:3/959: fsync d0/dc/d1a/d1d/d53/ff1 0 2024-04-05T03:02:57.708 INFO:tasks.workunit.client.1.smithi203.stdout:3/960: write d0/d41/f11e [819441,42048] 0 2024-04-05T03:02:57.713 INFO:tasks.workunit.client.1.smithi203.stdout:3/961: creat d0/d69/f120 x:0 0 0 2024-04-05T03:02:57.713 INFO:tasks.workunit.client.0.smithi153.stdout:9/569: dwrite d1/d10/d36/d57/d97/fa5 [0,4194304] 0 2024-04-05T03:02:57.713 INFO:tasks.workunit.client.0.smithi153.stdout:5/608: creat d0/d25/d4d/d9c/fd8 x:0 0 0 2024-04-05T03:02:57.716 INFO:tasks.workunit.client.1.smithi203.stdout:3/962: creat d0/dc/d1a/d1f/dac/d114/f121 x:0 0 0 2024-04-05T03:02:57.717 INFO:tasks.workunit.client.1.smithi203.stdout:3/963: chown d0/f23 9252416 1 2024-04-05T03:02:57.718 INFO:tasks.workunit.client.0.smithi153.stdout:5/609: chown d0/d9/d22/f7a 18465116 1 2024-04-05T03:02:57.718 INFO:tasks.workunit.client.0.smithi153.stdout:3/700: dwrite d1d/d32/f49 [4194304,4194304] 0 2024-04-05T03:02:57.722 INFO:tasks.workunit.client.0.smithi153.stdout:9/570: symlink d1/d10/d36/d57/d5c/lc6 0 2024-04-05T03:02:57.722 INFO:tasks.workunit.client.0.smithi153.stdout:6/824: truncate d4/dc/df/d16/d36/d52/f55 663266 0 2024-04-05T03:02:57.724 INFO:tasks.workunit.client.0.smithi153.stdout:7/823: write d2/da/d3a/f46 [1445606,78573] 0 2024-04-05T03:02:57.725 INFO:tasks.workunit.client.1.smithi203.stdout:3/964: dwrite d0/d21/f6c [0,4194304] 0 2024-04-05T03:02:57.726 INFO:tasks.workunit.client.0.smithi153.stdout:1/704: dwrite d5/d8/d20/d3f/f91 [0,4194304] 0 2024-04-05T03:02:57.726 INFO:tasks.workunit.client.0.smithi153.stdout:6/825: readlink d4/dc/d7a/le5 0 2024-04-05T03:02:57.728 INFO:tasks.workunit.client.0.smithi153.stdout:1/705: chown d5/d8/d20/d4b/d18/d16/d45/d68/da4/dc3 45963 1 2024-04-05T03:02:57.729 INFO:tasks.workunit.client.0.smithi153.stdout:3/701: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/ff7 x:0 0 0 2024-04-05T03:02:57.732 INFO:tasks.workunit.client.0.smithi153.stdout:1/706: write d5/d8/d20/d3f/f5c [7667575,16793] 0 2024-04-05T03:02:57.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.733+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f05ac1058d0 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:02:57.735 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 13, 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:02:57.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.734+0000 7f05a3fff700 1 -- 172.21.15.153:0/1512693990 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f05940587e0 con 0x7f05ac0fdfe0 2024-04-05T03:02:57.736 INFO:tasks.workunit.client.1.smithi203.stdout:3/965: dwrite d0/dc/d1a/d76/f94 [0,4194304] 0 2024-04-05T03:02:57.738 INFO:tasks.workunit.client.0.smithi153.stdout:5/610: creat d0/d2c/d3f/fd9 x:0 0 0 2024-04-05T03:02:57.739 INFO:tasks.workunit.client.0.smithi153.stdout:5/611: stat d0/d2c/d3f/da8/lac 0 2024-04-05T03:02:57.739 INFO:tasks.workunit.client.0.smithi153.stdout:9/571: creat d1/d10/d16/d22/d2c/d68/da9/fc7 x:0 0 0 2024-04-05T03:02:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.740+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 msgr2=0x7f059806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.740+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 0x7f059806d990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f059c009b80 tx=0x7f059c00b040 comp rx=0 tx=0).stop 2024-04-05T03:02:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.740+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 msgr2=0x7f05ac10ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.740+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0594013870 tx=0x7f0594009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.741+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 shutdown_connections 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.741+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f059806b4e0 0x7f059806d990 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.741+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05ac0fce70 0x7f05ac10d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.741+0000 7f05b1ba1700 1 --2- 172.21.15.153:0/1512693990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05ac0fdfe0 0x7f05ac10ddd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.741+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 >> 172.21.15.153:0/1512693990 conn(0x7f05ac0f8550 msgr2=0x7f05ac0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:57.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.742+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 shutdown_connections 2024-04-05T03:02:57.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.742+0000 7f05b1ba1700 1 -- 172.21.15.153:0/1512693990 wait complete. 2024-04-05T03:02:57.743 INFO:tasks.workunit.client.0.smithi153.stdout:6/826: rename d4/dc/df/d16/d36/d4d/d64/d110/df2/d118 to d4/dc/df/d16/d36/d10e/d11b 0 2024-04-05T03:02:57.744 INFO:tasks.workunit.client.0.smithi153.stdout:3/702: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/lf8 0 2024-04-05T03:02:57.746 INFO:tasks.workunit.client.0.smithi153.stdout:1/707: creat d5/d8/d20/d4b/d18/d16/d2c/d62/fd1 x:0 0 0 2024-04-05T03:02:57.748 INFO:tasks.workunit.client.1.smithi203.stdout:4/981: sync 2024-04-05T03:02:57.750 INFO:tasks.workunit.client.0.smithi153.stdout:5/612: chown d0/d9/d22/c7d 1 1 2024-04-05T03:02:57.753 INFO:tasks.workunit.client.0.smithi153.stdout:3/703: mkdir d1d/d32/df9 0 2024-04-05T03:02:57.756 INFO:tasks.workunit.client.0.smithi153.stdout:1/708: mknod d5/d8/d20/d4b/d51/db5/cd2 0 2024-04-05T03:02:57.756 INFO:tasks.workunit.client.0.smithi153.stdout:6/827: dwrite d4/f6 [0,4194304] 0 2024-04-05T03:02:57.761 INFO:tasks.workunit.client.0.smithi153.stdout:8/584: dread d7/fd [0,4194304] 0 2024-04-05T03:02:57.762 INFO:tasks.workunit.client.0.smithi153.stdout:9/572: dwrite d1/d10/d36/f62 [0,4194304] 0 2024-04-05T03:02:57.764 INFO:tasks.workunit.client.0.smithi153.stdout:3/704: write d1d/d27/f64 [3715519,54206] 0 2024-04-05T03:02:57.765 INFO:tasks.workunit.client.0.smithi153.stdout:6/828: mkdir d4/d11c 0 2024-04-05T03:02:57.769 INFO:tasks.workunit.client.0.smithi153.stdout:8/585: creat d7/d13/d19/d1b/d21/fbf x:0 0 0 2024-04-05T03:02:57.772 INFO:tasks.workunit.client.0.smithi153.stdout:1/709: creat d5/d8/d20/d2f/db8/fd3 x:0 0 0 2024-04-05T03:02:57.772 INFO:tasks.workunit.client.0.smithi153.stdout:9/573: creat d1/d10/d36/d57/d97/fc8 x:0 0 0 2024-04-05T03:02:57.773 INFO:tasks.workunit.client.0.smithi153.stdout:3/705: unlink d1d/d27/d2b/f4e 0 2024-04-05T03:02:57.778 INFO:tasks.workunit.client.0.smithi153.stdout:5/613: fsync d0/d2c/d3f/fd9 0 2024-04-05T03:02:57.778 INFO:tasks.workunit.client.0.smithi153.stdout:6/829: creat d4/dc/df/d16/d36/f11d x:0 0 0 2024-04-05T03:02:57.781 INFO:tasks.workunit.client.0.smithi153.stdout:8/586: dwrite d7/d13/d19/d1b/d6c/fb0 [0,4194304] 0 2024-04-05T03:02:57.788 INFO:tasks.workunit.client.0.smithi153.stdout:7/824: truncate d2/da/d47/d5d/d7c/f81 1500397 0 2024-04-05T03:02:57.789 INFO:tasks.workunit.client.0.smithi153.stdout:6/830: symlink d4/l11e 0 2024-04-05T03:02:57.790 INFO:tasks.workunit.client.0.smithi153.stdout:7/825: stat d2/da/d3a/d41/ddd/de1/lf3 0 2024-04-05T03:02:57.791 INFO:tasks.workunit.client.0.smithi153.stdout:1/710: rename d5/d8/d20/d4b/d51/fa9 to d5/d8/d20/d4b/d18/d16/d33/d83/fd4 0 2024-04-05T03:02:57.793 INFO:tasks.workunit.client.0.smithi153.stdout:5/614: symlink d0/d2c/d3f/lda 0 2024-04-05T03:02:57.795 INFO:tasks.workunit.client.0.smithi153.stdout:5/615: chown d0/f16 3715 1 2024-04-05T03:02:57.800 INFO:tasks.workunit.client.0.smithi153.stdout:9/574: dwrite d1/d10/d16/d22/d2c/d68/d81/fbb [0,4194304] 0 2024-04-05T03:02:57.800 INFO:tasks.workunit.client.0.smithi153.stdout:5/616: chown d0/d25/d4d/d62/c4a 69948069 1 2024-04-05T03:02:57.801 INFO:tasks.workunit.client.0.smithi153.stdout:5/617: chown d0/d25/d4d/d62/d5f/d86/f77 24297914 1 2024-04-05T03:02:57.801 INFO:tasks.workunit.client.0.smithi153.stdout:8/587: getdents d7/d13/d19/d1b/d6d 0 2024-04-05T03:02:57.803 INFO:tasks.workunit.client.0.smithi153.stdout:6/831: mknod d4/dc/df/d16/d24/d5e/c11f 0 2024-04-05T03:02:57.804 INFO:tasks.workunit.client.0.smithi153.stdout:7/826: rmdir d2/da/d11/d88 39 2024-04-05T03:02:57.806 INFO:tasks.workunit.client.0.smithi153.stdout:3/706: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/l6c to d1d/d32/d93/d3c/d3f/d90/d92/lfa 0 2024-04-05T03:02:57.809 INFO:tasks.workunit.client.0.smithi153.stdout:5/618: unlink d0/d25/d4d/d62/d5f/d86/cc6 0 2024-04-05T03:02:57.817 INFO:tasks.workunit.client.0.smithi153.stdout:5/619: mknod d0/cdb 0 2024-04-05T03:02:57.821 INFO:tasks.workunit.client.0.smithi153.stdout:5/620: unlink d0/f16 0 2024-04-05T03:02:57.821 INFO:tasks.workunit.client.0.smithi153.stdout:3/707: link d1d/d27/d2b/l65 d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/lfb 0 2024-04-05T03:02:57.821 INFO:tasks.workunit.client.0.smithi153.stdout:7/827: write d2/da/d47/fa1 [445157,51341] 0 2024-04-05T03:02:57.823 INFO:tasks.workunit.client.0.smithi153.stdout:1/711: truncate d5/d8/d20/d4b/d18/d16/d33/d83/f8e 79540 0 2024-04-05T03:02:57.825 INFO:tasks.workunit.client.0.smithi153.stdout:7/828: truncate d2/fd6 596390 0 2024-04-05T03:02:57.826 INFO:tasks.workunit.client.0.smithi153.stdout:8/588: write d7/d85/f78 [943144,80499] 0 2024-04-05T03:02:57.832 INFO:tasks.workunit.client.0.smithi153.stdout:8/589: rename d7/f2b to d7/d13/dab/fc0 0 2024-04-05T03:02:57.833 INFO:tasks.workunit.client.0.smithi153.stdout:3/708: read d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f76 [1265235,92098] 0 2024-04-05T03:02:57.838 INFO:tasks.workunit.client.0.smithi153.stdout:9/575: dwrite d1/d9/d1b/f28 [0,4194304] 0 2024-04-05T03:02:57.839 INFO:tasks.workunit.client.0.smithi153.stdout:3/709: fdatasync d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fa5 0 2024-04-05T03:02:57.841 INFO:tasks.workunit.client.0.smithi153.stdout:3/710: readlink d1d/d32/d93/d3c/d3f/d9c/la2 0 2024-04-05T03:02:57.846 INFO:tasks.workunit.client.0.smithi153.stdout:2/726: sync 2024-04-05T03:02:57.850 INFO:tasks.workunit.client.0.smithi153.stdout:8/590: dread d7/d85/f78 [0,4194304] 0 2024-04-05T03:02:57.852 INFO:tasks.workunit.client.0.smithi153.stdout:1/712: dwrite d5/d8/d20/f38 [0,4194304] 0 2024-04-05T03:02:57.854 INFO:tasks.workunit.client.0.smithi153.stdout:9/576: rename d1/d10/f5f to d1/d10/d16/d22/fc9 0 2024-04-05T03:02:57.856 INFO:tasks.workunit.client.0.smithi153.stdout:9/577: stat d1/d6c/f82 0 2024-04-05T03:02:57.856 INFO:tasks.workunit.client.0.smithi153.stdout:7/829: dread d2/da/d47/f92 [0,4194304] 0 2024-04-05T03:02:57.858 INFO:tasks.workunit.client.0.smithi153.stdout:2/727: rmdir d9/da/d19/d2e/d7e/d7a 39 2024-04-05T03:02:57.860 INFO:tasks.workunit.client.0.smithi153.stdout:2/728: stat d9/d68/l3b 0 2024-04-05T03:02:57.860 INFO:tasks.workunit.client.0.smithi153.stdout:1/713: rename d5/fbb to d5/d8/d20/d2f/d40/fd5 0 2024-04-05T03:02:57.861 INFO:tasks.workunit.client.0.smithi153.stdout:9/578: creat d1/d10/d16/d22/d2c/d68/fca x:0 0 0 2024-04-05T03:02:57.863 INFO:tasks.workunit.client.0.smithi153.stdout:2/729: truncate d9/d68/db9/fa0 4558709 0 2024-04-05T03:02:57.864 INFO:tasks.workunit.client.0.smithi153.stdout:5/621: dread d0/d2c/f2d [0,4194304] 0 2024-04-05T03:02:57.867 INFO:tasks.workunit.client.0.smithi153.stdout:1/714: mkdir d5/d8/d20/d4b/dd6 0 2024-04-05T03:02:57.868 INFO:tasks.workunit.client.0.smithi153.stdout:5/622: rmdir d0/d25/d68 39 2024-04-05T03:02:57.870 INFO:tasks.workunit.client.0.smithi153.stdout:2/730: mkdir d9/da/df9 0 2024-04-05T03:02:57.872 INFO:tasks.workunit.client.0.smithi153.stdout:1/715: symlink d5/dd/d1b/ld7 0 2024-04-05T03:02:57.873 INFO:tasks.workunit.client.0.smithi153.stdout:9/579: rmdir d1/d10/d16/d22/d2c/d68 39 2024-04-05T03:02:57.874 INFO:tasks.workunit.client.0.smithi153.stdout:8/591: write d7/d85/f32 [3366817,121458] 0 2024-04-05T03:02:57.875 INFO:tasks.workunit.client.0.smithi153.stdout:2/731: symlink d9/da/d94/dca/df7/lfa 0 2024-04-05T03:02:57.876 INFO:tasks.workunit.client.0.smithi153.stdout:9/580: truncate d1/d10/d16/d22/fc3 852884 0 2024-04-05T03:02:57.877 INFO:tasks.workunit.client.0.smithi153.stdout:5/623: rmdir d0/d25/d4d/d62/d37/d38/dcd 0 2024-04-05T03:02:57.878 INFO:tasks.workunit.client.0.smithi153.stdout:1/716: fsync d5/d8/d20/d2f/d40/f4c 0 2024-04-05T03:02:57.879 INFO:tasks.workunit.client.0.smithi153.stdout:8/592: fdatasync d7/d46/fa7 0 2024-04-05T03:02:57.881 INFO:tasks.workunit.client.0.smithi153.stdout:2/732: symlink d9/da/d37/d8a/lfb 0 2024-04-05T03:02:57.881 INFO:tasks.workunit.client.0.smithi153.stdout:8/593: dread d7/d85/f78 [0,4194304] 0 2024-04-05T03:02:57.881 INFO:tasks.workunit.client.0.smithi153.stdout:9/581: creat d1/d3a/fcb x:0 0 0 2024-04-05T03:02:57.883 INFO:tasks.workunit.client.0.smithi153.stdout:9/582: chown d1/d10/d36/d57/f6f 59071469 1 2024-04-05T03:02:57.883 INFO:tasks.workunit.client.0.smithi153.stdout:8/594: chown d7/d13/d19/d1b/d21/l4b 1310818484 1 2024-04-05T03:02:57.886 INFO:tasks.workunit.client.0.smithi153.stdout:1/717: rename d5/cce to d5/d8/d20/d4b/d18/d16/d33/d83/cd8 0 2024-04-05T03:02:57.888 INFO:tasks.workunit.client.0.smithi153.stdout:2/733: creat d9/da/d94/ffc x:0 0 0 2024-04-05T03:02:57.890 INFO:tasks.workunit.client.0.smithi153.stdout:5/624: getdents d0/d4e/db6 0 2024-04-05T03:02:57.890 INFO:tasks.workunit.client.0.smithi153.stdout:9/583: creat d1/d10/d16/d22/fcc x:0 0 0 2024-04-05T03:02:57.890 INFO:tasks.workunit.client.0.smithi153.stdout:1/718: symlink d5/d8/d20/d4b/d1e/ld9 0 2024-04-05T03:02:57.895 INFO:tasks.workunit.client.0.smithi153.stdout:5/625: creat d0/d25/d4d/d9c/dc4/dcb/fdc x:0 0 0 2024-04-05T03:02:57.897 INFO:tasks.workunit.client.0.smithi153.stdout:2/734: rename d9/da/d37/l5d to d9/da/df9/lfd 0 2024-04-05T03:02:57.899 INFO:tasks.workunit.client.0.smithi153.stdout:8/595: getdents d7/d13/d19/d1b/d4f 0 2024-04-05T03:02:57.907 INFO:tasks.workunit.client.0.smithi153.stdout:9/584: rename d1/d9/d1b/f7b to d1/fcd 0 2024-04-05T03:02:57.910 INFO:tasks.workunit.client.0.smithi153.stdout:1/719: dwrite d5/d8/d20/d2f/d40/fd5 [0,4194304] 0 2024-04-05T03:02:57.910 INFO:tasks.workunit.client.0.smithi153.stdout:5/626: write d0/d25/d68/f99 [731061,84281] 0 2024-04-05T03:02:57.914 INFO:tasks.workunit.client.0.smithi153.stdout:8/596: dwrite d7/f74 [0,4194304] 0 2024-04-05T03:02:57.919 INFO:tasks.workunit.client.0.smithi153.stdout:9/585: creat d1/d10/d36/d57/d97/fce x:0 0 0 2024-04-05T03:02:57.922 INFO:tasks.workunit.client.0.smithi153.stdout:5/627: rmdir d0/d25/d68 39 2024-04-05T03:02:57.923 INFO:tasks.workunit.client.0.smithi153.stdout:2/735: dwrite d9/da/d19/d2e/d4f/f86 [0,4194304] 0 2024-04-05T03:02:57.926 INFO:tasks.workunit.client.0.smithi153.stdout:2/736: fsync d9/da/d19/d2e/dc7/fcc 0 2024-04-05T03:02:57.927 INFO:tasks.workunit.client.0.smithi153.stdout:2/737: readlink d9/da/d37/lb0 0 2024-04-05T03:02:57.927 INFO:tasks.workunit.client.0.smithi153.stdout:8/597: mknod d7/d13/d19/d1b/d4f/d68/dbc/cc1 0 2024-04-05T03:02:57.929 INFO:tasks.workunit.client.0.smithi153.stdout:8/598: stat d7/d2a/f37 0 2024-04-05T03:02:57.929 INFO:tasks.workunit.client.0.smithi153.stdout:1/720: dread d5/d8/d20/f88 [4194304,4194304] 0 2024-04-05T03:02:57.931 INFO:tasks.workunit.client.0.smithi153.stdout:5/628: write d0/d25/d4d/d62/d5f/d86/f74 [2762320,36437] 0 2024-04-05T03:02:57.934 INFO:tasks.workunit.client.0.smithi153.stdout:9/586: rename d1/d10/d36/d71/lbf to d1/d6c/lcf 0 2024-04-05T03:02:57.935 INFO:tasks.workunit.client.0.smithi153.stdout:8/599: mknod d7/d13/d19/d1b/d6d/d8c/cc2 0 2024-04-05T03:02:57.937 INFO:tasks.workunit.client.0.smithi153.stdout:9/587: fdatasync d1/d10/d36/d57/d97/fce 0 2024-04-05T03:02:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 -- 172.21.15.153:0/3924231088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0072820 msgr2=0x7f90f0072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 --2- 172.21.15.153:0/3924231088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0072820 0x7f90f0072c30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f90e0004940 tx=0x7f90e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 -- 172.21.15.153:0/3924231088 shutdown_connections 2024-04-05T03:02:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 --2- 172.21.15.153:0/3924231088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0073170 0x7f90f0073b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 --2- 172.21.15.153:0/3924231088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0072820 0x7f90f0072c30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.938+0000 7f90f4939700 1 -- 172.21.15.153:0/3924231088 >> 172.21.15.153:0/3924231088 conn(0x7f90f00fa950 msgr2=0x7f90f00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:57.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.939+0000 7f90f4939700 1 -- 172.21.15.153:0/3924231088 shutdown_connections 2024-04-05T03:02:57.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.939+0000 7f90f4939700 1 -- 172.21.15.153:0/3924231088 wait complete. 2024-04-05T03:02:57.939 INFO:tasks.workunit.client.0.smithi153.stdout:5/629: truncate d0/d9/f20 1231564 0 2024-04-05T03:02:57.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 Processor -- start 2024-04-05T03:02:57.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 -- start start 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0114550 0x7f90f01153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0114a10 con 0x7f90f0072820 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.940+0000 7f90f4939700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0114b50 con 0x7f90f0114550 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39806/0 (socket says 172.21.15.153:39806) 2024-04-05T03:02:57.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 -- 172.21.15.153:0/176896510 learned_addr learned my addr 172.21.15.153:0/176896510 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90e5d9b700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0114550 0x7f90f01153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 -- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0114550 msgr2=0x7f90f01153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0114550 0x7f90f01153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.941+0000 7f90ee59c700 1 -- 172.21.15.153:0/176896510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e0005040 con 0x7f90f0072820 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90ee59c700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f90e000b0d0 tx=0x7f90e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e000e070 con 0x7f90f0072820 2024-04-05T03:02:57.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90e000b410 con 0x7f90f0072820 2024-04-05T03:02:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e0012600 con 0x7f90f0072820 2024-04-05T03:02:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90f4939700 1 -- 172.21.15.153:0/176896510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f01159a0 con 0x7f90f0072820 2024-04-05T03:02:57.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.942+0000 7f90f4939700 1 -- 172.21.15.153:0/176896510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f0115e20 con 0x7f90f0072820 2024-04-05T03:02:57.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.944+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f005f5f0 con 0x7f90f0072820 2024-04-05T03:02:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.945+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f90e000ba60 con 0x7f90f0072820 2024-04-05T03:02:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.945+0000 7f90e77fe700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 0x7f90dc06dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:02:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.946+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f90e008af00 con 0x7f90f0072820 2024-04-05T03:02:57.946 INFO:tasks.workunit.client.0.smithi153.stdout:5/630: rename d0/d25/d4d/d62/d37/d38/cb8 to d0/d25/d4d/d9c/dc7/dc9/cdd 0 2024-04-05T03:02:57.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.946+0000 7f90e5d9b700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 0x7f90dc06dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:02:57.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.947+0000 7f90e5d9b700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 0x7f90dc06dac0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f90d8004600 tx=0x7f90d800b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:02:57.947 INFO:tasks.workunit.client.0.smithi153.stdout:9/588: dread d1/d10/d16/d22/d2c/d68/d81/fbb [0,4194304] 0 2024-04-05T03:02:57.950 INFO:tasks.workunit.client.0.smithi153.stdout:5/631: creat d0/d25/d4d/d9c/dc7/fde x:0 0 0 2024-04-05T03:02:57.951 INFO:tasks.workunit.client.0.smithi153.stdout:5/632: dread - d0/d2c/f8f zero size 2024-04-05T03:02:57.953 INFO:tasks.workunit.client.0.smithi153.stdout:5/633: fdatasync d0/d25/f36 0 2024-04-05T03:02:57.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:57.955+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f90e00587d0 con 0x7f90f0072820 2024-04-05T03:02:57.955 INFO:tasks.workunit.client.1.smithi203.stdout:4/982: mkdir da/d3c/d12a/d144 0 2024-04-05T03:02:57.956 INFO:tasks.workunit.client.1.smithi203.stdout:4/983: fdatasync da/d29/d2e/db0/dbe/f13f 0 2024-04-05T03:02:57.958 INFO:tasks.workunit.client.1.smithi203.stdout:4/984: read - da/d29/d2e/d57/da7/fd9 zero size 2024-04-05T03:02:57.959 INFO:tasks.workunit.client.1.smithi203.stdout:3/966: write d0/dc/d1a/d1f/dac/fc5 [205442,11871] 0 2024-04-05T03:02:57.961 INFO:tasks.workunit.client.0.smithi153.stdout:5/634: rmdir d0/d2c/d5e/da7 0 2024-04-05T03:02:57.964 INFO:tasks.workunit.client.0.smithi153.stdout:1/721: write d5/dd/f13 [141530,53488] 0 2024-04-05T03:02:57.966 INFO:tasks.workunit.client.0.smithi153.stdout:2/738: dread d9/d68/fa7 [0,4194304] 0 2024-04-05T03:02:57.968 INFO:tasks.workunit.client.1.smithi203.stdout:4/985: creat da/f145 x:0 0 0 2024-04-05T03:02:57.970 INFO:tasks.workunit.client.0.smithi153.stdout:8/600: truncate d7/fc 331126 0 2024-04-05T03:02:57.972 INFO:tasks.workunit.client.0.smithi153.stdout:2/739: rmdir d9/da/d37 39 2024-04-05T03:02:57.976 INFO:tasks.workunit.client.0.smithi153.stdout:1/722: rename d5/d8/d20/d4b/d18/d16/d48/l67 to d5/d8/d20/d4b/d18/d16/d33/lda 0 2024-04-05T03:02:57.978 INFO:tasks.workunit.client.0.smithi153.stdout:5/635: dwrite d0/d25/d68/fae [0,4194304] 0 2024-04-05T03:02:57.979 INFO:tasks.workunit.client.1.smithi203.stdout:4/986: dwrite da/d29/d2e/d57/da5/d127/fd6 [0,4194304] 0 2024-04-05T03:02:57.979 INFO:tasks.workunit.client.0.smithi153.stdout:8/601: truncate d7/d2a/f77 523834 0 2024-04-05T03:02:57.980 INFO:tasks.workunit.client.0.smithi153.stdout:2/740: mknod d9/d68/d98/de9/cfe 0 2024-04-05T03:02:57.982 INFO:tasks.workunit.client.0.smithi153.stdout:1/723: unlink d5/d8/d20/d4b/d18/cb7 0 2024-04-05T03:02:57.982 INFO:tasks.workunit.client.1.smithi203.stdout:4/987: chown da/c2a 6343 1 2024-04-05T03:02:57.983 INFO:tasks.workunit.client.0.smithi153.stdout:8/602: stat d7/c24 0 2024-04-05T03:02:57.985 INFO:tasks.workunit.client.0.smithi153.stdout:5/636: creat d0/d25/d4d/d62/d5f/db2/fdf x:0 0 0 2024-04-05T03:02:57.986 INFO:tasks.workunit.client.1.smithi203.stdout:4/988: mknod da/d71/c146 0 2024-04-05T03:02:57.987 INFO:tasks.workunit.client.0.smithi153.stdout:1/724: rmdir d5/dd 39 2024-04-05T03:02:57.990 INFO:tasks.workunit.client.0.smithi153.stdout:8/603: creat d7/d13/d19/d1b/d4f/d5f/d54/fc3 x:0 0 0 2024-04-05T03:02:57.992 INFO:tasks.workunit.client.1.smithi203.stdout:4/989: mknod da/d29/d2e/d57/d96/ddc/de1/c147 0 2024-04-05T03:02:57.993 INFO:tasks.workunit.client.0.smithi153.stdout:5/637: rename d0/d25/d68/f99 to d0/d25/d4d/d9c/dc4/dcb/fe0 0 2024-04-05T03:02:57.997 INFO:tasks.workunit.client.1.smithi203.stdout:4/990: symlink da/d3c/d4c/l148 0 2024-04-05T03:02:57.998 INFO:tasks.workunit.client.0.smithi153.stdout:2/741: chown d9/da/d37/d8a/da8/led 78 1 2024-04-05T03:02:57.999 INFO:tasks.workunit.client.0.smithi153.stdout:2/742: write d9/da/d19/d2e/dc7/fc0 [2079567,91193] 0 2024-04-05T03:02:58.000 INFO:tasks.workunit.client.0.smithi153.stdout:8/604: symlink d7/d13/d19/d1b/d4f/d72/lc4 0 2024-04-05T03:02:58.005 INFO:tasks.workunit.client.1.smithi203.stdout:3/967: rename d0/dc/d1a/d1d/led to d0/dc/d1a/d8d/dc9/l122 0 2024-04-05T03:02:58.005 INFO:tasks.workunit.client.1.smithi203.stdout:4/991: dwrite f9 [8388608,4194304] 0 2024-04-05T03:02:58.006 INFO:tasks.workunit.client.0.smithi153.stdout:5/638: unlink d0/d25/d68/fae 0 2024-04-05T03:02:58.007 INFO:tasks.workunit.client.1.smithi203.stdout:3/968: readlink d0/dc/d1a/d8d/dc9/lf9 0 2024-04-05T03:02:58.007 INFO:tasks.workunit.client.1.smithi203.stdout:4/992: write f9 [3530285,74577] 0 2024-04-05T03:02:58.009 INFO:tasks.workunit.client.1.smithi203.stdout:4/993: readlink da/d29/d2e/d57/le5 0 2024-04-05T03:02:58.009 INFO:tasks.workunit.client.0.smithi153.stdout:8/605: symlink d7/d13/d19/d1b/d4f/d68/dbc/lc5 0 2024-04-05T03:02:58.009 INFO:tasks.workunit.client.0.smithi153.stdout:1/725: fdatasync d5/dd/d1b/fae 0 2024-04-05T03:02:58.013 INFO:tasks.workunit.client.1.smithi203.stdout:3/969: creat d0/dc/d1a/d1f/da1/daf/ddc/f123 x:0 0 0 2024-04-05T03:02:58.014 INFO:tasks.workunit.client.0.smithi153.stdout:1/726: read d5/d8/d20/d4b/d18/d16/d2c/f7d [3299514,13708] 0 2024-04-05T03:02:58.018 INFO:tasks.workunit.client.0.smithi153.stdout:5/639: unlink d0/d9/c71 0 2024-04-05T03:02:58.018 INFO:tasks.workunit.client.1.smithi203.stdout:3/970: symlink d0/dc/d1a/d1d/l124 0 2024-04-05T03:02:58.023 INFO:tasks.workunit.client.0.smithi153.stdout:5/640: mknod d0/d9/ce1 0 2024-04-05T03:02:58.023 INFO:tasks.workunit.client.1.smithi203.stdout:3/971: rename d0/dc/d1a/d1f/dac/d114 to d0/d69/d10f/d125 0 2024-04-05T03:02:58.024 INFO:tasks.workunit.client.1.smithi203.stdout:3/972: write d0/d21/d81/f59 [750515,4132] 0 2024-04-05T03:02:58.024 INFO:tasks.workunit.client.0.smithi153.stdout:8/606: write d7/d46/fa7 [2446997,94956] 0 2024-04-05T03:02:58.026 INFO:tasks.workunit.client.0.smithi153.stdout:8/607: fsync d7/d13/d19/d1b/d21/fbf 0 2024-04-05T03:02:58.026 INFO:tasks.workunit.client.0.smithi153.stdout:8/608: chown d7/d13/d19/d95 47 1 2024-04-05T03:02:58.029 INFO:tasks.workunit.client.0.smithi153.stdout:2/743: dread d9/da/d19/d2e/f8f [0,4194304] 0 2024-04-05T03:02:58.029 INFO:tasks.workunit.client.1.smithi203.stdout:3/973: creat d0/dc/d1a/d1f/da1/d10c/f126 x:0 0 0 2024-04-05T03:02:58.031 INFO:tasks.workunit.client.0.smithi153.stdout:5/641: getdents d0/d9 0 2024-04-05T03:02:58.031 INFO:tasks.workunit.client.1.smithi203.stdout:3/974: readlink d0/dc/d1a/d76/d35/l7a 0 2024-04-05T03:02:58.031 INFO:tasks.workunit.client.1.smithi203.stdout:4/994: truncate da/d29/d2e/d53/f113 3557473 0 2024-04-05T03:02:58.032 INFO:tasks.workunit.client.0.smithi153.stdout:1/727: truncate d5/f76 148109 0 2024-04-05T03:02:58.032 INFO:tasks.workunit.client.0.smithi153.stdout:8/609: write d7/d13/d19/d1b/d4f/d96/fb7 [262660,83210] 0 2024-04-05T03:02:58.034 INFO:tasks.workunit.client.1.smithi203.stdout:3/975: mkdir d0/dc/d1a/ddf/d127 0 2024-04-05T03:02:58.035 INFO:tasks.workunit.client.0.smithi153.stdout:2/744: readlink d9/da/d36/l66 0 2024-04-05T03:02:58.035 INFO:tasks.workunit.client.0.smithi153.stdout:8/610: readlink d7/d13/d19/d1b/d4f/d72/d73/l9e 0 2024-04-05T03:02:58.036 INFO:tasks.workunit.client.0.smithi153.stdout:1/728: symlink d5/d8/d20/d4b/d51/da2/ldb 0 2024-04-05T03:02:58.036 INFO:tasks.workunit.client.0.smithi153.stdout:5/642: symlink d0/d25/d4d/d62/d5f/d86/le2 0 2024-04-05T03:02:58.037 INFO:tasks.workunit.client.0.smithi153.stdout:8/611: mknod d7/d13/d19/d1b/d6d/cc6 0 2024-04-05T03:02:58.038 INFO:tasks.workunit.client.1.smithi203.stdout:4/995: getdents da/d29/d2e/d57/da5/d127 0 2024-04-05T03:02:58.040 INFO:tasks.workunit.client.0.smithi153.stdout:5/643: truncate d0/d25/d4d/d9c/dc4/dcb/fdc 201095 0 2024-04-05T03:02:58.041 INFO:tasks.workunit.client.0.smithi153.stdout:8/612: mknod d7/d13/d19/d1b/d33/cc7 0 2024-04-05T03:02:58.043 INFO:tasks.workunit.client.0.smithi153.stdout:1/729: rmdir d5/d8/d20/d4b/d51 39 2024-04-05T03:02:58.043 INFO:tasks.workunit.client.0.smithi153.stdout:8/613: fsync d7/d13/d19/d1b/d4f/d5f/fb2 0 2024-04-05T03:02:58.045 INFO:tasks.workunit.client.0.smithi153.stdout:5/644: creat d0/d4e/fe3 x:0 0 0 2024-04-05T03:02:58.045 INFO:tasks.workunit.client.0.smithi153.stdout:1/730: chown d5/d8/d20/d4b/d18/d16/f22 127202873 1 2024-04-05T03:02:58.046 INFO:tasks.workunit.client.1.smithi203.stdout:3/976: truncate d0/dc/d38/f8f 2657738 0 2024-04-05T03:02:58.047 INFO:tasks.workunit.client.1.smithi203.stdout:4/996: write da/d3c/d4c/fb7 [930628,64341] 0 2024-04-05T03:02:58.048 INFO:tasks.workunit.client.0.smithi153.stdout:8/614: mkdir d7/d13/dc8 0 2024-04-05T03:02:58.050 INFO:tasks.workunit.client.1.smithi203.stdout:3/977: creat d0/dc/d1a/d1f/dac/f128 x:0 0 0 2024-04-05T03:02:58.051 INFO:tasks.workunit.client.1.smithi203.stdout:4/997: symlink da/d29/d2e/d57/da5/l149 0 2024-04-05T03:02:58.051 INFO:tasks.workunit.client.1.smithi203.stdout:4/998: chown da/d3c/d12a 2065 1 2024-04-05T03:02:58.052 INFO:tasks.workunit.client.0.smithi153.stdout:5/645: rename d0/d25/d4d/d9c/dc7/dc9/cdd to d0/d25/d4d/d9c/dc4/ce4 0 2024-04-05T03:02:58.053 INFO:tasks.workunit.client.0.smithi153.stdout:2/745: dread d9/da/d19/d2e/d7e/d53/fe3 [0,4194304] 0 2024-04-05T03:02:58.054 INFO:tasks.workunit.client.0.smithi153.stdout:1/731: link d5/d8/f6e d5/d8/d20/d4b/d18/d16/d45/d68/fdc 0 2024-04-05T03:02:58.055 INFO:tasks.workunit.client.1.smithi203.stdout:4/999: mkdir da/d29/d2e/d57/d96/dbc/df5/d138/d14a 0 2024-04-05T03:02:58.055 INFO:tasks.workunit.client.0.smithi153.stdout:5/646: write d0/d25/d4d/d9c/db7/fd5 [97330,6860] 0 2024-04-05T03:02:58.056 INFO:tasks.workunit.client.0.smithi153.stdout:2/746: symlink d9/da/d19/d2e/d7e/d53/lff 0 2024-04-05T03:02:58.057 INFO:tasks.workunit.client.1.smithi203.stdout:3/978: rename d0/dc/d1a/d8d/dc9/l122 to d0/dc/d38/d5d/l129 0 2024-04-05T03:02:58.057 INFO:tasks.workunit.client.0.smithi153.stdout:2/747: readlink d9/da/l25 0 2024-04-05T03:02:58.058 INFO:tasks.workunit.client.0.smithi153.stdout:8/615: link d7/d2a/l3c d7/d13/d19/d1b/d21/lc9 0 2024-04-05T03:02:58.059 INFO:tasks.workunit.client.0.smithi153.stdout:8/616: readlink d7/d13/lb6 0 2024-04-05T03:02:58.059 INFO:tasks.workunit.client.0.smithi153.stdout:1/732: creat d5/d8/d20/d4b/d18/d16/d45/fdd x:0 0 0 2024-04-05T03:02:58.060 INFO:tasks.workunit.client.1.smithi203.stdout:3/979: truncate d0/dc/d1a/d8d/d96/f6a 3906744 0 2024-04-05T03:02:58.060 INFO:tasks.workunit.client.0.smithi153.stdout:2/748: mknod d9/d68/d98/de9/c100 0 2024-04-05T03:02:58.060 INFO:tasks.workunit.client.1.smithi203.stdout:3/980: fsync d0/d41/fd4 0 2024-04-05T03:02:58.063 INFO:tasks.workunit.client.0.smithi153.stdout:8/617: read d7/d13/d19/d1b/d4f/f86 [3904849,80378] 0 2024-04-05T03:02:58.063 INFO:tasks.workunit.client.0.smithi153.stdout:5/647: dwrite d0/d25/f6e [0,4194304] 0 2024-04-05T03:02:58.063 INFO:tasks.workunit.client.1.smithi203.stdout:3/981: rmdir d0/d21/d81 39 2024-04-05T03:02:58.071 INFO:tasks.workunit.client.0.smithi153.stdout:1/733: rename d5/d8/d20/d2f/c7c to d5/d8/d20/d4b/d18/d16/d48/cde 0 2024-04-05T03:02:58.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:57 smithi153 ceph-mon[29381]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:58.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:57 smithi153 ceph-mon[29381]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:58.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:57 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1512693990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:02:58.072 INFO:tasks.workunit.client.0.smithi153.stdout:5/648: dwrite d0/d2c/fbe [0,4194304] 0 2024-04-05T03:02:58.073 INFO:tasks.workunit.client.0.smithi153.stdout:1/734: chown d5/dd/f5d 9431371 1 2024-04-05T03:02:58.073 INFO:tasks.workunit.client.0.smithi153.stdout:8/618: mknod d7/d13/d19/d1b/cca 0 2024-04-05T03:02:58.074 INFO:tasks.workunit.client.0.smithi153.stdout:5/649: dread - d0/d25/d4d/d9c/dc7/dc9/fcc zero size 2024-04-05T03:02:58.075 INFO:tasks.workunit.client.0.smithi153.stdout:2/749: getdents d9/da/d19/d2e/d7e/d7a/d59 0 2024-04-05T03:02:58.076 INFO:tasks.workunit.client.0.smithi153.stdout:2/750: rename d9/d68/d98 to d9/d68/d98/da3/d101 22 2024-04-05T03:02:58.076 INFO:tasks.workunit.client.0.smithi153.stdout:1/735: mknod d5/d8/d20/d4b/d18/d16/d45/cdf 0 2024-04-05T03:02:58.077 INFO:tasks.workunit.client.0.smithi153.stdout:2/751: stat d9/da/d37 0 2024-04-05T03:02:58.078 INFO:tasks.workunit.client.0.smithi153.stdout:5/650: mknod d0/d25/d4d/d9c/dc4/ce5 0 2024-04-05T03:02:58.081 INFO:tasks.workunit.client.0.smithi153.stdout:1/736: unlink d5/d8/d20/d4b/d18/d16/d2c/cc6 0 2024-04-05T03:02:58.082 INFO:tasks.workunit.client.0.smithi153.stdout:2/752: mkdir d9/da/d19/d2e/d7e/d102 0 2024-04-05T03:02:58.082 INFO:tasks.workunit.client.0.smithi153.stdout:1/737: stat d5/d8/d20/d4b/d18/d16/d48/d96/fbe 0 2024-04-05T03:02:58.083 INFO:tasks.workunit.client.0.smithi153.stdout:5/651: getdents d0/d25/d4d/d62/d5f/d86/dd7 0 2024-04-05T03:02:58.085 INFO:tasks.workunit.client.0.smithi153.stdout:2/753: mkdir d9/da/d36/def/d103 0 2024-04-05T03:02:58.085 INFO:tasks.workunit.client.0.smithi153.stdout:1/738: truncate d5/d8/d20/d4b/d18/d16/d45/fa3 662360 0 2024-04-05T03:02:58.086 INFO:tasks.workunit.client.0.smithi153.stdout:2/754: read d9/da/d19/d2e/d4f/f86 [1035425,131068] 0 2024-04-05T03:02:58.088 INFO:tasks.workunit.client.0.smithi153.stdout:1/739: write d5/d8/d20/d4b/d18/d16/d45/d68/f77 [991343,35381] 0 2024-04-05T03:02:58.090 INFO:tasks.workunit.client.0.smithi153.stdout:2/755: symlink d9/da/d36/l104 0 2024-04-05T03:02:58.090 INFO:tasks.workunit.client.0.smithi153.stdout:1/740: write d5/d8/d20/d4b/d18/d16/f36 [2250696,14202] 0 2024-04-05T03:02:58.091 INFO:tasks.workunit.client.0.smithi153.stdout:2/756: chown d9/da/d19/c6c 1841527 1 2024-04-05T03:02:58.092 INFO:tasks.workunit.client.0.smithi153.stdout:2/757: write d9/da/d19/d2e/d7e/de6/f6b [732872,113730] 0 2024-04-05T03:02:58.092 INFO:tasks.workunit.client.0.smithi153.stdout:1/741: mknod d5/d8/d20/d2f/db8/ce0 0 2024-04-05T03:02:58.094 INFO:tasks.workunit.client.0.smithi153.stdout:1/742: write d5/d8/d20/d4b/d18/d16/d48/f99 [1108764,18526] 0 2024-04-05T03:02:58.094 INFO:tasks.workunit.client.0.smithi153.stdout:2/758: creat d9/d68/d98/f105 x:0 0 0 2024-04-05T03:02:58.095 INFO:tasks.workunit.client.0.smithi153.stdout:2/759: chown d9/da/d37/d8a 4128 1 2024-04-05T03:02:58.097 INFO:tasks.workunit.client.0.smithi153.stdout:2/760: getdents d9/da/df9 0 2024-04-05T03:02:58.097 INFO:tasks.workunit.client.0.smithi153.stdout:2/761: fsync d9/da/f64 0 2024-04-05T03:02:58.100 INFO:tasks.workunit.client.0.smithi153.stdout:2/762: getdents d9 0 2024-04-05T03:02:58.100 INFO:tasks.workunit.client.0.smithi153.stdout:2/763: readlink d9/da/d19/d2e/l31 0 2024-04-05T03:02:58.102 INFO:tasks.workunit.client.0.smithi153.stdout:0/662: sync 2024-04-05T03:02:58.102 INFO:tasks.workunit.client.0.smithi153.stdout:4/806: sync 2024-04-05T03:02:58.106 INFO:tasks.workunit.client.1.smithi203.stdout:3/982: mknod d0/d21/c12a 0 2024-04-05T03:02:58.106 INFO:tasks.workunit.client.0.smithi153.stdout:4/807: read - da/db/d20/fda zero size 2024-04-05T03:02:58.107 INFO:tasks.workunit.client.1.smithi203.stdout:3/983: chown d0/c55 1 1 2024-04-05T03:02:58.107 INFO:tasks.workunit.client.1.smithi203.stdout:3/984: fdatasync d0/d21/d83/fd7 0 2024-04-05T03:02:58.108 INFO:tasks.workunit.client.1.smithi203.stdout:3/985: chown d0/dc/d1a 56873 1 2024-04-05T03:02:58.111 INFO:tasks.workunit.client.1.smithi203.stdout:3/986: creat d0/d21/d83/f12b x:0 0 0 2024-04-05T03:02:58.112 INFO:tasks.workunit.client.1.smithi203.stdout:3/987: chown d0/dc/d38/dca/fd1 256530 1 2024-04-05T03:02:58.113 INFO:tasks.workunit.client.0.smithi153.stdout:0/663: creat d9/db/d42/d7e/db6/d1e/d45/d62/fcc x:0 0 0 2024-04-05T03:02:58.118 INFO:tasks.workunit.client.0.smithi153.stdout:0/664: mknod d9/d14/ccd 0 2024-04-05T03:02:58.118 INFO:tasks.workunit.client.0.smithi153.stdout:2/764: dwrite d9/da/d19/d2e/d7e/d53/f83 [0,4194304] 0 2024-04-05T03:02:58.119 INFO:tasks.workunit.client.0.smithi153.stdout:8/619: write d7/f16 [1406525,18441] 0 2024-04-05T03:02:58.124 INFO:tasks.workunit.client.0.smithi153.stdout:1/743: dread d5/d8/d20/d4b/f11 [0,4194304] 0 2024-04-05T03:02:58.124 INFO:tasks.workunit.client.1.smithi203.stdout:3/988: truncate d0/d41/f9a 1162796 0 2024-04-05T03:02:58.125 INFO:tasks.workunit.client.0.smithi153.stdout:4/808: dread da/de/da1/fc0 [4194304,4194304] 0 2024-04-05T03:02:58.128 INFO:tasks.workunit.client.0.smithi153.stdout:2/765: dread d9/da/d19/d2e/d7e/d7a/f57 [0,4194304] 0 2024-04-05T03:02:58.129 INFO:tasks.workunit.client.0.smithi153.stdout:8/620: mkdir d7/d13/d19/d95/dcb 0 2024-04-05T03:02:58.131 INFO:tasks.workunit.client.0.smithi153.stdout:5/652: dwrite d0/f75 [0,4194304] 0 2024-04-05T03:02:58.131 INFO:tasks.workunit.client.0.smithi153.stdout:2/766: chown d9/d77/c4c 31263 1 2024-04-05T03:02:58.131 INFO:tasks.workunit.client.1.smithi203.stdout:3/989: unlink d0/d21/d81/le0 0 2024-04-05T03:02:58.134 INFO:tasks.workunit.client.0.smithi153.stdout:4/809: creat da/de/d66/d111/d86/f119 x:0 0 0 2024-04-05T03:02:58.137 INFO:tasks.workunit.client.0.smithi153.stdout:2/767: creat d9/d68/db9/f106 x:0 0 0 2024-04-05T03:02:58.140 INFO:tasks.workunit.client.0.smithi153.stdout:4/810: symlink da/de/d56/d78/d8f/l11a 0 2024-04-05T03:02:58.140 INFO:tasks.workunit.client.1.smithi203.stdout:3/990: link d0/dc/d1a/d1d/c1e d0/dc/d38/d5d/d66/c12c 0 2024-04-05T03:02:58.141 INFO:tasks.workunit.client.0.smithi153.stdout:8/621: creat d7/d13/dc8/fcc x:0 0 0 2024-04-05T03:02:58.141 INFO:tasks.workunit.client.0.smithi153.stdout:5/653: dwrite d0/d25/d4d/d9c/dc4/dcb/fdc [0,4194304] 0 2024-04-05T03:02:58.145 INFO:tasks.workunit.client.0.smithi153.stdout:8/622: write d7/d2a/f2d [931316,76484] 0 2024-04-05T03:02:58.145 INFO:tasks.workunit.client.0.smithi153.stdout:2/768: creat d9/da/d19/d2e/d7e/d102/f107 x:0 0 0 2024-04-05T03:02:58.146 INFO:tasks.workunit.client.0.smithi153.stdout:4/811: truncate da/db/d20/d35/d38/d76/f115 884318 0 2024-04-05T03:02:58.146 INFO:tasks.workunit.client.0.smithi153.stdout:8/623: chown d7/d13/d19/d1b/d33/c93 0 1 2024-04-05T03:02:58.147 INFO:tasks.workunit.client.0.smithi153.stdout:4/812: readlink da/db/d20/d35/d41/d67/l8d 0 2024-04-05T03:02:58.147 INFO:tasks.workunit.client.0.smithi153.stdout:4/813: fsync da/de/d66/ffd 0 2024-04-05T03:02:58.150 INFO:tasks.workunit.client.0.smithi153.stdout:2/769: creat d9/da/d94/dca/df7/f108 x:0 0 0 2024-04-05T03:02:58.151 INFO:tasks.workunit.client.0.smithi153.stdout:2/770: write d9/d68/fba [5098600,88] 0 2024-04-05T03:02:58.152 INFO:tasks.workunit.client.0.smithi153.stdout:8/624: chown d7/d85/f4a 28 1 2024-04-05T03:02:58.152 INFO:tasks.workunit.client.0.smithi153.stdout:2/771: stat d9/d68/d98/da3 0 2024-04-05T03:02:58.155 INFO:tasks.workunit.client.0.smithi153.stdout:8/625: mknod d7/d13/d19/d1b/d6c/ccd 0 2024-04-05T03:02:58.159 INFO:tasks.workunit.client.0.smithi153.stdout:1/744: truncate d5/d8/d20/d4b/d1e/f8f 4038815 0 2024-04-05T03:02:58.159 INFO:tasks.workunit.client.0.smithi153.stdout:4/814: dwrite da/de/f1e [0,4194304] 0 2024-04-05T03:02:58.160 INFO:tasks.workunit.client.0.smithi153.stdout:5/654: truncate d0/d9/f1e 255612 0 2024-04-05T03:02:58.162 INFO:tasks.workunit.client.0.smithi153.stdout:2/772: dwrite d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:02:58.164 INFO:tasks.workunit.client.0.smithi153.stdout:2/773: readlink d9/da/d94/ld2 0 2024-04-05T03:02:58.165 INFO:tasks.workunit.client.0.smithi153.stdout:5/655: write d0/f14 [2967361,40568] 0 2024-04-05T03:02:58.170 INFO:tasks.workunit.client.0.smithi153.stdout:8/626: dwrite d7/d13/dc8/fcc [0,4194304] 0 2024-04-05T03:02:58.170 INFO:tasks.workunit.client.0.smithi153.stdout:2/774: readlink d9/da/df9/lfd 0 2024-04-05T03:02:58.170 INFO:tasks.workunit.client.0.smithi153.stdout:1/745: chown d5/d8/d20/d4b/d18/d16/d33/d83/f8e 7 1 2024-04-05T03:02:58.173 INFO:tasks.workunit.client.0.smithi153.stdout:2/775: unlink d9/d77/c41 0 2024-04-05T03:02:58.174 INFO:tasks.workunit.client.0.smithi153.stdout:4/815: dwrite da/db/d20/d45/f4e [4194304,4194304] 0 2024-04-05T03:02:58.175 INFO:tasks.workunit.client.0.smithi153.stdout:2/776: write d9/da/d19/d2e/d7e/d7a/fda [1030548,125229] 0 2024-04-05T03:02:58.177 INFO:tasks.workunit.client.0.smithi153.stdout:5/656: fsync d0/d25/d4d/d62/d5f/d86/da4/fa1 0 2024-04-05T03:02:58.183 INFO:tasks.workunit.client.0.smithi153.stdout:4/816: symlink da/db/d20/d35/d38/l11b 0 2024-04-05T03:02:58.186 INFO:tasks.workunit.client.0.smithi153.stdout:5/657: getdents d0/d25 0 2024-04-05T03:02:58.187 INFO:tasks.workunit.client.0.smithi153.stdout:5/658: write d0/d25/d4d/d9c/dc7/fde [987595,59511] 0 2024-04-05T03:02:58.189 INFO:tasks.workunit.client.0.smithi153.stdout:5/659: rmdir d0/d25/d4d/d62/d37/d38 39 2024-04-05T03:02:58.195 INFO:tasks.workunit.client.0.smithi153.stdout:5/660: dwrite d0/d25/f6e [0,4194304] 0 2024-04-05T03:02:58.196 INFO:tasks.workunit.client.0.smithi153.stdout:5/661: chown d0/d25/d68/f9b 9446 1 2024-04-05T03:02:58.198 INFO:tasks.workunit.client.0.smithi153.stdout:5/662: symlink d0/d25/le6 0 2024-04-05T03:02:58.205 INFO:tasks.workunit.client.0.smithi153.stdout:4/817: write da/f1d [177382,67891] 0 2024-04-05T03:02:58.210 INFO:tasks.workunit.client.0.smithi153.stdout:5/663: mknod d0/d25/d68/ce7 0 2024-04-05T03:02:58.210 INFO:tasks.workunit.client.0.smithi153.stdout:1/746: dwrite d5/d8/d20/d4b/d18/d16/d45/d68/fdc [0,4194304] 0 2024-04-05T03:02:58.215 INFO:tasks.workunit.client.0.smithi153.stdout:1/747: rmdir d5/d8/d20/d4b/d18/d61 39 2024-04-05T03:02:58.219 INFO:tasks.workunit.client.0.smithi153.stdout:5/664: dwrite d0/f75 [4194304,4194304] 0 2024-04-05T03:02:58.220 INFO:tasks.workunit.client.0.smithi153.stdout:4/818: rename da/db/d20/d35/d38/d76/d90/fd6 to da/de/d56/d6a/d2d/f11c 0 2024-04-05T03:02:58.221 INFO:tasks.workunit.client.0.smithi153.stdout:4/819: read - da/db/d20/d35/d41/f62 zero size 2024-04-05T03:02:58.226 INFO:tasks.workunit.client.1.smithi203.stdout:3/991: write d0/dc/d1a/d8d/d96/f90 [279769,112402] 0 2024-04-05T03:02:58.228 INFO:tasks.workunit.client.1.smithi203.stdout:3/992: truncate d0/d69/d10f/d125/f121 1036727 0 2024-04-05T03:02:58.231 INFO:tasks.workunit.client.1.smithi203.stdout:3/993: fdatasync d0/d21/f45 0 2024-04-05T03:02:58.235 INFO:tasks.workunit.client.0.smithi153.stdout:4/820: truncate da/de/d56/d6a/d2d/f11c 1173117 0 2024-04-05T03:02:58.237 INFO:tasks.workunit.client.1.smithi203.stdout:3/994: rename d0/dc/d1a/d8d/d96 to d0/dc/d38/d5d/d66/d12d 0 2024-04-05T03:02:58.238 INFO:tasks.workunit.client.0.smithi153.stdout:4/821: creat da/db/d20/d35/d38/d76/f11d x:0 0 0 2024-04-05T03:02:58.240 INFO:tasks.workunit.client.1.smithi203.stdout:3/995: dread d0/dc/ff3 [0,4194304] 0 2024-04-05T03:02:58.241 INFO:tasks.workunit.client.0.smithi153.stdout:4/822: creat da/de/d66/d111/d86/f11e x:0 0 0 2024-04-05T03:02:58.241 INFO:tasks.workunit.client.1.smithi203.stdout:3/996: chown d0/dc/d38/d5d/cf5 0 1 2024-04-05T03:02:58.241 INFO:tasks.workunit.client.0.smithi153.stdout:4/823: write da/f1d [1723401,83770] 0 2024-04-05T03:02:58.242 INFO:tasks.workunit.client.1.smithi203.stdout:3/997: write d0/d21/d81/f7b [838062,52701] 0 2024-04-05T03:02:58.242 INFO:tasks.workunit.client.0.smithi153.stdout:4/824: fdatasync da/db/f79 0 2024-04-05T03:02:58.243 INFO:tasks.workunit.client.0.smithi153.stdout:4/825: write da/f103 [79831,53097] 0 2024-04-05T03:02:58.244 INFO:tasks.workunit.client.1.smithi203.stdout:3/998: creat d0/dc/d1a/d8d/dc9/f12e x:0 0 0 2024-04-05T03:02:58.245 INFO:tasks.workunit.client.1.smithi203.stdout:3/999: stat d0/dc/d1a/d1d/l11d 0 2024-04-05T03:02:58.246 INFO:tasks.workunit.client.0.smithi153.stdout:4/826: unlink da/db/d20/f32 0 2024-04-05T03:02:58.248 INFO:tasks.workunit.client.0.smithi153.stdout:4/827: stat da/de/d56/d78/d8f/d68/ff2 0 2024-04-05T03:02:58.249 INFO:tasks.workunit.client.1.smithi203.stderr:+ rm -rf -- ./tmp.ZzKx9mB6wg 2024-04-05T03:02:58.251 INFO:tasks.workunit.client.0.smithi153.stdout:4/828: getdents da/db/d20/d3c/d91/dd5 0 2024-04-05T03:02:58.253 INFO:tasks.workunit.client.0.smithi153.stdout:4/829: chown da/db/d20/d3c/d81/lf9 0 1 2024-04-05T03:02:58.255 INFO:tasks.workunit.client.0.smithi153.stdout:8/627: dread d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:02:58.255 INFO:tasks.workunit.client.0.smithi153.stdout:4/830: mkdir da/db/d8b/d10f/d11f 0 2024-04-05T03:02:58.260 INFO:tasks.workunit.client.0.smithi153.stdout:4/831: creat da/db/d8b/d10f/d11f/f120 x:0 0 0 2024-04-05T03:02:58.262 INFO:tasks.workunit.client.0.smithi153.stdout:1/748: dread f4 [8388608,4194304] 0 2024-04-05T03:02:58.264 INFO:tasks.workunit.client.0.smithi153.stdout:8/628: dwrite d7/fa [4194304,4194304] 0 2024-04-05T03:02:58.265 INFO:tasks.workunit.client.0.smithi153.stdout:4/832: rmdir da/db/d20/d35 39 2024-04-05T03:02:58.266 INFO:tasks.workunit.client.0.smithi153.stdout:4/833: readlink da/db/d20/d3c/led 0 2024-04-05T03:02:58.267 INFO:tasks.workunit.client.0.smithi153.stdout:1/749: chown d5/d8/d20/d4b/d18/d16/d45/d68/l8a 324095 1 2024-04-05T03:02:58.269 INFO:tasks.workunit.client.0.smithi153.stdout:5/665: dread d0/d2c/f3a [0,4194304] 0 2024-04-05T03:02:58.271 INFO:tasks.workunit.client.0.smithi153.stdout:1/750: mkdir d5/d8/d20/d4b/de1 0 2024-04-05T03:02:58.273 INFO:tasks.workunit.client.0.smithi153.stdout:4/834: write da/de/d56/d6a/d2d/f2e [594795,10268] 0 2024-04-05T03:02:58.274 INFO:tasks.workunit.client.0.smithi153.stdout:5/666: chown d0/d25/d4d/d62/dbb 79 1 2024-04-05T03:02:58.274 INFO:tasks.workunit.client.0.smithi153.stdout:8/629: dwrite d7/f16 [0,4194304] 0 2024-04-05T03:02:58.276 INFO:tasks.workunit.client.0.smithi153.stdout:1/751: fsync d5/d8/d20/d4b/d51/da2/f92 0 2024-04-05T03:02:58.279 INFO:tasks.workunit.client.0.smithi153.stdout:1/752: fsync d5/d8/d20/d4b/d18/d16/d45/d68/fdc 0 2024-04-05T03:02:58.280 INFO:tasks.workunit.client.0.smithi153.stdout:1/753: chown d5/d8/d20/d2f 986363 1 2024-04-05T03:02:58.280 INFO:tasks.workunit.client.0.smithi153.stdout:5/667: mknod d0/d25/d4d/d9c/ce8 0 2024-04-05T03:02:58.281 INFO:tasks.workunit.client.0.smithi153.stdout:5/668: chown d0/d9/d22/c7d 3476105 1 2024-04-05T03:02:58.282 INFO:tasks.workunit.client.0.smithi153.stdout:8/630: getdents d7/d13/d19/d95 0 2024-04-05T03:02:58.284 INFO:tasks.workunit.client.0.smithi153.stdout:6/832: sync 2024-04-05T03:02:58.284 INFO:tasks.workunit.client.0.smithi153.stdout:8/631: write d7/d13/dc8/fcc [5142219,117907] 0 2024-04-05T03:02:58.285 INFO:tasks.workunit.client.0.smithi153.stdout:1/754: mkdir d5/de2 0 2024-04-05T03:02:58.289 INFO:tasks.workunit.client.0.smithi153.stdout:5/669: dread d0/d25/d4d/f8d [4194304,4194304] 0 2024-04-05T03:02:58.294 INFO:tasks.workunit.client.0.smithi153.stdout:5/670: dread d0/d9/d22/fc2 [0,4194304] 0 2024-04-05T03:02:58.294 INFO:tasks.workunit.client.0.smithi153.stdout:1/755: link d5/dd/f13 d5/d8/d20/d4b/d18/d16/d45/d68/da4/fe3 0 2024-04-05T03:02:58.295 INFO:tasks.workunit.client.0.smithi153.stdout:5/671: rmdir d0/d2c/d3f/da8 39 2024-04-05T03:02:58.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:57 smithi203 ceph-mon[39911]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:58.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:57 smithi203 ceph-mon[39911]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:02:58.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:57 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1512693990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:02:58.297 INFO:tasks.workunit.client.0.smithi153.stdout:8/632: write d7/f8 [1421110,73988] 0 2024-04-05T03:02:58.297 INFO:tasks.workunit.client.0.smithi153.stdout:6/833: truncate d4/d38/d4c/fa5 1393911 0 2024-04-05T03:02:58.298 INFO:tasks.workunit.client.0.smithi153.stdout:5/672: mknod d0/d2c/d5e/ce9 0 2024-04-05T03:02:58.300 INFO:tasks.workunit.client.0.smithi153.stdout:5/673: chown d0/d25/d4d/d9c/fd8 26815 1 2024-04-05T03:02:58.300 INFO:tasks.workunit.client.0.smithi153.stdout:1/756: mknod d5/d8/d20/d4b/d18/d16/d45/d68/da4/ce4 0 2024-04-05T03:02:58.304 INFO:tasks.workunit.client.0.smithi153.stdout:1/757: dread - d5/d8/d20/d4b/d18/fb2 zero size 2024-04-05T03:02:58.312 INFO:tasks.workunit.client.0.smithi153.stdout:5/674: dwrite d0/f27 [0,4194304] 0 2024-04-05T03:02:58.315 INFO:tasks.workunit.client.0.smithi153.stdout:6/834: dread d4/dc/df/fb6 [0,4194304] 0 2024-04-05T03:02:58.316 INFO:tasks.workunit.client.0.smithi153.stdout:5/675: symlink d0/d25/d4d/d9c/dc7/lea 0 2024-04-05T03:02:58.316 INFO:tasks.workunit.client.0.smithi153.stdout:8/633: dwrite d7/d2a/fa2 [0,4194304] 0 2024-04-05T03:02:58.320 INFO:tasks.workunit.client.0.smithi153.stdout:8/634: write d7/d13/dc8/fcc [3277980,1490] 0 2024-04-05T03:02:58.321 INFO:tasks.workunit.client.0.smithi153.stdout:5/676: symlink d0/d25/d4d/d9c/db7/leb 0 2024-04-05T03:02:58.322 INFO:tasks.workunit.client.0.smithi153.stdout:6/835: symlink d4/dc/df/d16/d36/d4d/l120 0 2024-04-05T03:02:58.324 INFO:tasks.workunit.client.0.smithi153.stdout:8/635: creat d7/d13/d19/d1b/d4f/d68/fce x:0 0 0 2024-04-05T03:02:58.326 INFO:tasks.workunit.client.0.smithi153.stdout:5/677: creat d0/d25/d4d/fec x:0 0 0 2024-04-05T03:02:58.330 INFO:tasks.workunit.client.0.smithi153.stdout:6/836: creat d4/dc/df/d16/d44/f121 x:0 0 0 2024-04-05T03:02:58.330 INFO:tasks.workunit.client.0.smithi153.stdout:8/636: creat d7/d2a/fcf x:0 0 0 2024-04-05T03:02:58.334 INFO:tasks.workunit.client.0.smithi153.stdout:8/637: truncate d7/d13/d19/d1b/d21/fbf 29698 0 2024-04-05T03:02:58.334 INFO:tasks.workunit.client.0.smithi153.stdout:5/678: rename d0/d2c/f3a to d0/fed 0 2024-04-05T03:02:58.336 INFO:tasks.workunit.client.0.smithi153.stdout:1/758: dread d5/dd/d1b/f74 [0,4194304] 0 2024-04-05T03:02:58.344 INFO:tasks.workunit.client.0.smithi153.stdout:8/638: link d7/d2a/l40 d7/d13/d19/d1b/d4f/d5f/d54/ld0 0 2024-04-05T03:02:58.345 INFO:tasks.workunit.client.0.smithi153.stdout:6/837: dwrite d4/dc/df/d16/d24/d4e/fa9 [0,4194304] 0 2024-04-05T03:02:58.348 INFO:tasks.workunit.client.0.smithi153.stdout:5/679: dwrite d0/d25/d4d/d62/f56 [0,4194304] 0 2024-04-05T03:02:58.351 INFO:tasks.workunit.client.0.smithi153.stdout:6/838: creat d4/dc/df/d16/d36/d10e/d11b/f122 x:0 0 0 2024-04-05T03:02:58.353 INFO:tasks.workunit.client.0.smithi153.stdout:5/680: fsync d0/d2c/f2d 0 2024-04-05T03:02:58.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.354+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90f004bed0 con 0x7f90dc06b610 2024-04-05T03:02:58.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.357+0000 7f90e77fe700 1 -- 172.21.15.153:0/176896510 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f90f004bed0 con 0x7f90dc06b610 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "", 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:02:58.357 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:02:58.358 INFO:tasks.workunit.client.0.smithi153.stdout:8/639: dwrite d7/d85/f6a [0,4194304] 0 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 msgr2=0x7f90dc06dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 0x7f90dc06dac0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f90d8004600 tx=0x7f90d800b040 comp rx=0 tx=0).stop 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 msgr2=0x7f90f0114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f90e000b0d0 tx=0x7f90e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 shutdown_connections 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90f0072820 0x7f90f0114010 secure :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f90e000b0d0 tx=0x7f90e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f90dc06b610 0x7f90dc06dac0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:58.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 --2- 172.21.15.153:0/176896510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90f0114550 0x7f90f01153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:02:58.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.361+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 >> 172.21.15.153:0/176896510 conn(0x7f90f00fa950 msgr2=0x7f90f00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:02:58.362 INFO:tasks.workunit.client.0.smithi153.stdout:6/839: mknod d4/d38/d4c/c123 0 2024-04-05T03:02:58.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.362+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 shutdown_connections 2024-04-05T03:02:58.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:02:58.362+0000 7f90e4d99700 1 -- 172.21.15.153:0/176896510 wait complete. 2024-04-05T03:02:58.365 INFO:tasks.workunit.client.0.smithi153.stdout:8/640: unlink d7/d2a/f98 0 2024-04-05T03:02:58.366 INFO:tasks.workunit.client.0.smithi153.stdout:5/681: mknod d0/d25/d4d/d62/d5f/d86/dd7/cee 0 2024-04-05T03:02:58.371 INFO:tasks.workunit.client.0.smithi153.stdout:5/682: read d0/d25/d4d/d9c/dc4/dcb/fe0 [554905,8933] 0 2024-04-05T03:02:58.372 INFO:tasks.workunit.client.0.smithi153.stdout:8/641: truncate d7/d13/d19/f4e 3294976 0 2024-04-05T03:02:58.373 INFO:tasks.workunit.client.0.smithi153.stdout:6/840: read d4/dc/df/d16/d36/fb2 [3067815,98432] 0 2024-04-05T03:02:58.374 INFO:tasks.workunit.client.0.smithi153.stdout:8/642: stat d7/d2a/d50/c88 0 2024-04-05T03:02:58.374 INFO:tasks.workunit.client.0.smithi153.stdout:5/683: write d0/d25/d4d/d62/d5f/d86/da4/fa1 [790435,66171] 0 2024-04-05T03:02:58.376 INFO:tasks.workunit.client.0.smithi153.stdout:6/841: write d4/dc/df/d16/d44/fd9 [967590,22693] 0 2024-04-05T03:02:58.378 INFO:tasks.workunit.client.0.smithi153.stdout:6/842: fdatasync d4/f6 0 2024-04-05T03:02:58.381 INFO:tasks.workunit.client.0.smithi153.stdout:5/684: creat d0/d25/d4d/d9c/fef x:0 0 0 2024-04-05T03:02:58.382 INFO:tasks.workunit.client.0.smithi153.stdout:5/685: readlink d0/d25/d4d/d62/dbb/lbd 0 2024-04-05T03:02:58.383 INFO:tasks.workunit.client.0.smithi153.stdout:6/843: rename d4/dc/df/d16/c10a to d4/dc/df/d16/d24/d5e/de8/dfa/c124 0 2024-04-05T03:02:58.388 INFO:tasks.workunit.client.0.smithi153.stdout:5/686: getdents d0/d25/d4d/d9c/db7 0 2024-04-05T03:02:58.395 INFO:tasks.workunit.client.0.smithi153.stdout:8/643: dread d7/d13/d19/d1b/d33/f3a [0,4194304] 0 2024-04-05T03:02:58.398 INFO:tasks.workunit.client.0.smithi153.stdout:8/644: fsync d7/ff 0 2024-04-05T03:02:58.402 INFO:tasks.workunit.client.0.smithi153.stdout:8/645: rename d7/d2a/c55 to d7/d13/d19/d95/dcb/cd1 0 2024-04-05T03:02:58.404 INFO:tasks.workunit.client.0.smithi153.stdout:6/844: dread d4/dc/df/d16/d5f/fb7 [4194304,4194304] 0 2024-04-05T03:02:58.404 INFO:tasks.workunit.client.0.smithi153.stdout:8/646: creat d7/d2a/d8b/fd2 x:0 0 0 2024-04-05T03:02:58.405 INFO:tasks.workunit.client.0.smithi153.stdout:8/647: fdatasync f3 0 2024-04-05T03:02:58.407 INFO:tasks.workunit.client.0.smithi153.stdout:8/648: write d7/d13/d19/d1b/d4f/d68/fce [495985,115549] 0 2024-04-05T03:02:58.412 INFO:tasks.workunit.client.0.smithi153.stdout:8/649: link f6 d7/d13/d19/d1b/d4f/d68/dbc/fd3 0 2024-04-05T03:02:58.413 INFO:tasks.workunit.client.0.smithi153.stdout:8/650: dread - d7/d13/d19/d1b/d6d/fae zero size 2024-04-05T03:02:58.418 INFO:tasks.workunit.client.0.smithi153.stdout:8/651: symlink d7/d13/d19/d1b/d4f/d68/dbc/ld4 0 2024-04-05T03:02:58.420 INFO:tasks.workunit.client.0.smithi153.stdout:6/845: dwrite d4/dc/df/d16/d44/fa1 [0,4194304] 0 2024-04-05T03:02:58.423 INFO:tasks.workunit.client.0.smithi153.stdout:8/652: dread d7/d2a/fa2 [0,4194304] 0 2024-04-05T03:02:58.425 INFO:tasks.workunit.client.0.smithi153.stdout:8/653: dread - d7/d2a/f37 zero size 2024-04-05T03:02:58.426 INFO:tasks.workunit.client.0.smithi153.stdout:8/654: write d7/d13/d19/d1b/d4f/d5f/d54/fc3 [318664,90208] 0 2024-04-05T03:02:58.427 INFO:tasks.workunit.client.0.smithi153.stdout:6/846: mkdir d4/dc/df/d16/d24/d4e/d125 0 2024-04-05T03:02:58.430 INFO:tasks.workunit.client.0.smithi153.stdout:6/847: truncate d4/dc/df/d16/fcc 235347 0 2024-04-05T03:02:58.433 INFO:tasks.workunit.client.0.smithi153.stdout:6/848: creat d4/dc/df/dcb/d119/f126 x:0 0 0 2024-04-05T03:02:58.437 INFO:tasks.workunit.client.0.smithi153.stdout:6/849: link d4/d38/ce4 d4/dc/df/d16/c127 0 2024-04-05T03:02:58.438 INFO:tasks.workunit.client.0.smithi153.stdout:8/655: truncate d7/d13/d19/d1b/d4f/d68/f75 1713138 0 2024-04-05T03:02:58.440 INFO:tasks.workunit.client.0.smithi153.stdout:6/850: creat d4/dc/df/d16/d33/f128 x:0 0 0 2024-04-05T03:02:58.445 INFO:tasks.workunit.client.0.smithi153.stdout:6/851: rename d4/dc/df/dcb/d119 to d4/dc/df/d16/d129 0 2024-04-05T03:02:58.447 INFO:tasks.workunit.client.0.smithi153.stdout:6/852: chown d4/dc/d7a/d9c/fc3 95465 1 2024-04-05T03:02:58.448 INFO:tasks.workunit.client.0.smithi153.stdout:6/853: stat d4/dc/df/d16/d36/d4d/l120 0 2024-04-05T03:02:58.449 INFO:tasks.workunit.client.0.smithi153.stdout:3/711: sync 2024-04-05T03:02:58.451 INFO:tasks.workunit.client.0.smithi153.stdout:8/656: write d7/d13/d19/d1b/d33/f3a [2238059,52342] 0 2024-04-05T03:02:58.453 INFO:tasks.workunit.client.0.smithi153.stdout:6/854: write d4/dc/df/d16/d24/f29 [2586869,122706] 0 2024-04-05T03:02:58.455 INFO:tasks.workunit.client.0.smithi153.stdout:8/657: mkdir d7/dd5 0 2024-04-05T03:02:58.457 INFO:tasks.workunit.client.0.smithi153.stdout:3/712: rmdir d1d/d32/d93/d3c/d3f/d90/da4/ddb 0 2024-04-05T03:02:58.457 INFO:tasks.workunit.client.0.smithi153.stdout:3/713: stat d1d/d32/f56 0 2024-04-05T03:02:58.457 INFO:tasks.workunit.client.0.smithi153.stdout:6/855: creat d4/dc/df/d16/d17/d54/f12a x:0 0 0 2024-04-05T03:02:58.458 INFO:tasks.workunit.client.0.smithi153.stdout:8/658: creat d7/d13/d19/d1b/d6d/fd6 x:0 0 0 2024-04-05T03:02:58.460 INFO:tasks.workunit.client.0.smithi153.stdout:8/659: dread - d7/d13/d19/d1b/d6d/fd6 zero size 2024-04-05T03:02:58.465 INFO:tasks.workunit.client.0.smithi153.stdout:3/714: dwrite d1d/d27/f74 [0,4194304] 0 2024-04-05T03:02:58.467 INFO:tasks.workunit.client.0.smithi153.stdout:8/660: dread d7/d2a/f2d [0,4194304] 0 2024-04-05T03:02:58.468 INFO:tasks.workunit.client.0.smithi153.stdout:6/856: dwrite d4/dc/df/d16/d44/fdb [4194304,4194304] 0 2024-04-05T03:02:58.471 INFO:tasks.workunit.client.0.smithi153.stdout:8/661: dread d7/fd [0,4194304] 0 2024-04-05T03:02:58.476 INFO:tasks.workunit.client.0.smithi153.stdout:8/662: creat d7/d13/d19/d1b/db9/fd7 x:0 0 0 2024-04-05T03:02:58.480 INFO:tasks.workunit.client.0.smithi153.stdout:8/663: unlink d7/d2a/l40 0 2024-04-05T03:02:58.481 INFO:tasks.workunit.client.0.smithi153.stdout:6/857: dwrite d4/dc/df/f13 [0,4194304] 0 2024-04-05T03:02:58.482 INFO:tasks.workunit.client.0.smithi153.stdout:8/664: chown d7/d13/d19/d1b/c45 23683284 1 2024-04-05T03:02:58.487 INFO:tasks.workunit.client.0.smithi153.stdout:6/858: rename d4/dc/df/d16/d36/d4d/d64/f106 to d4/dc/df/d16/d24/d4e/d125/f12b 0 2024-04-05T03:02:58.488 INFO:tasks.workunit.client.0.smithi153.stdout:8/665: dread d7/d13/d19/d1b/d4f/d68/f75 [0,4194304] 0 2024-04-05T03:02:58.493 INFO:tasks.workunit.client.0.smithi153.stdout:8/666: rename d7/d2a/fa1 to d7/d13/d19/d1b/d4f/d72/d73/fd8 0 2024-04-05T03:02:58.495 INFO:tasks.workunit.client.0.smithi153.stdout:6/859: truncate d4/dc/df/d16/d17/f3a 1842827 0 2024-04-05T03:02:58.498 INFO:tasks.workunit.client.0.smithi153.stdout:3/715: dread d1d/d32/f49 [0,4194304] 0 2024-04-05T03:02:58.498 INFO:tasks.workunit.client.0.smithi153.stdout:3/716: chown d1d/ce5 257 1 2024-04-05T03:02:58.503 INFO:tasks.workunit.client.0.smithi153.stdout:3/717: symlink d1d/d32/d93/dc2/lfc 0 2024-04-05T03:02:58.506 INFO:tasks.workunit.client.0.smithi153.stdout:6/860: symlink d4/dc/df/l12c 0 2024-04-05T03:02:58.507 INFO:tasks.workunit.client.0.smithi153.stdout:3/718: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/ffd x:0 0 0 2024-04-05T03:02:58.509 INFO:tasks.workunit.client.0.smithi153.stdout:6/861: mknod d4/dc/df/d16/d36/d10e/c12d 0 2024-04-05T03:02:58.510 INFO:tasks.workunit.client.0.smithi153.stdout:3/719: mknod d1d/ddf/cfe 0 2024-04-05T03:02:58.510 INFO:tasks.workunit.client.0.smithi153.stdout:6/862: fsync d4/dc/df/d16/d36/d4d/dad/ff1 0 2024-04-05T03:02:58.511 INFO:tasks.workunit.client.0.smithi153.stdout:3/720: fsync d1d/d32/d93/d3c/d3f/d7c/fa8 0 2024-04-05T03:02:58.512 INFO:tasks.workunit.client.0.smithi153.stdout:3/721: write d1d/d27/d2b/f86 [76417,104609] 0 2024-04-05T03:02:58.517 INFO:tasks.workunit.client.0.smithi153.stdout:8/667: dread d7/d13/d19/d1b/d4f/d68/f84 [0,4194304] 0 2024-04-05T03:02:58.518 INFO:tasks.workunit.client.0.smithi153.stdout:8/668: stat d7/d13/d19/d1b/d4f/d72/lc4 0 2024-04-05T03:02:58.523 INFO:tasks.workunit.client.0.smithi153.stdout:3/722: dwrite d1d/fcf [0,4194304] 0 2024-04-05T03:02:58.528 INFO:tasks.workunit.client.0.smithi153.stdout:8/669: dwrite d7/d13/d19/d1b/d33/f59 [0,4194304] 0 2024-04-05T03:02:58.530 INFO:tasks.workunit.client.0.smithi153.stdout:3/723: rename d1d/d32/d93/d3c/cf5 to d1d/d32/d93/d3c/d3f/d7c/de7/cff 0 2024-04-05T03:02:58.533 INFO:tasks.workunit.client.0.smithi153.stdout:3/724: mknod d1d/d32/d93/d3c/d94/c100 0 2024-04-05T03:02:58.535 INFO:tasks.workunit.client.0.smithi153.stdout:3/725: mknod d1d/d27/df4/c101 0 2024-04-05T03:02:58.538 INFO:tasks.workunit.client.0.smithi153.stdout:3/726: creat d1d/d32/d93/d3c/d3f/d9c/f102 x:0 0 0 2024-04-05T03:02:58.540 INFO:tasks.workunit.client.0.smithi153.stdout:3/727: unlink d1d/d27/d2b/l66 0 2024-04-05T03:02:58.542 INFO:tasks.workunit.client.0.smithi153.stdout:3/728: mknod d1d/d27/d2b/c103 0 2024-04-05T03:02:58.548 INFO:tasks.workunit.client.0.smithi153.stdout:3/729: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f81 [0,4194304] 0 2024-04-05T03:02:58.550 INFO:tasks.workunit.client.0.smithi153.stdout:3/730: rmdir d1d/d32/d93/d3c/d3f/d90/d92 39 2024-04-05T03:02:58.553 INFO:tasks.workunit.client.0.smithi153.stdout:3/731: mkdir d1d/d32/df9/d104 0 2024-04-05T03:02:58.556 INFO:tasks.workunit.client.0.smithi153.stdout:3/732: creat d1d/d32/df9/d104/f105 x:0 0 0 2024-04-05T03:02:58.556 INFO:tasks.workunit.client.0.smithi153.stdout:3/733: dread - d1d/d27/feb zero size 2024-04-05T03:02:58.567 INFO:tasks.workunit.client.0.smithi153.stdout:8/670: dread d7/d85/f32 [0,4194304] 0 2024-04-05T03:02:58.568 INFO:tasks.workunit.client.0.smithi153.stdout:3/734: dwrite d1d/d27/d2b/f86 [0,4194304] 0 2024-04-05T03:02:58.580 INFO:tasks.workunit.client.0.smithi153.stdout:3/735: truncate d1d/d32/d93/fb1 793335 0 2024-04-05T03:02:58.580 INFO:tasks.workunit.client.0.smithi153.stdout:8/671: chown d7/d13/d19/d1b/d4f/d5f/d54/c79 4 1 2024-04-05T03:02:58.582 INFO:tasks.workunit.client.0.smithi153.stdout:8/672: rmdir d7/d13/dc8 39 2024-04-05T03:02:58.584 INFO:tasks.workunit.client.0.smithi153.stdout:8/673: symlink d7/d13/d19/d1b/d33/ld9 0 2024-04-05T03:02:58.648 INFO:tasks.workunit.client.0.smithi153.stdout:7/830: sync 2024-04-05T03:02:58.651 INFO:tasks.workunit.client.0.smithi153.stdout:7/831: getdents d2/da/d3a 0 2024-04-05T03:02:58.652 INFO:tasks.workunit.client.0.smithi153.stdout:7/832: dread - d2/da/d47/d5d/d7c/d8d/df2/f112 zero size 2024-04-05T03:02:58.658 INFO:tasks.workunit.client.0.smithi153.stdout:7/833: creat d2/da/d3a/d41/d7a/d54/d85/df6/f114 x:0 0 0 2024-04-05T03:02:58.659 INFO:tasks.workunit.client.0.smithi153.stdout:7/834: truncate d2/da/d3a/d41/d7a/fd3 1081 0 2024-04-05T03:02:58.663 INFO:tasks.workunit.client.0.smithi153.stdout:7/835: getdents d2/da/d47/d5d/d7c/d8d/d95/dce/dc8/dde 0 2024-04-05T03:02:58.667 INFO:tasks.workunit.client.0.smithi153.stdout:7/836: read d2/da/d11/d88/da6/fc2 [12936,9012] 0 2024-04-05T03:02:58.669 INFO:tasks.workunit.client.0.smithi153.stdout:7/837: stat d2/da/d3a/d41/d7a/d54/d85/df6/d93/f4b 0 2024-04-05T03:02:58.671 INFO:tasks.workunit.client.0.smithi153.stdout:7/838: creat d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106/f115 x:0 0 0 2024-04-05T03:02:58.673 INFO:tasks.workunit.client.0.smithi153.stdout:7/839: mknod d2/da/d3a/d41/d7a/d54/c116 0 2024-04-05T03:02:58.678 INFO:tasks.workunit.client.0.smithi153.stdout:7/840: write d2/da/d3a/d41/d7a/f3c [717707,58430] 0 2024-04-05T03:02:58.683 INFO:tasks.workunit.client.0.smithi153.stdout:7/841: link d2/da/d3a/d41/d7a/d54/d85/df6/d93/fe4 d2/da/d3a/d41/d7a/d54/d85/df6/d77/f117 0 2024-04-05T03:02:58.684 INFO:tasks.workunit.client.0.smithi153.stdout:7/842: write d2/da/d3a/d41/d7a/d54/ff0 [2007086,43578] 0 2024-04-05T03:02:58.686 INFO:tasks.workunit.client.0.smithi153.stdout:7/843: read d2/f9 [2487936,25597] 0 2024-04-05T03:02:58.688 INFO:tasks.workunit.client.0.smithi153.stdout:7/844: chown d2/da/d3a/d41/d7a/d54/c6b 1 1 2024-04-05T03:02:58.690 INFO:tasks.workunit.client.0.smithi153.stdout:7/845: mkdir d2/da/d47/d5d/d7c/d118 0 2024-04-05T03:02:58.695 INFO:tasks.workunit.client.0.smithi153.stdout:7/846: rename d2/da/d47/d5d/d7c/d8d to d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119 0 2024-04-05T03:02:58.697 INFO:tasks.workunit.client.0.smithi153.stdout:7/847: symlink d2/d9b/d6f/l11a 0 2024-04-05T03:02:58.709 INFO:tasks.workunit.client.0.smithi153.stdout:7/848: dread d2/da/d47/d5d/f69 [0,4194304] 0 2024-04-05T03:02:58.715 INFO:tasks.workunit.client.0.smithi153.stdout:7/849: rmdir d2/da 39 2024-04-05T03:02:58.717 INFO:tasks.workunit.client.0.smithi153.stdout:7/850: truncate d2/f6 3339006 0 2024-04-05T03:02:58.723 INFO:tasks.workunit.client.0.smithi153.stdout:7/851: dwrite d2/da/d47/f9e [0,4194304] 0 2024-04-05T03:02:58.727 INFO:tasks.workunit.client.0.smithi153.stdout:7/852: dread d2/da/f68 [0,4194304] 0 2024-04-05T03:02:58.728 INFO:tasks.workunit.client.0.smithi153.stdout:7/853: read d2/d9b/f61 [2855834,82429] 0 2024-04-05T03:02:58.729 INFO:tasks.workunit.client.0.smithi153.stdout:7/854: chown d2/da/d3a/d41/d7a/d54 15 1 2024-04-05T03:02:58.732 INFO:tasks.workunit.client.0.smithi153.stdout:7/855: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d93/c11b 0 2024-04-05T03:02:58.736 INFO:tasks.workunit.client.0.smithi153.stdout:7/856: creat d2/da/d3a/d41/d7a/f11c x:0 0 0 2024-04-05T03:02:58.754 INFO:tasks.workunit.client.0.smithi153.stdout:7/857: dread d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/fe2 [0,4194304] 0 2024-04-05T03:02:58.758 INFO:tasks.workunit.client.0.smithi153.stdout:7/858: rename d2/da/d3a/f46 to d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/f11d 0 2024-04-05T03:02:58.760 INFO:tasks.workunit.client.0.smithi153.stdout:7/859: write d2/da/d3a/d41/d7a/d54/ff0 [3606115,96615] 0 2024-04-05T03:02:58.762 INFO:tasks.workunit.client.0.smithi153.stdout:7/860: fsync d2/da/d3a/d41/ddd/f107 0 2024-04-05T03:02:58.763 INFO:tasks.workunit.client.0.smithi153.stdout:7/861: write d2/da/d3a/d41/ff1 [831321,127224] 0 2024-04-05T03:02:58.772 INFO:tasks.workunit.client.0.smithi153.stdout:7/862: dwrite d2/da/d11/d88/fca [0,4194304] 0 2024-04-05T03:02:58.773 INFO:tasks.workunit.client.0.smithi153.stdout:7/863: dread - d2/da/d47/fda zero size 2024-04-05T03:02:58.782 INFO:tasks.workunit.client.0.smithi153.stdout:7/864: dwrite d2/da/d3a/d41/d7a/f4e [0,4194304] 0 2024-04-05T03:02:58.786 INFO:tasks.workunit.client.0.smithi153.stdout:7/865: rename d2/da/d3a/fd4 to d2/da/d47/d5d/d7c/d118/f11e 0 2024-04-05T03:02:58.786 INFO:tasks.workunit.client.0.smithi153.stdout:7/866: read - d2/da/d3a/d41/ddd/fef zero size 2024-04-05T03:02:58.813 INFO:tasks.workunit.client.0.smithi153.stdout:9/589: sync 2024-04-05T03:02:58.814 INFO:tasks.workunit.client.0.smithi153.stdout:7/867: dread d2/da/d11/f24 [4194304,4194304] 0 2024-04-05T03:02:58.818 INFO:tasks.workunit.client.0.smithi153.stdout:7/868: truncate d2/da/d3a/d41/d31/f37 800578 0 2024-04-05T03:02:58.821 INFO:tasks.workunit.client.0.smithi153.stdout:9/590: write d1/d10/d16/d22/f3e [215497,101271] 0 2024-04-05T03:02:58.822 INFO:tasks.workunit.client.0.smithi153.stdout:7/869: getdents d2/d9b 0 2024-04-05T03:02:58.823 INFO:tasks.workunit.client.0.smithi153.stdout:7/870: chown d2/da/c8f 75220518 1 2024-04-05T03:02:58.827 INFO:tasks.workunit.client.0.smithi153.stdout:9/591: dwrite d1/d3a/fb5 [0,4194304] 0 2024-04-05T03:02:58.828 INFO:tasks.workunit.client.0.smithi153.stdout:9/592: chown d1/d9/d1b/f7d 56401271 1 2024-04-05T03:02:58.830 INFO:tasks.workunit.client.0.smithi153.stdout:9/593: fsync d1/d10/d36/f56 0 2024-04-05T03:02:58.834 INFO:tasks.workunit.client.0.smithi153.stdout:9/594: unlink d1/d10/d16/f1a 0 2024-04-05T03:02:58.835 INFO:tasks.workunit.client.0.smithi153.stdout:9/595: read - d1/d10/d16/d22/d2c/d68/fa7 zero size 2024-04-05T03:02:58.838 INFO:tasks.workunit.client.0.smithi153.stdout:7/871: dread d2/da/d11/f13 [4194304,4194304] 0 2024-04-05T03:02:58.838 INFO:tasks.workunit.client.0.smithi153.stdout:9/596: rmdir d1/d10/d16/d22/d2c 39 2024-04-05T03:02:58.842 INFO:tasks.workunit.client.0.smithi153.stdout:9/597: rename d1/d10/d16/d22/l85 to d1/d10/d36/d57/d97/ld0 0 2024-04-05T03:02:58.845 INFO:tasks.workunit.client.0.smithi153.stdout:9/598: creat d1/d10/d16/d22/d2c/d68/d81/fd1 x:0 0 0 2024-04-05T03:02:58.847 INFO:tasks.workunit.client.0.smithi153.stdout:7/872: dread d2/da/d11/d39/f76 [0,4194304] 0 2024-04-05T03:02:58.847 INFO:tasks.workunit.client.0.smithi153.stdout:9/599: mkdir d1/d10/d36/dd2 0 2024-04-05T03:02:58.848 INFO:tasks.workunit.client.0.smithi153.stdout:7/873: read d2/da/d3a/d41/ddd/f107 [6025220,47678] 0 2024-04-05T03:02:58.851 INFO:tasks.workunit.client.0.smithi153.stdout:9/600: creat d1/d10/d36/fd3 x:0 0 0 2024-04-05T03:02:58.852 INFO:tasks.workunit.client.0.smithi153.stdout:7/874: rename d2/da/d47/f9e to d2/da/d47/f11f 0 2024-04-05T03:02:58.854 INFO:tasks.workunit.client.0.smithi153.stdout:9/601: creat d1/d10/d16/d22/d2c/d68/da9/fd4 x:0 0 0 2024-04-05T03:02:58.855 INFO:tasks.workunit.client.0.smithi153.stdout:7/875: readlink d2/da/d11/lea 0 2024-04-05T03:02:58.858 INFO:tasks.workunit.client.0.smithi153.stdout:9/602: rmdir d1/d10/d36/dd2 0 2024-04-05T03:02:58.862 INFO:tasks.workunit.client.0.smithi153.stdout:9/603: truncate d1/d10/d16/d22/d2c/f58 5027781 0 2024-04-05T03:02:58.862 INFO:tasks.workunit.client.0.smithi153.stdout:7/876: dwrite d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/f105 [0,4194304] 0 2024-04-05T03:02:58.864 INFO:tasks.workunit.client.0.smithi153.stdout:7/877: chown d2/da/d3a/d41/d7a/d54/d85/df6/d93/f74 26 1 2024-04-05T03:02:58.872 INFO:tasks.workunit.client.0.smithi153.stdout:7/878: unlink d2/da/f68 0 2024-04-05T03:02:58.875 INFO:tasks.workunit.client.0.smithi153.stdout:7/879: mknod d2/d9b/d6f/d10a/c120 0 2024-04-05T03:02:58.875 INFO:tasks.workunit.client.0.smithi153.stdout:9/604: write d1/fcd [900063,52804] 0 2024-04-05T03:02:58.877 INFO:tasks.workunit.client.0.smithi153.stdout:9/605: dread - d1/d10/d16/d22/fcc zero size 2024-04-05T03:02:58.880 INFO:tasks.workunit.client.0.smithi153.stdout:7/880: read d2/da/d11/d39/f63 [1017405,94516] 0 2024-04-05T03:02:58.881 INFO:tasks.workunit.client.0.smithi153.stdout:7/881: chown d2/da/d11/f24 422621 1 2024-04-05T03:02:58.883 INFO:tasks.workunit.client.0.smithi153.stdout:7/882: symlink d2/da/d3a/d41/ddd/l121 0 2024-04-05T03:02:58.886 INFO:tasks.workunit.client.0.smithi153.stdout:7/883: unlink d2/da/d47/d5d/c90 0 2024-04-05T03:02:58.888 INFO:tasks.workunit.client.0.smithi153.stdout:7/884: mknod d2/da/c122 0 2024-04-05T03:02:58.890 INFO:tasks.workunit.client.0.smithi153.stdout:7/885: mkdir d2/d123 0 2024-04-05T03:02:58.890 INFO:tasks.workunit.client.0.smithi153.stdout:7/886: stat d2/f1d 0 2024-04-05T03:02:58.891 INFO:tasks.workunit.client.0.smithi153.stdout:7/887: stat d2/da/d47/da9 0 2024-04-05T03:02:58.907 INFO:tasks.workunit.client.0.smithi153.stdout:7/888: dwrite d2/da/d3a/d41/f19 [0,4194304] 0 2024-04-05T03:02:58.910 INFO:tasks.workunit.client.0.smithi153.stdout:7/889: read d2/da/d3a/d41/d7a/d54/d85/df6/d59/fcf [2659283,59794] 0 2024-04-05T03:02:58.915 INFO:tasks.workunit.client.0.smithi153.stdout:7/890: write d2/da/f7d [40851,90624] 0 2024-04-05T03:02:58.922 INFO:tasks.workunit.client.0.smithi153.stdout:7/891: write d2/da/d3a/d41/d31/f37 [1211711,9467] 0 2024-04-05T03:02:58.926 INFO:tasks.workunit.client.0.smithi153.stdout:7/892: rename d2/da/d47/d5d/d7c/cbd to d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/c124 0 2024-04-05T03:02:58.937 INFO:tasks.workunit.client.0.smithi153.stdout:7/893: dread d2/f6 [0,4194304] 0 2024-04-05T03:02:58.944 INFO:tasks.workunit.client.0.smithi153.stdout:7/894: truncate d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/fe2 384295 0 2024-04-05T03:02:58.945 INFO:tasks.workunit.client.0.smithi153.stdout:7/895: dread - d2/da/d47/d5d/f103 zero size 2024-04-05T03:02:59.016 INFO:tasks.workunit.client.0.smithi153.stdout:7/896: dread - d2/da/d47/d5d/fc9 zero size 2024-04-05T03:02:59.035 INFO:tasks.workunit.client.0.smithi153.stdout:7/897: link d2/da/d11/f2d d2/da/d47/d5d/d7c/d118/f125 0 2024-04-05T03:02:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:02:58 smithi153 ceph-mon[29381]: pgmap v14: 65 pgs: 65 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 149 MiB/s rd, 398 MiB/s wr, 768 op/s 2024-04-05T03:02:59.082 INFO:tasks.workunit.client.0.smithi153.stdout:7/898: rename d2/da/d3a/d41/f8a to d2/da/d3a/d41/d31/f126 0 2024-04-05T03:02:59.092 INFO:tasks.workunit.client.0.smithi153.stdout:0/665: sync 2024-04-05T03:02:59.094 INFO:tasks.workunit.client.0.smithi153.stdout:0/666: creat d9/d68/fce x:0 0 0 2024-04-05T03:02:59.099 INFO:tasks.workunit.client.0.smithi153.stdout:0/667: write d9/db/d42/d7e/f8c [352655,69256] 0 2024-04-05T03:02:59.102 INFO:tasks.workunit.client.0.smithi153.stdout:0/668: truncate d9/db/d42/d7e/f8c 564427 0 2024-04-05T03:02:59.103 INFO:tasks.workunit.client.0.smithi153.stdout:7/899: dwrite d2/da/d3a/d41/d7a/fd3 [0,4194304] 0 2024-04-05T03:02:59.104 INFO:tasks.workunit.client.0.smithi153.stdout:7/900: stat d2/da/d11/d88/da6/l100 0 2024-04-05T03:02:59.107 INFO:tasks.workunit.client.0.smithi153.stdout:0/669: creat d9/db/d42/d7e/db6/d1d/d61/d8a/fcf x:0 0 0 2024-04-05T03:02:59.108 INFO:tasks.workunit.client.0.smithi153.stdout:7/901: rename d2/da/d47/ff4 to d2/da/d3a/d41/da8/f127 0 2024-04-05T03:02:59.109 INFO:tasks.workunit.client.0.smithi153.stdout:0/670: mkdir d9/db/d42/d7e/db6/daf/dd0 0 2024-04-05T03:02:59.113 INFO:tasks.workunit.client.0.smithi153.stdout:7/902: rename d2/da/lac to d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/dc8/dde/l128 0 2024-04-05T03:02:59.114 INFO:tasks.workunit.client.0.smithi153.stdout:7/903: chown d2/da/d11/d88/da6/l100 1755 1 2024-04-05T03:02:59.117 INFO:tasks.workunit.client.0.smithi153.stdout:7/904: creat d2/da/d11/d88/f129 x:0 0 0 2024-04-05T03:02:59.123 INFO:tasks.workunit.client.0.smithi153.stdout:7/905: dwrite d2/da/d47/d5d/f103 [0,4194304] 0 2024-04-05T03:02:59.132 INFO:tasks.workunit.client.0.smithi153.stdout:0/671: dread d9/db/d42/d7e/db6/d1d/d61/d8a/f7b [0,4194304] 0 2024-04-05T03:02:59.134 INFO:tasks.workunit.client.0.smithi153.stdout:7/906: dwrite d2/da/d3a/d41/d31/f126 [4194304,4194304] 0 2024-04-05T03:02:59.135 INFO:tasks.workunit.client.0.smithi153.stdout:0/672: write d9/db/d42/d7e/db6/d1e/d45/d62/d79/fc6 [986677,109664] 0 2024-04-05T03:02:59.136 INFO:tasks.workunit.client.0.smithi153.stdout:7/907: write d2/da/f7d [1362336,44510] 0 2024-04-05T03:02:59.143 INFO:tasks.workunit.client.0.smithi153.stdout:0/673: dwrite d9/db/d42/d7e/db6/f59 [0,4194304] 0 2024-04-05T03:02:59.148 INFO:tasks.workunit.client.0.smithi153.stdout:7/908: link d2/da/d3a/d41/d7a/d54/d85/df6/d93/l10f d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/l12a 0 2024-04-05T03:02:59.150 INFO:tasks.workunit.client.0.smithi153.stdout:0/674: unlink d9/db/d42/d7e/db6/d1e/d45/d62/cbc 0 2024-04-05T03:02:59.155 INFO:tasks.workunit.client.0.smithi153.stdout:7/909: dwrite d2/ffd [0,4194304] 0 2024-04-05T03:02:59.158 INFO:tasks.workunit.client.0.smithi153.stdout:7/910: mkdir d2/da/d11/d39/d12b 0 2024-04-05T03:02:59.161 INFO:tasks.workunit.client.0.smithi153.stdout:7/911: unlink d2/c30 0 2024-04-05T03:02:59.162 INFO:tasks.workunit.client.0.smithi153.stdout:7/912: readlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/l7f 0 2024-04-05T03:02:59.163 INFO:tasks.workunit.client.0.smithi153.stdout:0/675: write f7 [1936628,96561] 0 2024-04-05T03:02:59.169 INFO:tasks.workunit.client.0.smithi153.stdout:0/676: unlink d9/db/f13 0 2024-04-05T03:02:59.176 INFO:tasks.workunit.client.0.smithi153.stdout:0/677: dwrite d9/db/d42/d7e/db6/d1e/d45/fc2 [0,4194304] 0 2024-04-05T03:02:59.178 INFO:tasks.workunit.client.0.smithi153.stdout:0/678: creat d9/d14/fd1 x:0 0 0 2024-04-05T03:02:59.179 INFO:tasks.workunit.client.0.smithi153.stdout:0/679: chown d9/db/d42/d43/fa1 5 1 2024-04-05T03:02:59.182 INFO:tasks.workunit.client.0.smithi153.stdout:0/680: mkdir d9/db/d42/d7e/db6/d1e/d45/d62/dd2 0 2024-04-05T03:02:59.198 INFO:tasks.workunit.client.0.smithi153.stdout:7/913: dread d2/da/d3a/d41/f8c [0,4194304] 0 2024-04-05T03:02:59.200 INFO:tasks.workunit.client.0.smithi153.stdout:0/681: dwrite d9/db/d42/d43/f4a [0,4194304] 0 2024-04-05T03:02:59.201 INFO:tasks.workunit.client.0.smithi153.stdout:0/682: chown d9/db/d42/d7e/db6/d1d/c87 87493 1 2024-04-05T03:02:59.205 INFO:tasks.workunit.client.0.smithi153.stdout:7/914: dread d2/da/d11/f72 [0,4194304] 0 2024-04-05T03:02:59.206 INFO:tasks.workunit.client.0.smithi153.stdout:2/777: sync 2024-04-05T03:02:59.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/915: fsync d2/da/d3a/d41/d31/f45 0 2024-04-05T03:02:59.208 INFO:tasks.workunit.client.0.smithi153.stdout:2/778: chown d9/d62/c76 64372 1 2024-04-05T03:02:59.211 INFO:tasks.workunit.client.0.smithi153.stdout:0/683: dwrite d9/d68/f9b [0,4194304] 0 2024-04-05T03:02:59.216 INFO:tasks.workunit.client.0.smithi153.stdout:7/916: creat d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/dc8/dde/f12c x:0 0 0 2024-04-05T03:02:59.218 INFO:tasks.workunit.client.0.smithi153.stdout:2/779: write d9/da/d19/d2e/d7e/fdc [29948,58480] 0 2024-04-05T03:02:59.220 INFO:tasks.workunit.client.0.smithi153.stdout:0/684: write d9/db/d42/d7e/db6/d1d/f3b [2184627,109487] 0 2024-04-05T03:02:59.220 INFO:tasks.workunit.client.0.smithi153.stdout:2/780: stat d9/da/d19/d2e/d7e/d53 0 2024-04-05T03:02:59.223 INFO:tasks.workunit.client.0.smithi153.stdout:7/917: link d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/df2/f112 d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/f12d 0 2024-04-05T03:02:59.227 INFO:tasks.workunit.client.0.smithi153.stdout:2/781: rename d9/da/d19/d2e/d7e/d7a/lee to d9/da/d19/d2e/d7e/d7a/d6d/de8/l109 0 2024-04-05T03:02:59.230 INFO:tasks.workunit.client.0.smithi153.stdout:7/918: rename d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/dc8/dde to d2/da/d3a/d41/da8/d12e 0 2024-04-05T03:02:59.231 INFO:tasks.workunit.client.0.smithi153.stdout:7/919: chown d2/da/c8f 163 1 2024-04-05T03:02:59.231 INFO:tasks.workunit.client.0.smithi153.stdout:2/782: creat d9/da/d19/de2/f10a x:0 0 0 2024-04-05T03:02:59.232 INFO:tasks.workunit.client.0.smithi153.stdout:7/920: dread - d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/f12d zero size 2024-04-05T03:02:59.233 INFO:tasks.workunit.client.0.smithi153.stdout:7/921: chown d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106/d110 2014 1 2024-04-05T03:02:59.235 INFO:tasks.workunit.client.0.smithi153.stdout:7/922: write d2/da/d3a/d41/ff1 [1612803,53987] 0 2024-04-05T03:02:59.236 INFO:tasks.workunit.client.0.smithi153.stdout:2/783: symlink d9/da/d94/dca/df7/l10b 0 2024-04-05T03:02:59.237 INFO:tasks.workunit.client.0.smithi153.stdout:0/685: write d9/db/f4b [1086260,42575] 0 2024-04-05T03:02:59.237 INFO:tasks.workunit.client.0.smithi153.stdout:7/923: write d2/da/d3a/d41/d7a/d54/d85/ff7 [1075966,43007] 0 2024-04-05T03:02:59.242 INFO:tasks.workunit.client.0.smithi153.stdout:0/686: creat d9/d68/d9e/fd3 x:0 0 0 2024-04-05T03:02:59.243 INFO:tasks.workunit.client.0.smithi153.stdout:0/687: chown d9/db/l31 16041 1 2024-04-05T03:02:59.245 INFO:tasks.workunit.client.0.smithi153.stdout:2/784: link d9/da/d19/d2e/d4f/l72 d9/da/df9/l10c 0 2024-04-05T03:02:59.245 INFO:tasks.workunit.client.0.smithi153.stdout:2/785: chown d9/d68/d98 464919010 1 2024-04-05T03:02:59.246 INFO:tasks.workunit.client.0.smithi153.stdout:0/688: mkdir d9/db/d42/dd4 0 2024-04-05T03:02:59.247 INFO:tasks.workunit.client.0.smithi153.stdout:2/786: fdatasync d9/da/d19/d2e/d7e/d7a/d6d/f6e 0 2024-04-05T03:02:59.250 INFO:tasks.workunit.client.0.smithi153.stdout:0/689: rename d9/db/d42/d7e/db6/lad to d9/d68/d6d/ld5 0 2024-04-05T03:02:59.251 INFO:tasks.workunit.client.0.smithi153.stdout:7/924: dread d2/da/d47/d5d/f10d [0,4194304] 0 2024-04-05T03:02:59.252 INFO:tasks.workunit.client.0.smithi153.stdout:0/690: fsync d9/db/f37 0 2024-04-05T03:02:59.252 INFO:tasks.workunit.client.0.smithi153.stdout:7/925: write d2/da/d47/fbc [2642244,41827] 0 2024-04-05T03:02:59.255 INFO:tasks.workunit.client.0.smithi153.stdout:7/926: write d2/da/d11/f24 [8013009,70538] 0 2024-04-05T03:02:59.260 INFO:tasks.workunit.client.0.smithi153.stdout:0/691: write d9/db/d42/d7e/db6/d1e/f52 [604142,86779] 0 2024-04-05T03:02:59.274 INFO:tasks.workunit.client.0.smithi153.stdout:0/692: dread d9/db/fd [0,4194304] 0 2024-04-05T03:02:59.276 INFO:tasks.workunit.client.0.smithi153.stdout:0/693: creat d9/db/fd6 x:0 0 0 2024-04-05T03:02:59.278 INFO:tasks.workunit.client.0.smithi153.stdout:0/694: truncate d9/db/d42/d43/fa1 65495 0 2024-04-05T03:02:59.278 INFO:tasks.workunit.client.0.smithi153.stdout:0/695: readlink d9/d14/lbd 0 2024-04-05T03:02:59.283 INFO:tasks.workunit.client.0.smithi153.stdout:0/696: dwrite d9/db/d42/d7e/db6/f18 [0,4194304] 0 2024-04-05T03:02:59.284 INFO:tasks.workunit.client.0.smithi153.stdout:0/697: write d9/d68/fce [692443,118170] 0 2024-04-05T03:02:59.291 INFO:tasks.workunit.client.0.smithi153.stdout:0/698: link d9/db/d42/l88 d9/db/d42/d7e/db6/d1d/d61/ld7 0 2024-04-05T03:02:59.294 INFO:tasks.workunit.client.0.smithi153.stdout:0/699: getdents d9/db/d42/d7e/db6/d1e 0 2024-04-05T03:02:59.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:02:58 smithi203 ceph-mon[39911]: pgmap v14: 65 pgs: 65 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 149 MiB/s rd, 398 MiB/s wr, 768 op/s 2024-04-05T03:02:59.298 INFO:tasks.workunit.client.0.smithi153.stdout:0/700: dread d9/db/d42/d7e/db6/f17 [0,4194304] 0 2024-04-05T03:02:59.305 INFO:tasks.workunit.client.0.smithi153.stdout:0/701: dwrite d9/d14/fd1 [0,4194304] 0 2024-04-05T03:02:59.306 INFO:tasks.workunit.client.0.smithi153.stdout:0/702: dread - d9/d14/f7a zero size 2024-04-05T03:02:59.308 INFO:tasks.workunit.client.0.smithi153.stdout:0/703: write d9/db/d42/d7e/db6/d1e/d45/fba [81269,17682] 0 2024-04-05T03:02:59.310 INFO:tasks.workunit.client.0.smithi153.stdout:0/704: chown d9/db/d42/d7e/db6/d1e/d45/fc2 1006452489 1 2024-04-05T03:02:59.313 INFO:tasks.workunit.client.0.smithi153.stdout:0/705: mknod d9/db/d42/d7e/db6/daf/cd8 0 2024-04-05T03:02:59.315 INFO:tasks.workunit.client.0.smithi153.stdout:0/706: mkdir d9/dd9 0 2024-04-05T03:02:59.318 INFO:tasks.workunit.client.0.smithi153.stdout:0/707: rmdir d9/dd9 0 2024-04-05T03:02:59.321 INFO:tasks.workunit.client.0.smithi153.stdout:0/708: symlink d9/db/d42/d7e/db6/d1e/d45/d62/d79/lda 0 2024-04-05T03:02:59.323 INFO:tasks.workunit.client.0.smithi153.stdout:0/709: chown d9/db/d42/d7e/db6/d1d/d61/l8d 320 1 2024-04-05T03:02:59.325 INFO:tasks.workunit.client.0.smithi153.stdout:0/710: symlink d9/d14/d67/d6f/ldb 0 2024-04-05T03:02:59.326 INFO:tasks.workunit.client.0.smithi153.stdout:0/711: rmdir d9/db/d42/d7e/db6/d1d/d61 39 2024-04-05T03:02:59.332 INFO:tasks.workunit.client.0.smithi153.stdout:0/712: dwrite d9/d14/d67/d6f/f7c [0,4194304] 0 2024-04-05T03:02:59.333 INFO:tasks.workunit.client.0.smithi153.stdout:0/713: read - d9/d14/f7a zero size 2024-04-05T03:02:59.337 INFO:tasks.workunit.client.0.smithi153.stdout:0/714: symlink d9/db/d42/d7e/db6/d1e/d45/d62/d79/ldc 0 2024-04-05T03:02:59.344 INFO:tasks.workunit.client.0.smithi153.stdout:0/715: dwrite d9/db/d42/d7e/db6/d1e/d45/d62/d79/fc6 [0,4194304] 0 2024-04-05T03:02:59.347 INFO:tasks.workunit.client.0.smithi153.stdout:0/716: rmdir d9/db/d42/d7e/db6/d1e/d45/dae 39 2024-04-05T03:02:59.353 INFO:tasks.workunit.client.0.smithi153.stdout:0/717: dread d9/db/f4b [0,4194304] 0 2024-04-05T03:02:59.374 INFO:tasks.workunit.client.0.smithi153.stdout:0/718: dread d9/db/d42/d7e/db6/d1e/f54 [0,4194304] 0 2024-04-05T03:02:59.376 INFO:tasks.workunit.client.0.smithi153.stdout:0/719: creat d9/d14/fdd x:0 0 0 2024-04-05T03:02:59.395 INFO:tasks.workunit.client.0.smithi153.stdout:4/835: sync 2024-04-05T03:02:59.396 INFO:tasks.workunit.client.0.smithi153.stdout:4/836: write da/de/d66/d111/ffa [573401,51236] 0 2024-04-05T03:02:59.399 INFO:tasks.workunit.client.0.smithi153.stdout:4/837: rename da/de/f14 to da/db/d20/d3c/d81/db6/dc7/de1/f121 0 2024-04-05T03:02:59.400 INFO:tasks.workunit.client.0.smithi153.stdout:4/838: fdatasync da/de/d66/d111/d86/f119 0 2024-04-05T03:02:59.403 INFO:tasks.workunit.client.0.smithi153.stdout:4/839: rename da/db/d8b/db7/cc4 to da/de/d56/d78/c122 0 2024-04-05T03:02:59.404 INFO:tasks.workunit.client.0.smithi153.stdout:4/840: write da/db/d20/d3c/d81/db6/f50 [1262630,117746] 0 2024-04-05T03:02:59.442 INFO:tasks.workunit.client.0.smithi153.stdout:4/841: rename da/db/d20/d3c/d81/db6/dc7/de1/f121 to da/db/d20/d35/d38/f123 0 2024-04-05T03:02:59.444 INFO:tasks.workunit.client.0.smithi153.stdout:4/842: unlink da/de/d56/d78/d8f/d68/ff2 0 2024-04-05T03:02:59.445 INFO:tasks.workunit.client.0.smithi153.stdout:4/843: chown da/db/f96 141877 1 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:1/759: sync 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:8/674: sync 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:9/606: sync 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:3/736: sync 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:6/863: sync 2024-04-05T03:02:59.584 INFO:tasks.workunit.client.0.smithi153.stdout:5/687: sync 2024-04-05T03:02:59.587 INFO:tasks.workunit.client.0.smithi153.stdout:5/688: rmdir d0 39 2024-04-05T03:02:59.589 INFO:tasks.workunit.client.0.smithi153.stdout:3/737: dread d1d/d32/d93/d3c/d94/fd1 [0,4194304] 0 2024-04-05T03:02:59.591 INFO:tasks.workunit.client.0.smithi153.stdout:9/607: creat d1/d10/d16/d22/d2c/d68/fd5 x:0 0 0 2024-04-05T03:02:59.593 INFO:tasks.workunit.client.0.smithi153.stdout:1/760: rename d5/d8/d20/d4b/d18/d16/d48/f50 to d5/d8/d20/d3f/fe5 0 2024-04-05T03:02:59.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/738: symlink d1d/d27/df4/l106 0 2024-04-05T03:02:59.597 INFO:tasks.workunit.client.0.smithi153.stdout:1/761: write d5/d8/d20/d4b/d18/d16/d45/d93/fcc [518253,42019] 0 2024-04-05T03:02:59.598 INFO:tasks.workunit.client.0.smithi153.stdout:9/608: dwrite d1/d1d/f5a [0,4194304] 0 2024-04-05T03:02:59.602 INFO:tasks.workunit.client.0.smithi153.stdout:8/675: link d7/c2e d7/d13/d19/d95/cda 0 2024-04-05T03:02:59.606 INFO:tasks.workunit.client.0.smithi153.stdout:9/609: mknod d1/d10/d16/d22/d2c/cd6 0 2024-04-05T03:02:59.607 INFO:tasks.workunit.client.0.smithi153.stdout:3/739: dwrite d1d/d27/f96 [4194304,4194304] 0 2024-04-05T03:02:59.608 INFO:tasks.workunit.client.0.smithi153.stdout:9/610: truncate d1/fc4 697608 0 2024-04-05T03:02:59.608 INFO:tasks.workunit.client.0.smithi153.stdout:5/689: rename d0/d25/d4d/d62/l15 to d0/d25/d4d/d62/d5f/d86/lf0 0 2024-04-05T03:02:59.609 INFO:tasks.workunit.client.0.smithi153.stdout:3/740: chown d1d/d32/f53 10321 1 2024-04-05T03:02:59.609 INFO:tasks.workunit.client.0.smithi153.stdout:8/676: rmdir d7/d13/d19/d1b/d4f/d5f/d54 39 2024-04-05T03:02:59.611 INFO:tasks.workunit.client.0.smithi153.stdout:1/762: dread d5/d8/d20/d4b/d18/d16/d45/d93/fcc [0,4194304] 0 2024-04-05T03:02:59.612 INFO:tasks.workunit.client.0.smithi153.stdout:9/611: creat d1/d10/d16/d22/d2c/d68/d81/fd7 x:0 0 0 2024-04-05T03:02:59.613 INFO:tasks.workunit.client.0.smithi153.stdout:3/741: creat d1d/d32/d93/d3c/d3f/d9c/f107 x:0 0 0 2024-04-05T03:02:59.614 INFO:tasks.workunit.client.0.smithi153.stdout:5/690: creat d0/d25/d4d/d62/d5f/d86/da4/ff1 x:0 0 0 2024-04-05T03:02:59.618 INFO:tasks.workunit.client.0.smithi153.stdout:1/763: creat d5/d8/d20/d4b/d18/d16/fe6 x:0 0 0 2024-04-05T03:02:59.623 INFO:tasks.workunit.client.0.smithi153.stdout:8/677: unlink d7/d13/d19/d1b/d21/c25 0 2024-04-05T03:02:59.627 INFO:tasks.workunit.client.0.smithi153.stdout:3/742: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/l108 0 2024-04-05T03:02:59.627 INFO:tasks.workunit.client.0.smithi153.stdout:1/764: dwrite d5/d8/d20/d3f/f64 [4194304,4194304] 0 2024-04-05T03:02:59.627 INFO:tasks.workunit.client.0.smithi153.stdout:9/612: dwrite d1/d10/d36/d57/d5c/f89 [0,4194304] 0 2024-04-05T03:02:59.632 INFO:tasks.workunit.client.0.smithi153.stdout:8/678: rmdir d7/d13/d19/d1b 39 2024-04-05T03:02:59.634 INFO:tasks.workunit.client.0.smithi153.stdout:9/613: symlink d1/d10/d36/d57/d5c/ld8 0 2024-04-05T03:02:59.636 INFO:tasks.workunit.client.0.smithi153.stdout:9/614: write d1/d9/d1b/f28 [5023796,60014] 0 2024-04-05T03:02:59.638 INFO:tasks.workunit.client.0.smithi153.stdout:3/743: rename d1d/d32/d93/d3c/d94/c100 to d1d/d32/df9/c109 0 2024-04-05T03:02:59.638 INFO:tasks.workunit.client.0.smithi153.stdout:8/679: dread d7/d2a/f65 [0,4194304] 0 2024-04-05T03:02:59.639 INFO:tasks.workunit.client.0.smithi153.stdout:8/680: chown d7/d13/c9c 0 1 2024-04-05T03:02:59.644 INFO:tasks.workunit.client.0.smithi153.stdout:3/744: mkdir d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a 0 2024-04-05T03:02:59.645 INFO:tasks.workunit.client.0.smithi153.stdout:3/745: stat d1d/d32/d93/d3c/c50 0 2024-04-05T03:02:59.646 INFO:tasks.workunit.client.0.smithi153.stdout:1/765: dread d5/d8/f9d [0,4194304] 0 2024-04-05T03:02:59.647 INFO:tasks.workunit.client.0.smithi153.stdout:1/766: chown d5/d8/d20/d4b/d51/da2/f92 2053 1 2024-04-05T03:02:59.649 INFO:tasks.workunit.client.0.smithi153.stdout:3/746: creat d1d/d32/d93/f10b x:0 0 0 2024-04-05T03:02:59.652 INFO:tasks.workunit.client.0.smithi153.stdout:8/681: dread d7/fe [0,4194304] 0 2024-04-05T03:02:59.653 INFO:tasks.workunit.client.0.smithi153.stdout:1/767: unlink d5/d8/d20/d4b/d18/d16/d33/d83/cd8 0 2024-04-05T03:02:59.653 INFO:tasks.workunit.client.0.smithi153.stdout:5/691: truncate d0/d25/d4d/d62/d5f/d86/f74 142824 0 2024-04-05T03:02:59.657 INFO:tasks.workunit.client.0.smithi153.stdout:3/747: dwrite d1d/d27/d2b/f7e [0,4194304] 0 2024-04-05T03:02:59.670 INFO:tasks.workunit.client.0.smithi153.stdout:1/768: truncate d5/dd/f9f 6096720 0 2024-04-05T03:02:59.671 INFO:tasks.workunit.client.0.smithi153.stdout:9/615: dread d1/d9/d1b/f3f [0,4194304] 0 2024-04-05T03:02:59.672 INFO:tasks.workunit.client.0.smithi153.stdout:8/682: truncate d7/d13/d19/d1b/d6d/f7a 1067492 0 2024-04-05T03:02:59.672 INFO:tasks.workunit.client.0.smithi153.stdout:9/616: read - d1/d10/d16/d22/d2c/d68/fca zero size 2024-04-05T03:02:59.677 INFO:tasks.workunit.client.0.smithi153.stdout:1/769: creat d5/d8/d20/d4b/dd6/fe7 x:0 0 0 2024-04-05T03:02:59.681 INFO:tasks.workunit.client.0.smithi153.stdout:8/683: dwrite d7/d46/fa7 [4194304,4194304] 0 2024-04-05T03:02:59.682 INFO:tasks.workunit.client.0.smithi153.stdout:1/770: creat d5/d8/d20/d4b/d18/d16/d33/d83/fe8 x:0 0 0 2024-04-05T03:02:59.683 INFO:tasks.workunit.client.0.smithi153.stdout:8/684: readlink d7/d13/d19/d1b/d4f/d72/d73/l9e 0 2024-04-05T03:02:59.686 INFO:tasks.workunit.client.0.smithi153.stdout:8/685: symlink d7/ldb 0 2024-04-05T03:02:59.686 INFO:tasks.workunit.client.0.smithi153.stdout:8/686: readlink d7/d13/d19/d1b/d33/ld9 0 2024-04-05T03:02:59.692 INFO:tasks.workunit.client.0.smithi153.stdout:5/692: dread d0/d25/d4d/faf [0,4194304] 0 2024-04-05T03:02:59.694 INFO:tasks.workunit.client.0.smithi153.stdout:5/693: fsync d0/f27 0 2024-04-05T03:02:59.695 INFO:tasks.workunit.client.0.smithi153.stdout:3/748: dread d1d/d27/d2b/f2d [0,4194304] 0 2024-04-05T03:02:59.695 INFO:tasks.workunit.client.0.smithi153.stdout:8/687: dwrite d7/d13/dc8/fcc [0,4194304] 0 2024-04-05T03:02:59.696 INFO:tasks.workunit.client.0.smithi153.stdout:8/688: stat d7/c24 0 2024-04-05T03:02:59.696 INFO:tasks.workunit.client.0.smithi153.stdout:5/694: read - d0/d25/d4d/d9c/db7/fd6 zero size 2024-04-05T03:02:59.699 INFO:tasks.workunit.client.0.smithi153.stdout:3/749: mkdir d1d/d75/d10c 0 2024-04-05T03:02:59.704 INFO:tasks.workunit.client.0.smithi153.stdout:3/750: chown d1d/cdc 1 1 2024-04-05T03:02:59.708 INFO:tasks.workunit.client.0.smithi153.stdout:9/617: dread d1/d9/d1b/f6a [0,4194304] 0 2024-04-05T03:02:59.709 INFO:tasks.workunit.client.0.smithi153.stdout:3/751: getdents d1d 0 2024-04-05T03:02:59.710 INFO:tasks.workunit.client.0.smithi153.stdout:2/787: sync 2024-04-05T03:02:59.711 INFO:tasks.workunit.client.0.smithi153.stdout:9/618: readlink d1/d9/d1b/l61 0 2024-04-05T03:02:59.711 INFO:tasks.workunit.client.0.smithi153.stdout:9/619: chown d1/d10 496 1 2024-04-05T03:02:59.712 INFO:tasks.workunit.client.0.smithi153.stdout:3/752: chown d1d/d32/d93/d3c/d94/lcd 31891341 1 2024-04-05T03:02:59.713 INFO:tasks.workunit.client.0.smithi153.stdout:9/620: readlink d1/d10/d36/d57/d97/lac 0 2024-04-05T03:02:59.714 INFO:tasks.workunit.client.0.smithi153.stdout:9/621: truncate d1/d10/d16/d22/d2c/d68/fa1 306982 0 2024-04-05T03:02:59.716 INFO:tasks.workunit.client.0.smithi153.stdout:8/689: write d7/d13/d19/d1b/d6d/fae [443153,42236] 0 2024-04-05T03:02:59.716 INFO:tasks.workunit.client.0.smithi153.stdout:3/753: read d1d/d27/f64 [1150664,56285] 0 2024-04-05T03:02:59.718 INFO:tasks.workunit.client.0.smithi153.stdout:5/695: write d0/d25/d4d/d62/d37/d38/f87 [1617233,88120] 0 2024-04-05T03:02:59.719 INFO:tasks.workunit.client.0.smithi153.stdout:3/754: mkdir d1d/d10d 0 2024-04-05T03:02:59.720 INFO:tasks.workunit.client.0.smithi153.stdout:9/622: write d1/d10/d16/d22/d2c/d68/d81/f45 [3434338,75455] 0 2024-04-05T03:02:59.722 INFO:tasks.workunit.client.0.smithi153.stdout:8/690: creat d7/d13/dab/fdc x:0 0 0 2024-04-05T03:02:59.724 INFO:tasks.workunit.client.0.smithi153.stdout:5/696: read d0/d9/f1e [62731,115445] 0 2024-04-05T03:02:59.726 INFO:tasks.workunit.client.0.smithi153.stdout:9/623: dread d1/d10/d36/d71/fa0 [0,4194304] 0 2024-04-05T03:02:59.727 INFO:tasks.workunit.client.0.smithi153.stdout:8/691: truncate d7/d13/d19/d1b/d4f/d5f/fb2 2046586 0 2024-04-05T03:02:59.733 INFO:tasks.workunit.client.0.smithi153.stdout:2/788: dread d9/da/d19/d2e/d7e/d7a/d6d/f6e [0,4194304] 0 2024-04-05T03:02:59.739 INFO:tasks.workunit.client.0.smithi153.stdout:2/789: getdents d9/da/d19/d2e/d4f 0 2024-04-05T03:02:59.741 INFO:tasks.workunit.client.0.smithi153.stdout:5/697: dread d0/d25/d4d/d62/d5f/d86/da4/fa1 [0,4194304] 0 2024-04-05T03:02:59.741 INFO:tasks.workunit.client.0.smithi153.stdout:8/692: dread d7/fa [0,4194304] 0 2024-04-05T03:02:59.745 INFO:tasks.workunit.client.0.smithi153.stdout:8/693: creat d7/d13/d19/d1b/d33/fdd x:0 0 0 2024-04-05T03:02:59.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/698: mknod d0/d25/d4d/d62/d5f/d86/dd7/cf2 0 2024-04-05T03:02:59.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/699: stat d0/d9/d22/l92 0 2024-04-05T03:02:59.751 INFO:tasks.workunit.client.0.smithi153.stdout:5/700: getdents d0/d25 0 2024-04-05T03:02:59.756 INFO:tasks.workunit.client.0.smithi153.stdout:5/701: write d0/d2c/f3d [1292333,10458] 0 2024-04-05T03:02:59.756 INFO:tasks.workunit.client.0.smithi153.stdout:9/624: dread d1/d10/f64 [0,4194304] 0 2024-04-05T03:02:59.759 INFO:tasks.workunit.client.0.smithi153.stdout:8/694: dwrite d7/fd [4194304,4194304] 0 2024-04-05T03:02:59.759 INFO:tasks.workunit.client.0.smithi153.stdout:9/625: truncate d1/d9/f8e 498382 0 2024-04-05T03:02:59.761 INFO:tasks.workunit.client.0.smithi153.stdout:9/626: dread - d1/d6c/fc0 zero size 2024-04-05T03:02:59.761 INFO:tasks.workunit.client.0.smithi153.stdout:5/702: unlink d0/d25/d4d/d62/d5f/d86/f85 0 2024-04-05T03:02:59.762 INFO:tasks.workunit.client.0.smithi153.stdout:2/790: dread d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:02:59.763 INFO:tasks.workunit.client.0.smithi153.stdout:2/791: write d9/d68/fba [2634660,73814] 0 2024-04-05T03:02:59.770 INFO:tasks.workunit.client.0.smithi153.stdout:8/695: mknod d7/d13/d19/d1b/d4f/d5f/cde 0 2024-04-05T03:02:59.772 INFO:tasks.workunit.client.0.smithi153.stdout:9/627: rename d1/d10/d16/d22/l79 to d1/d10/d16/d22/d2c/ld9 0 2024-04-05T03:02:59.773 INFO:tasks.workunit.client.0.smithi153.stdout:8/696: symlink d7/d13/d19/ldf 0 2024-04-05T03:02:59.776 INFO:tasks.workunit.client.0.smithi153.stdout:9/628: creat d1/d3a/d91/d96/dbd/dbe/fda x:0 0 0 2024-04-05T03:02:59.777 INFO:tasks.workunit.client.0.smithi153.stdout:9/629: write d1/fc4 [1180261,11094] 0 2024-04-05T03:02:59.777 INFO:tasks.workunit.client.0.smithi153.stdout:8/697: mknod d7/d2a/d50/ce0 0 2024-04-05T03:02:59.780 INFO:tasks.workunit.client.0.smithi153.stdout:9/630: symlink d1/d10/d16/d22/d2c/d68/da9/ldb 0 2024-04-05T03:02:59.782 INFO:tasks.workunit.client.0.smithi153.stdout:8/698: creat d7/d85/fe1 x:0 0 0 2024-04-05T03:02:59.782 INFO:tasks.workunit.client.0.smithi153.stdout:9/631: stat d1/d10/d16/d22/d2c/la8 0 2024-04-05T03:02:59.782 INFO:tasks.workunit.client.0.smithi153.stdout:8/699: read - d7/d2a/d8b/fa6 zero size 2024-04-05T03:02:59.783 INFO:tasks.workunit.client.0.smithi153.stdout:2/792: dwrite d9/da/fe [0,4194304] 0 2024-04-05T03:02:59.787 INFO:tasks.workunit.client.0.smithi153.stdout:9/632: truncate d1/d10/d36/d71/fa0 1350201 0 2024-04-05T03:02:59.791 INFO:tasks.workunit.client.0.smithi153.stdout:9/633: creat d1/d10/d16/fdc x:0 0 0 2024-04-05T03:02:59.794 INFO:tasks.workunit.client.0.smithi153.stdout:9/634: chown d1/f2a 81428625 1 2024-04-05T03:02:59.795 INFO:tasks.workunit.client.0.smithi153.stdout:9/635: dread - d1/d10/d16/d22/d2c/d68/d81/fd7 zero size 2024-04-05T03:02:59.797 INFO:tasks.workunit.client.0.smithi153.stdout:2/793: write d9/da/d19/d2e/d7e/d7a/f57 [2045480,47622] 0 2024-04-05T03:02:59.802 INFO:tasks.workunit.client.0.smithi153.stdout:8/700: dwrite d7/d13/d19/d1b/d4f/f94 [0,4194304] 0 2024-04-05T03:02:59.804 INFO:tasks.workunit.client.0.smithi153.stdout:2/794: rename d9/da/d94/le7 to d9/da/d19/de2/l10d 0 2024-04-05T03:02:59.806 INFO:tasks.workunit.client.0.smithi153.stdout:9/636: link d1/d10/d16/d22/d2c/d68/d81/l44 d1/d9/ldd 0 2024-04-05T03:02:59.808 INFO:tasks.workunit.client.0.smithi153.stdout:8/701: rename d7/d13/d19/d95/dcb to d7/d13/d19/d1b/d4f/de2 0 2024-04-05T03:02:59.810 INFO:tasks.workunit.client.0.smithi153.stdout:9/637: rmdir d1/d10/d16/d22/d2c/d68/da9 39 2024-04-05T03:02:59.811 INFO:tasks.workunit.client.0.smithi153.stdout:8/702: readlink d7/d13/l34 0 2024-04-05T03:02:59.813 INFO:tasks.workunit.client.0.smithi153.stdout:2/795: dread d9/da/d37/fb6 [0,4194304] 0 2024-04-05T03:02:59.817 INFO:tasks.workunit.client.0.smithi153.stdout:8/703: stat d7/d13/d19/d1b/d4f/d68/dbc/fbe 0 2024-04-05T03:02:59.817 INFO:tasks.workunit.client.0.smithi153.stdout:2/796: mknod d9/da/d19/d2e/d7e/c10e 0 2024-04-05T03:02:59.819 INFO:tasks.workunit.client.0.smithi153.stdout:9/638: link d1/d10/d16/d22/c29 d1/d10/d16/d22/d2c/d68/cde 0 2024-04-05T03:02:59.819 INFO:tasks.workunit.client.0.smithi153.stdout:2/797: write d9/da/d19/d2e/d7e/de6/f33 [3995250,57283] 0 2024-04-05T03:02:59.820 INFO:tasks.workunit.client.0.smithi153.stdout:8/704: dread - d7/d13/d19/d1b/d4f/d5f/f87 zero size 2024-04-05T03:02:59.822 INFO:tasks.workunit.client.0.smithi153.stdout:2/798: mknod d9/d68/d9b/c10f 0 2024-04-05T03:02:59.823 INFO:tasks.workunit.client.0.smithi153.stdout:9/639: creat d1/d10/d16/d22/d2c/d68/da9/fdf x:0 0 0 2024-04-05T03:02:59.824 INFO:tasks.workunit.client.0.smithi153.stdout:9/640: stat d1/d10/d16/l95 0 2024-04-05T03:02:59.825 INFO:tasks.workunit.client.0.smithi153.stdout:8/705: creat d7/d13/d19/d1b/d4f/d5f/fe3 x:0 0 0 2024-04-05T03:02:59.826 INFO:tasks.workunit.client.0.smithi153.stdout:2/799: rename d9/da/d94/dbf/cec to d9/da/d19/d2e/d7e/d7a/c110 0 2024-04-05T03:02:59.828 INFO:tasks.workunit.client.0.smithi153.stdout:9/641: creat d1/fe0 x:0 0 0 2024-04-05T03:02:59.828 INFO:tasks.workunit.client.0.smithi153.stdout:2/800: read - d9/da/d94/dca/df7/f108 zero size 2024-04-05T03:02:59.828 INFO:tasks.workunit.client.0.smithi153.stdout:9/642: stat d1/d3a/l77 0 2024-04-05T03:02:59.829 INFO:tasks.workunit.client.0.smithi153.stdout:8/706: rename d7/d13/d19/d1b/d33/f59 to d7/d46/db1/fe4 0 2024-04-05T03:02:59.830 INFO:tasks.workunit.client.0.smithi153.stdout:2/801: readlink d9/da/d36/lbe 0 2024-04-05T03:02:59.831 INFO:tasks.workunit.client.0.smithi153.stdout:8/707: truncate d7/d9a/fbd 671365 0 2024-04-05T03:02:59.832 INFO:tasks.workunit.client.0.smithi153.stdout:9/643: rmdir d1/d10/d16/d22/d2c/d68/d81 39 2024-04-05T03:02:59.832 INFO:tasks.workunit.client.0.smithi153.stdout:8/708: readlink d7/d13/d19/d1b/d4f/d68/dbc/ld4 0 2024-04-05T03:02:59.832 INFO:tasks.workunit.client.0.smithi153.stdout:9/644: fdatasync d1/d3a/fb5 0 2024-04-05T03:02:59.833 INFO:tasks.workunit.client.0.smithi153.stdout:2/802: symlink d9/da/d94/dca/df7/l111 0 2024-04-05T03:02:59.835 INFO:tasks.workunit.client.0.smithi153.stdout:8/709: creat d7/d9a/fe5 x:0 0 0 2024-04-05T03:02:59.838 INFO:tasks.workunit.client.0.smithi153.stdout:2/803: creat d9/da/d37/f112 x:0 0 0 2024-04-05T03:02:59.840 INFO:tasks.workunit.client.0.smithi153.stdout:2/804: chown d9/da/d94/fc9 949490081 1 2024-04-05T03:02:59.842 INFO:tasks.workunit.client.0.smithi153.stdout:2/805: stat d9/da/d19/d2e/d7e 0 2024-04-05T03:02:59.844 INFO:tasks.workunit.client.0.smithi153.stdout:9/645: dwrite d1/d10/d16/d22/d2c/d68/d81/fd7 [0,4194304] 0 2024-04-05T03:02:59.845 INFO:tasks.workunit.client.0.smithi153.stdout:2/806: write d9/da/d19/d2e/dc7/fc0 [2353170,75500] 0 2024-04-05T03:02:59.847 INFO:tasks.workunit.client.0.smithi153.stdout:9/646: rmdir d1/d9 39 2024-04-05T03:02:59.849 INFO:tasks.workunit.client.0.smithi153.stdout:9/647: read d1/d3a/d91/fc5 [346449,58877] 0 2024-04-05T03:02:59.850 INFO:tasks.workunit.client.0.smithi153.stdout:2/807: write d9/da/d37/fc2 [1252680,15071] 0 2024-04-05T03:02:59.852 INFO:tasks.workunit.client.0.smithi153.stdout:8/710: dread d7/f74 [0,4194304] 0 2024-04-05T03:02:59.853 INFO:tasks.workunit.client.0.smithi153.stdout:2/808: mknod d9/c113 0 2024-04-05T03:02:59.856 INFO:tasks.workunit.client.0.smithi153.stdout:9/648: getdents d1/d10/d36/d57 0 2024-04-05T03:02:59.862 INFO:tasks.workunit.client.0.smithi153.stdout:2/809: dread d9/da/d19/f22 [0,4194304] 0 2024-04-05T03:02:59.863 INFO:tasks.workunit.client.0.smithi153.stdout:2/810: truncate d9/da/d19/d2e/d7e/d102/f107 687060 0 2024-04-05T03:02:59.903 INFO:tasks.workunit.client.0.smithi153.stdout:9/649: fsync d1/d10/d16/d22/d2c/d68/d81/fd7 0 2024-04-05T03:02:59.903 INFO:tasks.workunit.client.0.smithi153.stdout:7/927: sync 2024-04-05T03:02:59.903 INFO:tasks.workunit.client.0.smithi153.stdout:4/844: sync 2024-04-05T03:02:59.903 INFO:tasks.workunit.client.0.smithi153.stdout:0/720: sync 2024-04-05T03:02:59.903 INFO:tasks.workunit.client.0.smithi153.stdout:6/864: sync 2024-04-05T03:02:59.906 INFO:tasks.workunit.client.0.smithi153.stdout:6/865: dread - d4/dc/df/dcb/f116 zero size 2024-04-05T03:02:59.907 INFO:tasks.workunit.client.0.smithi153.stdout:4/845: fdatasync da/db/d20/d35/d38/d76/fc6 0 2024-04-05T03:02:59.908 INFO:tasks.workunit.client.0.smithi153.stdout:7/928: symlink d2/da/d3a/d41/da8/l12f 0 2024-04-05T03:02:59.910 INFO:tasks.workunit.client.0.smithi153.stdout:0/721: unlink d9/db/d42/d7e/db6/d1e/f50 0 2024-04-05T03:02:59.911 INFO:tasks.workunit.client.0.smithi153.stdout:9/650: readlink d1/d10/d16/d22/d2c/d68/d81/l44 0 2024-04-05T03:02:59.913 INFO:tasks.workunit.client.0.smithi153.stdout:4/846: mknod da/de/d56/d6a/d2d/ddb/c124 0 2024-04-05T03:02:59.913 INFO:tasks.workunit.client.0.smithi153.stdout:2/811: getdents d9/da/d19/d2e/d7e/d7a/d59 0 2024-04-05T03:02:59.914 INFO:tasks.workunit.client.0.smithi153.stdout:7/929: creat d2/da/d47/da9/f130 x:0 0 0 2024-04-05T03:02:59.915 INFO:tasks.workunit.client.0.smithi153.stdout:6/866: creat d4/dc/df/dcb/dfe/f12e x:0 0 0 2024-04-05T03:02:59.917 INFO:tasks.workunit.client.0.smithi153.stdout:0/722: rename d9/d68 to d9/db/d42/d7e/db6/d1e/d45/d62/dde 0 2024-04-05T03:02:59.921 INFO:tasks.workunit.client.0.smithi153.stdout:6/867: rmdir d4/dc/df/d16/d44/ded 39 2024-04-05T03:02:59.924 INFO:tasks.workunit.client.0.smithi153.stdout:6/868: dread - d4/dc/df/d16/d129/f126 zero size 2024-04-05T03:02:59.926 INFO:tasks.workunit.client.0.smithi153.stdout:4/847: truncate da/de/d56/d78/d8f/f112 364925 0 2024-04-05T03:02:59.927 INFO:tasks.workunit.client.0.smithi153.stdout:0/723: dread d9/db/d42/d7e/db6/f16 [4194304,4194304] 0 2024-04-05T03:02:59.928 INFO:tasks.workunit.client.0.smithi153.stdout:7/930: dwrite d2/da/d3a/d41/da8/d12e/f12c [0,4194304] 0 2024-04-05T03:02:59.931 INFO:tasks.workunit.client.0.smithi153.stdout:0/724: readlink d9/db/l9c 0 2024-04-05T03:02:59.934 INFO:tasks.workunit.client.0.smithi153.stdout:6/869: chown d4/dc/df/d16/d24/d5e/de8/dfa/c108 5 1 2024-04-05T03:02:59.935 INFO:tasks.workunit.client.0.smithi153.stdout:9/651: rename d1/d10/d36/d57/d97/ld0 to d1/d10/le1 0 2024-04-05T03:02:59.938 INFO:tasks.workunit.client.0.smithi153.stdout:2/812: getdents d9/d77 0 2024-04-05T03:02:59.942 INFO:tasks.workunit.client.0.smithi153.stdout:2/813: creat d9/da/d37/d8a/f114 x:0 0 0 2024-04-05T03:02:59.943 INFO:tasks.workunit.client.0.smithi153.stdout:7/931: write d2/f6 [1197769,76592] 0 2024-04-05T03:02:59.945 INFO:tasks.workunit.client.0.smithi153.stdout:6/870: mknod d4/dc/df/d16/d5f/c12f 0 2024-04-05T03:02:59.945 INFO:tasks.workunit.client.0.smithi153.stdout:9/652: dwrite d1/d10/d36/d57/f78 [0,4194304] 0 2024-04-05T03:02:59.947 INFO:tasks.workunit.client.0.smithi153.stdout:4/848: link da/db/d20/d3c/l55 da/db/d20/l125 0 2024-04-05T03:02:59.947 INFO:tasks.workunit.client.0.smithi153.stdout:4/849: readlink l6 0 2024-04-05T03:02:59.954 INFO:tasks.workunit.client.0.smithi153.stdout:9/653: unlink d1/d10/d16/d22/d2c/d68/fad 0 2024-04-05T03:02:59.957 INFO:tasks.workunit.client.0.smithi153.stdout:9/654: dread d1/d10/d16/d22/d2c/f58 [4194304,4194304] 0 2024-04-05T03:02:59.964 INFO:tasks.workunit.client.0.smithi153.stdout:2/814: truncate d9/da/d19/d2e/d7e/f5e 2507583 0 2024-04-05T03:02:59.964 INFO:tasks.workunit.client.0.smithi153.stdout:7/932: dwrite d2/da/d47/d5d/d7c/f81 [0,4194304] 0 2024-04-05T03:02:59.964 INFO:tasks.workunit.client.0.smithi153.stdout:9/655: mkdir d1/d9/d1b/de2 0 2024-04-05T03:02:59.972 INFO:tasks.workunit.client.0.smithi153.stdout:9/656: write d1/d3a/d91/f9c [233840,30401] 0 2024-04-05T03:02:59.974 INFO:tasks.workunit.client.0.smithi153.stdout:0/725: dread d9/db/d42/d7e/f9a [4194304,4194304] 0 2024-04-05T03:02:59.976 INFO:tasks.workunit.client.0.smithi153.stdout:7/933: mkdir d2/da/d3a/d41/d131 0 2024-04-05T03:02:59.976 INFO:tasks.workunit.client.0.smithi153.stdout:2/815: dwrite d9/da/d19/d2e/d7e/d7a/f57 [0,4194304] 0 2024-04-05T03:02:59.979 INFO:tasks.workunit.client.0.smithi153.stdout:4/850: dread f7 [0,4194304] 0 2024-04-05T03:02:59.986 INFO:tasks.workunit.client.0.smithi153.stdout:0/726: mkdir d9/db/ddf 0 2024-04-05T03:02:59.988 INFO:tasks.workunit.client.0.smithi153.stdout:9/657: write d1/d3a/f83 [302924,31318] 0 2024-04-05T03:02:59.990 INFO:tasks.workunit.client.0.smithi153.stdout:7/934: rename d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/f105 to d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106/f132 0 2024-04-05T03:02:59.992 INFO:tasks.workunit.client.0.smithi153.stdout:7/935: stat d2/da/d11/d88/da6/fc2 0 2024-04-05T03:02:59.992 INFO:tasks.workunit.client.0.smithi153.stdout:2/816: creat d9/da/d19/d2e/d7e/f115 x:0 0 0 2024-04-05T03:02:59.992 INFO:tasks.workunit.client.0.smithi153.stdout:2/817: chown d9/da/df9 25115248 1 2024-04-05T03:02:59.996 INFO:tasks.workunit.client.0.smithi153.stdout:9/658: mknod d1/d10/d36/d57/ce3 0 2024-04-05T03:03:00.000 INFO:tasks.workunit.client.0.smithi153.stdout:4/851: getdents da/db 0 2024-04-05T03:03:00.010 INFO:tasks.workunit.client.0.smithi153.stdout:4/852: symlink da/db/dcf/l126 0 2024-04-05T03:03:00.010 INFO:tasks.workunit.client.0.smithi153.stdout:4/853: fdatasync da/de/d66/d111/d86/f119 0 2024-04-05T03:03:00.010 INFO:tasks.workunit.client.0.smithi153.stdout:0/727: link d9/db/d42/f74 d9/db/d42/d7e/db6/d1d/d61/fe0 0 2024-04-05T03:03:00.010 INFO:tasks.workunit.client.0.smithi153.stdout:7/936: truncate d2/da/d3a/d41/d7a/f4e 3080666 0 2024-04-05T03:03:00.012 INFO:tasks.workunit.client.0.smithi153.stdout:9/659: write d1/d10/d16/d22/d2c/f46 [242439,96079] 0 2024-04-05T03:03:00.012 INFO:tasks.workunit.client.0.smithi153.stdout:4/854: symlink da/db/d20/l127 0 2024-04-05T03:03:00.015 INFO:tasks.workunit.client.0.smithi153.stdout:7/937: mkdir d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/df2/d133 0 2024-04-05T03:03:00.017 INFO:tasks.workunit.client.0.smithi153.stdout:0/728: getdents d9/db/d42/d7e 0 2024-04-05T03:03:00.020 INFO:tasks.workunit.client.0.smithi153.stdout:9/660: link d1/d3a/d91/f9b d1/d10/d16/d22/d2c/d68/d81/fe4 0 2024-04-05T03:03:00.023 INFO:tasks.workunit.client.0.smithi153.stdout:4/855: dwrite da/de/d66/d111/d86/fe2 [0,4194304] 0 2024-04-05T03:03:00.024 INFO:tasks.workunit.client.0.smithi153.stdout:2/818: dread d9/da/d36/f4d [0,4194304] 0 2024-04-05T03:03:00.028 INFO:tasks.workunit.client.0.smithi153.stdout:9/661: rename d1/d9/d1b/f30 to d1/d3a/fe5 0 2024-04-05T03:03:00.035 INFO:tasks.workunit.client.0.smithi153.stdout:2/819: mkdir d9/da/d37/d8a/da8/de5/d116 0 2024-04-05T03:03:00.037 INFO:tasks.workunit.client.0.smithi153.stdout:7/938: truncate d2/da/d3a/d41/ff1 779953 0 2024-04-05T03:03:00.039 INFO:tasks.workunit.client.0.smithi153.stdout:0/729: truncate d9/db/d42/d7e/db6/f18 2786654 0 2024-04-05T03:03:00.047 INFO:tasks.workunit.client.0.smithi153.stdout:7/939: dwrite d2/da/ff5 [0,4194304] 0 2024-04-05T03:03:00.051 INFO:tasks.workunit.client.0.smithi153.stdout:2/820: dwrite d9/da/d37/fb6 [0,4194304] 0 2024-04-05T03:03:00.052 INFO:tasks.workunit.client.0.smithi153.stdout:7/940: mkdir d2/da/d11/d39/d134 0 2024-04-05T03:03:00.056 INFO:tasks.workunit.client.0.smithi153.stdout:4/856: dread da/de/d56/d78/f7e [0,4194304] 0 2024-04-05T03:03:00.059 INFO:tasks.workunit.client.0.smithi153.stdout:2/821: dread d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:03:00.062 INFO:tasks.workunit.client.0.smithi153.stdout:4/857: creat da/de/d56/d6a/d2d/ddb/f128 x:0 0 0 2024-04-05T03:03:00.064 INFO:tasks.workunit.client.0.smithi153.stdout:7/941: truncate d2/da/d47/d5d/d7c/d118/f125 1240063 0 2024-04-05T03:03:00.065 INFO:tasks.workunit.client.0.smithi153.stdout:7/942: chown d2/d9b 35 1 2024-04-05T03:03:00.066 INFO:tasks.workunit.client.0.smithi153.stdout:4/858: mknod da/db/d20/d35/d38/df7/c129 0 2024-04-05T03:03:00.067 INFO:tasks.workunit.client.0.smithi153.stdout:9/662: dread d1/d6c/f6e [0,4194304] 0 2024-04-05T03:03:00.068 INFO:tasks.workunit.client.0.smithi153.stdout:7/943: read - d2/da/d3a/d41/da8/d12e/ffc zero size 2024-04-05T03:03:00.069 INFO:tasks.workunit.client.0.smithi153.stdout:2/822: dwrite d9/da/d37/f112 [0,4194304] 0 2024-04-05T03:03:00.070 INFO:tasks.workunit.client.0.smithi153.stdout:4/859: mkdir da/de/d56/d12a 0 2024-04-05T03:03:00.071 INFO:tasks.workunit.client.0.smithi153.stdout:9/663: rmdir d1/d10/d36/d71 39 2024-04-05T03:03:00.073 INFO:tasks.workunit.client.0.smithi153.stdout:7/944: read d2/da/d3a/d41/da8/fad [61854,63335] 0 2024-04-05T03:03:00.074 INFO:tasks.workunit.client.0.smithi153.stdout:4/860: stat da/de/d56/d78/d8f/d68/f95 0 2024-04-05T03:03:00.075 INFO:tasks.workunit.client.0.smithi153.stdout:9/664: mkdir d1/d3a/d91/d96/de6 0 2024-04-05T03:03:00.077 INFO:tasks.workunit.client.0.smithi153.stdout:9/665: write d1/d9/d1b/f26 [2643829,115064] 0 2024-04-05T03:03:00.078 INFO:tasks.workunit.client.0.smithi153.stdout:7/945: creat d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/f135 x:0 0 0 2024-04-05T03:03:00.078 INFO:tasks.workunit.client.0.smithi153.stdout:9/666: stat d1/d10/d36/d57/d97/fce 0 2024-04-05T03:03:00.080 INFO:tasks.workunit.client.0.smithi153.stdout:7/946: write d2/da/d3a/d41/f19 [2642720,108894] 0 2024-04-05T03:03:00.087 INFO:tasks.workunit.client.0.smithi153.stdout:7/947: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/dc8/c136 0 2024-04-05T03:03:00.092 INFO:tasks.workunit.client.0.smithi153.stdout:9/667: write d1/d10/d16/f17 [689180,100230] 0 2024-04-05T03:03:00.094 INFO:tasks.workunit.client.0.smithi153.stdout:2/823: dread d9/da/f15 [0,4194304] 0 2024-04-05T03:03:00.094 INFO:tasks.workunit.client.0.smithi153.stdout:7/948: dwrite d2/f5 [4194304,4194304] 0 2024-04-05T03:03:00.095 INFO:tasks.workunit.client.0.smithi153.stdout:2/824: readlink d9/da/d19/l1b 0 2024-04-05T03:03:00.097 INFO:tasks.workunit.client.0.smithi153.stdout:9/668: creat d1/d3a/d91/d96/fe7 x:0 0 0 2024-04-05T03:03:00.098 INFO:tasks.workunit.client.0.smithi153.stdout:7/949: creat d2/da/d3a/d41/d131/f137 x:0 0 0 2024-04-05T03:03:00.101 INFO:tasks.workunit.client.0.smithi153.stdout:2/825: rename d9/da/d19/f6a to d9/da/d37/d8a/da8/de5/d116/f117 0 2024-04-05T03:03:00.105 INFO:tasks.workunit.client.0.smithi153.stdout:9/669: fsync d1/d9/d1b/f73 0 2024-04-05T03:03:00.106 INFO:tasks.workunit.client.0.smithi153.stdout:7/950: creat d2/da/d10e/f138 x:0 0 0 2024-04-05T03:03:00.107 INFO:tasks.workunit.client.0.smithi153.stdout:7/951: fdatasync d2/da/d47/d5d/d7c/fec 0 2024-04-05T03:03:00.107 INFO:tasks.workunit.client.0.smithi153.stdout:2/826: symlink d9/d68/d98/l118 0 2024-04-05T03:03:00.108 INFO:tasks.workunit.client.0.smithi153.stdout:7/952: readlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/l7f 0 2024-04-05T03:03:00.109 INFO:tasks.workunit.client.0.smithi153.stdout:9/670: mknod d1/d9/d1b/ce8 0 2024-04-05T03:03:00.110 INFO:tasks.workunit.client.0.smithi153.stdout:2/827: mknod d9/d68/db9/c119 0 2024-04-05T03:03:00.111 INFO:tasks.workunit.client.0.smithi153.stdout:7/953: symlink d2/da/d11/d39/d12b/l139 0 2024-04-05T03:03:00.112 INFO:tasks.workunit.client.0.smithi153.stdout:7/954: stat d2/da/d47/l89 0 2024-04-05T03:03:00.112 INFO:tasks.workunit.client.0.smithi153.stdout:2/828: fdatasync d9/d68/fab 0 2024-04-05T03:03:00.115 INFO:tasks.workunit.client.0.smithi153.stdout:7/955: truncate d2/d9b/f61 7331683 0 2024-04-05T03:03:00.117 INFO:tasks.workunit.client.0.smithi153.stdout:2/829: creat d9/da/d19/d2e/d7e/d53/da1/f11a x:0 0 0 2024-04-05T03:03:00.120 INFO:tasks.workunit.client.0.smithi153.stdout:2/830: mkdir d9/da/d19/d2e/d11b 0 2024-04-05T03:03:00.122 INFO:tasks.workunit.client.0.smithi153.stdout:2/831: chown d9/cb1 0 1 2024-04-05T03:03:00.124 INFO:tasks.workunit.client.0.smithi153.stdout:2/832: creat d9/da/d19/d2e/d7e/de6/f11c x:0 0 0 2024-04-05T03:03:00.131 INFO:tasks.workunit.client.0.smithi153.stdout:7/956: dread d2/da/d3a/d41/d31/f45 [4194304,4194304] 0 2024-04-05T03:03:00.133 INFO:tasks.workunit.client.0.smithi153.stdout:7/957: readlink d2/lc4 0 2024-04-05T03:03:00.136 INFO:tasks.workunit.client.0.smithi153.stdout:2/833: dwrite d9/da/d19/f22 [0,4194304] 0 2024-04-05T03:03:00.143 INFO:tasks.workunit.client.0.smithi153.stdout:7/958: dwrite d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/f135 [0,4194304] 0 2024-04-05T03:03:00.148 INFO:tasks.workunit.client.0.smithi153.stdout:7/959: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d93/d106/d110/c13a 0 2024-04-05T03:03:00.150 INFO:tasks.workunit.client.0.smithi153.stdout:7/960: dread - d2/da/d3a/d41/da8/d12e/ffc zero size 2024-04-05T03:03:00.151 INFO:tasks.workunit.client.0.smithi153.stdout:7/961: stat d2/da/d3a/d41/d7a/f11c 0 2024-04-05T03:03:00.157 INFO:tasks.workunit.client.0.smithi153.stdout:7/962: dwrite d2/da/d3a/d41/d7a/fd3 [0,4194304] 0 2024-04-05T03:03:00.160 INFO:tasks.workunit.client.0.smithi153.stdout:7/963: symlink d2/da/d47/da9/l13b 0 2024-04-05T03:03:00.162 INFO:tasks.workunit.client.0.smithi153.stdout:7/964: write d2/f5 [7828043,52953] 0 2024-04-05T03:03:00.164 INFO:tasks.workunit.client.0.smithi153.stdout:2/834: dread d9/da/d19/d2e/d7e/d7a/d59/f8d [0,4194304] 0 2024-04-05T03:03:00.168 INFO:tasks.workunit.client.0.smithi153.stdout:7/965: getdents d2 0 2024-04-05T03:03:00.169 INFO:tasks.workunit.client.0.smithi153.stdout:2/835: creat d9/da/d19/d2e/f11d x:0 0 0 2024-04-05T03:03:00.170 INFO:tasks.workunit.client.0.smithi153.stdout:2/836: chown d9/d68/ca2 18899 1 2024-04-05T03:03:00.171 INFO:tasks.workunit.client.0.smithi153.stdout:2/837: chown d9/da/d19/d2e/d7e/c73 10652655 1 2024-04-05T03:03:00.171 INFO:tasks.workunit.client.0.smithi153.stdout:7/966: creat d2/da/d47/d5d/d7c/f13c x:0 0 0 2024-04-05T03:03:00.172 INFO:tasks.workunit.client.0.smithi153.stdout:7/967: readlink d2/da/d11/d39/d12b/l139 0 2024-04-05T03:03:00.172 INFO:tasks.workunit.client.0.smithi153.stdout:2/838: write d9/da/d19/d2e/d7e/de6/f11c [596393,107324] 0 2024-04-05T03:03:00.174 INFO:tasks.workunit.client.0.smithi153.stdout:7/968: fsync d2/da/d47/d5d/d7c/f5b 0 2024-04-05T03:03:00.178 INFO:tasks.workunit.client.0.smithi153.stdout:2/839: write d9/da/f24 [5087161,915] 0 2024-04-05T03:03:00.180 INFO:tasks.workunit.client.0.smithi153.stdout:7/969: creat d2/da/d3a/d41/d7a/d54/d85/df6/d93/f13d x:0 0 0 2024-04-05T03:03:00.184 INFO:tasks.workunit.client.0.smithi153.stdout:7/970: getdents d2/da/d3a/d41/da8/d12e 0 2024-04-05T03:03:00.185 INFO:tasks.workunit.client.0.smithi153.stdout:2/840: dwrite d9/da/d19/d2e/dc7/fcc [0,4194304] 0 2024-04-05T03:03:00.190 INFO:tasks.workunit.client.0.smithi153.stdout:7/971: fdatasync d2/da/d3a/d41/d7a/d54/d85/df6/d93/f74 0 2024-04-05T03:03:00.190 INFO:tasks.workunit.client.0.smithi153.stdout:2/841: truncate d9/da/d19/d2e/d4f/f86 4559473 0 2024-04-05T03:03:00.193 INFO:tasks.workunit.client.0.smithi153.stdout:7/972: symlink d2/l13e 0 2024-04-05T03:03:00.205 INFO:tasks.workunit.client.0.smithi153.stdout:7/973: readlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/l64 0 2024-04-05T03:03:00.205 INFO:tasks.workunit.client.0.smithi153.stdout:2/842: dread d9/da/d19/d2e/d4f/f86 [0,4194304] 0 2024-04-05T03:03:00.207 INFO:tasks.workunit.client.0.smithi153.stdout:2/843: creat d9/d68/d9b/df0/f11e x:0 0 0 2024-04-05T03:03:00.208 INFO:tasks.workunit.client.0.smithi153.stdout:7/974: truncate d2/d9b/d6f/f78 3818564 0 2024-04-05T03:03:00.208 INFO:tasks.workunit.client.0.smithi153.stdout:2/844: fdatasync d9/da/f15 0 2024-04-05T03:03:00.211 INFO:tasks.workunit.client.0.smithi153.stdout:7/975: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/c13f 0 2024-04-05T03:03:00.212 INFO:tasks.workunit.client.0.smithi153.stdout:7/976: dread - d2/da/d3a/d41/d7a/d54/d85/df6/d93/f13d zero size 2024-04-05T03:03:00.214 INFO:tasks.workunit.client.0.smithi153.stdout:2/845: dwrite d9/da/d19/d2e/dc7/fcc [0,4194304] 0 2024-04-05T03:03:00.216 INFO:tasks.workunit.client.0.smithi153.stdout:2/846: dread - d9/da/d19/d2e/d7e/f115 zero size 2024-04-05T03:03:00.217 INFO:tasks.workunit.client.0.smithi153.stdout:7/977: symlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/l140 0 2024-04-05T03:03:00.217 INFO:tasks.workunit.client.0.smithi153.stdout:7/978: stat d2/da/d3a/d41/d131/f137 0 2024-04-05T03:03:00.226 INFO:tasks.workunit.client.0.smithi153.stdout:7/979: dwrite d2/f1d [0,4194304] 0 2024-04-05T03:03:00.228 INFO:tasks.workunit.client.0.smithi153.stdout:7/980: chown d2/da/d3a/ffb 906619 1 2024-04-05T03:03:00.229 INFO:tasks.workunit.client.0.smithi153.stdout:2/847: dwrite d9/d68/d9b/fb4 [0,4194304] 0 2024-04-05T03:03:00.235 INFO:tasks.workunit.client.0.smithi153.stdout:2/848: link d9/da/d19/d2e/d7e/f115 d9/d68/f11f 0 2024-04-05T03:03:00.237 INFO:tasks.workunit.client.0.smithi153.stdout:2/849: write d9/da/d19/de2/ff5 [268849,3313] 0 2024-04-05T03:03:00.240 INFO:tasks.workunit.client.0.smithi153.stdout:2/850: symlink d9/da/d19/d2e/d7e/d7a/d6d/l120 0 2024-04-05T03:03:00.243 INFO:tasks.workunit.client.0.smithi153.stdout:2/851: link d9/da/d19/d2e/dc7/fbb d9/da/f121 0 2024-04-05T03:03:00.262 INFO:tasks.workunit.client.0.smithi153.stdout:2/852: dread d9/d68/d98/fac [0,4194304] 0 2024-04-05T03:03:00.269 INFO:tasks.workunit.client.0.smithi153.stdout:2/853: dwrite d9/d68/f11f [0,4194304] 0 2024-04-05T03:03:00.270 INFO:tasks.workunit.client.0.smithi153.stdout:2/854: chown d9/da/fd0 107918 1 2024-04-05T03:03:00.271 INFO:tasks.workunit.client.0.smithi153.stdout:2/855: readlink d9/da/d36/l104 0 2024-04-05T03:03:00.273 INFO:tasks.workunit.client.0.smithi153.stdout:2/856: creat d9/da/d19/d2e/d7e/d102/f122 x:0 0 0 2024-04-05T03:03:00.274 INFO:tasks.workunit.client.0.smithi153.stdout:2/857: read d9/d68/fce [493326,100880] 0 2024-04-05T03:03:00.286 INFO:tasks.workunit.client.0.smithi153.stdout:2/858: write d9/da/d36/f55 [2096774,101391] 0 2024-04-05T03:03:00.291 INFO:tasks.workunit.client.0.smithi153.stdout:2/859: dread d9/da/d19/d2e/d7e/de6/f6b [0,4194304] 0 2024-04-05T03:03:00.294 INFO:tasks.workunit.client.0.smithi153.stdout:2/860: unlink d9/da/d19/d2e/d7e/d102/f122 0 2024-04-05T03:03:00.297 INFO:tasks.workunit.client.0.smithi153.stdout:2/861: getdents d9/da/d19/de2 0 2024-04-05T03:03:00.299 INFO:tasks.workunit.client.0.smithi153.stdout:2/862: unlink d9/d68/cc5 0 2024-04-05T03:03:00.301 INFO:tasks.workunit.client.0.smithi153.stdout:2/863: creat d9/da/d94/dca/f123 x:0 0 0 2024-04-05T03:03:00.302 INFO:tasks.workunit.client.0.smithi153.stdout:2/864: write d9/da/d37/d8a/ff1 [5015990,3638] 0 2024-04-05T03:03:00.304 INFO:tasks.workunit.client.0.smithi153.stdout:2/865: truncate d9/da/d19/d2e/d7e/d53/fe3 5441376 0 2024-04-05T03:03:00.461 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:00 smithi153 ceph-mon[29381]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:00.461 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:00 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:00.461 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:00 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:00 smithi203 ceph-mon[39911]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:00 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:00.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:00 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:00.549 INFO:tasks.workunit.client.0.smithi153.stdout:3/755: sync 2024-04-05T03:03:00.549 INFO:tasks.workunit.client.0.smithi153.stdout:1/771: sync 2024-04-05T03:03:00.551 INFO:tasks.workunit.client.0.smithi153.stdout:1/772: write d5/d8/d20/d4b/d18/d16/d33/f3c [505309,105700] 0 2024-04-05T03:03:00.553 INFO:tasks.workunit.client.0.smithi153.stdout:3/756: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/fa6 zero size 2024-04-05T03:03:00.554 INFO:tasks.workunit.client.0.smithi153.stdout:1/773: dread - d5/d8/d20/d3f/f86 zero size 2024-04-05T03:03:00.555 INFO:tasks.workunit.client.0.smithi153.stdout:3/757: mknod d1d/d10d/c10e 0 2024-04-05T03:03:00.558 INFO:tasks.workunit.client.0.smithi153.stdout:1/774: getdents d5/d8/d20/d4b/d18 0 2024-04-05T03:03:00.560 INFO:tasks.workunit.client.0.smithi153.stdout:3/758: link d1d/d75/cbc d1d/d75/c10f 0 2024-04-05T03:03:00.562 INFO:tasks.workunit.client.0.smithi153.stdout:3/759: fdatasync d1d/d27/d2b/f2f 0 2024-04-05T03:03:00.564 INFO:tasks.workunit.client.0.smithi153.stdout:1/775: link d5/d8/d20/d4b/d18/c9c d5/d8/d20/d2f/ce9 0 2024-04-05T03:03:00.566 INFO:tasks.workunit.client.0.smithi153.stdout:1/776: mknod d5/cea 0 2024-04-05T03:03:00.572 INFO:tasks.workunit.client.0.smithi153.stdout:1/777: mknod d5/d8/d20/d4b/d18/d16/d2c/ceb 0 2024-04-05T03:03:00.578 INFO:tasks.workunit.client.0.smithi153.stdout:3/760: dwrite d1d/d32/d93/f4c [0,4194304] 0 2024-04-05T03:03:00.588 INFO:tasks.workunit.client.0.smithi153.stdout:3/761: read d1d/d27/f8f [1231666,23973] 0 2024-04-05T03:03:00.589 INFO:tasks.workunit.client.0.smithi153.stdout:1/778: dread d5/dd/d1b/f74 [0,4194304] 0 2024-04-05T03:03:00.589 INFO:tasks.workunit.client.0.smithi153.stdout:3/762: write d1d/d32/d93/d3c/f5b [18062,9520] 0 2024-04-05T03:03:00.597 INFO:tasks.workunit.client.0.smithi153.stdout:3/763: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 [0,4194304] 0 2024-04-05T03:03:00.607 INFO:tasks.workunit.client.0.smithi153.stdout:3/764: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/l110 0 2024-04-05T03:03:00.608 INFO:tasks.workunit.client.0.smithi153.stdout:3/765: chown d1d/d32/d93/d3c/l52 16463514 1 2024-04-05T03:03:00.609 INFO:tasks.workunit.client.0.smithi153.stdout:3/766: rmdir d1d/d32/d93/d3c/d94 39 2024-04-05T03:03:00.610 INFO:tasks.workunit.client.0.smithi153.stdout:9/671: sync 2024-04-05T03:03:00.610 INFO:tasks.workunit.client.0.smithi153.stdout:5/703: sync 2024-04-05T03:03:00.610 INFO:tasks.workunit.client.0.smithi153.stdout:8/711: sync 2024-04-05T03:03:00.613 INFO:tasks.workunit.client.0.smithi153.stdout:8/712: unlink d7/d13/d19/ldf 0 2024-04-05T03:03:00.614 INFO:tasks.workunit.client.0.smithi153.stdout:9/672: read - d1/d10/f4c zero size 2024-04-05T03:03:00.615 INFO:tasks.workunit.client.0.smithi153.stdout:8/713: dread - d7/d13/d19/d1b/d4f/d5f/f87 zero size 2024-04-05T03:03:00.619 INFO:tasks.workunit.client.0.smithi153.stdout:3/767: rename d1d/c2e to d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/c111 0 2024-04-05T03:03:00.620 INFO:tasks.workunit.client.0.smithi153.stdout:9/673: unlink d1/d10/d16/d22/d2c/la8 0 2024-04-05T03:03:00.622 INFO:tasks.workunit.client.0.smithi153.stdout:5/704: getdents d0/d25 0 2024-04-05T03:03:00.624 INFO:tasks.workunit.client.0.smithi153.stdout:5/705: rmdir d0/d25/d4d/d62/d37/d38 39 2024-04-05T03:03:00.627 INFO:tasks.workunit.client.0.smithi153.stdout:5/706: truncate d0/d25/f70 5055505 0 2024-04-05T03:03:00.705 INFO:tasks.workunit.client.0.smithi153.stdout:5/707: truncate d0/d25/f36 235970 0 2024-04-05T03:03:00.709 INFO:tasks.workunit.client.0.smithi153.stdout:1/779: dwrite d5/f84 [0,4194304] 0 2024-04-05T03:03:00.712 INFO:tasks.workunit.client.0.smithi153.stdout:3/768: dread d1d/d27/f8f [0,4194304] 0 2024-04-05T03:03:00.714 INFO:tasks.workunit.client.0.smithi153.stdout:1/780: creat d5/d8/d20/d4b/d18/d16/d33/fec x:0 0 0 2024-04-05T03:03:00.722 INFO:tasks.workunit.client.0.smithi153.stdout:1/781: dwrite d5/d8/d20/d4b/d18/d16/f22 [4194304,4194304] 0 2024-04-05T03:03:00.722 INFO:tasks.workunit.client.0.smithi153.stdout:5/708: dread d0/d25/d4d/d62/d37/fa6 [0,4194304] 0 2024-04-05T03:03:00.725 INFO:tasks.workunit.client.0.smithi153.stdout:1/782: unlink d5/d8/d20/d4b/d51/da2/c8c 0 2024-04-05T03:03:00.726 INFO:tasks.workunit.client.0.smithi153.stdout:5/709: unlink d0/d25/l79 0 2024-04-05T03:03:00.730 INFO:tasks.workunit.client.0.smithi153.stdout:1/783: creat d5/d8/d20/fed x:0 0 0 2024-04-05T03:03:00.732 INFO:tasks.workunit.client.0.smithi153.stdout:5/710: link d0/d25/d4d/d9c/fd8 d0/d25/d4d/d9c/dc4/ff3 0 2024-04-05T03:03:00.734 INFO:tasks.workunit.client.0.smithi153.stdout:1/784: link d5/d8/d20/d4b/d18/d16/fe6 d5/d8/d20/d4b/d18/fee 0 2024-04-05T03:03:00.735 INFO:tasks.workunit.client.0.smithi153.stdout:5/711: creat d0/d25/d4d/d9c/dc4/dcb/ff4 x:0 0 0 2024-04-05T03:03:00.738 INFO:tasks.workunit.client.0.smithi153.stdout:5/712: creat d0/d25/d4d/d62/dbb/ff5 x:0 0 0 2024-04-05T03:03:00.740 INFO:tasks.workunit.client.0.smithi153.stdout:1/785: rename d5/d8/d20/d4b/d18/d16/d2c/c6f to d5/d8/d20/d4b/cef 0 2024-04-05T03:03:00.743 INFO:tasks.workunit.client.0.smithi153.stdout:5/713: link d0/d25/d4d/faf d0/d25/d4d/d9c/ff6 0 2024-04-05T03:03:00.747 INFO:tasks.workunit.client.0.smithi153.stdout:3/769: dread d1d/d32/d93/d3c/d3f/d7c/fa8 [0,4194304] 0 2024-04-05T03:03:00.748 INFO:tasks.workunit.client.0.smithi153.stdout:3/770: chown d1d/d27/df4/c101 2 1 2024-04-05T03:03:00.757 INFO:tasks.workunit.client.0.smithi153.stdout:1/786: write d5/d8/fc0 [1129660,81721] 0 2024-04-05T03:03:00.757 INFO:tasks.workunit.client.0.smithi153.stdout:5/714: write d0/d25/d4d/d62/f10 [1240169,71152] 0 2024-04-05T03:03:00.759 INFO:tasks.workunit.client.0.smithi153.stdout:5/715: dread - d0/d25/d4d/d9c/fef zero size 2024-04-05T03:03:00.761 INFO:tasks.workunit.client.0.smithi153.stdout:1/787: mknod d5/d8/d20/d4b/d51/db5/cf0 0 2024-04-05T03:03:00.762 INFO:tasks.workunit.client.0.smithi153.stdout:1/788: stat d5/d8/d20/d4b/d18/d16 0 2024-04-05T03:03:00.763 INFO:tasks.workunit.client.0.smithi153.stdout:5/716: mkdir d0/d2c/d3f/df7 0 2024-04-05T03:03:00.763 INFO:tasks.workunit.client.0.smithi153.stdout:1/789: stat d5/d8/d20/d2f/d40 0 2024-04-05T03:03:00.765 INFO:tasks.workunit.client.0.smithi153.stdout:1/790: creat d5/d8/d20/d2f/d40/ff1 x:0 0 0 2024-04-05T03:03:00.766 INFO:tasks.workunit.client.0.smithi153.stdout:5/717: creat d0/d25/d4d/d62/d5f/db2/ff8 x:0 0 0 2024-04-05T03:03:00.767 INFO:tasks.workunit.client.0.smithi153.stdout:3/771: dread d1d/d27/d2b/f62 [0,4194304] 0 2024-04-05T03:03:00.769 INFO:tasks.workunit.client.0.smithi153.stdout:5/718: mknod d0/d25/d4d/d62/d5f/db2/cf9 0 2024-04-05T03:03:00.770 INFO:tasks.workunit.client.0.smithi153.stdout:3/772: fsync d1d/fcf 0 2024-04-05T03:03:00.772 INFO:tasks.workunit.client.0.smithi153.stdout:5/719: truncate d0/d9/d22/f33 878738 0 2024-04-05T03:03:00.774 INFO:tasks.workunit.client.0.smithi153.stdout:3/773: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/c111 13952 1 2024-04-05T03:03:00.775 INFO:tasks.workunit.client.0.smithi153.stdout:3/774: stat d1d/d32/d93/d3c/d3f/l95 0 2024-04-05T03:03:00.779 INFO:tasks.workunit.client.0.smithi153.stdout:1/791: dwrite d5/d8/d20/d4b/d18/d16/d33/f70 [0,4194304] 0 2024-04-05T03:03:00.783 INFO:tasks.workunit.client.0.smithi153.stdout:1/792: creat d5/d8/d20/d2f/db8/ff2 x:0 0 0 2024-04-05T03:03:00.786 INFO:tasks.workunit.client.0.smithi153.stdout:5/720: dwrite d0/d25/d4d/d62/d37/f4b [0,4194304] 0 2024-04-05T03:03:00.787 INFO:tasks.workunit.client.0.smithi153.stdout:3/775: dwrite d1d/f6a [4194304,4194304] 0 2024-04-05T03:03:00.793 INFO:tasks.workunit.client.0.smithi153.stdout:5/721: creat d0/d25/d4d/d9c/dc7/ffa x:0 0 0 2024-04-05T03:03:00.793 INFO:tasks.workunit.client.0.smithi153.stdout:3/776: mkdir d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d112 0 2024-04-05T03:03:00.795 INFO:tasks.workunit.client.0.smithi153.stdout:5/722: truncate d0/d2c/d3f/fd9 821202 0 2024-04-05T03:03:00.795 INFO:tasks.workunit.client.0.smithi153.stdout:1/793: write d5/dd/f26 [1370958,77214] 0 2024-04-05T03:03:00.800 INFO:tasks.workunit.client.0.smithi153.stdout:3/777: creat d1d/d32/d93/d3c/d3f/d90/db6/f113 x:0 0 0 2024-04-05T03:03:00.806 INFO:tasks.workunit.client.0.smithi153.stdout:4/861: sync 2024-04-05T03:03:00.806 INFO:tasks.workunit.client.0.smithi153.stdout:2/866: sync 2024-04-05T03:03:00.806 INFO:tasks.workunit.client.0.smithi153.stdout:0/730: sync 2024-04-05T03:03:00.806 INFO:tasks.workunit.client.0.smithi153.stdout:7/981: sync 2024-04-05T03:03:00.807 INFO:tasks.workunit.client.0.smithi153.stdout:6/871: sync 2024-04-05T03:03:00.810 INFO:tasks.workunit.client.0.smithi153.stdout:1/794: dwrite d5/d8/d20/d3f/fe5 [0,4194304] 0 2024-04-05T03:03:00.810 INFO:tasks.workunit.client.0.smithi153.stdout:7/982: write d2/da/d11/f72 [1442184,75282] 0 2024-04-05T03:03:00.813 INFO:tasks.workunit.client.0.smithi153.stdout:4/862: creat da/db/d20/d35/d41/d67/f12b x:0 0 0 2024-04-05T03:03:00.816 INFO:tasks.workunit.client.0.smithi153.stdout:2/867: unlink d9/da/d19/d2e/dc7/fc0 0 2024-04-05T03:03:00.818 INFO:tasks.workunit.client.0.smithi153.stdout:6/872: creat d4/dc/df/d16/d24/f130 x:0 0 0 2024-04-05T03:03:00.822 INFO:tasks.workunit.client.0.smithi153.stdout:0/731: dread d9/f1a [0,4194304] 0 2024-04-05T03:03:00.825 INFO:tasks.workunit.client.0.smithi153.stdout:7/983: unlink d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/dce/dc8/c136 0 2024-04-05T03:03:00.826 INFO:tasks.workunit.client.0.smithi153.stdout:7/984: chown d2/da/d3a/d41/d7a/d54/d85/df6/f114 55874892 1 2024-04-05T03:03:00.827 INFO:tasks.workunit.client.0.smithi153.stdout:3/778: dread d1d/f21 [0,4194304] 0 2024-04-05T03:03:00.828 INFO:tasks.workunit.client.0.smithi153.stdout:0/732: symlink d9/db/d42/d7e/db6/d1e/d45/dae/le1 0 2024-04-05T03:03:00.829 INFO:tasks.workunit.client.0.smithi153.stdout:7/985: write d2/da/d3a/d41/d7a/d54/d85/df6/d77/fdf [1609238,111685] 0 2024-04-05T03:03:00.829 INFO:tasks.workunit.client.0.smithi153.stdout:1/795: rename d5/d8/d20/d4b/cab to d5/cf3 0 2024-04-05T03:03:00.829 INFO:tasks.workunit.client.0.smithi153.stdout:0/733: chown d9/d14/f40 229188 1 2024-04-05T03:03:00.834 INFO:tasks.workunit.client.0.smithi153.stdout:1/796: creat d5/de2/ff4 x:0 0 0 2024-04-05T03:03:00.837 INFO:tasks.workunit.client.0.smithi153.stdout:1/797: dread d5/f84 [0,4194304] 0 2024-04-05T03:03:00.841 INFO:tasks.workunit.client.0.smithi153.stdout:4/863: write da/de/d56/d78/d8f/fe8 [22037,116504] 0 2024-04-05T03:03:00.841 INFO:tasks.workunit.client.0.smithi153.stdout:2/868: write d9/d77/f90 [268300,114485] 0 2024-04-05T03:03:00.843 INFO:tasks.workunit.client.0.smithi153.stdout:2/869: write d9/da/d37/d8a/ff1 [13423221,82156] 0 2024-04-05T03:03:00.845 INFO:tasks.workunit.client.0.smithi153.stdout:1/798: rename d5/d8/d20/d4b/d18/d16 to d5/de2/df5 0 2024-04-05T03:03:00.846 INFO:tasks.workunit.client.0.smithi153.stdout:6/873: truncate d4/dc/df/d16/d44/fdb 7434362 0 2024-04-05T03:03:00.850 INFO:tasks.workunit.client.0.smithi153.stdout:1/799: mknod d5/d8/d20/d3f/da7/cf6 0 2024-04-05T03:03:00.852 INFO:tasks.workunit.client.0.smithi153.stdout:0/734: write d9/db/d42/d7e/db6/f17 [613635,19885] 0 2024-04-05T03:03:00.857 INFO:tasks.workunit.client.0.smithi153.stdout:6/874: creat d4/dc/df/d16/d17/f131 x:0 0 0 2024-04-05T03:03:00.858 INFO:tasks.workunit.client.0.smithi153.stdout:0/735: creat d9/db/d42/d43/fe2 x:0 0 0 2024-04-05T03:03:00.859 INFO:tasks.workunit.client.0.smithi153.stdout:1/800: creat d5/de2/df5/d48/d96/ff7 x:0 0 0 2024-04-05T03:03:00.859 INFO:tasks.workunit.client.0.smithi153.stdout:3/779: dwrite d1d/d32/d93/d3c/fd9 [0,4194304] 0 2024-04-05T03:03:00.865 INFO:tasks.workunit.client.0.smithi153.stdout:1/801: chown d5/d8/d20/d3f/da7/lc5 83098 1 2024-04-05T03:03:00.869 INFO:tasks.workunit.client.0.smithi153.stdout:3/780: write d1d/d32/d93/d3c/d3f/d90/d92/d99/fec [562933,119324] 0 2024-04-05T03:03:00.870 INFO:tasks.workunit.client.0.smithi153.stdout:2/870: dwrite d9/da/d36/f81 [8388608,4194304] 0 2024-04-05T03:03:00.870 INFO:tasks.workunit.client.0.smithi153.stdout:4/864: dwrite da/de/d56/d6a/d2d/fa5 [0,4194304] 0 2024-04-05T03:03:00.872 INFO:tasks.workunit.client.0.smithi153.stdout:1/802: mknod d5/d8/d20/d4b/d51/cf8 0 2024-04-05T03:03:00.873 INFO:tasks.workunit.client.0.smithi153.stdout:2/871: chown d9/d68/d98/de9/c100 1 1 2024-04-05T03:03:00.874 INFO:tasks.workunit.client.0.smithi153.stdout:3/781: creat d1d/d32/d93/d3c/d3f/d7c/f114 x:0 0 0 2024-04-05T03:03:00.875 INFO:tasks.workunit.client.0.smithi153.stdout:4/865: creat da/db/d20/d45/db1/f12c x:0 0 0 2024-04-05T03:03:00.888 INFO:tasks.workunit.client.0.smithi153.stdout:0/736: truncate d9/db/d42/d7e/db6/f16 2172342 0 2024-04-05T03:03:00.889 INFO:tasks.workunit.client.0.smithi153.stdout:0/737: stat d9/db/l9c 0 2024-04-05T03:03:00.892 INFO:tasks.workunit.client.0.smithi153.stdout:2/872: write d9/d68/fce [1128835,39037] 0 2024-04-05T03:03:00.892 INFO:tasks.workunit.client.0.smithi153.stdout:3/782: write d1d/d75/f77 [751274,62801] 0 2024-04-05T03:03:00.892 INFO:tasks.workunit.client.0.smithi153.stdout:6/875: dread d4/dc/df/d16/d5f/fac [0,4194304] 0 2024-04-05T03:03:00.892 INFO:tasks.workunit.client.0.smithi153.stdout:4/866: write da/db/d20/ff5 [415942,117647] 0 2024-04-05T03:03:00.896 INFO:tasks.workunit.client.0.smithi153.stdout:4/867: dread - da/de/d66/ffd zero size 2024-04-05T03:03:00.896 INFO:tasks.workunit.client.0.smithi153.stdout:2/873: read d9/d68/db9/f92 [2234793,91243] 0 2024-04-05T03:03:00.896 INFO:tasks.workunit.client.0.smithi153.stdout:3/783: creat d1d/d75/f115 x:0 0 0 2024-04-05T03:03:00.900 INFO:tasks.workunit.client.0.smithi153.stdout:3/784: mkdir d1d/d10d/d116 0 2024-04-05T03:03:00.904 INFO:tasks.workunit.client.0.smithi153.stdout:6/876: dwrite d4/dc/df/d16/d17/f131 [0,4194304] 0 2024-04-05T03:03:00.906 INFO:tasks.workunit.client.0.smithi153.stdout:4/868: write da/db/d8b/fa4 [850979,130969] 0 2024-04-05T03:03:00.911 INFO:tasks.workunit.client.0.smithi153.stdout:3/785: creat d1d/d32/d93/d3c/f117 x:0 0 0 2024-04-05T03:03:00.913 INFO:tasks.workunit.client.0.smithi153.stdout:6/877: creat d4/dc/d7a/d9c/f132 x:0 0 0 2024-04-05T03:03:00.918 INFO:tasks.workunit.client.0.smithi153.stdout:0/738: dwrite d9/db/d42/d7e/db6/d1e/d45/d62/d79/fc6 [4194304,4194304] 0 2024-04-05T03:03:00.918 INFO:tasks.workunit.client.0.smithi153.stdout:4/869: symlink da/db/d20/d45/db1/l12d 0 2024-04-05T03:03:00.922 INFO:tasks.workunit.client.0.smithi153.stdout:2/874: dwrite d9/da/d36/f4d [0,4194304] 0 2024-04-05T03:03:00.923 INFO:tasks.workunit.client.0.smithi153.stdout:1/803: dread f3 [0,4194304] 0 2024-04-05T03:03:00.926 INFO:tasks.workunit.client.0.smithi153.stdout:6/878: read d4/dc/d7a/da8/fe3 [878999,85204] 0 2024-04-05T03:03:00.927 INFO:tasks.workunit.client.0.smithi153.stdout:3/786: write d1d/d27/d2b/fc9 [396611,82659] 0 2024-04-05T03:03:00.928 INFO:tasks.workunit.client.0.smithi153.stdout:0/739: write d9/db/f66 [3557958,72338] 0 2024-04-05T03:03:00.931 INFO:tasks.workunit.client.0.smithi153.stdout:1/804: symlink d5/de2/df5/d2c/d62/lf9 0 2024-04-05T03:03:00.935 INFO:tasks.workunit.client.0.smithi153.stdout:1/805: fdatasync d5/de2/df5/f36 0 2024-04-05T03:03:00.936 INFO:tasks.workunit.client.0.smithi153.stdout:2/875: dwrite d9/da/f64 [4194304,4194304] 0 2024-04-05T03:03:00.938 INFO:tasks.workunit.client.0.smithi153.stdout:1/806: chown d5/de2/df5/d2c/ceb 298 1 2024-04-05T03:03:00.941 INFO:tasks.workunit.client.0.smithi153.stdout:1/807: mknod d5/de2/df5/d2c/d62/cfa 0 2024-04-05T03:03:00.942 INFO:tasks.workunit.client.0.smithi153.stdout:1/808: write d5/de2/df5/d33/d83/fe8 [301943,117130] 0 2024-04-05T03:03:00.943 INFO:tasks.workunit.client.0.smithi153.stdout:3/787: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f118 x:0 0 0 2024-04-05T03:03:00.948 INFO:tasks.workunit.client.0.smithi153.stdout:1/809: read d5/de2/df5/d45/d68/da4/fe3 [921044,60341] 0 2024-04-05T03:03:00.948 INFO:tasks.workunit.client.0.smithi153.stdout:3/788: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/c119 0 2024-04-05T03:03:00.949 INFO:tasks.workunit.client.0.smithi153.stdout:6/879: write d4/dc/df/d16/d24/d5e/fe2 [444230,109229] 0 2024-04-05T03:03:00.949 INFO:tasks.workunit.client.0.smithi153.stdout:2/876: link d9/da/d19/d2e/d7e/d53/f83 d9/da/d19/d2e/d7e/d7a/d6d/f124 0 2024-04-05T03:03:00.950 INFO:tasks.workunit.client.0.smithi153.stdout:1/810: chown d5/dd/l85 2343 1 2024-04-05T03:03:00.954 INFO:tasks.workunit.client.0.smithi153.stdout:6/880: fsync d4/dc/df/d16/d44/fa1 0 2024-04-05T03:03:00.955 INFO:tasks.workunit.client.0.smithi153.stdout:1/811: creat d5/de2/df5/d45/d68/ffb x:0 0 0 2024-04-05T03:03:00.957 INFO:tasks.workunit.client.0.smithi153.stdout:4/870: dread da/de/d56/d78/d8f/f99 [0,4194304] 0 2024-04-05T03:03:00.959 INFO:tasks.workunit.client.0.smithi153.stdout:3/789: dwrite d1d/fcf [4194304,4194304] 0 2024-04-05T03:03:00.962 INFO:tasks.workunit.client.0.smithi153.stdout:0/740: dwrite d9/db/d42/d7e/db6/d1d/d61/fe0 [0,4194304] 0 2024-04-05T03:03:00.963 INFO:tasks.workunit.client.0.smithi153.stdout:4/871: creat da/db/d8b/d10f/f12e x:0 0 0 2024-04-05T03:03:00.965 INFO:tasks.workunit.client.0.smithi153.stdout:6/881: write d4/d38/f8b [1340528,73353] 0 2024-04-05T03:03:00.965 INFO:tasks.workunit.client.0.smithi153.stdout:4/872: dread - da/db/d20/d35/d38/d76/d90/f10b zero size 2024-04-05T03:03:00.967 INFO:tasks.workunit.client.0.smithi153.stdout:2/877: link d9/da/d19/d2e/d7e/d53/da1/ccf d9/da/d19/df6/c125 0 2024-04-05T03:03:00.969 INFO:tasks.workunit.client.0.smithi153.stdout:3/790: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/c11a 0 2024-04-05T03:03:00.972 INFO:tasks.workunit.client.0.smithi153.stdout:3/791: chown d1d/d27/df4/c101 5505333 1 2024-04-05T03:03:00.973 INFO:tasks.workunit.client.0.smithi153.stdout:1/812: dwrite d5/d8/f65 [0,4194304] 0 2024-04-05T03:03:00.976 INFO:tasks.workunit.client.0.smithi153.stdout:1/813: chown d5/de2/df5/d2c/d62/lf9 7709 1 2024-04-05T03:03:00.980 INFO:tasks.workunit.client.0.smithi153.stdout:2/878: creat d9/f126 x:0 0 0 2024-04-05T03:03:00.981 INFO:tasks.workunit.client.0.smithi153.stdout:1/814: creat d5/d8/d20/d4b/de1/ffc x:0 0 0 2024-04-05T03:03:00.984 INFO:tasks.workunit.client.0.smithi153.stdout:0/741: dwrite d9/db/d42/d7e/db6/d1d/d61/f3d [0,4194304] 0 2024-04-05T03:03:00.985 INFO:tasks.workunit.client.0.smithi153.stdout:3/792: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/l11b 0 2024-04-05T03:03:00.985 INFO:tasks.workunit.client.0.smithi153.stdout:1/815: write d5/de2/df5/d45/f4a [3480542,101907] 0 2024-04-05T03:03:00.990 INFO:tasks.workunit.client.0.smithi153.stdout:2/879: creat d9/da/d37/d8a/da8/de5/f127 x:0 0 0 2024-04-05T03:03:00.991 INFO:tasks.workunit.client.0.smithi153.stdout:4/873: write da/de/d56/f9f [417798,12145] 0 2024-04-05T03:03:00.991 INFO:tasks.workunit.client.0.smithi153.stdout:3/793: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/f11c x:0 0 0 2024-04-05T03:03:00.995 INFO:tasks.workunit.client.0.smithi153.stdout:1/816: rename d5/d8/f9d to d5/ffd 0 2024-04-05T03:03:00.997 INFO:tasks.workunit.client.0.smithi153.stdout:3/794: write d1d/d75/fb4 [4769495,93839] 0 2024-04-05T03:03:00.999 INFO:tasks.workunit.client.0.smithi153.stdout:2/880: dwrite d9/da/d94/dca/df7/f108 [0,4194304] 0 2024-04-05T03:03:01.001 INFO:tasks.workunit.client.0.smithi153.stdout:0/742: rename d9/db/d42/d7e/db6/d1e/d45/c51 to d9/db/d42/d7e/db6/d1e/d45/d62/dde/d6d/ce3 0 2024-04-05T03:03:01.003 INFO:tasks.workunit.client.0.smithi153.stdout:2/881: truncate d9/da/d37/fcb 754310 0 2024-04-05T03:03:01.004 INFO:tasks.workunit.client.0.smithi153.stdout:1/817: creat d5/d8/d20/d2f/db8/ffe x:0 0 0 2024-04-05T03:03:01.006 INFO:tasks.workunit.client.0.smithi153.stdout:3/795: unlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/c111 0 2024-04-05T03:03:01.008 INFO:tasks.workunit.client.0.smithi153.stdout:4/874: rename da/db/f1a to da/db/d20/d3c/d91/f12f 0 2024-04-05T03:03:01.009 INFO:tasks.workunit.client.0.smithi153.stdout:4/875: truncate da/de/d66/d111/ffa 850500 0 2024-04-05T03:03:01.009 INFO:tasks.workunit.client.0.smithi153.stdout:0/743: mkdir d9/db/d42/d43/de4 0 2024-04-05T03:03:01.012 INFO:tasks.workunit.client.0.smithi153.stdout:1/818: write d5/de2/df5/d45/d93/f9a [415998,22958] 0 2024-04-05T03:03:01.015 INFO:tasks.workunit.client.0.smithi153.stdout:0/744: creat d9/db/d42/d7e/d8b/fe5 x:0 0 0 2024-04-05T03:03:01.017 INFO:tasks.workunit.client.0.smithi153.stdout:4/876: unlink da/de/d56/d78/d8f/l11a 0 2024-04-05T03:03:01.018 INFO:tasks.workunit.client.0.smithi153.stdout:1/819: chown d5/de2/df5/l80 0 1 2024-04-05T03:03:01.021 INFO:tasks.workunit.client.0.smithi153.stdout:6/882: dwrite d4/dc/df/d16/d17/f3a [0,4194304] 0 2024-04-05T03:03:01.024 INFO:tasks.workunit.client.0.smithi153.stdout:6/883: fsync d4/dc/df/d16/f23 0 2024-04-05T03:03:01.029 INFO:tasks.workunit.client.0.smithi153.stdout:6/884: readlink d4/dc/l30 0 2024-04-05T03:03:01.030 INFO:tasks.workunit.client.0.smithi153.stdout:3/796: dwrite d1d/d75/f84 [0,4194304] 0 2024-04-05T03:03:01.039 INFO:tasks.workunit.client.0.smithi153.stdout:3/797: dread - d1d/d32/d93/d3c/d3f/fb8 zero size 2024-04-05T03:03:01.041 INFO:tasks.workunit.client.0.smithi153.stdout:3/798: chown d1d/d75/f115 97780 1 2024-04-05T03:03:01.042 INFO:tasks.workunit.client.0.smithi153.stdout:1/820: dread d5/d8/d20/d4b/f79 [0,4194304] 0 2024-04-05T03:03:01.046 INFO:tasks.workunit.client.0.smithi153.stdout:4/877: link da/db/d20/d35/d38/d76/fc6 da/de/d56/d78/d8f/f130 0 2024-04-05T03:03:01.046 INFO:tasks.workunit.client.0.smithi153.stdout:2/882: dwrite d9/da/d19/d2e/d7e/d7a/d6d/f124 [0,4194304] 0 2024-04-05T03:03:01.049 INFO:tasks.workunit.client.0.smithi153.stdout:3/799: mkdir d1d/d32/d93/d3c/dc5/d11d 0 2024-04-05T03:03:01.049 INFO:tasks.workunit.client.0.smithi153.stdout:0/745: dread d9/db/d42/d43/f4a [0,4194304] 0 2024-04-05T03:03:01.053 INFO:tasks.workunit.client.0.smithi153.stdout:4/878: creat da/db/d20/d35/d38/d76/d90/da7/ddc/f131 x:0 0 0 2024-04-05T03:03:01.054 INFO:tasks.workunit.client.0.smithi153.stdout:3/800: mknod d1d/d32/d93/d3c/d3f/d7c/de7/c11e 0 2024-04-05T03:03:01.057 INFO:tasks.workunit.client.0.smithi153.stdout:3/801: write d1d/d32/d93/d3c/d3f/d90/d92/d99/ff1 [129655,10425] 0 2024-04-05T03:03:01.061 INFO:tasks.workunit.client.0.smithi153.stdout:6/885: write d4/dc/df/d16/d5f/f7c [793656,100497] 0 2024-04-05T03:03:01.062 INFO:tasks.workunit.client.0.smithi153.stdout:3/802: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f7f zero size 2024-04-05T03:03:01.062 INFO:tasks.workunit.client.0.smithi153.stdout:4/879: creat da/db/d20/d45/db1/f132 x:0 0 0 2024-04-05T03:03:01.067 INFO:tasks.workunit.client.0.smithi153.stdout:3/803: rename d1d/d75 to d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/d11f 0 2024-04-05T03:03:01.068 INFO:tasks.workunit.client.0.smithi153.stdout:1/821: write d5/de2/df5/fe6 [313035,107919] 0 2024-04-05T03:03:01.070 INFO:tasks.workunit.client.0.smithi153.stdout:4/880: getdents da/de 0 2024-04-05T03:03:01.073 INFO:tasks.workunit.client.0.smithi153.stdout:3/804: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fd6 zero size 2024-04-05T03:03:01.075 INFO:tasks.workunit.client.0.smithi153.stdout:0/746: write d9/d14/f3f [396360,56405] 0 2024-04-05T03:03:01.077 INFO:tasks.workunit.client.0.smithi153.stdout:4/881: mkdir da/db/d20/d35/d41/da8/dc1/d133 0 2024-04-05T03:03:01.078 INFO:tasks.workunit.client.0.smithi153.stdout:3/805: truncate d1d/d32/d93/d3c/f7b 2092802 0 2024-04-05T03:03:01.079 INFO:tasks.workunit.client.0.smithi153.stdout:0/747: chown d9/db/d42/d7e/db6/d1e/d84 861851 1 2024-04-05T03:03:01.082 INFO:tasks.workunit.client.0.smithi153.stdout:3/806: stat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f 0 2024-04-05T03:03:01.083 INFO:tasks.workunit.client.0.smithi153.stdout:1/822: write d5/d8/d20/d3f/f86 [423822,8837] 0 2024-04-05T03:03:01.085 INFO:tasks.workunit.client.0.smithi153.stdout:0/748: dread - d9/db/d42/d43/fa2 zero size 2024-04-05T03:03:01.086 INFO:tasks.workunit.client.0.smithi153.stdout:4/882: dwrite da/de/d56/d78/fce [0,4194304] 0 2024-04-05T03:03:01.088 INFO:tasks.workunit.client.0.smithi153.stdout:1/823: mkdir d5/d8/d20/d3f/dff 0 2024-04-05T03:03:01.088 INFO:tasks.workunit.client.0.smithi153.stdout:0/749: dread - d9/db/d42/d7e/db6/fc0 zero size 2024-04-05T03:03:01.092 INFO:tasks.workunit.client.0.smithi153.stdout:4/883: mkdir da/db/d20/d35/d38/d76/d90/da7/ddc/d134 0 2024-04-05T03:03:01.094 INFO:tasks.workunit.client.0.smithi153.stdout:0/750: read d9/db/d42/d43/f76 [1082078,33571] 0 2024-04-05T03:03:01.095 INFO:tasks.workunit.client.0.smithi153.stdout:4/884: write da/db/d20/d45/f6b [4458551,17453] 0 2024-04-05T03:03:01.100 INFO:tasks.workunit.client.0.smithi153.stdout:3/807: link d1d/d32/d93/d3c/d3f/d90/d92/led d1d/d32/d93/d3c/l120 0 2024-04-05T03:03:01.102 INFO:tasks.workunit.client.0.smithi153.stdout:0/751: symlink d9/db/d42/dd4/le6 0 2024-04-05T03:03:01.103 INFO:tasks.workunit.client.0.smithi153.stdout:0/752: chown d9/db/d42/d7e/db6/d1e/d45 378 1 2024-04-05T03:03:01.104 INFO:tasks.workunit.client.0.smithi153.stdout:4/885: unlink da/db/d20/d3c/c4b 0 2024-04-05T03:03:01.105 INFO:tasks.workunit.client.0.smithi153.stdout:0/753: chown d9/db/d42/d7e/db6/d1e/f25 1 1 2024-04-05T03:03:01.109 INFO:tasks.workunit.client.0.smithi153.stdout:6/886: dread d4/dc/f46 [0,4194304] 0 2024-04-05T03:03:01.116 INFO:tasks.workunit.client.0.smithi153.stdout:3/808: rmdir d1d/d32/d93/d3c/d3f/d90/db6/dd0 0 2024-04-05T03:03:01.119 INFO:tasks.workunit.client.0.smithi153.stdout:1/824: dwrite d5/de2/df5/d33/d83/f8e [0,4194304] 0 2024-04-05T03:03:01.121 INFO:tasks.workunit.client.0.smithi153.stdout:0/754: truncate d9/db/d42/d7e/db6/d1d/d61/f1b 2389792 0 2024-04-05T03:03:01.124 INFO:tasks.workunit.client.0.smithi153.stdout:3/809: write d1d/d32/d93/d3c/d3f/d90/d92/d99/ff1 [1152539,117954] 0 2024-04-05T03:03:01.130 INFO:tasks.workunit.client.0.smithi153.stdout:4/886: dwrite da/db/d20/f29 [0,4194304] 0 2024-04-05T03:03:01.135 INFO:tasks.workunit.client.0.smithi153.stdout:0/755: dwrite d9/d14/f3f [0,4194304] 0 2024-04-05T03:03:01.137 INFO:tasks.workunit.client.0.smithi153.stdout:1/825: creat d5/d8/d20/d4b/f100 x:0 0 0 2024-04-05T03:03:01.138 INFO:tasks.workunit.client.0.smithi153.stdout:1/826: readlink d5/d8/d20/d2f/d40/l46 0 2024-04-05T03:03:01.139 INFO:tasks.workunit.client.0.smithi153.stdout:4/887: mkdir da/db/d20/d35/d38/d76/d90/da7/ddc/d135 0 2024-04-05T03:03:01.142 INFO:tasks.workunit.client.0.smithi153.stdout:0/756: mkdir d9/db/d42/d7e/db6/d1e/d45/d62/de7 0 2024-04-05T03:03:01.142 INFO:tasks.workunit.client.0.smithi153.stdout:3/810: getdents d1d/d32/df9 0 2024-04-05T03:03:01.142 INFO:tasks.workunit.client.0.smithi153.stdout:3/811: readlink d1d/d32/d93/lc8 0 2024-04-05T03:03:01.146 INFO:tasks.workunit.client.0.smithi153.stdout:6/887: write d4/dc/df/d16/d5f/d73/f9e [1907232,22116] 0 2024-04-05T03:03:01.148 INFO:tasks.workunit.client.0.smithi153.stdout:1/827: rename d5/d8/d20/d4b/d51/f7f to d5/de2/df5/d2c/f101 0 2024-04-05T03:03:01.150 INFO:tasks.workunit.client.0.smithi153.stdout:4/888: creat da/db/d20/d3c/f136 x:0 0 0 2024-04-05T03:03:01.151 INFO:tasks.workunit.client.0.smithi153.stdout:1/828: dread - d5/d8/d20/d4b/d51/da2/f92 zero size 2024-04-05T03:03:01.152 INFO:tasks.workunit.client.0.smithi153.stdout:3/812: link d1d/d27/f96 d1d/d32/d93/d3c/d3f/d90/db6/f121 0 2024-04-05T03:03:01.152 INFO:tasks.workunit.client.0.smithi153.stdout:0/757: dwrite d9/db/f4b [0,4194304] 0 2024-04-05T03:03:01.154 INFO:tasks.workunit.client.0.smithi153.stdout:1/829: write d5/d8/d20/d2f/db8/fd3 [907027,49653] 0 2024-04-05T03:03:01.155 INFO:tasks.workunit.client.0.smithi153.stdout:6/888: symlink d4/dc/df/d16/d36/d4d/d64/l133 0 2024-04-05T03:03:01.157 INFO:tasks.workunit.client.0.smithi153.stdout:6/889: stat d4/dc/df/d16/d36/d10e/c12d 0 2024-04-05T03:03:01.159 INFO:tasks.workunit.client.0.smithi153.stdout:0/758: truncate d9/db/d42/d7e/db6/d1e/d45/d62/dde/d9e/fd3 896872 0 2024-04-05T03:03:01.165 INFO:tasks.workunit.client.0.smithi153.stdout:1/830: dwrite d5/d8/d20/d2f/db8/ff2 [0,4194304] 0 2024-04-05T03:03:01.171 INFO:tasks.workunit.client.0.smithi153.stdout:6/890: getdents d4/dc/df/d16/d36/d10e 0 2024-04-05T03:03:01.172 INFO:tasks.workunit.client.0.smithi153.stdout:0/759: getdents d9/d14/d67 0 2024-04-05T03:03:01.173 INFO:tasks.workunit.client.0.smithi153.stdout:1/831: creat d5/d8/d20/d2f/f102 x:0 0 0 2024-04-05T03:03:01.174 INFO:tasks.workunit.client.0.smithi153.stdout:6/891: fsync d4/f20 0 2024-04-05T03:03:01.177 INFO:tasks.workunit.client.0.smithi153.stdout:1/832: unlink d5/de2/df5/d45/d68/l8a 0 2024-04-05T03:03:01.179 INFO:tasks.workunit.client.0.smithi153.stdout:1/833: mkdir d5/de2/df5/d45/d103 0 2024-04-05T03:03:01.182 INFO:tasks.workunit.client.0.smithi153.stdout:6/892: link d4/dc/df/f26 d4/d38/f134 0 2024-04-05T03:03:01.182 INFO:tasks.workunit.client.0.smithi153.stdout:1/834: creat d5/d8/d20/d4b/de1/f104 x:0 0 0 2024-04-05T03:03:01.184 INFO:tasks.workunit.client.0.smithi153.stdout:0/760: link d9/db/d42/d7e/db6/d1e/d45/d62/dde/fce d9/db/d42/d7e/db6/fe8 0 2024-04-05T03:03:01.184 INFO:tasks.workunit.client.0.smithi153.stdout:4/889: dread da/f26 [0,4194304] 0 2024-04-05T03:03:01.187 INFO:tasks.workunit.client.0.smithi153.stdout:0/761: chown d9/db/d42/d7e/db6/d1e/d45/d62/dde/f9b 4185 1 2024-04-05T03:03:01.188 INFO:tasks.workunit.client.0.smithi153.stdout:0/762: write d9/db/d42/d43/f85 [394433,74266] 0 2024-04-05T03:03:01.189 INFO:tasks.workunit.client.0.smithi153.stdout:6/893: unlink d4/dc/df/d16/d33/f128 0 2024-04-05T03:03:01.190 INFO:tasks.workunit.client.0.smithi153.stdout:3/813: dread d1d/d27/fe3 [0,4194304] 0 2024-04-05T03:03:01.191 INFO:tasks.workunit.client.0.smithi153.stdout:0/763: write d9/db/d42/d7e/db6/d1e/d45/d62/dde/f9b [4807346,79160] 0 2024-04-05T03:03:01.194 INFO:tasks.workunit.client.0.smithi153.stdout:4/890: creat da/db/d20/d45/dc3/f137 x:0 0 0 2024-04-05T03:03:01.194 INFO:tasks.workunit.client.0.smithi153.stdout:4/891: chown da/db/d20/d3c/d81/lf9 4815668 1 2024-04-05T03:03:01.199 INFO:tasks.workunit.client.0.smithi153.stdout:3/814: mknod d1d/d32/d93/d3c/dc5/c122 0 2024-04-05T03:03:01.201 INFO:tasks.workunit.client.0.smithi153.stdout:6/894: dwrite d4/dc/df/d16/d36/fb2 [0,4194304] 0 2024-04-05T03:03:01.201 INFO:tasks.workunit.client.0.smithi153.stdout:0/764: write d9/db/d42/d7e/db6/d1d/d61/f2b [4053971,41642] 0 2024-04-05T03:03:01.203 INFO:tasks.workunit.client.0.smithi153.stdout:3/815: fsync d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 0 2024-04-05T03:03:01.203 INFO:tasks.workunit.client.0.smithi153.stdout:1/835: write d5/de2/df5/d2c/f7d [478699,86885] 0 2024-04-05T03:03:01.205 INFO:tasks.workunit.client.0.smithi153.stdout:6/895: mkdir d4/dc/df/dcb/d135 0 2024-04-05T03:03:01.211 INFO:tasks.workunit.client.0.smithi153.stdout:3/816: dread d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fcb [0,4194304] 0 2024-04-05T03:03:01.212 INFO:tasks.workunit.client.0.smithi153.stdout:1/836: creat d5/d8/d20/d4b/de1/f105 x:0 0 0 2024-04-05T03:03:01.213 INFO:tasks.workunit.client.0.smithi153.stdout:1/837: chown d5/d8/d20/d2f/db8/ff2 780 1 2024-04-05T03:03:01.217 INFO:tasks.workunit.client.0.smithi153.stdout:6/896: write d4/dc/df/d16/d24/f32 [2739827,90624] 0 2024-04-05T03:03:01.218 INFO:tasks.workunit.client.0.smithi153.stdout:3/817: mkdir d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d123 0 2024-04-05T03:03:01.218 INFO:tasks.workunit.client.0.smithi153.stdout:4/892: link da/de/d56/d78/d8f/f99 da/db/d20/d35/d38/d76/f138 0 2024-04-05T03:03:01.220 INFO:tasks.workunit.client.0.smithi153.stdout:3/818: readlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/lf8 0 2024-04-05T03:03:01.220 INFO:tasks.workunit.client.0.smithi153.stdout:1/838: symlink d5/de2/df5/d33/d83/l106 0 2024-04-05T03:03:01.222 INFO:tasks.workunit.client.0.smithi153.stdout:6/897: mknod d4/dc/df/d16/d129/c136 0 2024-04-05T03:03:01.222 INFO:tasks.workunit.client.0.smithi153.stdout:3/819: read d1d/d32/d93/d3c/d3f/d90/da4/fb0 [818553,112390] 0 2024-04-05T03:03:01.227 INFO:tasks.workunit.client.0.smithi153.stdout:1/839: dwrite d5/d8/d20/d3f/f91 [0,4194304] 0 2024-04-05T03:03:01.228 INFO:tasks.workunit.client.0.smithi153.stdout:1/840: write d5/d8/d20/d4b/d1e/f4f [755851,122294] 0 2024-04-05T03:03:01.229 INFO:tasks.workunit.client.0.smithi153.stdout:4/893: creat da/de/d56/f139 x:0 0 0 2024-04-05T03:03:01.231 INFO:tasks.workunit.client.0.smithi153.stdout:0/765: truncate d9/d14/f3f 2963588 0 2024-04-05T03:03:01.231 INFO:tasks.workunit.client.0.smithi153.stdout:6/898: truncate d4/dc/df/d16/d17/d54/fb8 1021906 0 2024-04-05T03:03:01.233 INFO:tasks.workunit.client.0.smithi153.stdout:3/820: creat d1d/d32/d93/d3c/f124 x:0 0 0 2024-04-05T03:03:01.234 INFO:tasks.workunit.client.0.smithi153.stdout:1/841: dread d5/de2/df5/fe6 [0,4194304] 0 2024-04-05T03:03:01.235 INFO:tasks.workunit.client.0.smithi153.stdout:6/899: read d4/dc/df/d16/fcf [2837317,112449] 0 2024-04-05T03:03:01.236 INFO:tasks.workunit.client.0.smithi153.stdout:4/894: mknod da/db/d20/d45/dc3/c13a 0 2024-04-05T03:03:01.239 INFO:tasks.workunit.client.0.smithi153.stdout:0/766: rename d9/db/f4b to d9/db/d42/d7e/db6/daf/fe9 0 2024-04-05T03:03:01.242 INFO:tasks.workunit.client.0.smithi153.stdout:1/842: unlink d5/d8/d20/d4b/d51/da2/f92 0 2024-04-05T03:03:01.245 INFO:tasks.workunit.client.0.smithi153.stdout:3/821: mknod d1d/d10d/d116/c125 0 2024-04-05T03:03:01.247 INFO:tasks.workunit.client.0.smithi153.stdout:8/714: sync 2024-04-05T03:03:01.247 INFO:tasks.workunit.client.0.smithi153.stdout:8/715: write f3 [3690983,3590] 0 2024-04-05T03:03:01.248 INFO:tasks.workunit.client.0.smithi153.stdout:8/716: chown d7 1 1 2024-04-05T03:03:01.248 INFO:tasks.workunit.client.0.smithi153.stdout:6/900: unlink d4/dc/df/d16/d36/d4d/d64/d110/l111 0 2024-04-05T03:03:01.250 INFO:tasks.workunit.client.0.smithi153.stdout:1/843: mkdir d5/de2/df5/d45/d68/d107 0 2024-04-05T03:03:01.250 INFO:tasks.workunit.client.0.smithi153.stdout:1/844: chown d5/d8/d20/d4b/de1 49279482 1 2024-04-05T03:03:01.391 INFO:tasks.workunit.client.0.smithi153.stdout:6/901: fdatasync d4/f89 0 2024-04-05T03:03:01.393 INFO:tasks.workunit.client.0.smithi153.stdout:3/822: truncate d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fae 3809683 0 2024-04-05T03:03:01.393 INFO:tasks.workunit.client.0.smithi153.stdout:1/845: dwrite d5/d8/d20/d4b/d51/da2/fd0 [0,4194304] 0 2024-04-05T03:03:01.396 INFO:tasks.workunit.client.0.smithi153.stdout:1/846: dread - d5/de2/ff4 zero size 2024-04-05T03:03:01.396 INFO:tasks.workunit.client.0.smithi153.stdout:3/823: rmdir d1d/d27/df4 39 2024-04-05T03:03:01.398 INFO:tasks.workunit.client.0.smithi153.stdout:4/895: dwrite da/db/f96 [4194304,4194304] 0 2024-04-05T03:03:01.400 INFO:tasks.workunit.client.0.smithi153.stdout:4/896: stat da/de/d56/d6a/d2d/ddb/f128 0 2024-04-05T03:03:01.401 INFO:tasks.workunit.client.0.smithi153.stdout:1/847: creat d5/de2/df5/d45/f108 x:0 0 0 2024-04-05T03:03:01.404 INFO:tasks.workunit.client.0.smithi153.stdout:3/824: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d123/l126 0 2024-04-05T03:03:01.406 INFO:tasks.workunit.client.0.smithi153.stdout:3/825: chown d1d/d27/d2b/le9 15 1 2024-04-05T03:03:01.407 INFO:tasks.workunit.client.0.smithi153.stdout:3/826: fsync d1d/d32/d93/f4c 0 2024-04-05T03:03:01.408 INFO:tasks.workunit.client.0.smithi153.stdout:4/897: write da/db/d20/d3c/d81/db6/f8c [2236130,87778] 0 2024-04-05T03:03:01.410 INFO:tasks.workunit.client.0.smithi153.stdout:3/827: dread - d1d/d32/d93/d3c/d3f/fb8 zero size 2024-04-05T03:03:01.410 INFO:tasks.workunit.client.0.smithi153.stdout:1/848: dwrite d5/d8/f6e [0,4194304] 0 2024-04-05T03:03:01.411 INFO:tasks.workunit.client.0.smithi153.stdout:1/849: fdatasync d5/de2/df5/d45/d68/da4/fac 0 2024-04-05T03:03:01.415 INFO:tasks.workunit.client.0.smithi153.stdout:1/850: creat d5/d8/d20/d4b/d51/db5/f109 x:0 0 0 2024-04-05T03:03:01.420 INFO:tasks.workunit.client.0.smithi153.stdout:1/851: dread d5/d8/d20/d2f/db8/ff2 [0,4194304] 0 2024-04-05T03:03:01.421 INFO:tasks.workunit.client.0.smithi153.stdout:4/898: fsync da/db/f96 0 2024-04-05T03:03:01.423 INFO:tasks.workunit.client.0.smithi153.stdout:1/852: unlink d5/d8/d20/d4b/d51/db5/cd2 0 2024-04-05T03:03:01.424 INFO:tasks.workunit.client.0.smithi153.stdout:1/853: readlink d5/d8/d20/d2f/d40/l46 0 2024-04-05T03:03:01.425 INFO:tasks.workunit.client.0.smithi153.stdout:4/899: symlink da/de/d56/d78/l13b 0 2024-04-05T03:03:01.426 INFO:tasks.workunit.client.0.smithi153.stdout:1/854: rmdir d5/de2/df5/d2c/d62 39 2024-04-05T03:03:01.428 INFO:tasks.workunit.client.0.smithi153.stdout:4/900: creat da/db/d20/d3c/d81/db6/f13c x:0 0 0 2024-04-05T03:03:01.432 INFO:tasks.workunit.client.0.smithi153.stdout:4/901: rename da/de/d66/d111/f10a to da/db/d8b/d10f/f13d 0 2024-04-05T03:03:01.434 INFO:tasks.workunit.client.0.smithi153.stdout:1/855: truncate d5/d8/d20/d4b/d51/da2/fd0 2913116 0 2024-04-05T03:03:01.435 INFO:tasks.workunit.client.0.smithi153.stdout:1/856: chown d5/de2/df5/d45/d103 153 1 2024-04-05T03:03:01.436 INFO:tasks.workunit.client.0.smithi153.stdout:1/857: dread - d5/de2/df5/d45/d68/da4/dc3/fcd zero size 2024-04-05T03:03:01.440 INFO:tasks.workunit.client.0.smithi153.stdout:1/858: dwrite d5/d8/f75 [0,4194304] 0 2024-04-05T03:03:01.443 INFO:tasks.workunit.client.0.smithi153.stdout:1/859: creat d5/de2/df5/d45/d68/f10a x:0 0 0 2024-04-05T03:03:01.447 INFO:tasks.workunit.client.0.smithi153.stdout:1/860: rename d5/de2/df5/d33/d83 to d5/de2/df5/d33/d10b 0 2024-04-05T03:03:01.447 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:01 smithi203 ceph-mon[39911]: pgmap v15: 65 pgs: 65 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 112 MiB/s rd, 259 MiB/s wr, 526 op/s 2024-04-05T03:03:01.447 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:01 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:01.447 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:01 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:01.450 INFO:tasks.workunit.client.0.smithi153.stdout:1/861: rename d5/dd/f5d to d5/d8/d20/d4b/d51/f10c 0 2024-04-05T03:03:01.469 INFO:tasks.workunit.client.0.smithi153.stdout:1/862: dwrite d5/d8/d20/d4b/d18/fee [0,4194304] 0 2024-04-05T03:03:01.470 INFO:tasks.workunit.client.0.smithi153.stdout:1/863: write d5/de2/df5/d48/d96/fbe [313733,117073] 0 2024-04-05T03:03:01.474 INFO:tasks.workunit.client.0.smithi153.stdout:1/864: rmdir d5/d8/d20/d3f/dff 0 2024-04-05T03:03:01.480 INFO:tasks.workunit.client.0.smithi153.stdout:1/865: dwrite d5/de2/df5/d45/f4a [0,4194304] 0 2024-04-05T03:03:01.480 INFO:tasks.workunit.client.0.smithi153.stdout:7/986: sync 2024-04-05T03:03:01.482 INFO:tasks.workunit.client.0.smithi153.stdout:1/866: creat d5/d8/d20/d4b/d1e/f10d x:0 0 0 2024-04-05T03:03:01.483 INFO:tasks.workunit.client.0.smithi153.stdout:2/883: sync 2024-04-05T03:03:01.483 INFO:tasks.workunit.client.0.smithi153.stdout:5/723: sync 2024-04-05T03:03:01.483 INFO:tasks.workunit.client.0.smithi153.stdout:9/674: sync 2024-04-05T03:03:01.484 INFO:tasks.workunit.client.0.smithi153.stdout:1/867: chown d5/de2/df5/d33/d10b/ca5 419522510 1 2024-04-05T03:03:01.485 INFO:tasks.workunit.client.0.smithi153.stdout:7/987: rename d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/d95/f135 to d2/da/d11/d39/d134/f141 0 2024-04-05T03:03:01.497 INFO:tasks.workunit.client.0.smithi153.stdout:5/724: symlink d0/d25/d4d/d9c/dc7/dc9/lfb 0 2024-04-05T03:03:01.498 INFO:tasks.workunit.client.0.smithi153.stdout:1/868: mkdir d5/de2/df5/d45/d93/d10e 0 2024-04-05T03:03:01.499 INFO:tasks.workunit.client.0.smithi153.stdout:9/675: write d1/f2a [666910,55226] 0 2024-04-05T03:03:01.502 INFO:tasks.workunit.client.0.smithi153.stdout:5/725: mknod d0/d25/d4d/d62/d37/cfc 0 2024-04-05T03:03:01.503 INFO:tasks.workunit.client.0.smithi153.stdout:1/869: mkdir d5/d8/d20/d4b/d51/db5/d10f 0 2024-04-05T03:03:01.507 INFO:tasks.workunit.client.0.smithi153.stdout:2/884: dwrite d9/da/f49 [0,4194304] 0 2024-04-05T03:03:01.509 INFO:tasks.workunit.client.0.smithi153.stdout:1/870: mknod d5/d8/d20/d4b/de1/c110 0 2024-04-05T03:03:01.511 INFO:tasks.workunit.client.0.smithi153.stdout:9/676: dwrite d1/d9/d1b/fb0 [0,4194304] 0 2024-04-05T03:03:01.512 INFO:tasks.workunit.client.0.smithi153.stdout:7/988: dread d2/da/d47/f11f [0,4194304] 0 2024-04-05T03:03:01.515 INFO:tasks.workunit.client.0.smithi153.stdout:2/885: mknod d9/da/d19/d2e/d7e/d7a/c128 0 2024-04-05T03:03:01.516 INFO:tasks.workunit.client.0.smithi153.stdout:5/726: truncate d0/d25/d4d/d62/d37/d38/f98 1985913 0 2024-04-05T03:03:01.519 INFO:tasks.workunit.client.0.smithi153.stdout:9/677: mkdir d1/d9/d1b/de9 0 2024-04-05T03:03:01.520 INFO:tasks.workunit.client.0.smithi153.stdout:1/871: dwrite d5/d8/d20/d4b/d1e/f10d [0,4194304] 0 2024-04-05T03:03:01.526 INFO:tasks.workunit.client.0.smithi153.stdout:1/872: mkdir d5/de2/df5/d45/d93/d111 0 2024-04-05T03:03:01.526 INFO:tasks.workunit.client.0.smithi153.stdout:7/989: creat d2/da/d3a/d41/d7a/f142 x:0 0 0 2024-04-05T03:03:01.527 INFO:tasks.workunit.client.0.smithi153.stdout:2/886: fdatasync d9/d68/fa7 0 2024-04-05T03:03:01.530 INFO:tasks.workunit.client.0.smithi153.stdout:5/727: rename d0/d25/d4d/d62/c5b to d0/cfd 0 2024-04-05T03:03:01.532 INFO:tasks.workunit.client.0.smithi153.stdout:2/887: symlink d9/da/d94/dca/l129 0 2024-04-05T03:03:01.536 INFO:tasks.workunit.client.0.smithi153.stdout:7/990: link d2/da/d3a/d41/d31/lba d2/d9b/d6f/d10a/l143 0 2024-04-05T03:03:01.536 INFO:tasks.workunit.client.0.smithi153.stdout:9/678: dread d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:03:01.536 INFO:tasks.workunit.client.0.smithi153.stdout:1/873: dwrite d5/de2/df5/d2c/f7d [0,4194304] 0 2024-04-05T03:03:01.537 INFO:tasks.workunit.client.0.smithi153.stdout:2/888: creat d9/d62/f12a x:0 0 0 2024-04-05T03:03:01.539 INFO:tasks.workunit.client.0.smithi153.stdout:7/991: fsync d2/da/d3a/d41/d31/f37 0 2024-04-05T03:03:01.543 INFO:tasks.workunit.client.0.smithi153.stdout:9/679: dread d1/d10/d16/d22/fc9 [0,4194304] 0 2024-04-05T03:03:01.544 INFO:tasks.workunit.client.0.smithi153.stdout:2/889: dread d9/da/d37/fcb [0,4194304] 0 2024-04-05T03:03:01.548 INFO:tasks.workunit.client.0.smithi153.stdout:7/992: rename d2/da/c36 to d2/da/d11/d39/dfe/c144 0 2024-04-05T03:03:01.549 INFO:tasks.workunit.client.0.smithi153.stdout:2/890: mknod d9/da/d37/c12b 0 2024-04-05T03:03:01.550 INFO:tasks.workunit.client.0.smithi153.stdout:2/891: dread - d9/da/d37/d8a/f114 zero size 2024-04-05T03:03:01.552 INFO:tasks.workunit.client.0.smithi153.stdout:1/874: fdatasync d5/dd/d1b/f1d 0 2024-04-05T03:03:01.555 INFO:tasks.workunit.client.0.smithi153.stdout:5/728: write d0/fed [2970365,42702] 0 2024-04-05T03:03:01.556 INFO:tasks.workunit.client.0.smithi153.stdout:2/892: dread - d9/da/f121 zero size 2024-04-05T03:03:01.557 INFO:tasks.workunit.client.0.smithi153.stdout:1/875: mkdir d5/de2/df5/d45/d68/d112 0 2024-04-05T03:03:01.557 INFO:tasks.workunit.client.0.smithi153.stdout:1/876: stat d5/de2/df5 0 2024-04-05T03:03:01.559 INFO:tasks.workunit.client.0.smithi153.stdout:9/680: creat d1/fea x:0 0 0 2024-04-05T03:03:01.559 INFO:tasks.workunit.client.0.smithi153.stdout:7/993: mknod d2/da/d3a/d41/d7a/d54/d85/df6/d93/d119/df2/d133/c145 0 2024-04-05T03:03:01.562 INFO:tasks.workunit.client.0.smithi153.stdout:2/893: dread d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:03:01.565 INFO:tasks.workunit.client.0.smithi153.stdout:9/681: symlink d1/d9/dbc/leb 0 2024-04-05T03:03:01.567 INFO:tasks.workunit.client.0.smithi153.stdout:2/894: rename d9/da/d19/d2e/dc7/fcc to d9/da/d37/d8a/da8/f12c 0 2024-04-05T03:03:01.568 INFO:tasks.workunit.client.0.smithi153.stdout:2/895: chown d9/d68/fab 395434712 1 2024-04-05T03:03:01.571 INFO:tasks.workunit.client.0.smithi153.stdout:9/682: rename d1/d10/d16/d22/d2c/d68/da9/ldb to d1/d10/d16/d22/d2c/d68/d81/lec 0 2024-04-05T03:03:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:01 smithi153 ceph-mon[29381]: pgmap v15: 65 pgs: 65 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 112 MiB/s rd, 259 MiB/s wr, 526 op/s 2024-04-05T03:03:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:01 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:01.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:01 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:01.572 INFO:tasks.workunit.client.0.smithi153.stdout:9/683: chown d1/d9/d1b/f1c 678 1 2024-04-05T03:03:01.578 INFO:tasks.workunit.client.0.smithi153.stdout:2/896: mknod d9/da/d19/d2e/d11b/c12d 0 2024-04-05T03:03:01.579 INFO:tasks.workunit.client.0.smithi153.stdout:5/729: truncate d0/d25/d4d/d9c/dc7/fde 785855 0 2024-04-05T03:03:01.581 INFO:tasks.workunit.client.0.smithi153.stdout:5/730: chown d0/d25/d4d/f8d 839 1 2024-04-05T03:03:01.581 INFO:tasks.workunit.client.0.smithi153.stdout:7/994: write d2/da/d47/d5d/d7c/d118/f125 [174019,54463] 0 2024-04-05T03:03:01.586 INFO:tasks.workunit.client.0.smithi153.stdout:1/877: dwrite d5/de2/df5/d48/d96/fa0 [0,4194304] 0 2024-04-05T03:03:01.590 INFO:tasks.workunit.client.0.smithi153.stdout:2/897: dwrite d9/da/d37/d8a/ff1 [8388608,4194304] 0 2024-04-05T03:03:01.593 INFO:tasks.workunit.client.0.smithi153.stdout:7/995: dwrite d2/da/d3a/d41/ddd/fef [0,4194304] 0 2024-04-05T03:03:01.609 INFO:tasks.workunit.client.0.smithi153.stdout:9/684: dread d1/d9/fc1 [0,4194304] 0 2024-04-05T03:03:01.613 INFO:tasks.workunit.client.0.smithi153.stdout:5/731: creat d0/d2c/ffe x:0 0 0 2024-04-05T03:03:01.614 INFO:tasks.workunit.client.0.smithi153.stdout:5/732: write d0/d25/d4d/fec [288959,64487] 0 2024-04-05T03:03:01.615 INFO:tasks.workunit.client.0.smithi153.stdout:2/898: mkdir d9/da/d19/d2e/d7e/d7a/d59/d12e 0 2024-04-05T03:03:01.620 INFO:tasks.workunit.client.0.smithi153.stdout:2/899: read d9/d77/f58 [3362744,65358] 0 2024-04-05T03:03:01.621 INFO:tasks.workunit.client.0.smithi153.stdout:5/733: getdents d0/d25/d4d/d62/d5f/d86 0 2024-04-05T03:03:01.621 INFO:tasks.workunit.client.0.smithi153.stdout:2/900: chown d9/d68/fce 1 1 2024-04-05T03:03:01.624 INFO:tasks.workunit.client.0.smithi153.stdout:5/734: rename d0/d9/f1e to d0/d25/d4d/d62/d5f/d86/fff 0 2024-04-05T03:03:01.626 INFO:tasks.workunit.client.0.smithi153.stdout:2/901: mknod d9/da/d19/df6/c12f 0 2024-04-05T03:03:01.629 INFO:tasks.workunit.client.0.smithi153.stdout:5/735: creat d0/d2c/d3f/f100 x:0 0 0 2024-04-05T03:03:01.631 INFO:tasks.workunit.client.0.smithi153.stdout:5/736: creat d0/d4e/db6/f101 x:0 0 0 2024-04-05T03:03:01.635 INFO:tasks.workunit.client.0.smithi153.stdout:5/737: mkdir d0/d2c/d3f/d102 0 2024-04-05T03:03:01.642 INFO:tasks.workunit.client.0.smithi153.stdout:5/738: dwrite d0/d2c/f2d [0,4194304] 0 2024-04-05T03:03:01.642 INFO:tasks.workunit.client.0.smithi153.stdout:5/739: readlink d0/d25/d4d/l7f 0 2024-04-05T03:03:01.642 INFO:tasks.workunit.client.0.smithi153.stdout:1/878: truncate d5/d8/d20/f5f 3806455 0 2024-04-05T03:03:01.645 INFO:tasks.workunit.client.0.smithi153.stdout:1/879: creat d5/de2/df5/d45/d68/da4/f113 x:0 0 0 2024-04-05T03:03:01.648 INFO:tasks.workunit.client.0.smithi153.stdout:1/880: creat d5/d8/d20/d3f/f114 x:0 0 0 2024-04-05T03:03:01.649 INFO:tasks.workunit.client.0.smithi153.stdout:7/996: write d2/da/d3a/d41/d7a/d54/d85/df6/f79 [4816912,91947] 0 2024-04-05T03:03:01.649 INFO:tasks.workunit.client.0.smithi153.stdout:9/685: write d1/d10/d16/f98 [589310,101] 0 2024-04-05T03:03:01.651 INFO:tasks.workunit.client.0.smithi153.stdout:1/881: write d5/d8/d20/d2f/d40/ff1 [998969,1732] 0 2024-04-05T03:03:01.653 INFO:tasks.workunit.client.0.smithi153.stdout:2/902: truncate d9/da/d19/d2e/d7e/d7a/d6d/f124 2121938 0 2024-04-05T03:03:01.654 INFO:tasks.workunit.client.0.smithi153.stdout:7/997: chown d2/f6 16334 1 2024-04-05T03:03:01.654 INFO:tasks.workunit.client.0.smithi153.stdout:7/998: stat d2/f6 0 2024-04-05T03:03:01.655 INFO:tasks.workunit.client.0.smithi153.stdout:9/686: mkdir d1/d10/d36/d57/d97/ded 0 2024-04-05T03:03:01.658 INFO:tasks.workunit.client.0.smithi153.stdout:1/882: symlink d5/de2/df5/d45/d93/l115 0 2024-04-05T03:03:01.659 INFO:tasks.workunit.client.0.smithi153.stdout:1/883: stat d5/de2/df5/d45/d68/da4/dc3/fcd 0 2024-04-05T03:03:01.660 INFO:tasks.workunit.client.0.smithi153.stdout:2/903: unlink d9/da/fd0 0 2024-04-05T03:03:01.662 INFO:tasks.workunit.client.0.smithi153.stdout:7/999: creat d2/d9b/d6f/d10a/f146 x:0 0 0 2024-04-05T03:03:01.663 INFO:tasks.workunit.client.0.smithi153.stdout:9/687: rename d1/d1d/l90 to d1/d1d/lee 0 2024-04-05T03:03:01.664 INFO:tasks.workunit.client.0.smithi153.stdout:5/740: getdents d0/d25/d4d/d62/d5f/d86 0 2024-04-05T03:03:01.666 INFO:tasks.workunit.client.0.smithi153.stdout:1/884: dread d5/d8/fc0 [0,4194304] 0 2024-04-05T03:03:01.669 INFO:tasks.workunit.client.0.smithi153.stdout:2/904: dwrite d9/da/d94/ffc [0,4194304] 0 2024-04-05T03:03:01.670 INFO:tasks.workunit.client.0.smithi153.stdout:1/885: write d5/d8/d20/d4b/d18/fb2 [551873,113876] 0 2024-04-05T03:03:01.673 INFO:tasks.workunit.client.0.smithi153.stdout:5/741: unlink d0/d25/f96 0 2024-04-05T03:03:01.674 INFO:tasks.workunit.client.0.smithi153.stdout:1/886: write d5/de2/df5/d33/f3c [2018157,1677] 0 2024-04-05T03:03:01.677 INFO:tasks.workunit.client.0.smithi153.stdout:2/905: truncate d9/da/fd5 803045 0 2024-04-05T03:03:01.680 INFO:tasks.workunit.client.0.smithi153.stdout:1/887: fsync d5/de2/df5/d45/d93/fcc 0 2024-04-05T03:03:01.686 INFO:tasks.workunit.client.0.smithi153.stdout:2/906: dwrite d9/d62/f12a [0,4194304] 0 2024-04-05T03:03:01.687 INFO:tasks.workunit.client.0.smithi153.stdout:9/688: dread d1/d9/fc [0,4194304] 0 2024-04-05T03:03:01.689 INFO:tasks.workunit.client.0.smithi153.stdout:1/888: creat d5/d8/f116 x:0 0 0 2024-04-05T03:03:01.690 INFO:tasks.workunit.client.0.smithi153.stdout:9/689: chown d1/d10/d36/d57/d5c/f89 1453885 1 2024-04-05T03:03:01.694 INFO:tasks.workunit.client.0.smithi153.stdout:5/742: dread d0/d25/f6e [0,4194304] 0 2024-04-05T03:03:01.695 INFO:tasks.workunit.client.0.smithi153.stdout:9/690: creat d1/d10/d16/d22/d2c/d68/d81/fef x:0 0 0 2024-04-05T03:03:01.697 INFO:tasks.workunit.client.0.smithi153.stdout:5/743: write d0/d25/d4d/d62/d37/f4f [3825749,41678] 0 2024-04-05T03:03:01.697 INFO:tasks.workunit.client.0.smithi153.stdout:2/907: dwrite d9/da/f49 [0,4194304] 0 2024-04-05T03:03:01.701 INFO:tasks.workunit.client.0.smithi153.stdout:5/744: readlink d0/d25/d4d/d62/d5f/d86/lf0 0 2024-04-05T03:03:01.702 INFO:tasks.workunit.client.0.smithi153.stdout:2/908: mkdir d9/d68/d98/d130 0 2024-04-05T03:03:01.702 INFO:tasks.workunit.client.0.smithi153.stdout:2/909: stat d9/d68/d98/fac 0 2024-04-05T03:03:01.706 INFO:tasks.workunit.client.0.smithi153.stdout:2/910: mknod d9/da/df9/c131 0 2024-04-05T03:03:01.708 INFO:tasks.workunit.client.0.smithi153.stdout:2/911: chown d9/da/d19/d2e/d7e/d7a/d6d/f124 87737 1 2024-04-05T03:03:01.709 INFO:tasks.workunit.client.0.smithi153.stdout:9/691: write d1/d10/f4c [722675,50958] 0 2024-04-05T03:03:01.713 INFO:tasks.workunit.client.0.smithi153.stdout:9/692: creat d1/d10/d16/d22/d2c/d68/da9/ff0 x:0 0 0 2024-04-05T03:03:01.716 INFO:tasks.workunit.client.0.smithi153.stdout:9/693: chown d1/d10/d16/d22/d2c/d68/l72 6725 1 2024-04-05T03:03:01.717 INFO:tasks.workunit.client.0.smithi153.stdout:2/912: dwrite d9/da/d19/d2e/d7e/d7a/fdb [0,4194304] 0 2024-04-05T03:03:01.722 INFO:tasks.workunit.client.0.smithi153.stdout:2/913: creat d9/da/d94/f132 x:0 0 0 2024-04-05T03:03:01.722 INFO:tasks.workunit.client.0.smithi153.stdout:2/914: dread - d9/da/d37/d8a/f114 zero size 2024-04-05T03:03:01.723 INFO:tasks.workunit.client.0.smithi153.stdout:5/745: dread d0/d25/d4d/d62/d37/f4f [0,4194304] 0 2024-04-05T03:03:01.726 INFO:tasks.workunit.client.0.smithi153.stdout:2/915: creat d9/da/d37/d8a/da8/de5/d116/f133 x:0 0 0 2024-04-05T03:03:01.729 INFO:tasks.workunit.client.0.smithi153.stdout:5/746: rmdir d0/d25 39 2024-04-05T03:03:01.729 INFO:tasks.workunit.client.0.smithi153.stdout:9/694: link d1/d10/d16/d22/d2c/d68/f7c d1/d10/d36/d71/ff1 0 2024-04-05T03:03:01.731 INFO:tasks.workunit.client.0.smithi153.stdout:5/747: chown d0/d25/d4d/d9c/fef 1 1 2024-04-05T03:03:01.733 INFO:tasks.workunit.client.0.smithi153.stdout:2/916: creat d9/da/d19/d2e/d7e/d7a/f134 x:0 0 0 2024-04-05T03:03:01.735 INFO:tasks.workunit.client.0.smithi153.stdout:2/917: write d9/da/d19/d2e/d7e/d7a/d6d/f6e [4568411,66375] 0 2024-04-05T03:03:01.735 INFO:tasks.workunit.client.0.smithi153.stdout:9/695: creat d1/d1d/db2/ff2 x:0 0 0 2024-04-05T03:03:01.736 INFO:tasks.workunit.client.0.smithi153.stdout:2/918: fsync d9/da/f64 0 2024-04-05T03:03:01.736 INFO:tasks.workunit.client.0.smithi153.stdout:5/748: creat d0/d2c/f103 x:0 0 0 2024-04-05T03:03:01.739 INFO:tasks.workunit.client.0.smithi153.stdout:9/696: mknod d1/d10/cf3 0 2024-04-05T03:03:01.741 INFO:tasks.workunit.client.0.smithi153.stdout:2/919: unlink d9/da/d94/dca/l129 0 2024-04-05T03:03:01.743 INFO:tasks.workunit.client.0.smithi153.stdout:5/749: creat d0/d25/d4d/d62/d5f/d86/f104 x:0 0 0 2024-04-05T03:03:01.743 INFO:tasks.workunit.client.0.smithi153.stdout:2/920: fsync d9/da/d37/d8a/da8/de5/d116/f133 0 2024-04-05T03:03:01.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/750: symlink d0/d2c/d3f/l105 0 2024-04-05T03:03:01.748 INFO:tasks.workunit.client.0.smithi153.stdout:2/921: creat d9/da/d19/d2e/d7e/d7a/d6d/de8/f135 x:0 0 0 2024-04-05T03:03:01.748 INFO:tasks.workunit.client.0.smithi153.stdout:5/751: write d0/d2c/ffe [926804,55799] 0 2024-04-05T03:03:01.750 INFO:tasks.workunit.client.0.smithi153.stdout:9/697: dwrite d1/d9/fc1 [0,4194304] 0 2024-04-05T03:03:01.750 INFO:tasks.workunit.client.0.smithi153.stdout:2/922: dread - d9/da/d37/d8a/da8/de5/f127 zero size 2024-04-05T03:03:01.751 INFO:tasks.workunit.client.0.smithi153.stdout:9/698: chown d1/d10/d36/f62 61 1 2024-04-05T03:03:01.756 INFO:tasks.workunit.client.0.smithi153.stdout:5/752: unlink d0/d9/f1f 0 2024-04-05T03:03:01.758 INFO:tasks.workunit.client.0.smithi153.stdout:2/923: creat d9/da/d19/d2e/d7e/d102/f136 x:0 0 0 2024-04-05T03:03:01.760 INFO:tasks.workunit.client.0.smithi153.stdout:9/699: rename d1/d9/d1b/c8c to d1/d6c/cf4 0 2024-04-05T03:03:01.761 INFO:tasks.workunit.client.0.smithi153.stdout:5/753: mknod d0/d2c/d3f/df7/c106 0 2024-04-05T03:03:01.763 INFO:tasks.workunit.client.0.smithi153.stdout:5/754: symlink d0/d2c/d5e/l107 0 2024-04-05T03:03:01.763 INFO:tasks.workunit.client.0.smithi153.stdout:9/700: fsync d1/d10/d36/d71/fa0 0 2024-04-05T03:03:01.764 INFO:tasks.workunit.client.0.smithi153.stdout:5/755: write d0/d25/d4d/d62/d5f/d86/f104 [1041358,56623] 0 2024-04-05T03:03:01.766 INFO:tasks.workunit.client.0.smithi153.stdout:5/756: creat d0/d25/d4d/f108 x:0 0 0 2024-04-05T03:03:01.769 INFO:tasks.workunit.client.0.smithi153.stdout:5/757: mkdir d0/d25/d4d/d9c/dc4/dcb/d109 0 2024-04-05T03:03:01.774 INFO:tasks.workunit.client.0.smithi153.stdout:5/758: dwrite d0/d25/d4d/d62/f56 [0,4194304] 0 2024-04-05T03:03:01.776 INFO:tasks.workunit.client.0.smithi153.stdout:5/759: mkdir d0/d25/d4d/d9c/dc4/dcb/d10a 0 2024-04-05T03:03:01.778 INFO:tasks.workunit.client.0.smithi153.stdout:5/760: creat d0/d25/d4d/d62/d5f/db2/f10b x:0 0 0 2024-04-05T03:03:01.785 INFO:tasks.workunit.client.0.smithi153.stdout:5/761: link d0/d9/d22/l92 d0/d2c/d3f/da8/l10c 0 2024-04-05T03:03:01.787 INFO:tasks.workunit.client.0.smithi153.stdout:5/762: mknod d0/d2c/d3f/d102/c10d 0 2024-04-05T03:03:01.802 INFO:tasks.workunit.client.0.smithi153.stdout:0/767: sync 2024-04-05T03:03:01.804 INFO:tasks.workunit.client.0.smithi153.stdout:0/768: write d9/db/d42/d7e/db6/d1e/d45/d62/d79/fc6 [6340642,108056] 0 2024-04-05T03:03:01.810 INFO:tasks.workunit.client.0.smithi153.stdout:0/769: dwrite d9/d14/d67/d6f/fab [0,4194304] 0 2024-04-05T03:03:01.815 INFO:tasks.workunit.client.0.smithi153.stdout:0/770: write d9/db/d42/d7e/db6/d1e/f80 [542882,70941] 0 2024-04-05T03:03:01.820 INFO:tasks.workunit.client.0.smithi153.stdout:0/771: truncate d9/db/d42/d7e/db6/d1e/d45/d62/d79/fb1 291174 0 2024-04-05T03:03:01.822 INFO:tasks.workunit.client.0.smithi153.stdout:0/772: mknod d9/cea 0 2024-04-05T03:03:01.824 INFO:tasks.workunit.client.0.smithi153.stdout:0/773: symlink d9/db/d42/d7e/db6/d1d/d61/d8a/leb 0 2024-04-05T03:03:01.827 INFO:tasks.workunit.client.0.smithi153.stdout:0/774: rename d9/db/d42/d7e/db6/d1e/d84/cc8 to d9/db/d42/d7e/db6/d1e/d45/dae/cec 0 2024-04-05T03:03:01.828 INFO:tasks.workunit.client.0.smithi153.stdout:0/775: dread - d9/db/d42/d7e/db6/d1d/d61/d8a/fcf zero size 2024-04-05T03:03:01.829 INFO:tasks.workunit.client.0.smithi153.stdout:0/776: readlink d9/db/d42/d7e/db6/d1d/d61/d8a/l75 0 2024-04-05T03:03:01.830 INFO:tasks.workunit.client.0.smithi153.stdout:0/777: stat d9/d14/d67/fbf 0 2024-04-05T03:03:01.830 INFO:tasks.workunit.client.0.smithi153.stdout:0/778: fdatasync d9/d14/fd1 0 2024-04-05T03:03:01.847 INFO:tasks.workunit.client.0.smithi153.stdout:0/779: dread d9/d14/f40 [0,4194304] 0 2024-04-05T03:03:01.849 INFO:tasks.workunit.client.0.smithi153.stdout:0/780: creat d9/db/d42/d7e/db6/fed x:0 0 0 2024-04-05T03:03:01.850 INFO:tasks.workunit.client.0.smithi153.stdout:0/781: dread - d9/d14/f7a zero size 2024-04-05T03:03:01.854 INFO:tasks.workunit.client.0.smithi153.stdout:0/782: getdents d9/db/d42/d7e/db6/d1e/d45/d62/dde 0 2024-04-05T03:03:01.856 INFO:tasks.workunit.client.0.smithi153.stdout:0/783: fdatasync d9/db/d42/d7e/db6/fe8 0 2024-04-05T03:03:01.857 INFO:tasks.workunit.client.0.smithi153.stdout:0/784: chown d9/db/d42/d7e/db6/d1d/d61/f3d 11 1 2024-04-05T03:03:01.860 INFO:tasks.workunit.client.0.smithi153.stdout:0/785: creat d9/db/d42/d43/d89/fee x:0 0 0 2024-04-05T03:03:01.863 INFO:tasks.workunit.client.0.smithi153.stdout:0/786: rename d9/db/d42/dd4/le6 to d9/db/lef 0 2024-04-05T03:03:01.865 INFO:tasks.workunit.client.0.smithi153.stdout:0/787: creat d9/db/d42/dd4/ff0 x:0 0 0 2024-04-05T03:03:01.868 INFO:tasks.workunit.client.0.smithi153.stdout:0/788: link d9/d14/d67/fc3 d9/ff1 0 2024-04-05T03:03:01.882 INFO:tasks.workunit.client.0.smithi153.stdout:0/789: dread d9/db/fd [0,4194304] 0 2024-04-05T03:03:01.883 INFO:tasks.workunit.client.0.smithi153.stdout:0/790: stat d9/db/d42/d7e/db6/d1d/d61/d8a/leb 0 2024-04-05T03:03:01.884 INFO:tasks.workunit.client.0.smithi153.stdout:0/791: fsync d9/db/d42/d7e/db6/d1e/d45/d62/dde/fc9 0 2024-04-05T03:03:01.890 INFO:tasks.workunit.client.0.smithi153.stdout:0/792: dwrite d9/db/d42/d43/f4a [0,4194304] 0 2024-04-05T03:03:01.898 INFO:tasks.workunit.client.0.smithi153.stdout:0/793: dwrite d9/db/fd6 [0,4194304] 0 2024-04-05T03:03:01.907 INFO:tasks.workunit.client.0.smithi153.stdout:0/794: rename d9/fc7 to d9/db/d42/d7e/db6/d1e/d45/ff2 0 2024-04-05T03:03:01.911 INFO:tasks.workunit.client.0.smithi153.stdout:0/795: read d9/db/f37 [3155026,68962] 0 2024-04-05T03:03:01.914 INFO:tasks.workunit.client.0.smithi153.stdout:0/796: mknod d9/db/d42/d7e/db6/d1e/d45/d62/de7/cf3 0 2024-04-05T03:03:01.926 INFO:tasks.workunit.client.0.smithi153.stdout:0/797: dwrite d9/db/d42/f71 [0,4194304] 0 2024-04-05T03:03:01.927 INFO:tasks.workunit.client.0.smithi153.stdout:0/798: stat d9/db/d42/d7e/db6/d1e/d45/d62/dde/d6d/l8e 0 2024-04-05T03:03:01.933 INFO:tasks.workunit.client.0.smithi153.stdout:0/799: rename d9/db/d42/d7e/db6/d1e/d45/d62/d79 to d9/db/d42/d7e/db6/d1e/d45/dae/df4 0 2024-04-05T03:03:01.939 INFO:tasks.workunit.client.0.smithi153.stdout:0/800: dwrite d9/db/d42/d7e/db6/d1e/fa7 [0,4194304] 0 2024-04-05T03:03:01.941 INFO:tasks.workunit.client.0.smithi153.stdout:0/801: chown d9/db/d42/d43/de4 46080 1 2024-04-05T03:03:01.943 INFO:tasks.workunit.client.0.smithi153.stdout:0/802: dread d9/db/d42/d43/fa1 [0,4194304] 0 2024-04-05T03:03:01.954 INFO:tasks.workunit.client.0.smithi153.stdout:0/803: dwrite d9/db/d42/d7e/db6/f15 [0,4194304] 0 2024-04-05T03:03:01.962 INFO:tasks.workunit.client.0.smithi153.stdout:0/804: dread d9/f69 [0,4194304] 0 2024-04-05T03:03:01.966 INFO:tasks.workunit.client.0.smithi153.stdout:0/805: symlink d9/db/d42/lf5 0 2024-04-05T03:03:01.969 INFO:tasks.workunit.client.0.smithi153.stdout:0/806: creat d9/db/d42/d43/de4/ff6 x:0 0 0 2024-04-05T03:03:01.980 INFO:tasks.workunit.client.0.smithi153.stdout:0/807: write f3 [124298,21049] 0 2024-04-05T03:03:01.984 INFO:tasks.workunit.client.0.smithi153.stdout:0/808: creat d9/db/d42/d7e/db6/d1e/ff7 x:0 0 0 2024-04-05T03:03:01.987 INFO:tasks.workunit.client.0.smithi153.stdout:0/809: mkdir d9/db/d42/d7e/db6/d1d/d61/d8a/df8 0 2024-04-05T03:03:01.991 INFO:tasks.workunit.client.0.smithi153.stdout:0/810: symlink d9/db/d42/d7e/db6/d1d/d61/d8a/df8/lf9 0 2024-04-05T03:03:01.991 INFO:tasks.workunit.client.0.smithi153.stdout:0/811: dread - d9/db/f44 zero size 2024-04-05T03:03:01.992 INFO:tasks.workunit.client.0.smithi153.stdout:0/812: stat d9/d14/ccd 0 2024-04-05T03:03:01.997 INFO:tasks.workunit.client.0.smithi153.stdout:0/813: rename d9/db/d42/d7e/db6/f16 to d9/db/d42/d43/de4/ffa 0 2024-04-05T03:03:02.000 INFO:tasks.workunit.client.0.smithi153.stdout:0/814: rmdir d9/db/d42/d43/d89 39 2024-04-05T03:03:02.001 INFO:tasks.workunit.client.0.smithi153.stdout:0/815: fdatasync d9/d14/d67/d6f/f7c 0 2024-04-05T03:03:02.003 INFO:tasks.workunit.client.0.smithi153.stdout:0/816: truncate d9/d14/d67/fc3 175808 0 2024-04-05T03:03:02.006 INFO:tasks.workunit.client.0.smithi153.stdout:0/817: fsync d9/db/d42/d7e/fa6 0 2024-04-05T03:03:02.020 INFO:tasks.workunit.client.0.smithi153.stdout:0/818: dwrite d9/db/d42/d7e/db6/d1d/d61/d8a/fa5 [0,4194304] 0 2024-04-05T03:03:02.023 INFO:tasks.workunit.client.0.smithi153.stdout:0/819: chown d9/db/d42/d7e/db6/d1e/fa7 222 1 2024-04-05T03:03:02.031 INFO:tasks.workunit.client.0.smithi153.stdout:0/820: link d9/db/d42/d43/cc5 d9/db/d42/d7e/db6/d1e/cfb 0 2024-04-05T03:03:02.050 INFO:tasks.workunit.client.0.smithi153.stdout:0/821: dwrite d9/db/d42/d7e/db6/d1e/d45/d62/dde/fa4 [0,4194304] 0 2024-04-05T03:03:02.050 INFO:tasks.workunit.client.0.smithi153.stdout:0/822: creat d9/db/d42/d43/de4/ffc x:0 0 0 2024-04-05T03:03:02.051 INFO:tasks.workunit.client.0.smithi153.stdout:0/823: read - d9/db/d42/d43/de4/ffc zero size 2024-04-05T03:03:02.052 INFO:tasks.workunit.client.0.smithi153.stdout:0/824: stat d9/db/f66 0 2024-04-05T03:03:02.057 INFO:tasks.workunit.client.0.smithi153.stdout:0/825: rename d9/db/d42/d7e/db6/d1d/d61/d8a/fcf to d9/d14/d67/d6f/ffd 0 2024-04-05T03:03:02.061 INFO:tasks.workunit.client.0.smithi153.stdout:0/826: getdents d9/db/d42/d7e/db6/daf 0 2024-04-05T03:03:02.064 INFO:tasks.workunit.client.0.smithi153.stdout:0/827: read d9/db/d42/f74 [2689533,34941] 0 2024-04-05T03:03:02.066 INFO:tasks.workunit.client.0.smithi153.stdout:0/828: write d9/db/d42/d7e/d8b/fe5 [813022,120870] 0 2024-04-05T03:03:02.068 INFO:tasks.workunit.client.0.smithi153.stdout:0/829: symlink d9/db/ddf/lfe 0 2024-04-05T03:03:02.070 INFO:tasks.workunit.client.0.smithi153.stdout:0/830: unlink d9/db/d42/l88 0 2024-04-05T03:03:02.071 INFO:tasks.workunit.client.0.smithi153.stdout:0/831: chown d9/db/d42/d7e/db6/d1e/d45/ccb 60 1 2024-04-05T03:03:02.074 INFO:tasks.workunit.client.0.smithi153.stdout:0/832: dread d9/db/d42/d7e/db6/d1e/d45/f6a [0,4194304] 0 2024-04-05T03:03:02.080 INFO:tasks.workunit.client.0.smithi153.stdout:0/833: link d9/db/d42/d43/f85 d9/db/d42/d7e/db6/d1e/d45/d62/fff 0 2024-04-05T03:03:02.080 INFO:tasks.workunit.client.0.smithi153.stdout:0/834: readlink d9/db/d42/d7e/db6/d1e/l2c 0 2024-04-05T03:03:02.087 INFO:tasks.workunit.client.0.smithi153.stdout:0/835: write d9/db/d42/d7e/db6/d1d/d61/d8a/f7b [2490098,90792] 0 2024-04-05T03:03:02.088 INFO:tasks.workunit.client.0.smithi153.stdout:0/836: chown d9/db/d42/d7e/fa6 46 1 2024-04-05T03:03:02.090 INFO:tasks.workunit.client.0.smithi153.stdout:0/837: chown d9/db/d42/d43/cc5 13 1 2024-04-05T03:03:02.092 INFO:tasks.workunit.client.0.smithi153.stdout:0/838: mknod d9/db/d42/d7e/d8b/c100 0 2024-04-05T03:03:02.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/717: sync 2024-04-05T03:03:02.100 INFO:tasks.workunit.client.0.smithi153.stdout:0/839: dwrite d9/db/f11 [4194304,4194304] 0 2024-04-05T03:03:02.101 INFO:tasks.workunit.client.0.smithi153.stdout:8/718: symlink d7/d13/d19/d1b/d6d/d8c/le6 0 2024-04-05T03:03:02.103 INFO:tasks.workunit.client.0.smithi153.stdout:0/840: mknod d9/db/d42/d7e/c101 0 2024-04-05T03:03:02.104 INFO:tasks.workunit.client.0.smithi153.stdout:8/719: creat d7/d9a/fe7 x:0 0 0 2024-04-05T03:03:02.104 INFO:tasks.workunit.client.0.smithi153.stdout:0/841: read d9/d14/d67/d6f/fab [2651235,90400] 0 2024-04-05T03:03:02.107 INFO:tasks.workunit.client.0.smithi153.stdout:8/720: symlink d7/d13/dab/le8 0 2024-04-05T03:03:02.110 INFO:tasks.workunit.client.0.smithi153.stdout:8/721: creat d7/d13/d19/d1b/d4f/d5f/fe9 x:0 0 0 2024-04-05T03:03:02.112 INFO:tasks.workunit.client.0.smithi153.stdout:8/722: mknod d7/d13/d19/d1b/d4f/d68/cea 0 2024-04-05T03:03:02.114 INFO:tasks.workunit.client.0.smithi153.stdout:8/723: write d7/d13/d19/d1b/d4f/f86 [1920670,119950] 0 2024-04-05T03:03:02.117 INFO:tasks.workunit.client.0.smithi153.stdout:8/724: creat d7/d13/d19/d1b/d4f/d68/dbc/feb x:0 0 0 2024-04-05T03:03:02.123 INFO:tasks.workunit.client.0.smithi153.stdout:8/725: dwrite d7/d13/d19/d1b/d33/fdd [0,4194304] 0 2024-04-05T03:03:02.126 INFO:tasks.workunit.client.0.smithi153.stdout:6/902: sync 2024-04-05T03:03:02.128 INFO:tasks.workunit.client.0.smithi153.stdout:8/726: link d7/d13/d19/fb5 d7/d13/d19/d1b/d4f/d5f/fec 0 2024-04-05T03:03:02.131 INFO:tasks.workunit.client.0.smithi153.stdout:8/727: dread d7/d13/f8e [0,4194304] 0 2024-04-05T03:03:02.132 INFO:tasks.workunit.client.0.smithi153.stdout:6/903: write d4/dc/df/f56 [4806800,20434] 0 2024-04-05T03:03:02.135 INFO:tasks.workunit.client.0.smithi153.stdout:6/904: truncate d4/dc/df/d16/d44/ded/fff 130568 0 2024-04-05T03:03:02.138 INFO:tasks.workunit.client.0.smithi153.stdout:6/905: unlink d4/dc/df/d16/d5f/fb7 0 2024-04-05T03:03:02.141 INFO:tasks.workunit.client.0.smithi153.stdout:6/906: rename d4/dc/df/d16/d24/c53 to d4/dc/df/d16/d36/d4d/dad/c137 0 2024-04-05T03:03:02.148 INFO:tasks.workunit.client.0.smithi153.stdout:8/728: dread d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:03:02.152 INFO:tasks.workunit.client.0.smithi153.stdout:8/729: rename d7/d13/d19/d1b/d4f/d68/fce to d7/d13/dc8/fed 0 2024-04-05T03:03:02.154 INFO:tasks.workunit.client.0.smithi153.stdout:6/907: dread d4/dc/df/f56 [0,4194304] 0 2024-04-05T03:03:02.157 INFO:tasks.workunit.client.0.smithi153.stdout:6/908: mknod d4/dc/df/d16/d5f/d73/c138 0 2024-04-05T03:03:02.161 INFO:tasks.workunit.client.0.smithi153.stdout:8/730: truncate f3 3131164 0 2024-04-05T03:03:02.161 INFO:tasks.workunit.client.0.smithi153.stdout:6/909: write d4/dc/df/d16/d24/d5e/f60 [971885,61065] 0 2024-04-05T03:03:02.164 INFO:tasks.workunit.client.0.smithi153.stdout:8/731: rename d7/d2a/d8b/fd2 to d7/d2a/d50/fee 0 2024-04-05T03:03:02.165 INFO:tasks.workunit.client.0.smithi153.stdout:6/910: mkdir d4/dc/df/d16/d24/d5e/de8/dfa/d139 0 2024-04-05T03:03:02.166 INFO:tasks.workunit.client.0.smithi153.stdout:6/911: chown d4/dc/df/c6e 7 1 2024-04-05T03:03:02.169 INFO:tasks.workunit.client.0.smithi153.stdout:6/912: rename d4/dc/ffd to d4/dc/df/d16/d24/f13a 0 2024-04-05T03:03:02.170 INFO:tasks.workunit.client.0.smithi153.stdout:6/913: chown d4/dc/d7a/le5 115 1 2024-04-05T03:03:02.175 INFO:tasks.workunit.client.0.smithi153.stdout:6/914: dwrite d4/dc/df/d16/d17/f131 [0,4194304] 0 2024-04-05T03:03:02.178 INFO:tasks.workunit.client.0.smithi153.stdout:6/915: symlink d4/dc/df/d16/d113/l13b 0 2024-04-05T03:03:02.180 INFO:tasks.workunit.client.0.smithi153.stdout:6/916: chown d4/dc/df/d16/d24/f13a 47258758 1 2024-04-05T03:03:02.187 INFO:tasks.workunit.client.0.smithi153.stdout:6/917: truncate d4/dc/df/f56 4396564 0 2024-04-05T03:03:02.190 INFO:tasks.workunit.client.0.smithi153.stdout:6/918: unlink d4/dc/df/c34 0 2024-04-05T03:03:02.316 INFO:tasks.workunit.client.0.smithi153.stdout:3/828: sync 2024-04-05T03:03:02.317 INFO:tasks.workunit.client.0.smithi153.stdout:3/829: write d1d/d32/d93/d3c/d3f/d90/db6/f113 [283298,3104] 0 2024-04-05T03:03:02.324 INFO:tasks.workunit.client.0.smithi153.stdout:3/830: dwrite d1d/d32/f49 [8388608,4194304] 0 2024-04-05T03:03:02.328 INFO:tasks.workunit.client.0.smithi153.stdout:3/831: creat d1d/ddf/f127 x:0 0 0 2024-04-05T03:03:02.334 INFO:tasks.workunit.client.0.smithi153.stdout:3/832: dwrite d1d/d27/f64 [0,4194304] 0 2024-04-05T03:03:02.338 INFO:tasks.workunit.client.0.smithi153.stdout:3/833: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/c128 0 2024-04-05T03:03:02.341 INFO:tasks.workunit.client.0.smithi153.stdout:3/834: mknod d1d/d32/d93/d3c/d3f/d90/db6/c129 0 2024-04-05T03:03:02.343 INFO:tasks.workunit.client.0.smithi153.stdout:3/835: mkdir d1d/d12a 0 2024-04-05T03:03:02.348 INFO:tasks.workunit.client.0.smithi153.stdout:3/836: dwrite d1d/f6a [0,4194304] 0 2024-04-05T03:03:02.359 INFO:tasks.workunit.client.0.smithi153.stdout:3/837: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f59 [619514,89319] 0 2024-04-05T03:03:02.360 INFO:tasks.workunit.client.0.smithi153.stdout:3/838: stat d1d/d32/d93/d3c/fd9 0 2024-04-05T03:03:02.361 INFO:tasks.workunit.client.0.smithi153.stdout:3/839: rmdir d1d/d32 39 2024-04-05T03:03:02.363 INFO:tasks.workunit.client.0.smithi153.stdout:3/840: chown d1d/d32/d93/c2a 5407448 1 2024-04-05T03:03:02.365 INFO:tasks.workunit.client.0.smithi153.stdout:3/841: read d1d/d32/d93/d3c/d3f/d9c/fc4 [3608604,118299] 0 2024-04-05T03:03:02.367 INFO:tasks.workunit.client.0.smithi153.stdout:3/842: dread - d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fea zero size 2024-04-05T03:03:02.368 INFO:tasks.workunit.client.0.smithi153.stdout:3/843: write d1d/d32/d93/d3c/d3f/d9c/f102 [909058,68256] 0 2024-04-05T03:03:02.369 INFO:tasks.workunit.client.0.smithi153.stdout:3/844: fsync d1d/d32/d93/d3c/d3f/d9c/f107 0 2024-04-05T03:03:02.370 INFO:tasks.workunit.client.0.smithi153.stdout:3/845: fdatasync d1d/fcf 0 2024-04-05T03:03:02.384 INFO:tasks.workunit.client.0.smithi153.stdout:3/846: dread d1d/fa7 [0,4194304] 0 2024-04-05T03:03:02.387 INFO:tasks.workunit.client.0.smithi153.stdout:3/847: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d123/c12b 0 2024-04-05T03:03:02.390 INFO:tasks.workunit.client.0.smithi153.stdout:3/848: unlink d1d/d32/d93/d3c/d3f/d7c/de7/c11e 0 2024-04-05T03:03:02.393 INFO:tasks.workunit.client.0.smithi153.stdout:3/849: mknod d1d/d32/d93/d3c/d94/c12c 0 2024-04-05T03:03:02.395 INFO:tasks.workunit.client.0.smithi153.stdout:3/850: mkdir d1d/d32/d93/d3c/d3f/d90/d12d 0 2024-04-05T03:03:02.396 INFO:tasks.workunit.client.0.smithi153.stdout:3/851: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/f9a 0 1 2024-04-05T03:03:02.407 INFO:tasks.workunit.client.0.smithi153.stdout:3/852: dwrite d1d/d32/d93/d3c/d3f/d7c/fef [0,4194304] 0 2024-04-05T03:03:02.408 INFO:tasks.workunit.client.0.smithi153.stdout:3/853: fsync d1d/d32/f49 0 2024-04-05T03:03:02.411 INFO:tasks.workunit.client.0.smithi153.stdout:3/854: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/d11f/cbc 0 1 2024-04-05T03:03:02.415 INFO:tasks.workunit.client.0.smithi153.stdout:3/855: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/l12e 0 2024-04-05T03:03:02.416 INFO:tasks.workunit.client.0.smithi153.stdout:3/856: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fa5 [3695581,55400] 0 2024-04-05T03:03:02.419 INFO:tasks.workunit.client.0.smithi153.stdout:3/857: creat d1d/d32/d93/d3c/d3f/f12f x:0 0 0 2024-04-05T03:03:02.470 INFO:tasks.workunit.client.0.smithi153.stdout:4/902: sync 2024-04-05T03:03:02.475 INFO:tasks.workunit.client.0.smithi153.stdout:4/903: rename da/de/d66/d111/fd3 to da/db/d20/d35/f13e 0 2024-04-05T03:03:02.614 INFO:tasks.workunit.client.0.smithi153.stdout:1/889: sync 2024-04-05T03:03:02.614 INFO:tasks.workunit.client.0.smithi153.stdout:1/890: chown d5/d8/f65 45582468 1 2024-04-05T03:03:02.616 INFO:tasks.workunit.client.0.smithi153.stdout:1/891: write d5/d8/d20/d4b/de1/f105 [823125,93700] 0 2024-04-05T03:03:02.617 INFO:tasks.workunit.client.0.smithi153.stdout:1/892: write d5/d8/f6e [1769094,107746] 0 2024-04-05T03:03:02.620 INFO:tasks.workunit.client.0.smithi153.stdout:1/893: symlink d5/de2/df5/d2c/l117 0 2024-04-05T03:03:02.625 INFO:tasks.workunit.client.0.smithi153.stdout:1/894: dwrite d5/f84 [0,4194304] 0 2024-04-05T03:03:02.630 INFO:tasks.workunit.client.0.smithi153.stdout:1/895: dread d5/de2/df5/d45/d93/fcc [0,4194304] 0 2024-04-05T03:03:02.630 INFO:tasks.workunit.client.0.smithi153.stdout:1/896: write d5/de2/df5/f36 [1864913,113477] 0 2024-04-05T03:03:02.631 INFO:tasks.workunit.client.0.smithi153.stdout:1/897: fdatasync d5/de2/df5/d48/d96/fa0 0 2024-04-05T03:03:02.636 INFO:tasks.workunit.client.0.smithi153.stdout:1/898: mknod d5/dd/c118 0 2024-04-05T03:03:02.639 INFO:tasks.workunit.client.0.smithi153.stdout:1/899: mknod d5/de2/df5/d45/d93/d111/c119 0 2024-04-05T03:03:02.640 INFO:tasks.workunit.client.0.smithi153.stdout:1/900: write d5/de2/df5/d45/d68/f10a [930331,57661] 0 2024-04-05T03:03:02.643 INFO:tasks.workunit.client.0.smithi153.stdout:1/901: creat d5/de2/df5/f11a x:0 0 0 2024-04-05T03:03:02.649 INFO:tasks.workunit.client.0.smithi153.stdout:1/902: dwrite d5/de2/df5/d45/d68/da4/f113 [0,4194304] 0 2024-04-05T03:03:02.652 INFO:tasks.workunit.client.0.smithi153.stdout:1/903: symlink d5/d8/d20/d4b/d51/l11b 0 2024-04-05T03:03:02.653 INFO:tasks.workunit.client.0.smithi153.stdout:1/904: write d5/d8/d20/d4b/dd6/fe7 [958145,118549] 0 2024-04-05T03:03:02.661 INFO:tasks.workunit.client.0.smithi153.stdout:1/905: dwrite d5/de2/df5/d45/d93/f9a [4194304,4194304] 0 2024-04-05T03:03:02.662 INFO:tasks.workunit.client.0.smithi153.stdout:1/906: chown d5/d8/d20/d4b/d51/db5/f109 710 1 2024-04-05T03:03:02.662 INFO:tasks.workunit.client.0.smithi153.stdout:1/907: write d5/f76 [1837991,62649] 0 2024-04-05T03:03:02.667 INFO:tasks.workunit.client.0.smithi153.stdout:1/908: rename d5/de2/df5/d45/d93/d10e to d5/d8/d20/d4b/de1/d11c 0 2024-04-05T03:03:02.670 INFO:tasks.workunit.client.0.smithi153.stdout:1/909: dread d5/d8/d20/d3f/f86 [0,4194304] 0 2024-04-05T03:03:02.671 INFO:tasks.workunit.client.0.smithi153.stdout:2/924: sync 2024-04-05T03:03:02.674 INFO:tasks.workunit.client.0.smithi153.stdout:2/925: rmdir d9/da/d19/d2e/d7e/d7a/d6d/de8 39 2024-04-05T03:03:02.674 INFO:tasks.workunit.client.0.smithi153.stdout:1/910: dread d5/d8/d20/d2f/d40/ff1 [0,4194304] 0 2024-04-05T03:03:02.677 INFO:tasks.workunit.client.0.smithi153.stdout:2/926: write d9/da/d37/d8a/da8/de5/d116/f117 [5184556,20397] 0 2024-04-05T03:03:02.678 INFO:tasks.workunit.client.0.smithi153.stdout:2/927: readlink d9/da/d19/l2a 0 2024-04-05T03:03:02.679 INFO:tasks.workunit.client.0.smithi153.stdout:1/911: link d5/d8/d20/d4b/d18/d61/l9e d5/de2/df5/d48/l11d 0 2024-04-05T03:03:02.682 INFO:tasks.workunit.client.0.smithi153.stdout:2/928: unlink d9/d68/fba 0 2024-04-05T03:03:02.683 INFO:tasks.workunit.client.0.smithi153.stdout:2/929: write d9/da/d94/dca/df7/f108 [4086982,46306] 0 2024-04-05T03:03:02.683 INFO:tasks.workunit.client.0.smithi153.stdout:2/930: dread - d9/d68/d98/f105 zero size 2024-04-05T03:03:02.688 INFO:tasks.workunit.client.0.smithi153.stdout:1/912: rmdir d5/d8/d20 39 2024-04-05T03:03:02.692 INFO:tasks.workunit.client.0.smithi153.stdout:2/931: dread d9/da/fd5 [0,4194304] 0 2024-04-05T03:03:02.694 INFO:tasks.workunit.client.0.smithi153.stdout:1/913: rename d5/d8/c30 to d5/d8/d20/d4b/d1e/c11e 0 2024-04-05T03:03:02.696 INFO:tasks.workunit.client.0.smithi153.stdout:2/932: mkdir d9/da/d19/d2e/d7e/d7a/d6d/de8/d137 0 2024-04-05T03:03:02.697 INFO:tasks.workunit.client.0.smithi153.stdout:1/914: mkdir d5/d8/d20/d4b/d11f 0 2024-04-05T03:03:02.698 INFO:tasks.workunit.client.0.smithi153.stdout:2/933: creat d9/da/d94/dca/df7/f138 x:0 0 0 2024-04-05T03:03:02.698 INFO:tasks.workunit.client.0.smithi153.stdout:1/915: readlink d5/d8/d20/d4b/lc1 0 2024-04-05T03:03:02.699 INFO:tasks.workunit.client.0.smithi153.stdout:1/916: dread - d5/de2/df5/d33/fec zero size 2024-04-05T03:03:02.700 INFO:tasks.workunit.client.0.smithi153.stdout:2/934: fsync d9/d68/fab 0 2024-04-05T03:03:02.700 INFO:tasks.workunit.client.0.smithi153.stdout:1/917: stat d5/de2/df5/d45/c98 0 2024-04-05T03:03:02.701 INFO:tasks.workunit.client.0.smithi153.stdout:1/918: write d5/d8/d20/d2f/d40/fd5 [2833507,7750] 0 2024-04-05T03:03:02.702 INFO:tasks.workunit.client.0.smithi153.stdout:2/935: dread - d9/ff3 zero size 2024-04-05T03:03:02.703 INFO:tasks.workunit.client.0.smithi153.stdout:2/936: fsync d9/da/d37/d8a/ff1 0 2024-04-05T03:03:02.707 INFO:tasks.workunit.client.0.smithi153.stdout:1/919: dwrite d5/d8/f116 [0,4194304] 0 2024-04-05T03:03:02.708 INFO:tasks.workunit.client.0.smithi153.stdout:2/937: mkdir d9/da/d19/d2e/d7e/d7a/d59/d12e/d139 0 2024-04-05T03:03:02.713 INFO:tasks.workunit.client.0.smithi153.stdout:2/938: mkdir d9/da/d19/d2e/d11b/d13a 0 2024-04-05T03:03:02.714 INFO:tasks.workunit.client.0.smithi153.stdout:1/920: dwrite d5/d8/d20/d4b/d1e/f4f [0,4194304] 0 2024-04-05T03:03:02.716 INFO:tasks.workunit.client.0.smithi153.stdout:1/921: write d5/d8/d20/d4b/de1/f104 [981449,40271] 0 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:4/904: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:5/763: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:6/919: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:3/858: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:8/732: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:9/701: sync 2024-04-05T03:03:02.728 INFO:tasks.workunit.client.0.smithi153.stdout:0/842: sync 2024-04-05T03:03:02.729 INFO:tasks.workunit.client.0.smithi153.stdout:9/702: readlink d1/d10/l2f 0 2024-04-05T03:03:02.732 INFO:tasks.workunit.client.0.smithi153.stdout:0/843: fsync d9/db/d42/d43/de4/ff6 0 2024-04-05T03:03:02.734 INFO:tasks.workunit.client.0.smithi153.stdout:8/733: dread - d7/d13/d19/d1b/db9/fd7 zero size 2024-04-05T03:03:02.734 INFO:tasks.workunit.client.0.smithi153.stdout:6/920: readlink d4/dc/df/d16/d24/d5e/de8/l100 0 2024-04-05T03:03:02.734 INFO:tasks.workunit.client.0.smithi153.stdout:2/939: dread d9/d68/d9b/fb4 [0,4194304] 0 2024-04-05T03:03:02.735 INFO:tasks.workunit.client.0.smithi153.stdout:3/859: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f81 [2121566,95283] 0 2024-04-05T03:03:02.736 INFO:tasks.workunit.client.0.smithi153.stdout:1/922: truncate d5/d8/d20/d4b/d1e/f4f 2849884 0 2024-04-05T03:03:02.736 INFO:tasks.workunit.client.0.smithi153.stdout:9/703: creat d1/d3a/d91/d96/dbd/ff5 x:0 0 0 2024-04-05T03:03:02.737 INFO:tasks.workunit.client.0.smithi153.stdout:2/940: fsync d9/da/d36/f81 0 2024-04-05T03:03:02.739 INFO:tasks.workunit.client.0.smithi153.stdout:8/734: chown d7/d13/d19/d1b/d4f/d72/lc4 13 1 2024-04-05T03:03:02.772 INFO:tasks.workunit.client.0.smithi153.stdout:9/704: creat d1/d3a/d91/d96/de6/ff6 x:0 0 0 2024-04-05T03:03:02.774 INFO:tasks.workunit.client.0.smithi153.stdout:8/735: read d7/d13/d19/d1b/d4f/d96/fb7 [128464,81062] 0 2024-04-05T03:03:02.779 INFO:tasks.workunit.client.0.smithi153.stdout:9/705: mkdir d1/d3a/df7 0 2024-04-05T03:03:02.782 INFO:tasks.workunit.client.0.smithi153.stdout:0/844: truncate d9/db/d42/d7e/db6/d1e/f54 2001240 0 2024-04-05T03:03:02.784 INFO:tasks.workunit.client.0.smithi153.stdout:1/923: dwrite d5/d8/d20/d2f/db8/ffe [0,4194304] 0 2024-04-05T03:03:02.785 INFO:tasks.workunit.client.0.smithi153.stdout:2/941: dwrite d9/da/d19/d2e/d7e/fdc [0,4194304] 0 2024-04-05T03:03:02.785 INFO:tasks.workunit.client.0.smithi153.stdout:8/736: chown d7/d13/d19/l29 53392 1 2024-04-05T03:03:02.785 INFO:tasks.workunit.client.0.smithi153.stdout:5/764: dwrite d0/d4e/f9e [0,4194304] 0 2024-04-05T03:03:02.788 INFO:tasks.workunit.client.0.smithi153.stdout:5/765: stat d0/d2c/fbe 0 2024-04-05T03:03:02.790 INFO:tasks.workunit.client.0.smithi153.stdout:9/706: creat d1/d1d/ff8 x:0 0 0 2024-04-05T03:03:02.792 INFO:tasks.workunit.client.0.smithi153.stdout:3/860: dread d1d/d27/f74 [0,4194304] 0 2024-04-05T03:03:02.792 INFO:tasks.workunit.client.0.smithi153.stdout:1/924: fdatasync d5/de2/df5/f11a 0 2024-04-05T03:03:02.795 INFO:tasks.workunit.client.0.smithi153.stdout:5/766: write d0/d25/d4d/d9c/dc4/dcb/ff4 [312370,74706] 0 2024-04-05T03:03:02.797 INFO:tasks.workunit.client.0.smithi153.stdout:8/737: creat d7/d13/d19/d1b/d4f/d68/fef x:0 0 0 2024-04-05T03:03:02.797 INFO:tasks.workunit.client.0.smithi153.stdout:4/905: dread da/db/d20/f29 [0,4194304] 0 2024-04-05T03:03:02.802 INFO:tasks.workunit.client.0.smithi153.stdout:2/942: dwrite d9/da/d37/d8a/da8/de5/d116/f133 [0,4194304] 0 2024-04-05T03:03:02.804 INFO:tasks.workunit.client.0.smithi153.stdout:9/707: mknod d1/d10/d16/d22/d2c/d68/cf9 0 2024-04-05T03:03:02.804 INFO:tasks.workunit.client.0.smithi153.stdout:1/925: dread - d5/de2/df5/d48/d96/fbd zero size 2024-04-05T03:03:02.805 INFO:tasks.workunit.client.0.smithi153.stdout:3/861: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 221 1 2024-04-05T03:03:02.806 INFO:tasks.workunit.client.0.smithi153.stdout:4/906: unlink da/db/f96 0 2024-04-05T03:03:02.808 INFO:tasks.workunit.client.0.smithi153.stdout:0/845: truncate d9/db/d42/d7e/db6/d1e/d45/dae/df4/fb1 1285872 0 2024-04-05T03:03:02.810 INFO:tasks.workunit.client.0.smithi153.stdout:1/926: creat d5/d8/d20/d3f/da7/f120 x:0 0 0 2024-04-05T03:03:02.813 INFO:tasks.workunit.client.0.smithi153.stdout:5/767: link d0/d2c/fbe d0/d25/d4d/d62/dbb/f10e 0 2024-04-05T03:03:02.813 INFO:tasks.workunit.client.0.smithi153.stdout:3/862: dread d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f59 [0,4194304] 0 2024-04-05T03:03:02.813 INFO:tasks.workunit.client.0.smithi153.stdout:9/708: mknod d1/d10/d36/d71/cfa 0 2024-04-05T03:03:02.815 INFO:tasks.workunit.client.0.smithi153.stdout:2/943: mknod d9/da/d36/def/d103/c13b 0 2024-04-05T03:03:02.816 INFO:tasks.workunit.client.0.smithi153.stdout:4/907: fdatasync da/db/d20/d45/db1/f110 0 2024-04-05T03:03:02.818 INFO:tasks.workunit.client.0.smithi153.stdout:0/846: mknod d9/d14/c102 0 2024-04-05T03:03:02.820 INFO:tasks.workunit.client.0.smithi153.stdout:8/738: creat d7/d13/d19/d1b/d4f/ff0 x:0 0 0 2024-04-05T03:03:02.823 INFO:tasks.workunit.client.0.smithi153.stdout:1/927: write d5/d8/d20/d4b/f11 [3012012,113074] 0 2024-04-05T03:03:02.824 INFO:tasks.workunit.client.0.smithi153.stdout:5/768: mkdir d0/d25/d4d/d9c/d10f 0 2024-04-05T03:03:02.826 INFO:tasks.workunit.client.0.smithi153.stdout:9/709: symlink d1/d10/d36/d57/lfb 0 2024-04-05T03:03:02.827 INFO:tasks.workunit.client.0.smithi153.stdout:3/863: truncate d1d/d27/d2b/f2d 4484926 0 2024-04-05T03:03:02.829 INFO:tasks.workunit.client.0.smithi153.stdout:2/944: write d9/d68/d9b/fb4 [5101014,1624] 0 2024-04-05T03:03:02.830 INFO:tasks.workunit.client.0.smithi153.stdout:0/847: unlink d9/db/d42/d7e/db6/fe8 0 2024-04-05T03:03:02.835 INFO:tasks.workunit.client.0.smithi153.stdout:1/928: rename d5/d8/d20/d2f/d40/ff1 to d5/d8/d20/d4b/de1/f121 0 2024-04-05T03:03:02.836 INFO:tasks.workunit.client.0.smithi153.stdout:9/710: mkdir d1/d3a/d91/d96/dfc 0 2024-04-05T03:03:02.837 INFO:tasks.workunit.client.0.smithi153.stdout:3/864: creat d1d/d32/d93/d3c/d3f/d90/da4/f130 x:0 0 0 2024-04-05T03:03:02.838 INFO:tasks.workunit.client.0.smithi153.stdout:4/908: mkdir da/de/da1/d13f 0 2024-04-05T03:03:02.841 INFO:tasks.workunit.client.0.smithi153.stdout:0/848: unlink d9/db/d42/d7e/db6/d1e/d45/d62/cb8 0 2024-04-05T03:03:02.842 INFO:tasks.workunit.client.0.smithi153.stdout:5/769: unlink d0/d9/ca2 0 2024-04-05T03:03:02.846 INFO:tasks.workunit.client.0.smithi153.stdout:2/945: rename d9/da/d94/dca/f123 to d9/da/d19/d2e/d7e/d7a/d59/f13c 0 2024-04-05T03:03:02.849 INFO:tasks.workunit.client.0.smithi153.stdout:4/909: unlink da/db/d20/d45/f82 0 2024-04-05T03:03:02.852 INFO:tasks.workunit.client.0.smithi153.stdout:9/711: mknod d1/d10/d16/cfd 0 2024-04-05T03:03:02.853 INFO:tasks.workunit.client.0.smithi153.stdout:2/946: creat d9/da/d19/d2e/d7e/d102/f13d x:0 0 0 2024-04-05T03:03:02.854 INFO:tasks.workunit.client.0.smithi153.stdout:2/947: chown d9/da/d94/dca/df7 12 1 2024-04-05T03:03:02.857 INFO:tasks.workunit.client.0.smithi153.stdout:8/739: unlink d7/d13/d19/d1b/d4f/d5f/d54/c57 0 2024-04-05T03:03:02.858 INFO:tasks.workunit.client.0.smithi153.stdout:4/910: truncate da/db/f109 891739 0 2024-04-05T03:03:02.859 INFO:tasks.workunit.client.0.smithi153.stdout:1/929: creat d5/dd/f122 x:0 0 0 2024-04-05T03:03:02.867 INFO:tasks.workunit.client.0.smithi153.stdout:8/740: symlink d7/d46/db1/lf1 0 2024-04-05T03:03:02.868 INFO:tasks.workunit.client.0.smithi153.stdout:8/741: write d7/d13/d19/d1b/d4f/d5f/fe3 [542301,62235] 0 2024-04-05T03:03:02.870 INFO:tasks.workunit.client.0.smithi153.stdout:9/712: creat d1/d9/d1b/de2/ffe x:0 0 0 2024-04-05T03:03:02.870 INFO:tasks.workunit.client.0.smithi153.stdout:1/930: dread d5/d8/d20/f88 [4194304,4194304] 0 2024-04-05T03:03:02.872 INFO:tasks.workunit.client.0.smithi153.stdout:0/849: getdents d9/db 0 2024-04-05T03:03:02.873 INFO:tasks.workunit.client.0.smithi153.stdout:8/742: fsync d7/d85/f78 0 2024-04-05T03:03:02.876 INFO:tasks.workunit.client.0.smithi153.stdout:8/743: fdatasync d7/d13/d19/d1b/d21/fbf 0 2024-04-05T03:03:02.876 INFO:tasks.workunit.client.0.smithi153.stdout:9/713: symlink d1/d9/lff 0 2024-04-05T03:03:02.877 INFO:tasks.workunit.client.0.smithi153.stdout:0/850: dread d9/db/d42/d43/f4a [0,4194304] 0 2024-04-05T03:03:02.880 INFO:tasks.workunit.client.0.smithi153.stdout:5/770: dwrite d0/d9/d22/fc2 [0,4194304] 0 2024-04-05T03:03:02.882 INFO:tasks.workunit.client.0.smithi153.stdout:9/714: fsync d1/d3a/d91/fc5 0 2024-04-05T03:03:02.883 INFO:tasks.workunit.client.0.smithi153.stdout:4/911: write da/de/d56/f61 [1075776,92833] 0 2024-04-05T03:03:02.887 INFO:tasks.workunit.client.0.smithi153.stdout:1/931: write d5/f9 [3827151,19569] 0 2024-04-05T03:03:02.889 INFO:tasks.workunit.client.0.smithi153.stdout:3/865: dwrite d1d/d27/d2b/f2d [0,4194304] 0 2024-04-05T03:03:02.889 INFO:tasks.workunit.client.0.smithi153.stdout:2/948: dwrite d9/d68/fa7 [0,4194304] 0 2024-04-05T03:03:02.890 INFO:tasks.workunit.client.0.smithi153.stdout:9/715: write d1/d10/d16/f17 [2727785,91174] 0 2024-04-05T03:03:02.891 INFO:tasks.workunit.client.0.smithi153.stdout:8/744: write d7/fe [3401811,61298] 0 2024-04-05T03:03:02.896 INFO:tasks.workunit.client.0.smithi153.stdout:0/851: mkdir d9/db/d42/d7e/db6/d1d/d61/d8a/d103 0 2024-04-05T03:03:02.900 INFO:tasks.workunit.client.0.smithi153.stdout:2/949: creat d9/d68/d98/da3/f13e x:0 0 0 2024-04-05T03:03:02.901 INFO:tasks.workunit.client.0.smithi153.stdout:4/912: dwrite da/f97 [0,4194304] 0 2024-04-05T03:03:02.903 INFO:tasks.workunit.client.0.smithi153.stdout:3/866: rename d1d/f21 to d1d/d32/df9/f131 0 2024-04-05T03:03:02.903 INFO:tasks.workunit.client.0.smithi153.stdout:8/745: dread - d7/d46/faa zero size 2024-04-05T03:03:02.906 INFO:tasks.workunit.client.0.smithi153.stdout:4/913: fdatasync da/db/d20/f29 0 2024-04-05T03:03:02.910 INFO:tasks.workunit.client.0.smithi153.stdout:1/932: getdents d5/d8/d20/d4b/d18 0 2024-04-05T03:03:02.912 INFO:tasks.workunit.client.0.smithi153.stdout:8/746: readlink d7/d13/d19/d1b/d4f/d72/d73/l82 0 2024-04-05T03:03:02.915 INFO:tasks.workunit.client.0.smithi153.stdout:4/914: mknod da/de/d56/d78/d8f/d68/c140 0 2024-04-05T03:03:02.920 INFO:tasks.workunit.client.0.smithi153.stdout:9/716: dwrite d1/d10/d16/f23 [0,4194304] 0 2024-04-05T03:03:02.923 INFO:tasks.workunit.client.0.smithi153.stdout:8/747: rename d7/d13/d19/d1b/d33/f3a to d7/d9a/ff2 0 2024-04-05T03:03:02.924 INFO:tasks.workunit.client.0.smithi153.stdout:4/915: chown da/db/d20/d35/d41/d67 184862 1 2024-04-05T03:03:02.926 INFO:tasks.workunit.client.0.smithi153.stdout:1/933: dwrite d5/de2/df5/d33/f3c [0,4194304] 0 2024-04-05T03:03:02.928 INFO:tasks.workunit.client.0.smithi153.stdout:8/748: mknod d7/d13/d19/d1b/d4f/cf3 0 2024-04-05T03:03:02.929 INFO:tasks.workunit.client.0.smithi153.stdout:9/717: rmdir d1/d3a/df7 0 2024-04-05T03:03:02.931 INFO:tasks.workunit.client.0.smithi153.stdout:3/867: dread d1d/d27/d2b/f71 [0,4194304] 0 2024-04-05T03:03:02.933 INFO:tasks.workunit.client.0.smithi153.stdout:8/749: dread - d7/d13/d19/d95/fb3 zero size 2024-04-05T03:03:02.943 INFO:tasks.workunit.client.0.smithi153.stdout:1/934: write d5/de2/df5/f7a [932897,26900] 0 2024-04-05T03:03:02.945 INFO:tasks.workunit.client.0.smithi153.stdout:4/916: dwrite da/db/d20/d35/d38/d76/d90/fca [0,4194304] 0 2024-04-05T03:03:02.947 INFO:tasks.workunit.client.0.smithi153.stdout:0/852: dread d9/db/d42/d7e/db6/f46 [0,4194304] 0 2024-04-05T03:03:02.948 INFO:tasks.workunit.client.0.smithi153.stdout:9/718: write d1/d10/d36/d71/ff1 [3284716,122491] 0 2024-04-05T03:03:02.949 INFO:tasks.workunit.client.0.smithi153.stdout:3/868: write d1d/d32/d93/d3c/d3f/d90/d92/d99/fa6 [2508,100891] 0 2024-04-05T03:03:02.949 INFO:tasks.workunit.client.0.smithi153.stdout:1/935: chown d5/d8/d20/d2f/f102 703 1 2024-04-05T03:03:02.951 INFO:tasks.workunit.client.0.smithi153.stdout:0/853: dread d9/ff1 [0,4194304] 0 2024-04-05T03:03:02.954 INFO:tasks.workunit.client.0.smithi153.stdout:4/917: creat da/db/d20/d35/d38/df7/f141 x:0 0 0 2024-04-05T03:03:02.959 INFO:tasks.workunit.client.0.smithi153.stdout:9/719: symlink d1/d10/d36/l100 0 2024-04-05T03:03:02.959 INFO:tasks.workunit.client.0.smithi153.stdout:9/720: chown d1/d10 1890089 1 2024-04-05T03:03:02.960 INFO:tasks.workunit.client.0.smithi153.stdout:4/918: symlink da/db/d20/d3c/l142 0 2024-04-05T03:03:02.962 INFO:tasks.workunit.client.0.smithi153.stdout:3/869: link d1d/ddf/cfe d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d112/c132 0 2024-04-05T03:03:02.964 INFO:tasks.workunit.client.0.smithi153.stdout:9/721: getdents d1/d10/d16 0 2024-04-05T03:03:02.969 INFO:tasks.workunit.client.0.smithi153.stdout:2/950: dread d9/da/d19/d2e/f43 [0,4194304] 0 2024-04-05T03:03:02.969 INFO:tasks.workunit.client.0.smithi153.stdout:9/722: symlink d1/d9/d1b/de9/l101 0 2024-04-05T03:03:02.970 INFO:tasks.workunit.client.0.smithi153.stdout:3/870: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/fcb [132197,118521] 0 2024-04-05T03:03:02.974 INFO:tasks.workunit.client.0.smithi153.stdout:2/951: write d9/da/d37/d8a/da8/de5/d116/f117 [769101,71528] 0 2024-04-05T03:03:02.974 INFO:tasks.workunit.client.0.smithi153.stdout:9/723: creat d1/d10/d16/d22/d2c/d68/da9/f102 x:0 0 0 2024-04-05T03:03:02.976 INFO:tasks.workunit.client.0.smithi153.stdout:3/871: mknod d1d/d27/df4/c133 0 2024-04-05T03:03:02.979 INFO:tasks.workunit.client.0.smithi153.stdout:0/854: dread d9/db/d42/f60 [0,4194304] 0 2024-04-05T03:03:02.980 INFO:tasks.workunit.client.0.smithi153.stdout:0/855: chown d9/d14/c4f 1 1 2024-04-05T03:03:02.981 INFO:tasks.workunit.client.0.smithi153.stdout:2/952: unlink d9/da/d37/d8a/da8/led 0 2024-04-05T03:03:02.982 INFO:tasks.workunit.client.0.smithi153.stdout:9/724: symlink d1/d10/d36/d57/d97/ded/l103 0 2024-04-05T03:03:02.983 INFO:tasks.workunit.client.0.smithi153.stdout:3/872: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/d11f/d10c/c134 0 2024-04-05T03:03:02.984 INFO:tasks.workunit.client.0.smithi153.stdout:0/856: mknod d9/db/d42/d7e/db6/d1e/d45/d62/c104 0 2024-04-05T03:03:02.985 INFO:tasks.workunit.client.0.smithi153.stdout:9/725: chown d1/d10/d16/d22/d2c/cd6 55947 1 2024-04-05T03:03:02.986 INFO:tasks.workunit.client.0.smithi153.stdout:2/953: truncate d9/da/d36/f63 1892368 0 2024-04-05T03:03:02.987 INFO:tasks.workunit.client.0.smithi153.stdout:8/750: dread d7/d13/d19/d1b/d4f/f94 [0,4194304] 0 2024-04-05T03:03:02.987 INFO:tasks.workunit.client.0.smithi153.stdout:3/873: creat d1d/d27/f135 x:0 0 0 2024-04-05T03:03:02.988 INFO:tasks.workunit.client.0.smithi153.stdout:0/857: symlink d9/db/ddf/l105 0 2024-04-05T03:03:02.988 INFO:tasks.workunit.client.0.smithi153.stdout:9/726: fdatasync d1/d10/d16/f7f 0 2024-04-05T03:03:02.991 INFO:tasks.workunit.client.0.smithi153.stdout:0/858: chown d9/db/d42/d7e/db6/d1d/d61/d8a/f7b 56 1 2024-04-05T03:03:02.994 INFO:tasks.workunit.client.0.smithi153.stdout:9/727: rename d1/d6c/f86 to d1/d6c/f104 0 2024-04-05T03:03:02.996 INFO:tasks.workunit.client.0.smithi153.stdout:8/751: symlink d7/d13/d19/d1b/d4f/d5f/d54/lf4 0 2024-04-05T03:03:03.006 INFO:tasks.workunit.client.0.smithi153.stdout:9/728: dwrite d1/f1f [0,4194304] 0 2024-04-05T03:03:03.010 INFO:tasks.workunit.client.0.smithi153.stdout:3/874: creat d1d/d32/d93/d3c/d3f/d90/f136 x:0 0 0 2024-04-05T03:03:03.012 INFO:tasks.workunit.client.0.smithi153.stdout:3/875: fdatasync d1d/d27/d2b/f7e 0 2024-04-05T03:03:03.016 INFO:tasks.workunit.client.0.smithi153.stdout:3/876: truncate d1d/d32/d93/d3c/d3f/d9c/f107 191632 0 2024-04-05T03:03:03.018 INFO:tasks.workunit.client.0.smithi153.stdout:0/859: write d9/d14/d67/d6f/fab [506894,88563] 0 2024-04-05T03:03:03.025 INFO:tasks.workunit.client.0.smithi153.stdout:8/752: dwrite d7/d13/d19/d1b/d4f/d72/f99 [0,4194304] 0 2024-04-05T03:03:03.031 INFO:tasks.workunit.client.0.smithi153.stdout:8/753: mkdir d7/d13/d19/d1b/d4f/d5f/df5 0 2024-04-05T03:03:03.033 INFO:tasks.workunit.client.0.smithi153.stdout:0/860: truncate d9/db/d42/f71 3696246 0 2024-04-05T03:03:03.038 INFO:tasks.workunit.client.0.smithi153.stdout:0/861: fdatasync d9/db/d42/d7e/db6/d1e/d45/d62/fa0 0 2024-04-05T03:03:03.038 INFO:tasks.workunit.client.0.smithi153.stdout:8/754: dread d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:03:03.041 INFO:tasks.workunit.client.0.smithi153.stdout:3/877: dread d1d/d32/d93/d3c/f7b [0,4194304] 0 2024-04-05T03:03:03.041 INFO:tasks.workunit.client.0.smithi153.stdout:8/755: rmdir d7/d13/d19/d1b/db9 39 2024-04-05T03:03:03.047 INFO:tasks.workunit.client.0.smithi153.stdout:3/878: rename d1d/d27/f96 to d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/d11f/d10c/f137 0 2024-04-05T03:03:03.048 INFO:tasks.workunit.client.0.smithi153.stdout:9/729: dread d1/d9/d1b/f6a [0,4194304] 0 2024-04-05T03:03:03.049 INFO:tasks.workunit.client.0.smithi153.stdout:9/730: fdatasync d1/d10/d36/f56 0 2024-04-05T03:03:03.054 INFO:tasks.workunit.client.0.smithi153.stdout:8/756: link d7/d2a/f2c d7/d13/d19/d1b/db9/ff6 0 2024-04-05T03:03:03.056 INFO:tasks.workunit.client.0.smithi153.stdout:0/862: write d9/db/d42/d7e/db6/f59 [3235092,108945] 0 2024-04-05T03:03:03.058 INFO:tasks.workunit.client.0.smithi153.stdout:3/879: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d10a/d11f to d1d/d12a/d138 0 2024-04-05T03:03:03.063 INFO:tasks.workunit.client.0.smithi153.stdout:0/863: fsync d9/db/d42/d7e/db6/d1e/d45/d62/dde/fce 0 2024-04-05T03:03:03.066 INFO:tasks.workunit.client.0.smithi153.stdout:0/864: mknod d9/db/ddf/c106 0 2024-04-05T03:03:03.067 INFO:tasks.workunit.client.0.smithi153.stdout:9/731: dread d1/d10/f39 [0,4194304] 0 2024-04-05T03:03:03.071 INFO:tasks.workunit.client.0.smithi153.stdout:9/732: truncate d1/d9/d1b/f3f 5139234 0 2024-04-05T03:03:03.071 INFO:tasks.workunit.client.0.smithi153.stdout:3/880: write d1d/d27/d2b/f71 [989254,238] 0 2024-04-05T03:03:03.072 INFO:tasks.workunit.client.0.smithi153.stdout:0/865: mknod d9/db/d42/d7e/c107 0 2024-04-05T03:03:03.074 INFO:tasks.workunit.client.0.smithi153.stdout:9/733: dread - d1/d10/d16/d22/d2c/d68/d81/fef zero size 2024-04-05T03:03:03.075 INFO:tasks.workunit.client.0.smithi153.stdout:3/881: fsync d1d/d27/f8f 0 2024-04-05T03:03:03.078 INFO:tasks.workunit.client.0.smithi153.stdout:0/866: mknod d9/db/d42/d7e/db6/d1e/d45/d62/dde/c108 0 2024-04-05T03:03:03.078 INFO:tasks.workunit.client.0.smithi153.stdout:9/734: creat d1/d1d/f105 x:0 0 0 2024-04-05T03:03:03.079 INFO:tasks.workunit.client.0.smithi153.stdout:9/735: chown d1/d3a/fb5 7656 1 2024-04-05T03:03:03.081 INFO:tasks.workunit.client.0.smithi153.stdout:3/882: mkdir d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d139 0 2024-04-05T03:03:03.082 INFO:tasks.workunit.client.0.smithi153.stdout:9/736: chown d1/d10/d36/d71/fa0 2 1 2024-04-05T03:03:03.087 INFO:tasks.workunit.client.0.smithi153.stdout:0/867: rename d9/db/d42/f60 to d9/db/d42/f109 0 2024-04-05T03:03:03.088 INFO:tasks.workunit.client.0.smithi153.stdout:0/868: stat d9/d14/d67 0 2024-04-05T03:03:03.090 INFO:tasks.workunit.client.0.smithi153.stdout:9/737: rename d1/d10/d16/d22/d2c/f46 to d1/d3a/d91/d96/dbd/dbe/f106 0 2024-04-05T03:03:03.092 INFO:tasks.workunit.client.0.smithi153.stdout:9/738: read d1/d9/d1b/f73 [16402,100990] 0 2024-04-05T03:03:03.092 INFO:tasks.workunit.client.0.smithi153.stdout:0/869: creat d9/db/d42/d7e/db6/daf/dd0/f10a x:0 0 0 2024-04-05T03:03:03.097 INFO:tasks.workunit.client.0.smithi153.stdout:0/870: dread d9/db/d42/d7e/db6/d1e/d45/d62/dde/fa4 [0,4194304] 0 2024-04-05T03:03:03.100 INFO:tasks.workunit.client.0.smithi153.stdout:0/871: mkdir d9/d14/d10b 0 2024-04-05T03:03:03.106 INFO:tasks.workunit.client.0.smithi153.stdout:0/872: dwrite d9/db/d42/d7e/db6/d1e/d45/fba [0,4194304] 0 2024-04-05T03:03:03.108 INFO:tasks.workunit.client.0.smithi153.stdout:0/873: dread - d9/db/d42/d7e/db6/daf/dd0/f10a zero size 2024-04-05T03:03:03.110 INFO:tasks.workunit.client.0.smithi153.stdout:9/739: dwrite d1/d3a/d91/d96/fa3 [0,4194304] 0 2024-04-05T03:03:03.114 INFO:tasks.workunit.client.0.smithi153.stdout:9/740: creat d1/d9/d1b/f107 x:0 0 0 2024-04-05T03:03:03.117 INFO:tasks.workunit.client.0.smithi153.stdout:0/874: truncate d9/db/d42/d7e/db6/d1e/d45/d62/fff 963821 0 2024-04-05T03:03:03.120 INFO:tasks.workunit.client.0.smithi153.stdout:9/741: rename d1/d10/d16/d22/d2c/d68/d81/l44 to d1/d10/d36/d57/l108 0 2024-04-05T03:03:03.121 INFO:tasks.workunit.client.0.smithi153.stdout:9/742: chown d1/d3a/f83 46 1 2024-04-05T03:03:03.121 INFO:tasks.workunit.client.0.smithi153.stdout:0/875: creat d9/db/d42/d7e/db6/d1e/d45/d62/dde/d9e/f10c x:0 0 0 2024-04-05T03:03:03.125 INFO:tasks.workunit.client.0.smithi153.stdout:9/743: dread d1/d3a/d91/d96/fa3 [0,4194304] 0 2024-04-05T03:03:03.134 INFO:tasks.workunit.client.0.smithi153.stdout:9/744: write d1/d10/d36/d57/f6f [3808392,63714] 0 2024-04-05T03:03:03.137 INFO:tasks.workunit.client.0.smithi153.stdout:0/876: dread d9/db/d42/d7e/db6/d1e/f32 [0,4194304] 0 2024-04-05T03:03:03.138 INFO:tasks.workunit.client.0.smithi153.stdout:0/877: readlink d9/db/l9c 0 2024-04-05T03:03:03.140 INFO:tasks.workunit.client.0.smithi153.stdout:9/745: rename d1/d10/d36/f48 to d1/d10/d16/d22/f109 0 2024-04-05T03:03:03.141 INFO:tasks.workunit.client.0.smithi153.stdout:0/878: creat d9/db/d42/d7e/db6/d1e/d45/d62/de7/f10d x:0 0 0 2024-04-05T03:03:03.141 INFO:tasks.workunit.client.0.smithi153.stdout:0/879: chown d9/l3c 174699009 1 2024-04-05T03:03:03.147 INFO:tasks.workunit.client.0.smithi153.stdout:9/746: truncate d1/d10/f64 2527897 0 2024-04-05T03:03:03.147 INFO:tasks.workunit.client.0.smithi153.stdout:9/747: stat d1/d3a/f93 0 2024-04-05T03:03:03.151 INFO:tasks.workunit.client.0.smithi153.stdout:9/748: creat d1/d10/d36/f10a x:0 0 0 2024-04-05T03:03:03.152 INFO:tasks.workunit.client.0.smithi153.stdout:0/880: dwrite d9/db/d42/d7e/db6/d1d/d61/f1b [0,4194304] 0 2024-04-05T03:03:03.154 INFO:tasks.workunit.client.0.smithi153.stdout:9/749: rename d1/d10/d36/d57/d5c/lc6 to d1/d1d/l10b 0 2024-04-05T03:03:03.156 INFO:tasks.workunit.client.0.smithi153.stdout:0/881: fdatasync d9/db/d42/d7e/d8b/fe5 0 2024-04-05T03:03:03.158 INFO:tasks.workunit.client.0.smithi153.stdout:0/882: write d9/db/d42/d43/fe2 [364370,98929] 0 2024-04-05T03:03:03.161 INFO:tasks.workunit.client.0.smithi153.stdout:9/750: dwrite d1/d10/d36/d57/f78 [0,4194304] 0 2024-04-05T03:03:03.162 INFO:tasks.workunit.client.0.smithi153.stdout:0/883: getdents d9/db 0 2024-04-05T03:03:03.163 INFO:tasks.workunit.client.0.smithi153.stdout:9/751: stat d1/d3a/d91/d96/fa3 0 2024-04-05T03:03:03.166 INFO:tasks.workunit.client.0.smithi153.stdout:0/884: write d9/d14/d67/d6f/ffd [69922,45696] 0 2024-04-05T03:03:03.171 INFO:tasks.workunit.client.0.smithi153.stdout:9/752: write d1/d6c/f104 [1358361,94837] 0 2024-04-05T03:03:03.171 INFO:tasks.workunit.client.0.smithi153.stdout:9/753: stat d1/d9/d1b/ce8 0 2024-04-05T03:03:03.174 INFO:tasks.workunit.client.0.smithi153.stdout:9/754: creat d1/d10/d36/d57/d97/ded/f10c x:0 0 0 2024-04-05T03:03:03.176 INFO:tasks.workunit.client.0.smithi153.stdout:9/755: unlink d1/d10/d16/d22/d2c/cd6 0 2024-04-05T03:03:03.177 INFO:tasks.workunit.client.0.smithi153.stdout:9/756: truncate d1/d10/f39 2589129 0 2024-04-05T03:03:03.179 INFO:tasks.workunit.client.0.smithi153.stdout:9/757: symlink d1/d10/d36/d57/d5c/l10d 0 2024-04-05T03:03:03.187 INFO:tasks.workunit.client.0.smithi153.stdout:9/758: dwrite d1/d10/d36/d71/fa0 [0,4194304] 0 2024-04-05T03:03:03.190 INFO:tasks.workunit.client.0.smithi153.stdout:9/759: creat d1/d10/d36/d71/f10e x:0 0 0 2024-04-05T03:03:03.196 INFO:tasks.workunit.client.0.smithi153.stdout:9/760: dwrite d1/d10/d36/d57/d97/fc8 [0,4194304] 0 2024-04-05T03:03:03.202 INFO:tasks.workunit.client.0.smithi153.stdout:9/761: creat d1/f10f x:0 0 0 2024-04-05T03:03:03.205 INFO:tasks.workunit.client.0.smithi153.stdout:9/762: creat d1/f110 x:0 0 0 2024-04-05T03:03:03.206 INFO:tasks.workunit.client.0.smithi153.stdout:9/763: fdatasync d1/d9/d1b/f73 0 2024-04-05T03:03:03.211 INFO:tasks.workunit.client.0.smithi153.stdout:9/764: dwrite d1/d10/d36/f62 [0,4194304] 0 2024-04-05T03:03:03.214 INFO:tasks.workunit.client.0.smithi153.stdout:9/765: rename d1/d10/f5b to d1/d10/d16/f111 0 2024-04-05T03:03:03.223 INFO:tasks.workunit.client.0.smithi153.stdout:9/766: chown d1/d10/d16/l52 0 1 2024-04-05T03:03:03.254 INFO:tasks.workunit.client.0.smithi153.stdout:6/921: sync 2024-04-05T03:03:03.270 INFO:tasks.workunit.client.0.smithi153.stdout:6/922: dread d4/f7 [0,4194304] 0 2024-04-05T03:03:03.271 INFO:tasks.workunit.client.0.smithi153.stdout:6/923: fsync d4/dc/df/d16/d44/fa1 0 2024-04-05T03:03:03.274 INFO:tasks.workunit.client.0.smithi153.stdout:6/924: getdents d4/dc/df/d16/d5f/d73 0 2024-04-05T03:03:03.277 INFO:tasks.workunit.client.0.smithi153.stdout:6/925: creat d4/dc/df/d16/d24/d8d/f13c x:0 0 0 2024-04-05T03:03:03.278 INFO:tasks.workunit.client.0.smithi153.stdout:6/926: stat d4/dc/df/d16/d24/d4e/fa9 0 2024-04-05T03:03:03.280 INFO:tasks.workunit.client.0.smithi153.stdout:6/927: mkdir d4/dc/df/d16/d129/d13d 0 2024-04-05T03:03:03.282 INFO:tasks.workunit.client.0.smithi153.stdout:6/928: mknod d4/dc/df/c13e 0 2024-04-05T03:03:03.284 INFO:tasks.workunit.client.0.smithi153.stdout:6/929: chown d4/dc/lfb 3127 1 2024-04-05T03:03:03.285 INFO:tasks.workunit.client.0.smithi153.stdout:6/930: dread - d4/dc/df/d16/d24/f130 zero size 2024-04-05T03:03:03.286 INFO:tasks.workunit.client.0.smithi153.stdout:6/931: read d4/f7 [3188343,12960] 0 2024-04-05T03:03:03.288 INFO:tasks.workunit.client.0.smithi153.stdout:6/932: mkdir d4/dc/d101/d13f 0 2024-04-05T03:03:03.288 INFO:tasks.workunit.client.0.smithi153.stdout:6/933: dread - d4/dc/df/d16/d44/f121 zero size 2024-04-05T03:03:03.294 INFO:tasks.workunit.client.0.smithi153.stdout:6/934: dwrite d4/d38/f8b [0,4194304] 0 2024-04-05T03:03:03.298 INFO:tasks.workunit.client.0.smithi153.stdout:6/935: rename d4/dc/df/d16/d17/d54/f11a to d4/dc/df/d16/d36/d10e/f140 0 2024-04-05T03:03:03.302 INFO:tasks.workunit.client.0.smithi153.stdout:6/936: symlink d4/dc/df/d16/d36/d10e/d11b/l141 0 2024-04-05T03:03:03.317 INFO:tasks.workunit.client.0.smithi153.stdout:6/937: dwrite d4/f89 [0,4194304] 0 2024-04-05T03:03:03.327 INFO:tasks.workunit.client.0.smithi153.stdout:6/938: dwrite d4/d38/d4c/fb0 [0,4194304] 0 2024-04-05T03:03:03.330 INFO:tasks.workunit.client.0.smithi153.stdout:6/939: mknod d4/c142 0 2024-04-05T03:03:03.337 INFO:tasks.workunit.client.0.smithi153.stdout:6/940: rename d4/dc/l19 to d4/dc/df/d16/d33/l143 0 2024-04-05T03:03:03.339 INFO:tasks.workunit.client.0.smithi153.stdout:6/941: mkdir d4/dc/df/d144 0 2024-04-05T03:03:03.342 INFO:tasks.workunit.client.0.smithi153.stdout:6/942: creat d4/dc/df/d16/d24/d4e/f145 x:0 0 0 2024-04-05T03:03:03.344 INFO:tasks.workunit.client.0.smithi153.stdout:6/943: mkdir d4/dc/d101/d13f/d146 0 2024-04-05T03:03:03.347 INFO:tasks.workunit.client.0.smithi153.stdout:6/944: creat d4/f147 x:0 0 0 2024-04-05T03:03:03.365 INFO:tasks.workunit.client.0.smithi153.stdout:6/945: dread d4/dc/df/d16/d24/f29 [0,4194304] 0 2024-04-05T03:03:03.377 INFO:tasks.workunit.client.0.smithi153.stdout:6/946: dread d4/dc/df/d16/d36/d4d/d64/f6c [0,4194304] 0 2024-04-05T03:03:03.378 INFO:tasks.workunit.client.0.smithi153.stdout:6/947: symlink d4/dc/df/d16/d129/d13d/l148 0 2024-04-05T03:03:03.380 INFO:tasks.workunit.client.0.smithi153.stdout:6/948: symlink d4/d11c/l149 0 2024-04-05T03:03:03.383 INFO:tasks.workunit.client.0.smithi153.stdout:6/949: symlink d4/dc/d101/d13f/d146/l14a 0 2024-04-05T03:03:03.384 INFO:tasks.workunit.client.0.smithi153.stdout:6/950: chown d4/dc/df/d16/d5f/d73/fb1 118 1 2024-04-05T03:03:03.384 INFO:tasks.workunit.client.0.smithi153.stdout:6/951: stat d4/dc/df/c11 0 2024-04-05T03:03:03.385 INFO:tasks.workunit.client.0.smithi153.stdout:6/952: write d4/dc/df/d16/d44/fa1 [261819,84633] 0 2024-04-05T03:03:03.386 INFO:tasks.workunit.client.0.smithi153.stdout:6/953: chown d4/dc/df/c11 623180 1 2024-04-05T03:03:03.395 INFO:tasks.workunit.client.0.smithi153.stdout:6/954: dread d4/dc/d7a/da8/fe3 [0,4194304] 0 2024-04-05T03:03:03.400 INFO:tasks.workunit.client.0.smithi153.stdout:6/955: rename d4/dc/l50 to d4/dc/df/d16/d24/d4e/l14b 0 2024-04-05T03:03:03.400 INFO:tasks.workunit.client.0.smithi153.stdout:6/956: dread - d4/dc/df/d16/d44/f121 zero size 2024-04-05T03:03:03.403 INFO:tasks.workunit.client.0.smithi153.stdout:6/957: rename d4/dc/df/d16/d36/d4d/f91 to d4/dc/df/d117/f14c 0 2024-04-05T03:03:03.405 INFO:tasks.workunit.client.0.smithi153.stdout:6/958: creat d4/d38/d10f/f14d x:0 0 0 2024-04-05T03:03:03.410 INFO:tasks.workunit.client.0.smithi153.stdout:6/959: rename d4/dc/df/d16/d36/df9 to d4/dc/df/d16/d36/d14e 0 2024-04-05T03:03:03.411 INFO:tasks.workunit.client.0.smithi153.stdout:6/960: read d4/f51 [1233413,40368] 0 2024-04-05T03:03:03.413 INFO:tasks.workunit.client.0.smithi153.stdout:6/961: truncate d4/dc/df/d16/d129/f126 319088 0 2024-04-05T03:03:03.414 INFO:tasks.workunit.client.0.smithi153.stdout:6/962: write d4/dc/df/d16/d44/ded/fff [582690,85872] 0 2024-04-05T03:03:03.418 INFO:tasks.workunit.client.0.smithi153.stdout:6/963: rename d4/f6 to d4/dc/df/d16/d24/df0/f14f 0 2024-04-05T03:03:03.427 INFO:tasks.workunit.client.0.smithi153.stdout:6/964: truncate d4/dc/df/d16/d36/d4d/d64/f6c 3451012 0 2024-04-05T03:03:03.430 INFO:tasks.workunit.client.0.smithi153.stdout:6/965: rename d4/dc/d101/d13f/d146/l14a to d4/dc/df/d16/d113/l150 0 2024-04-05T03:03:03.431 INFO:tasks.workunit.client.0.smithi153.stdout:6/966: fdatasync d4/dc/df/d16/d24/d4e/f8a 0 2024-04-05T03:03:03.488 INFO:tasks.workunit.client.0.smithi153.stdout:6/967: dwrite d4/f15 [0,4194304] 0 2024-04-05T03:03:03.497 INFO:tasks.workunit.client.0.smithi153.stdout:6/968: dwrite d4/dc/df/d16/d36/d4d/f87 [0,4194304] 0 2024-04-05T03:03:03.501 INFO:tasks.workunit.client.0.smithi153.stdout:6/969: creat d4/dc/f151 x:0 0 0 2024-04-05T03:03:03.506 INFO:tasks.workunit.client.0.smithi153.stdout:6/970: truncate d4/dc/df/d16/d24/fb9 378978 0 2024-04-05T03:03:03.507 INFO:tasks.workunit.client.0.smithi153.stdout:6/971: write d4/dc/df/d16/d24/f32 [4644980,89052] 0 2024-04-05T03:03:03.526 INFO:tasks.workunit.client.0.smithi153.stdout:6/972: dwrite d4/f10d [0,4194304] 0 2024-04-05T03:03:03.534 INFO:tasks.workunit.client.0.smithi153.stdout:6/973: dwrite d4/d38/f134 [0,4194304] 0 2024-04-05T03:03:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:03 smithi203 ceph-mon[39911]: pgmap v16: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 122 MiB/s rd, 280 MiB/s wr, 608 op/s 2024-04-05T03:03:03.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:03 smithi153 ceph-mon[29381]: pgmap v16: 65 pgs: 65 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 122 MiB/s rd, 280 MiB/s wr, 608 op/s 2024-04-05T03:03:03.583 INFO:tasks.workunit.client.0.smithi153.stdout:5/771: sync 2024-04-05T03:03:03.584 INFO:tasks.workunit.client.0.smithi153.stdout:5/772: creat d0/d2c/f110 x:0 0 0 2024-04-05T03:03:03.677 INFO:tasks.workunit.client.0.smithi153.stdout:5/773: dwrite d0/d2c/f3d [0,4194304] 0 2024-04-05T03:03:03.679 INFO:tasks.workunit.client.0.smithi153.stdout:5/774: write d0/d25/d4d/d62/d37/f4f [1775405,21197] 0 2024-04-05T03:03:03.681 INFO:tasks.workunit.client.0.smithi153.stdout:5/775: creat d0/d2c/d3f/d102/f111 x:0 0 0 2024-04-05T03:03:03.740 INFO:tasks.workunit.client.0.smithi153.stdout:5/776: write d0/d25/d4d/d62/d5f/fbf [459630,88495] 0 2024-04-05T03:03:03.741 INFO:tasks.workunit.client.0.smithi153.stdout:5/777: stat d0/d9/c23 0 2024-04-05T03:03:03.742 INFO:tasks.workunit.client.0.smithi153.stdout:5/778: chown d0/d25/d4d/d9c/fd8 2031811 1 2024-04-05T03:03:03.744 INFO:tasks.workunit.client.0.smithi153.stdout:5/779: fsync d0/d9/d22/f7a 0 2024-04-05T03:03:03.745 INFO:tasks.workunit.client.0.smithi153.stdout:5/780: dread - d0/d25/d4d/d9c/faa zero size 2024-04-05T03:03:03.746 INFO:tasks.workunit.client.0.smithi153.stdout:5/781: dread - d0/d2c/d3f/d102/f111 zero size 2024-04-05T03:03:03.750 INFO:tasks.workunit.client.0.smithi153.stdout:5/782: write d0/d2c/fbe [1939530,50788] 0 2024-04-05T03:03:03.754 INFO:tasks.workunit.client.0.smithi153.stdout:5/783: link d0/d2c/d3f/da8/l10c d0/d25/d4d/d62/d5f/d86/da4/l112 0 2024-04-05T03:03:03.755 INFO:tasks.workunit.client.0.smithi153.stdout:5/784: symlink d0/d25/d4d/d9c/dc4/dcb/l113 0 2024-04-05T03:03:03.758 INFO:tasks.workunit.client.0.smithi153.stdout:5/785: truncate d0/d25/d4d/d62/d37/d38/f98 265379 0 2024-04-05T03:03:03.771 INFO:tasks.workunit.client.0.smithi153.stdout:5/786: dread d0/d9/f21 [0,4194304] 0 2024-04-05T03:03:03.772 INFO:tasks.workunit.client.0.smithi153.stdout:5/787: fdatasync d0/d25/f31 0 2024-04-05T03:03:03.774 INFO:tasks.workunit.client.0.smithi153.stdout:5/788: unlink d0/d25/c89 0 2024-04-05T03:03:03.776 INFO:tasks.workunit.client.0.smithi153.stdout:5/789: read d0/d25/d4d/d62/f10 [3830852,62432] 0 2024-04-05T03:03:03.777 INFO:tasks.workunit.client.0.smithi153.stdout:5/790: write d0/d25/d4d/d9c/dc7/ffa [141040,80990] 0 2024-04-05T03:03:03.780 INFO:tasks.workunit.client.0.smithi153.stdout:5/791: dread d0/d25/d4d/d62/d5f/d86/fff [0,4194304] 0 2024-04-05T03:03:03.781 INFO:tasks.workunit.client.0.smithi153.stdout:5/792: stat d0/d4e/db6 0 2024-04-05T03:03:03.781 INFO:tasks.workunit.client.0.smithi153.stdout:5/793: dread - d0/d25/d4d/d62/d5f/db2/f10b zero size 2024-04-05T03:03:03.782 INFO:tasks.workunit.client.0.smithi153.stdout:5/794: write d0/d4e/db6/f101 [822829,48900] 0 2024-04-05T03:03:03.784 INFO:tasks.workunit.client.0.smithi153.stdout:5/795: mknod d0/d25/d4d/d62/dbb/c114 0 2024-04-05T03:03:03.784 INFO:tasks.workunit.client.0.smithi153.stdout:5/796: fsync d0/f27 0 2024-04-05T03:03:03.787 INFO:tasks.workunit.client.0.smithi153.stdout:5/797: rename d0/d2c/d3f/l105 to d0/d25/d4d/d62/d5f/d86/da4/l115 0 2024-04-05T03:03:03.788 INFO:tasks.workunit.client.0.smithi153.stdout:5/798: write d0/d25/d4d/d62/f56 [2736173,66215] 0 2024-04-05T03:03:03.789 INFO:tasks.workunit.client.0.smithi153.stdout:5/799: stat d0/d25/d4d/l7f 0 2024-04-05T03:03:03.791 INFO:tasks.workunit.client.0.smithi153.stdout:5/800: stat d0/d9/d22/l92 0 2024-04-05T03:03:03.791 INFO:tasks.workunit.client.0.smithi153.stdout:5/801: dread - d0/d25/fb1 zero size 2024-04-05T03:03:03.925 INFO:tasks.workunit.client.0.smithi153.stdout:1/936: sync 2024-04-05T03:03:03.926 INFO:tasks.workunit.client.0.smithi153.stdout:4/919: sync 2024-04-05T03:03:03.928 INFO:tasks.workunit.client.0.smithi153.stdout:4/920: unlink da/l8e 0 2024-04-05T03:03:03.929 INFO:tasks.workunit.client.0.smithi153.stdout:1/937: unlink d5/d8/d20/d3f/f114 0 2024-04-05T03:03:03.930 INFO:tasks.workunit.client.0.smithi153.stdout:4/921: fsync da/db/d20/d3c/d91/fbb 0 2024-04-05T03:03:03.931 INFO:tasks.workunit.client.0.smithi153.stdout:4/922: read da/f97 [3771162,123550] 0 2024-04-05T03:03:03.933 INFO:tasks.workunit.client.0.smithi153.stdout:1/938: fsync d5/f8d 0 2024-04-05T03:03:03.939 INFO:tasks.workunit.client.0.smithi153.stdout:1/939: fsync d5/dd/d1b/f66 0 2024-04-05T03:03:03.941 INFO:tasks.workunit.client.0.smithi153.stdout:1/940: truncate d5/d8/d20/d3f/f64 9354990 0 2024-04-05T03:03:03.946 INFO:tasks.workunit.client.0.smithi153.stdout:1/941: write d5/de2/df5/d33/d10b/fd4 [1103520,126302] 0 2024-04-05T03:03:03.951 INFO:tasks.workunit.client.0.smithi153.stdout:4/923: dread da/db/d20/d35/d38/f59 [4194304,4194304] 0 2024-04-05T03:03:03.952 INFO:tasks.workunit.client.0.smithi153.stdout:4/924: chown da/db/d20/c92 77477 1 2024-04-05T03:03:03.954 INFO:tasks.workunit.client.0.smithi153.stdout:4/925: chown da/de/d56/d78/d8f/fe8 207146 1 2024-04-05T03:03:04.022 INFO:tasks.workunit.client.0.smithi153.stdout:8/757: sync 2024-04-05T03:03:04.022 INFO:tasks.workunit.client.0.smithi153.stdout:3/883: sync 2024-04-05T03:03:04.023 INFO:tasks.workunit.client.0.smithi153.stdout:2/954: sync 2024-04-05T03:03:04.023 INFO:tasks.workunit.client.0.smithi153.stdout:6/974: sync 2024-04-05T03:03:04.023 INFO:tasks.workunit.client.0.smithi153.stdout:0/885: sync 2024-04-05T03:03:04.023 INFO:tasks.workunit.client.0.smithi153.stdout:5/802: sync 2024-04-05T03:03:04.023 INFO:tasks.workunit.client.0.smithi153.stdout:9/767: sync 2024-04-05T03:03:04.025 INFO:tasks.workunit.client.0.smithi153.stdout:6/975: fsync d4/dc/f151 0 2024-04-05T03:03:04.029 INFO:tasks.workunit.client.0.smithi153.stdout:3/884: read - d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f7f zero size 2024-04-05T03:03:04.032 INFO:tasks.workunit.client.0.smithi153.stdout:0/886: mkdir d9/d14/d67/d6f/d10e 0 2024-04-05T03:03:04.033 INFO:tasks.workunit.client.0.smithi153.stdout:9/768: symlink d1/d9/d1b/l112 0 2024-04-05T03:03:04.034 INFO:tasks.workunit.client.0.smithi153.stdout:0/887: fsync d9/d14/fd1 0 2024-04-05T03:03:04.036 INFO:tasks.workunit.client.0.smithi153.stdout:8/758: dwrite d7/d9a/fbd [0,4194304] 0 2024-04-05T03:03:04.038 INFO:tasks.workunit.client.0.smithi153.stdout:6/976: fsync d4/dc/df/d16/d44/f96 0 2024-04-05T03:03:04.042 INFO:tasks.workunit.client.0.smithi153.stdout:8/759: read d7/d13/dc8/fcc [1983152,57596] 0 2024-04-05T03:03:04.050 INFO:tasks.workunit.client.0.smithi153.stdout:2/955: dwrite d9/da/d19/d2e/d7e/d53/f9c [0,4194304] 0 2024-04-05T03:03:04.053 INFO:tasks.workunit.client.0.smithi153.stdout:2/956: write d9/da/f49 [1145202,43808] 0 2024-04-05T03:03:04.055 INFO:tasks.workunit.client.0.smithi153.stdout:3/885: mknod d1d/c13a 0 2024-04-05T03:03:04.057 INFO:tasks.workunit.client.0.smithi153.stdout:0/888: dwrite d9/db/d42/d7e/db6/d1e/d45/dae/df4/fc6 [8388608,4194304] 0 2024-04-05T03:03:04.065 INFO:tasks.workunit.client.0.smithi153.stdout:9/769: truncate d1/d9/d1b/f75 4213658 0 2024-04-05T03:03:04.071 INFO:tasks.workunit.client.0.smithi153.stdout:6/977: write d4/dc/df/d16/d36/d10e/f140 [203485,123261] 0 2024-04-05T03:03:04.075 INFO:tasks.workunit.client.0.smithi153.stdout:3/886: mknod d1d/d12a/c13b 0 2024-04-05T03:03:04.082 INFO:tasks.workunit.client.0.smithi153.stdout:8/760: symlink d7/d85/lf7 0 2024-04-05T03:03:04.085 INFO:tasks.workunit.client.0.smithi153.stdout:6/978: unlink d4/dc/df/c13e 0 2024-04-05T03:03:04.087 INFO:tasks.workunit.client.0.smithi153.stdout:2/957: truncate d9/d68/db9/fbc 2124599 0 2024-04-05T03:03:04.094 INFO:tasks.workunit.client.0.smithi153.stdout:3/887: creat d1d/d10d/d116/f13c x:0 0 0 2024-04-05T03:03:04.094 INFO:tasks.workunit.client.0.smithi153.stdout:9/770: dwrite d1/d10/d16/d22/d2c/f31 [0,4194304] 0 2024-04-05T03:03:04.094 INFO:tasks.workunit.client.0.smithi153.stdout:6/979: truncate d4/dc/df/d16/d5f/d73/f9e 2721172 0 2024-04-05T03:03:04.094 INFO:tasks.workunit.client.0.smithi153.stdout:2/958: chown d9/d68/d9b/c10f 501 1 2024-04-05T03:03:04.097 INFO:tasks.workunit.client.0.smithi153.stdout:0/889: truncate d9/db/d42/d7e/db6/f15 1606755 0 2024-04-05T03:03:04.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/761: creat d7/d13/d19/d1b/d4f/d72/ff8 x:0 0 0 2024-04-05T03:03:04.100 INFO:tasks.workunit.client.0.smithi153.stdout:8/762: readlink d7/d13/d19/d1b/d6d/d8c/le6 0 2024-04-05T03:03:04.101 INFO:tasks.workunit.client.0.smithi153.stdout:6/980: mkdir d4/dc/df/d16/d129/d152 0 2024-04-05T03:03:04.108 INFO:tasks.workunit.client.0.smithi153.stdout:9/771: creat d1/d3a/d91/d96/dbd/f113 x:0 0 0 2024-04-05T03:03:04.109 INFO:tasks.workunit.client.0.smithi153.stdout:5/803: getdents d0/d25/d4d/d62/d5f/d86/dd7 0 2024-04-05T03:03:04.110 INFO:tasks.workunit.client.0.smithi153.stdout:0/890: readlink d9/db/d42/d7e/db6/d1e/l3a 0 2024-04-05T03:03:04.112 INFO:tasks.workunit.client.0.smithi153.stdout:9/772: truncate d1/d3a/f83 457287 0 2024-04-05T03:03:04.120 INFO:tasks.workunit.client.0.smithi153.stdout:6/981: truncate d4/dc/df/d16/d44/fd9 154392 0 2024-04-05T03:03:04.123 INFO:tasks.workunit.client.0.smithi153.stdout:9/773: creat d1/d10/d36/d57/d97/f114 x:0 0 0 2024-04-05T03:03:04.127 INFO:tasks.workunit.client.0.smithi153.stdout:6/982: dread - d4/dc/df/d16/d24/d4e/fd8 zero size 2024-04-05T03:03:04.128 INFO:tasks.workunit.client.0.smithi153.stdout:6/983: read - d4/dc/df/d16/d24/f130 zero size 2024-04-05T03:03:04.129 INFO:tasks.workunit.client.0.smithi153.stdout:6/984: chown d4/dc/df/d16/d44 153418 1 2024-04-05T03:03:04.130 INFO:tasks.workunit.client.0.smithi153.stdout:6/985: write d4/d38/f8b [862312,53537] 0 2024-04-05T03:03:04.146 INFO:tasks.workunit.client.0.smithi153.stdout:4/926: write da/f103 [250250,27431] 0 2024-04-05T03:03:04.150 INFO:tasks.workunit.client.0.smithi153.stdout:1/942: rename d5/d8/d20/d3f/f7b to d5/d8/d20/d4b/d18/d61/f123 0 2024-04-05T03:03:04.154 INFO:tasks.workunit.client.0.smithi153.stdout:9/774: rename d1/d9/fc to d1/d10/d36/d57/d5c/f115 0 2024-04-05T03:03:04.156 INFO:tasks.workunit.client.0.smithi153.stdout:9/775: write d1/d10/d36/d57/d97/fc8 [2549059,103092] 0 2024-04-05T03:03:04.156 INFO:tasks.workunit.client.0.smithi153.stdout:1/943: creat d5/d8/d20/d4b/d18/f124 x:0 0 0 2024-04-05T03:03:04.159 INFO:tasks.workunit.client.0.smithi153.stdout:1/944: chown d5/de2/df5/d45/d93/f9a 3026071 1 2024-04-05T03:03:04.163 INFO:tasks.workunit.client.0.smithi153.stdout:9/776: mknod d1/d10/d36/d71/c116 0 2024-04-05T03:03:04.164 INFO:tasks.workunit.client.0.smithi153.stdout:9/777: truncate d1/d3a/d91/d96/dbd/dbe/fda 942758 0 2024-04-05T03:03:04.165 INFO:tasks.workunit.client.0.smithi153.stdout:9/778: write d1/d1d/f5a [7282436,71361] 0 2024-04-05T03:03:04.166 INFO:tasks.workunit.client.0.smithi153.stdout:1/945: creat d5/d8/d20/d4b/d51/f125 x:0 0 0 2024-04-05T03:03:04.168 INFO:tasks.workunit.client.0.smithi153.stdout:1/946: write d5/de2/df5/d33/d10b/f8e [1994911,48810] 0 2024-04-05T03:03:04.170 INFO:tasks.workunit.client.0.smithi153.stdout:4/927: dread da/db/d20/d35/f13e [0,4194304] 0 2024-04-05T03:03:04.174 INFO:tasks.workunit.client.0.smithi153.stdout:9/779: creat d1/d3a/d91/d96/dfc/f117 x:0 0 0 2024-04-05T03:03:04.178 INFO:tasks.workunit.client.0.smithi153.stdout:4/928: creat da/db/d20/d35/d38/df7/f143 x:0 0 0 2024-04-05T03:03:04.180 INFO:tasks.workunit.client.0.smithi153.stdout:9/780: unlink d1/d1d/ff8 0 2024-04-05T03:03:04.182 INFO:tasks.workunit.client.0.smithi153.stdout:1/947: dread d5/de2/df5/f36 [0,4194304] 0 2024-04-05T03:03:04.186 INFO:tasks.workunit.client.0.smithi153.stdout:9/781: creat d1/d10/d36/d71/f118 x:0 0 0 2024-04-05T03:03:04.187 INFO:tasks.workunit.client.0.smithi153.stdout:9/782: readlink d1/d9/d1b/de9/l101 0 2024-04-05T03:03:04.189 INFO:tasks.workunit.client.0.smithi153.stdout:1/948: link d5/de2/df5/d45/d68/da4/fac d5/de2/df5/d45/d103/f126 0 2024-04-05T03:03:04.191 INFO:tasks.workunit.client.0.smithi153.stdout:9/783: creat d1/d10/d36/d71/f119 x:0 0 0 2024-04-05T03:03:04.199 INFO:tasks.workunit.client.0.smithi153.stdout:1/949: dwrite d5/d8/d20/f25 [0,4194304] 0 2024-04-05T03:03:04.203 INFO:tasks.workunit.client.0.smithi153.stdout:1/950: chown d5/d8/d20/d4b/cef 5 1 2024-04-05T03:03:04.205 INFO:tasks.workunit.client.0.smithi153.stdout:1/951: dread - d5/de2/df5/d45/d68/ffb zero size 2024-04-05T03:03:04.207 INFO:tasks.workunit.client.0.smithi153.stdout:1/952: fsync d5/f8d 0 2024-04-05T03:03:04.211 INFO:tasks.workunit.client.0.smithi153.stdout:3/888: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff2 [404265,5481] 0 2024-04-05T03:03:04.215 INFO:tasks.workunit.client.0.smithi153.stdout:1/953: link d5/d8/d20/d4b/d18/cc2 d5/de2/df5/d45/d68/d112/c127 0 2024-04-05T03:03:04.215 INFO:tasks.workunit.client.0.smithi153.stdout:2/959: dwrite d9/da/d19/d2e/d7e/d53/f83 [0,4194304] 0 2024-04-05T03:03:04.217 INFO:tasks.workunit.client.0.smithi153.stdout:8/763: truncate d7/f74 1616264 0 2024-04-05T03:03:04.221 INFO:tasks.workunit.client.0.smithi153.stdout:2/960: read d9/da/d37/d8a/da8/de5/d116/f133 [2053409,55769] 0 2024-04-05T03:03:04.221 INFO:tasks.workunit.client.0.smithi153.stdout:3/889: rmdir d1d/d32/d93/d3c/d3f 39 2024-04-05T03:03:04.221 INFO:tasks.workunit.client.0.smithi153.stdout:0/891: dwrite d9/db/d42/d7e/db6/f46 [0,4194304] 0 2024-04-05T03:03:04.223 INFO:tasks.workunit.client.0.smithi153.stdout:0/892: chown d9/db/d42/d7e/db6/d1d/d61/d8a/df8 461 1 2024-04-05T03:03:04.223 INFO:tasks.workunit.client.0.smithi153.stdout:9/784: getdents d1/d10/d36/d57/d97 0 2024-04-05T03:03:04.230 INFO:tasks.workunit.client.0.smithi153.stdout:1/954: dwrite d5/dd/d1b/f74 [0,4194304] 0 2024-04-05T03:03:04.236 INFO:tasks.workunit.client.0.smithi153.stdout:9/785: unlink d1/d1d/l50 0 2024-04-05T03:03:04.238 INFO:tasks.workunit.client.0.smithi153.stdout:9/786: write d1/d3a/d91/d96/de6/ff6 [83452,58473] 0 2024-04-05T03:03:04.240 INFO:tasks.workunit.client.0.smithi153.stdout:8/764: mknod d7/d13/d19/d1b/d4f/d5f/d54/cf9 0 2024-04-05T03:03:04.245 INFO:tasks.workunit.client.0.smithi153.stdout:3/890: rename c1c to d1d/d32/df9/c13d 0 2024-04-05T03:03:04.246 INFO:tasks.workunit.client.0.smithi153.stdout:0/893: getdents d9/d14/d67/d6f/d10e 0 2024-04-05T03:03:04.248 INFO:tasks.workunit.client.0.smithi153.stdout:6/986: dwrite d4/dc/d7a/da8/fca [0,4194304] 0 2024-04-05T03:03:04.254 INFO:tasks.workunit.client.0.smithi153.stdout:9/787: rename d1/d1d/lb8 to d1/d3a/d91/d96/de6/l11a 0 2024-04-05T03:03:04.263 INFO:tasks.workunit.client.0.smithi153.stdout:3/891: getdents d1d/d32/d93/d3c 0 2024-04-05T03:03:04.265 INFO:tasks.workunit.client.0.smithi153.stdout:3/892: symlink d1d/d32/d93/d3c/d94/l13e 0 2024-04-05T03:03:04.268 INFO:tasks.workunit.client.0.smithi153.stdout:5/804: mknod d0/d2c/c116 0 2024-04-05T03:03:04.269 INFO:tasks.workunit.client.0.smithi153.stdout:4/929: write da/db/d20/d35/d38/d76/f11d [986306,93348] 0 2024-04-05T03:03:04.271 INFO:tasks.workunit.client.0.smithi153.stdout:5/805: chown d0/d9/f21 55288 1 2024-04-05T03:03:04.272 INFO:tasks.workunit.client.0.smithi153.stdout:4/930: write da/db/dbd/ffc [5745704,79599] 0 2024-04-05T03:03:04.273 INFO:tasks.workunit.client.0.smithi153.stdout:3/893: rename d1d/d32/d93/d3c/d3f/fb8 to d1d/d32/d93/d3c/d3f/d7c/f13f 0 2024-04-05T03:03:04.275 INFO:tasks.workunit.client.0.smithi153.stdout:5/806: fsync d0/d4e/f9e 0 2024-04-05T03:03:04.277 INFO:tasks.workunit.client.0.smithi153.stdout:9/788: dread d1/d1d/f5a [4194304,4194304] 0 2024-04-05T03:03:04.280 INFO:tasks.workunit.client.0.smithi153.stdout:9/789: read - d1/d10/d16/d22/d2c/d68/fca zero size 2024-04-05T03:03:04.280 INFO:tasks.workunit.client.0.smithi153.stdout:3/894: creat d1d/d32/f140 x:0 0 0 2024-04-05T03:03:04.280 INFO:tasks.workunit.client.0.smithi153.stdout:0/894: dread d9/db/f11 [0,4194304] 0 2024-04-05T03:03:04.281 INFO:tasks.workunit.client.0.smithi153.stdout:5/807: mkdir d0/d2c/d117 0 2024-04-05T03:03:04.281 INFO:tasks.workunit.client.0.smithi153.stdout:3/895: chown d1d/f6a 50086956 1 2024-04-05T03:03:04.287 INFO:tasks.workunit.client.0.smithi153.stdout:9/790: rmdir d1/d3a/d91/d96/dbd/dbe 39 2024-04-05T03:03:04.288 INFO:tasks.workunit.client.0.smithi153.stdout:2/961: write d9/da/d19/d2e/dc7/fbb [469450,42507] 0 2024-04-05T03:03:04.289 INFO:tasks.workunit.client.0.smithi153.stdout:5/808: mknod d0/d4e/db6/c118 0 2024-04-05T03:03:04.291 INFO:tasks.workunit.client.0.smithi153.stdout:3/896: mkdir d1d/d10d/d116/d141 0 2024-04-05T03:03:04.291 INFO:tasks.workunit.client.0.smithi153.stdout:8/765: rmdir d7/d13/d19 39 2024-04-05T03:03:04.296 INFO:tasks.workunit.client.0.smithi153.stdout:1/955: truncate f3 4058235 0 2024-04-05T03:03:04.298 INFO:tasks.workunit.client.0.smithi153.stdout:0/895: dwrite d9/db/d42/d7e/f9a [4194304,4194304] 0 2024-04-05T03:03:04.299 INFO:tasks.workunit.client.0.smithi153.stdout:6/987: write d4/fda [1479788,3464] 0 2024-04-05T03:03:04.300 INFO:tasks.workunit.client.0.smithi153.stdout:5/809: read d0/d25/d4d/faf [138808,110122] 0 2024-04-05T03:03:04.302 INFO:tasks.workunit.client.0.smithi153.stdout:0/896: write d9/db/d42/d7e/db6/daf/dd0/f10a [602367,56604] 0 2024-04-05T03:03:04.306 INFO:tasks.workunit.client.0.smithi153.stdout:9/791: rename d1/f1f to d1/d3a/d91/d96/dbd/dbe/f11b 0 2024-04-05T03:03:04.307 INFO:tasks.workunit.client.0.smithi153.stdout:1/956: rmdir d5/de2/df5/d45/d93/d111 39 2024-04-05T03:03:04.310 INFO:tasks.workunit.client.0.smithi153.stdout:2/962: mkdir d9/da/d19/d2e/d7e/d7a/d59/d12e/d139/d13f 0 2024-04-05T03:03:04.314 INFO:tasks.workunit.client.0.smithi153.stdout:6/988: chown d4/d38/ce4 111051 1 2024-04-05T03:03:04.314 INFO:tasks.workunit.client.0.smithi153.stdout:5/810: rmdir d0/d4e 39 2024-04-05T03:03:04.314 INFO:tasks.workunit.client.0.smithi153.stdout:2/963: read d9/da/d37/d8a/da8/f12c [1441416,5092] 0 2024-04-05T03:03:04.322 INFO:tasks.workunit.client.0.smithi153.stdout:1/957: mkdir d5/d8/d20/d2f/d40/d128 0 2024-04-05T03:03:04.325 INFO:tasks.workunit.client.0.smithi153.stdout:8/766: dread d7/d13/d19/d1b/d33/fdd [0,4194304] 0 2024-04-05T03:03:04.329 INFO:tasks.workunit.client.0.smithi153.stdout:4/931: dwrite da/de/d66/d111/f33 [8388608,4194304] 0 2024-04-05T03:03:04.329 INFO:tasks.workunit.client.0.smithi153.stdout:0/897: dwrite d9/db/d42/f74 [0,4194304] 0 2024-04-05T03:03:04.337 INFO:tasks.workunit.client.0.smithi153.stdout:6/989: creat d4/dc/df/d16/d44/f153 x:0 0 0 2024-04-05T03:03:04.340 INFO:tasks.workunit.client.0.smithi153.stdout:5/811: dread d0/d25/d4d/d62/d5f/d86/f74 [0,4194304] 0 2024-04-05T03:03:04.343 INFO:tasks.workunit.client.0.smithi153.stdout:3/897: dwrite d1d/f28 [4194304,4194304] 0 2024-04-05T03:03:04.343 INFO:tasks.workunit.client.0.smithi153.stdout:8/767: dwrite d7/d13/d19/d1b/d6d/fae [0,4194304] 0 2024-04-05T03:03:04.344 INFO:tasks.workunit.client.0.smithi153.stdout:0/898: rmdir d9/d14 39 2024-04-05T03:03:04.347 INFO:tasks.workunit.client.0.smithi153.stdout:9/792: getdents d1/d9/d1b/de9 0 2024-04-05T03:03:04.347 INFO:tasks.workunit.client.0.smithi153.stdout:1/958: mkdir d5/de2/df5/d45/d68/d112/d129 0 2024-04-05T03:03:04.352 INFO:tasks.workunit.client.0.smithi153.stdout:4/932: rename da/db/d20/d35/d41/c64 to da/de/d56/d6a/d2d/ddb/c144 0 2024-04-05T03:03:04.352 INFO:tasks.workunit.client.0.smithi153.stdout:8/768: truncate d7/d9a/fe7 889466 0 2024-04-05T03:03:04.357 INFO:tasks.workunit.client.0.smithi153.stdout:2/964: dwrite d9/d77/f88 [0,4194304] 0 2024-04-05T03:03:04.358 INFO:tasks.workunit.client.0.smithi153.stdout:9/793: chown d1/d10/d16/d22/d2c/d68/da9/ff0 65943225 1 2024-04-05T03:03:04.358 INFO:tasks.workunit.client.0.smithi153.stdout:1/959: creat d5/d8/d20/d4b/de1/d11c/f12a x:0 0 0 2024-04-05T03:03:04.361 INFO:tasks.workunit.client.0.smithi153.stdout:6/990: unlink d4/dc/df/d16/d44/fd9 0 2024-04-05T03:03:04.363 INFO:tasks.workunit.client.0.smithi153.stdout:5/812: creat d0/d25/d4d/d9c/d10f/f119 x:0 0 0 2024-04-05T03:03:04.365 INFO:tasks.workunit.client.0.smithi153.stdout:3/898: mknod d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d123/c142 0 2024-04-05T03:03:04.365 INFO:tasks.workunit.client.0.smithi153.stdout:6/991: stat d4/dc/df/d16/d5f 0 2024-04-05T03:03:04.367 INFO:tasks.workunit.client.0.smithi153.stdout:3/899: chown d1d/d32 5878513 1 2024-04-05T03:03:04.368 INFO:tasks.workunit.client.0.smithi153.stdout:1/960: mkdir d5/de2/df5/d45/d103/d12b 0 2024-04-05T03:03:04.369 INFO:tasks.workunit.client.0.smithi153.stdout:8/769: dwrite d7/d85/fe1 [0,4194304] 0 2024-04-05T03:03:04.369 INFO:tasks.workunit.client.0.smithi153.stdout:2/965: mkdir d9/da/d19/df6/d140 0 2024-04-05T03:03:04.372 INFO:tasks.workunit.client.0.smithi153.stdout:0/899: rename d9/db/d42/d7e/db6/d1d/c87 to d9/d14/c10f 0 2024-04-05T03:03:04.378 INFO:tasks.workunit.client.0.smithi153.stdout:3/900: write d1d/d12a/d138/f115 [845170,4557] 0 2024-04-05T03:03:04.378 INFO:tasks.workunit.client.0.smithi153.stdout:9/794: dwrite d1/d3a/d91/d96/dfc/f117 [0,4194304] 0 2024-04-05T03:03:04.381 INFO:tasks.workunit.client.0.smithi153.stdout:3/901: write d1d/d32/d93/d3c/d3f/d9c/f102 [572562,97375] 0 2024-04-05T03:03:04.383 INFO:tasks.workunit.client.0.smithi153.stdout:6/992: fdatasync d4/dc/df/d16/d44/fdb 0 2024-04-05T03:03:04.388 INFO:tasks.workunit.client.0.smithi153.stdout:2/966: creat d9/da/d19/d2e/d11b/d13a/f141 x:0 0 0 2024-04-05T03:03:04.388 INFO:tasks.workunit.client.0.smithi153.stdout:6/993: chown d4/d38/d4c/fa0 1553 1 2024-04-05T03:03:04.390 INFO:tasks.workunit.client.0.smithi153.stdout:1/961: getdents d5/de2 0 2024-04-05T03:03:04.394 INFO:tasks.workunit.client.0.smithi153.stdout:4/933: dwrite da/de/d56/d6a/f87 [0,4194304] 0 2024-04-05T03:03:04.396 INFO:tasks.workunit.client.0.smithi153.stdout:5/813: write d0/d25/d4d/d62/d5f/f7e [531746,24120] 0 2024-04-05T03:03:04.398 INFO:tasks.workunit.client.0.smithi153.stdout:2/967: creat d9/da/d19/d2e/dc7/f142 x:0 0 0 2024-04-05T03:03:04.401 INFO:tasks.workunit.client.0.smithi153.stdout:0/900: rename d9/d14/c10f to d9/d14/d67/d6f/d10e/c110 0 2024-04-05T03:03:04.403 INFO:tasks.workunit.client.0.smithi153.stdout:6/994: creat d4/dc/df/d16/d129/d13d/f154 x:0 0 0 2024-04-05T03:03:04.404 INFO:tasks.workunit.client.0.smithi153.stdout:8/770: dread d7/f8 [0,4194304] 0 2024-04-05T03:03:04.405 INFO:tasks.workunit.client.0.smithi153.stdout:3/902: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f67 [0,4194304] 0 2024-04-05T03:03:04.405 INFO:tasks.workunit.client.0.smithi153.stdout:9/795: truncate d1/d9/d1b/fb0 4108063 0 2024-04-05T03:03:04.409 INFO:tasks.workunit.client.0.smithi153.stdout:2/968: chown d9/l79 1342600 1 2024-04-05T03:03:04.409 INFO:tasks.workunit.client.0.smithi153.stdout:1/962: dwrite d5/de2/df5/d45/f108 [0,4194304] 0 2024-04-05T03:03:04.414 INFO:tasks.workunit.client.0.smithi153.stdout:6/995: chown d4/dc/d7a/d9c/fc3 904 1 2024-04-05T03:03:04.415 INFO:tasks.workunit.client.0.smithi153.stdout:5/814: mkdir d0/d2c/d5e/d11a 0 2024-04-05T03:03:04.417 INFO:tasks.workunit.client.0.smithi153.stdout:6/996: dread - d4/f147 zero size 2024-04-05T03:03:04.418 INFO:tasks.workunit.client.0.smithi153.stdout:0/901: creat d9/db/d42/d7e/db6/d1e/d45/dae/f111 x:0 0 0 2024-04-05T03:03:04.420 INFO:tasks.workunit.client.0.smithi153.stdout:5/815: dread - d0/d25/d4d/d9c/faa zero size 2024-04-05T03:03:04.421 INFO:tasks.workunit.client.0.smithi153.stdout:0/902: dread - d9/db/d42/d7e/db6/fed zero size 2024-04-05T03:03:04.422 INFO:tasks.workunit.client.0.smithi153.stdout:6/997: write d4/dc/f151 [236400,79371] 0 2024-04-05T03:03:04.422 INFO:tasks.workunit.client.0.smithi153.stdout:5/816: chown d0/d25/d4d/d62 7616 1 2024-04-05T03:03:04.422 INFO:tasks.workunit.client.0.smithi153.stdout:1/963: mkdir d5/de2/df5/d48/d96/d12c 0 2024-04-05T03:03:04.424 INFO:tasks.workunit.client.0.smithi153.stdout:6/998: chown d4/dc/d101/d13f/d146 8 1 2024-04-05T03:03:04.430 INFO:tasks.workunit.client.0.smithi153.stdout:3/903: chown d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/c9b 14978 1 2024-04-05T03:03:04.431 INFO:tasks.workunit.client.0.smithi153.stdout:1/964: mkdir d5/d8/d20/d4b/d51/da2/d12d 0 2024-04-05T03:03:04.431 INFO:tasks.workunit.client.0.smithi153.stdout:9/796: link d1/d6c/cf4 d1/d10/d16/c11c 0 2024-04-05T03:03:04.432 INFO:tasks.workunit.client.0.smithi153.stdout:5/817: rename d0/l3e to d0/d25/d4d/d62/dbb/l11b 0 2024-04-05T03:03:04.435 INFO:tasks.workunit.client.0.smithi153.stdout:3/904: chown d1d/d32/d93/d3c/f73 13643263 1 2024-04-05T03:03:04.435 INFO:tasks.workunit.client.0.smithi153.stdout:5/818: chown d0/d2c/d3f/df7 0 1 2024-04-05T03:03:04.435 INFO:tasks.workunit.client.0.smithi153.stdout:4/934: dread da/de/da1/fc0 [0,4194304] 0 2024-04-05T03:03:04.439 INFO:tasks.workunit.client.0.smithi153.stdout:1/965: mknod d5/d8/d20/d4b/c12e 0 2024-04-05T03:03:04.439 INFO:tasks.workunit.client.0.smithi153.stdout:9/797: write d1/d3a/d91/d96/dbd/dbe/fda [793077,16986] 0 2024-04-05T03:03:04.440 INFO:tasks.workunit.client.0.smithi153.stdout:2/969: dwrite d9/da/d19/d2e/d7e/d7a/d59/f13c [0,4194304] 0 2024-04-05T03:03:04.440 INFO:tasks.workunit.client.0.smithi153.stdout:5/819: rmdir d0/d25/d4d/d9c/dc4/dcb 39 2024-04-05T03:03:04.443 INFO:tasks.workunit.client.0.smithi153.stdout:1/966: fdatasync d5/de2/df5/f7a 0 2024-04-05T03:03:04.445 INFO:tasks.workunit.client.0.smithi153.stdout:8/771: truncate d7/d85/fe1 1796628 0 2024-04-05T03:03:04.447 INFO:tasks.workunit.client.0.smithi153.stdout:4/935: creat da/de/d56/d6a/f145 x:0 0 0 2024-04-05T03:03:04.454 INFO:tasks.workunit.client.0.smithi153.stdout:3/905: dwrite d1d/d27/d2b/f62 [0,4194304] 0 2024-04-05T03:03:04.455 INFO:tasks.workunit.client.0.smithi153.stdout:5/820: creat d0/d4e/db6/f11c x:0 0 0 2024-04-05T03:03:04.457 INFO:tasks.workunit.client.0.smithi153.stdout:2/970: creat d9/da/d19/df6/d140/f143 x:0 0 0 2024-04-05T03:03:04.457 INFO:tasks.workunit.client.0.smithi153.stdout:9/798: chown d1/d10/d16/d22/fc9 9919843 1 2024-04-05T03:03:04.457 INFO:tasks.workunit.client.0.smithi153.stdout:0/903: dread d9/db/d42/f71 [0,4194304] 0 2024-04-05T03:03:04.457 INFO:tasks.workunit.client.0.smithi153.stdout:1/967: symlink d5/de2/df5/d45/d93/d111/l12f 0 2024-04-05T03:03:04.460 INFO:tasks.workunit.client.0.smithi153.stdout:3/906: fsync d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f118 0 2024-04-05T03:03:04.461 INFO:tasks.workunit.client.0.smithi153.stdout:4/936: rename da/de/d56/f9f to da/db/d20/d35/d38/f146 0 2024-04-05T03:03:04.461 INFO:tasks.workunit.client.0.smithi153.stdout:2/971: dread - d9/da/d19/d2e/d7e/d53/da1/f11a zero size 2024-04-05T03:03:04.464 INFO:tasks.workunit.client.0.smithi153.stdout:9/799: symlink d1/d3a/d91/l11d 0 2024-04-05T03:03:04.465 INFO:tasks.workunit.client.0.smithi153.stdout:1/968: unlink d5/de2/df5/d2c/c31 0 2024-04-05T03:03:04.466 INFO:tasks.workunit.client.0.smithi153.stdout:6/999: dread d4/dc/df/d16/d5f/d73/fb1 [4194304,4194304] 0 2024-04-05T03:03:04.466 INFO:tasks.workunit.client.0.smithi153.stdout:8/772: rename d7/d46/db1/lf1 to d7/d13/d19/d1b/d4f/d68/dbc/lfa 0 2024-04-05T03:03:04.467 INFO:tasks.workunit.client.0.smithi153.stdout:4/937: rmdir da/de/d56/d78/d8f/d68 39 2024-04-05T03:03:04.470 INFO:tasks.workunit.client.0.smithi153.stdout:2/972: chown d9/da/d19/d2e/d7e/de6/f6b 0 1 2024-04-05T03:03:04.471 INFO:tasks.workunit.client.0.smithi153.stdout:5/821: dwrite d0/fed [0,4194304] 0 2024-04-05T03:03:04.472 INFO:tasks.workunit.client.0.smithi153.stdout:0/904: rename d9/db/d42/d7e/db6/d1e/d45/d62/dde/d9e to d9/db/d42/d7e/d112 0 2024-04-05T03:03:04.474 INFO:tasks.workunit.client.0.smithi153.stdout:8/773: unlink d7/d13/d19/d1b/d21/c3f 0 2024-04-05T03:03:04.475 INFO:tasks.workunit.client.0.smithi153.stdout:5/822: dread d0/d25/d4d/d9c/dc7/ffa [0,4194304] 0 2024-04-05T03:03:04.479 INFO:tasks.workunit.client.0.smithi153.stdout:4/938: mkdir da/db/d20/d35/d147 0 2024-04-05T03:03:04.479 INFO:tasks.workunit.client.0.smithi153.stdout:2/973: rmdir d9/da/d19/d2e/dc7 39 2024-04-05T03:03:04.480 INFO:tasks.workunit.client.0.smithi153.stdout:2/974: stat d9/da 0 2024-04-05T03:03:04.481 INFO:tasks.workunit.client.0.smithi153.stdout:4/939: chown da/de/da1/d13f 0 1 2024-04-05T03:03:04.483 INFO:tasks.workunit.client.0.smithi153.stdout:5/823: rmdir d0/d2c/d3f/da8 39 2024-04-05T03:03:04.483 INFO:tasks.workunit.client.0.smithi153.stdout:0/905: dread d9/f1a [0,4194304] 0 2024-04-05T03:03:04.488 INFO:tasks.workunit.client.0.smithi153.stdout:4/940: creat da/db/d20/d3c/d91/dd5/d10d/f148 x:0 0 0 2024-04-05T03:03:04.488 INFO:tasks.workunit.client.0.smithi153.stdout:4/941: readlink da/de/d66/ld1 0 2024-04-05T03:03:04.491 INFO:tasks.workunit.client.0.smithi153.stdout:0/906: unlink d9/db/d42/d43/fa2 0 2024-04-05T03:03:04.492 INFO:tasks.workunit.client.0.smithi153.stdout:0/907: stat d9/db/d42/d43/fa1 0 2024-04-05T03:03:04.493 INFO:tasks.workunit.client.0.smithi153.stdout:8/774: getdents d7/d13/d19/d1b/db9 0 2024-04-05T03:03:04.496 INFO:tasks.workunit.client.0.smithi153.stdout:4/942: mknod da/db/d20/d3c/d81/db6/dc7/de1/c149 0 2024-04-05T03:03:04.496 INFO:tasks.workunit.client.0.smithi153.stdout:0/908: truncate d9/db/d42/d7e/db6/d1e/d45/ff2 234052 0 2024-04-05T03:03:04.498 INFO:tasks.workunit.client.0.smithi153.stdout:0/909: write d9/db/d42/d7e/f9a [8755087,8665] 0 2024-04-05T03:03:04.501 INFO:tasks.workunit.client.0.smithi153.stdout:3/907: dread d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 [0,4194304] 0 2024-04-05T03:03:04.501 INFO:tasks.workunit.client.0.smithi153.stdout:8/775: unlink d7/d13/d19/d1b/d4f/d68/dbc/feb 0 2024-04-05T03:03:04.503 INFO:tasks.workunit.client.0.smithi153.stdout:1/969: dread d5/dd/d1b/f66 [8388608,4194304] 0 2024-04-05T03:03:04.503 INFO:tasks.workunit.client.0.smithi153.stdout:4/943: creat da/db/de6/f14a x:0 0 0 2024-04-05T03:03:04.506 INFO:tasks.workunit.client.0.smithi153.stdout:0/910: rename d9/d14/f57 to d9/db/d42/dd4/f113 0 2024-04-05T03:03:04.510 INFO:tasks.workunit.client.0.smithi153.stdout:8/776: dread d7/d2a/f65 [0,4194304] 0 2024-04-05T03:03:04.512 INFO:tasks.workunit.client.0.smithi153.stdout:5/824: truncate d0/d25/d4d/d62/d37/f4b 2483197 0 2024-04-05T03:03:04.512 INFO:tasks.workunit.client.0.smithi153.stdout:1/970: dread d5/de2/df5/f36 [0,4194304] 0 2024-04-05T03:03:04.515 INFO:tasks.workunit.client.0.smithi153.stdout:3/908: dwrite d1d/d27/f64 [0,4194304] 0 2024-04-05T03:03:04.520 INFO:tasks.workunit.client.0.smithi153.stdout:1/971: creat d5/de2/df5/d45/d68/f130 x:0 0 0 2024-04-05T03:03:04.521 INFO:tasks.workunit.client.0.smithi153.stdout:4/944: dwrite da/db/d20/d35/d41/d67/f12b [0,4194304] 0 2024-04-05T03:03:04.523 INFO:tasks.workunit.client.0.smithi153.stdout:3/909: creat d1d/d12a/d138/f143 x:0 0 0 2024-04-05T03:03:04.524 INFO:tasks.workunit.client.0.smithi153.stdout:2/975: dread d9/ff [8388608,4194304] 0 2024-04-05T03:03:04.524 INFO:tasks.workunit.client.0.smithi153.stdout:0/911: truncate d9/db/d42/d7e/db6/d1e/d45/d62/dde/fc9 604919 0 2024-04-05T03:03:04.529 INFO:tasks.workunit.client.0.smithi153.stdout:2/976: mknod d9/da/d94/dca/df7/c144 0 2024-04-05T03:03:04.530 INFO:tasks.workunit.client.0.smithi153.stdout:3/910: dread - d1d/d32/d93/d3c/f117 zero size 2024-04-05T03:03:04.531 INFO:tasks.workunit.client.0.smithi153.stdout:1/972: creat d5/de2/df5/d48/d96/d12c/f131 x:0 0 0 2024-04-05T03:03:04.533 INFO:tasks.workunit.client.0.smithi153.stdout:4/945: truncate da/db/d20/d35/d38/f5c 2553788 0 2024-04-05T03:03:04.534 INFO:tasks.workunit.client.0.smithi153.stdout:8/777: dwrite d7/d13/d19/d1b/d4f/d5f/fe3 [0,4194304] 0 2024-04-05T03:03:04.536 INFO:tasks.workunit.client.0.smithi153.stdout:0/912: dread d9/db/d42/f71 [0,4194304] 0 2024-04-05T03:03:04.536 INFO:tasks.workunit.client.0.smithi153.stdout:1/973: fsync d5/dd/f9f 0 2024-04-05T03:03:04.537 INFO:tasks.workunit.client.0.smithi153.stdout:2/977: truncate d9/da/d19/d2e/f43 7891441 0 2024-04-05T03:03:04.539 INFO:tasks.workunit.client.0.smithi153.stdout:1/974: fdatasync d5/d8/d20/d4b/de1/f105 0 2024-04-05T03:03:04.540 INFO:tasks.workunit.client.0.smithi153.stdout:1/975: stat d5/de2/df5/d45/d68/fdc 0 2024-04-05T03:03:04.543 INFO:tasks.workunit.client.0.smithi153.stdout:0/913: rename d9/db/f66 to d9/d14/d67/f114 0 2024-04-05T03:03:04.544 INFO:tasks.workunit.client.0.smithi153.stdout:2/978: symlink d9/da/d36/l145 0 2024-04-05T03:03:04.545 INFO:tasks.workunit.client.0.smithi153.stdout:0/914: truncate d9/db/d42/d43/f4a 4835154 0 2024-04-05T03:03:04.546 INFO:tasks.workunit.client.0.smithi153.stdout:1/976: write d5/d8/d20/d4b/de1/f121 [1512086,63784] 0 2024-04-05T03:03:04.547 INFO:tasks.workunit.client.0.smithi153.stdout:2/979: creat d9/d68/d98/f146 x:0 0 0 2024-04-05T03:03:04.548 INFO:tasks.workunit.client.0.smithi153.stdout:2/980: dread - d9/da/d37/d8a/da8/de5/f127 zero size 2024-04-05T03:03:04.549 INFO:tasks.workunit.client.0.smithi153.stdout:2/981: fdatasync d9/da/f121 0 2024-04-05T03:03:04.549 INFO:tasks.workunit.client.0.smithi153.stdout:1/977: mkdir d5/de2/df5/d45/d93/d132 0 2024-04-05T03:03:04.552 INFO:tasks.workunit.client.0.smithi153.stdout:1/978: write d5/de2/df5/d2c/f7d [2607447,69035] 0 2024-04-05T03:03:04.552 INFO:tasks.workunit.client.0.smithi153.stdout:2/982: creat d9/da/d19/d2e/d7e/d7a/d59/d12e/d139/f147 x:0 0 0 2024-04-05T03:03:04.555 INFO:tasks.workunit.client.0.smithi153.stdout:1/979: write d5/de2/df5/f36 [1417179,130718] 0 2024-04-05T03:03:04.561 INFO:tasks.workunit.client.0.smithi153.stdout:2/983: symlink d9/da/d19/d2e/d7e/d7a/d59/d12e/d139/l148 0 2024-04-05T03:03:04.561 INFO:tasks.workunit.client.0.smithi153.stdout:5/825: dwrite d0/d25/d68/fd0 [0,4194304] 0 2024-04-05T03:03:04.564 INFO:tasks.workunit.client.0.smithi153.stdout:4/946: write da/db/d20/d3c/fb8 [4379466,18222] 0 2024-04-05T03:03:04.565 INFO:tasks.workunit.client.0.smithi153.stdout:4/947: chown da/db/d20/fda 101760260 1 2024-04-05T03:03:04.566 INFO:tasks.workunit.client.0.smithi153.stdout:1/980: unlink d5/lad 0 2024-04-05T03:03:04.567 INFO:tasks.workunit.client.0.smithi153.stdout:1/981: chown f4 3222923 1 2024-04-05T03:03:04.567 INFO:tasks.workunit.client.0.smithi153.stdout:0/915: dread d9/db/d42/d43/f76 [0,4194304] 0 2024-04-05T03:03:04.568 INFO:tasks.workunit.client.0.smithi153.stdout:5/826: symlink d0/d25/l11d 0 2024-04-05T03:03:04.569 INFO:tasks.workunit.client.0.smithi153.stdout:3/911: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fd7 [361644,100691] 0 2024-04-05T03:03:04.571 INFO:tasks.workunit.client.0.smithi153.stdout:5/827: chown d0/d25/d4d/d9c/dc7/dc9/cd4 5 1 2024-04-05T03:03:04.575 INFO:tasks.workunit.client.0.smithi153.stdout:1/982: write d5/d8/d20/d4b/d51/f10c [1776771,89614] 0 2024-04-05T03:03:04.576 INFO:tasks.workunit.client.0.smithi153.stdout:8/778: dread d7/fa [0,4194304] 0 2024-04-05T03:03:04.576 INFO:tasks.workunit.client.0.smithi153.stdout:0/916: mknod d9/db/d42/dd4/c115 0 2024-04-05T03:03:04.577 INFO:tasks.workunit.client.0.smithi153.stdout:0/917: stat d9/db/d42/d7e 0 2024-04-05T03:03:04.577 INFO:tasks.workunit.client.0.smithi153.stdout:4/948: getdents da/de/da1/d13f 0 2024-04-05T03:03:04.584 INFO:tasks.workunit.client.0.smithi153.stdout:3/912: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/l144 0 2024-04-05T03:03:04.589 INFO:tasks.workunit.client.0.smithi153.stdout:0/918: symlink d9/d14/d10b/l116 0 2024-04-05T03:03:04.590 INFO:tasks.workunit.client.0.smithi153.stdout:0/919: fdatasync d9/db/d42/d7e/db6/f59 0 2024-04-05T03:03:04.592 INFO:tasks.workunit.client.0.smithi153.stdout:4/949: creat da/de/d56/d12a/f14b x:0 0 0 2024-04-05T03:03:04.592 INFO:tasks.workunit.client.0.smithi153.stdout:2/984: truncate d9/d68/fce 494378 0 2024-04-05T03:03:04.593 INFO:tasks.workunit.client.0.smithi153.stdout:5/828: symlink d0/d25/d4d/d9c/dc4/dcb/d109/l11e 0 2024-04-05T03:03:04.594 INFO:tasks.workunit.client.0.smithi153.stdout:5/829: chown d0/d25/d4d/fec 3824662 1 2024-04-05T03:03:04.595 INFO:tasks.workunit.client.0.smithi153.stdout:8/779: write d7/d13/dab/fc0 [1056107,74139] 0 2024-04-05T03:03:04.597 INFO:tasks.workunit.client.0.smithi153.stdout:8/780: dread - d7/d13/d19/d95/fb3 zero size 2024-04-05T03:03:04.598 INFO:tasks.workunit.client.0.smithi153.stdout:5/830: creat d0/d4e/db6/f11f x:0 0 0 2024-04-05T03:03:04.599 INFO:tasks.workunit.client.0.smithi153.stdout:1/983: getdents d5/de2/df5/d45/d68/d112 0 2024-04-05T03:03:04.599 INFO:tasks.workunit.client.0.smithi153.stdout:8/781: chown d7/d13/d19/d95/fb3 158 1 2024-04-05T03:03:04.600 INFO:tasks.workunit.client.0.smithi153.stdout:3/913: creat d1d/d32/d93/f145 x:0 0 0 2024-04-05T03:03:04.602 INFO:tasks.workunit.client.0.smithi153.stdout:2/985: link d9/da/d37/d8a/da8/de5/f127 d9/da/d19/df6/d140/f149 0 2024-04-05T03:03:04.604 INFO:tasks.workunit.client.0.smithi153.stdout:2/986: stat d9/d68/d9b/fb4 0 2024-04-05T03:03:04.605 INFO:tasks.workunit.client.0.smithi153.stdout:8/782: creat d7/d46/ffb x:0 0 0 2024-04-05T03:03:04.605 INFO:tasks.workunit.client.0.smithi153.stdout:2/987: chown d9/da/fe 11 1 2024-04-05T03:03:04.606 INFO:tasks.workunit.client.0.smithi153.stdout:5/831: dwrite d0/d2c/fbe [4194304,4194304] 0 2024-04-05T03:03:04.608 INFO:tasks.workunit.client.0.smithi153.stdout:3/914: fdatasync d1d/d32/d93/d3c/d3f/d9c/faa 0 2024-04-05T03:03:04.610 INFO:tasks.workunit.client.0.smithi153.stdout:1/984: mknod d5/de2/df5/d2c/d62/c133 0 2024-04-05T03:03:04.611 INFO:tasks.workunit.client.0.smithi153.stdout:8/783: mknod d7/d13/d19/d1b/d4f/d72/d73/cfc 0 2024-04-05T03:03:04.611 INFO:tasks.workunit.client.0.smithi153.stdout:2/988: symlink d9/da/d94/dbf/l14a 0 2024-04-05T03:03:04.613 INFO:tasks.workunit.client.0.smithi153.stdout:5/832: symlink d0/d25/d4d/d9c/l120 0 2024-04-05T03:03:04.615 INFO:tasks.workunit.client.0.smithi153.stdout:4/950: write da/de/d56/d78/d8f/f130 [793523,31610] 0 2024-04-05T03:03:04.617 INFO:tasks.workunit.client.0.smithi153.stdout:5/833: dread - d0/d25/d4d/d62/d5f/db2/f10b zero size 2024-04-05T03:03:04.618 INFO:tasks.workunit.client.0.smithi153.stdout:8/784: symlink d7/d46/db1/lfd 0 2024-04-05T03:03:04.619 INFO:tasks.workunit.client.0.smithi153.stdout:5/834: stat d0/d9/d22/f41 0 2024-04-05T03:03:04.620 INFO:tasks.workunit.client.0.smithi153.stdout:5/835: chown d0/d4e 16093778 1 2024-04-05T03:03:04.622 INFO:tasks.workunit.client.0.smithi153.stdout:8/785: chown d7/d13/d19/d1b/d6d/d8c 3423796 1 2024-04-05T03:03:04.622 INFO:tasks.workunit.client.0.smithi153.stdout:2/989: dwrite d9/da/d19/d2e/d7e/d7a/d6d/f124 [0,4194304] 0 2024-04-05T03:03:04.625 INFO:tasks.workunit.client.0.smithi153.stdout:4/951: write da/de/da1/fc0 [10745354,48409] 0 2024-04-05T03:03:04.625 INFO:tasks.workunit.client.0.smithi153.stdout:8/786: chown d7/d13/d19/d1b/d4f/cf3 1897908 1 2024-04-05T03:03:04.625 INFO:tasks.workunit.client.0.smithi153.stdout:5/836: unlink d0/d25/d4d/faf 0 2024-04-05T03:03:04.627 INFO:tasks.workunit.client.0.smithi153.stdout:1/985: dread d5/d8/d20/d4b/dd6/fe7 [0,4194304] 0 2024-04-05T03:03:04.628 INFO:tasks.workunit.client.0.smithi153.stdout:3/915: dread d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f76 [0,4194304] 0 2024-04-05T03:03:04.630 INFO:tasks.workunit.client.0.smithi153.stdout:5/837: dread - d0/d25/d4d/d9c/d10f/f119 zero size 2024-04-05T03:03:04.630 INFO:tasks.workunit.client.0.smithi153.stdout:4/952: rmdir da/db/dbd 39 2024-04-05T03:03:04.630 INFO:tasks.workunit.client.0.smithi153.stdout:2/990: dread d9/da/d19/d2e/d4f/fc4 [0,4194304] 0 2024-04-05T03:03:04.633 INFO:tasks.workunit.client.0.smithi153.stdout:8/787: unlink d7/d13/c1e 0 2024-04-05T03:03:04.638 INFO:tasks.workunit.client.0.smithi153.stdout:3/916: write d1d/d27/fa1 [4700904,97654] 0 2024-04-05T03:03:04.639 INFO:tasks.workunit.client.0.smithi153.stdout:5/838: mkdir d0/d2c/d5e/d11a/d121 0 2024-04-05T03:03:04.640 INFO:tasks.workunit.client.0.smithi153.stdout:8/788: rmdir d7/d13/d19/d1b/d4f/d72 39 2024-04-05T03:03:04.641 INFO:tasks.workunit.client.0.smithi153.stdout:5/839: stat d0/d2c/d3f 0 2024-04-05T03:03:04.642 INFO:tasks.workunit.client.0.smithi153.stdout:1/986: rename d5/d8/d20/d4b/d51/da2 to d5/dd/d134 0 2024-04-05T03:03:04.644 INFO:tasks.workunit.client.0.smithi153.stdout:3/917: symlink d1d/d32/d93/d3c/d3f/d90/d92/d99/l146 0 2024-04-05T03:03:04.647 INFO:tasks.workunit.client.0.smithi153.stdout:5/840: symlink d0/d25/d4d/d9c/db7/l122 0 2024-04-05T03:03:04.648 INFO:tasks.workunit.client.0.smithi153.stdout:1/987: mknod d5/d8/d20/d4b/d51/db5/c135 0 2024-04-05T03:03:04.650 INFO:tasks.workunit.client.0.smithi153.stdout:4/953: getdents da/de/d56/d6a/d2d/ddb 0 2024-04-05T03:03:04.651 INFO:tasks.workunit.client.0.smithi153.stdout:1/988: fdatasync d5/d8/d20/d4b/d18/f124 0 2024-04-05T03:03:04.651 INFO:tasks.workunit.client.0.smithi153.stdout:2/991: dwrite d9/da/d37/d8a/da8/de5/f127 [0,4194304] 0 2024-04-05T03:03:04.657 INFO:tasks.workunit.client.0.smithi153.stdout:4/954: mknod da/db/d20/d35/d38/d76/d90/c14c 0 2024-04-05T03:03:04.659 INFO:tasks.workunit.client.0.smithi153.stdout:1/989: symlink d5/de2/df5/d45/d93/d111/l136 0 2024-04-05T03:03:04.660 INFO:tasks.workunit.client.0.smithi153.stdout:1/990: fdatasync d5/de2/df5/d45/d68/f77 0 2024-04-05T03:03:04.661 INFO:tasks.workunit.client.0.smithi153.stdout:5/841: write d0/d9/d22/f7a [656049,122311] 0 2024-04-05T03:03:04.665 INFO:tasks.workunit.client.0.smithi153.stdout:5/842: readlink d0/d25/d4d/d62/l7c 0 2024-04-05T03:03:04.668 INFO:tasks.workunit.client.0.smithi153.stdout:5/843: fdatasync d0/d25/d68/fa0 0 2024-04-05T03:03:04.669 INFO:tasks.workunit.client.0.smithi153.stdout:2/992: dwrite d9/d68/fab [0,4194304] 0 2024-04-05T03:03:04.680 INFO:tasks.workunit.client.0.smithi153.stdout:5/844: creat d0/d25/d4d/d62/d5f/d86/f123 x:0 0 0 2024-04-05T03:03:04.681 INFO:tasks.workunit.client.0.smithi153.stdout:8/789: dread d7/d13/f14 [0,4194304] 0 2024-04-05T03:03:04.686 INFO:tasks.workunit.client.0.smithi153.stdout:2/993: dwrite d9/d68/d98/da3/f13e [0,4194304] 0 2024-04-05T03:03:04.696 INFO:tasks.workunit.client.0.smithi153.stdout:2/994: truncate d9/da/d19/d2e/d7e/d7a/fda 595854 0 2024-04-05T03:03:04.697 INFO:tasks.workunit.client.0.smithi153.stdout:2/995: chown d9/da/df9/c131 358537 1 2024-04-05T03:03:04.698 INFO:tasks.workunit.client.0.smithi153.stdout:2/996: write d9/da/d19/d2e/d7e/d7a/f84 [4285246,82935] 0 2024-04-05T03:03:04.708 INFO:tasks.workunit.client.0.smithi153.stdout:8/790: write d7/d85/f6a [1811379,30765] 0 2024-04-05T03:03:04.712 INFO:tasks.workunit.client.0.smithi153.stdout:5/845: write d0/d25/d4d/d62/d37/d38/f98 [121910,49864] 0 2024-04-05T03:03:04.718 INFO:tasks.workunit.client.0.smithi153.stdout:5/846: link d0/d25/d4d/d62/d37/fa6 d0/d2c/d5e/d11a/f124 0 2024-04-05T03:03:04.720 INFO:tasks.workunit.client.0.smithi153.stdout:5/847: write d0/d2c/d3f/fd9 [1403060,97829] 0 2024-04-05T03:03:04.721 INFO:tasks.workunit.client.0.smithi153.stdout:8/791: dwrite d7/d2a/f65 [0,4194304] 0 2024-04-05T03:03:04.722 INFO:tasks.workunit.client.0.smithi153.stdout:5/848: fsync d0/d25/d4d/d62/d37/d38/f98 0 2024-04-05T03:03:04.725 INFO:tasks.workunit.client.0.smithi153.stdout:5/849: write d0/d25/d4d/d62/d37/d38/f98 [647897,52370] 0 2024-04-05T03:03:04.726 INFO:tasks.workunit.client.0.smithi153.stdout:8/792: readlink d7/d85/l70 0 2024-04-05T03:03:04.732 INFO:tasks.workunit.client.0.smithi153.stdout:5/850: dwrite d0/d25/d4d/d62/d5f/fbf [0,4194304] 0 2024-04-05T03:03:04.738 INFO:tasks.workunit.client.0.smithi153.stdout:5/851: truncate d0/f75 6160651 0 2024-04-05T03:03:04.742 INFO:tasks.workunit.client.0.smithi153.stdout:5/852: rename d0/d2c/d5e/d11a/d121 to d0/d9/d22/d125 0 2024-04-05T03:03:04.744 INFO:tasks.workunit.client.0.smithi153.stdout:5/853: symlink d0/d4e/db6/l126 0 2024-04-05T03:03:04.747 INFO:tasks.workunit.client.0.smithi153.stdout:5/854: creat d0/d25/f127 x:0 0 0 2024-04-05T03:03:04.748 INFO:tasks.workunit.client.0.smithi153.stdout:5/855: fsync d0/d25/d4d/d62/d5f/d86/f123 0 2024-04-05T03:03:05.069 INFO:tasks.workunit.client.0.smithi153.stdout:9/800: sync 2024-04-05T03:03:05.070 INFO:tasks.workunit.client.0.smithi153.stdout:9/801: fsync d1/d10/d36/f56 0 2024-04-05T03:03:05.083 INFO:tasks.workunit.client.0.smithi153.stdout:9/802: write d1/d10/d16/d22/d2c/f58 [5617511,97149] 0 2024-04-05T03:03:05.084 INFO:tasks.workunit.client.0.smithi153.stdout:9/803: chown d1/f10f 230759 1 2024-04-05T03:03:05.099 INFO:tasks.workunit.client.0.smithi153.stdout:9/804: dread d1/d3a/fb5 [0,4194304] 0 2024-04-05T03:03:05.100 INFO:tasks.workunit.client.0.smithi153.stdout:9/805: chown d1/d10/d16/d22/d2c/d68/da9/f102 3745 1 2024-04-05T03:03:05.103 INFO:tasks.workunit.client.0.smithi153.stdout:9/806: creat d1/d10/d16/d22/f11e x:0 0 0 2024-04-05T03:03:05.104 INFO:tasks.workunit.client.0.smithi153.stdout:9/807: read d1/d10/d36/d57/d97/fa5 [2230712,110770] 0 2024-04-05T03:03:05.109 INFO:tasks.workunit.client.0.smithi153.stdout:9/808: write d1/d10/d16/d22/d2c/d68/fd5 [402841,80292] 0 2024-04-05T03:03:05.117 INFO:tasks.workunit.client.0.smithi153.stdout:9/809: dwrite d1/d10/d16/d22/d2c/d68/d81/fd7 [4194304,4194304] 0 2024-04-05T03:03:05.121 INFO:tasks.workunit.client.0.smithi153.stdout:9/810: symlink d1/d3a/d91/d96/dbd/l11f 0 2024-04-05T03:03:05.127 INFO:tasks.workunit.client.0.smithi153.stdout:9/811: write d1/d9/d1b/f7d [2160153,27616] 0 2024-04-05T03:03:05.129 INFO:tasks.workunit.client.0.smithi153.stdout:9/812: dread - d1/d10/d16/d22/d2c/d68/fca zero size 2024-04-05T03:03:05.131 INFO:tasks.workunit.client.0.smithi153.stdout:9/813: mkdir d1/d10/d16/d120 0 2024-04-05T03:03:05.134 INFO:tasks.workunit.client.0.smithi153.stdout:9/814: mknod d1/d3a/d91/d96/c121 0 2024-04-05T03:03:05.136 INFO:tasks.workunit.client.0.smithi153.stdout:9/815: fsync d1/d9/d1b/f73 0 2024-04-05T03:03:05.137 INFO:tasks.workunit.client.0.smithi153.stdout:9/816: chown d1/d9 8 1 2024-04-05T03:03:05.139 INFO:tasks.workunit.client.0.smithi153.stdout:9/817: mkdir d1/d9/dbc/d122 0 2024-04-05T03:03:05.140 INFO:tasks.workunit.client.0.smithi153.stdout:9/818: write d1/f18 [1258144,131059] 0 2024-04-05T03:03:05.143 INFO:tasks.workunit.client.0.smithi153.stdout:9/819: mkdir d1/d10/d36/d71/d123 0 2024-04-05T03:03:05.145 INFO:tasks.workunit.client.0.smithi153.stdout:9/820: fdatasync d1/d10/d36/d57/d5c/f89 0 2024-04-05T03:03:05.151 INFO:tasks.workunit.client.0.smithi153.stdout:9/821: write d1/d9/d1b/fb0 [5081138,27930] 0 2024-04-05T03:03:05.152 INFO:tasks.workunit.client.0.smithi153.stdout:9/822: write d1/f18 [1921820,44117] 0 2024-04-05T03:03:05.155 INFO:tasks.workunit.client.0.smithi153.stdout:9/823: creat d1/d10/d36/f124 x:0 0 0 2024-04-05T03:03:05.161 INFO:tasks.workunit.client.0.smithi153.stdout:9/824: write d1/d10/d16/d22/d2c/d68/d81/fbb [304495,23223] 0 2024-04-05T03:03:05.164 INFO:tasks.workunit.client.0.smithi153.stdout:9/825: truncate d1/d3a/d91/d96/de6/ff6 1121051 0 2024-04-05T03:03:05.166 INFO:tasks.workunit.client.0.smithi153.stdout:9/826: unlink d1/d10/d36/f56 0 2024-04-05T03:03:05.180 INFO:tasks.workunit.client.0.smithi153.stdout:9/827: dread d1/d10/d16/f23 [0,4194304] 0 2024-04-05T03:03:05.184 INFO:tasks.workunit.client.0.smithi153.stdout:9/828: rename d1/d10/d16/d22/d2c/d68/d81 to d1/d9/d1b/de2/d125 0 2024-04-05T03:03:05.187 INFO:tasks.workunit.client.0.smithi153.stdout:9/829: rename d1/d10/d16/d22/d2c/d68/da9 to d1/d10/d16/d120/d126 0 2024-04-05T03:03:05.190 INFO:tasks.workunit.client.0.smithi153.stdout:0/920: sync 2024-04-05T03:03:05.190 INFO:tasks.workunit.client.0.smithi153.stdout:9/830: mknod d1/d10/d36/d71/c127 0 2024-04-05T03:03:05.196 INFO:tasks.workunit.client.0.smithi153.stdout:0/921: read d9/db/d42/d7e/db6/d1e/d45/d62/f7d [72365,111424] 0 2024-04-05T03:03:05.197 INFO:tasks.workunit.client.0.smithi153.stdout:9/831: write d1/d10/d16/d22/f3e [1690460,28349] 0 2024-04-05T03:03:05.199 INFO:tasks.workunit.client.0.smithi153.stdout:9/832: mknod d1/d1d/c128 0 2024-04-05T03:03:05.201 INFO:tasks.workunit.client.0.smithi153.stdout:9/833: dread - d1/d9/d1b/de2/d125/fd1 zero size 2024-04-05T03:03:05.202 INFO:tasks.workunit.client.0.smithi153.stdout:9/834: write d1/fea [211967,11625] 0 2024-04-05T03:03:05.204 INFO:tasks.workunit.client.0.smithi153.stdout:9/835: symlink d1/d3a/l129 0 2024-04-05T03:03:05.207 INFO:tasks.workunit.client.0.smithi153.stdout:9/836: creat d1/d10/d36/d57/d97/ded/f12a x:0 0 0 2024-04-05T03:03:05.208 INFO:tasks.workunit.client.0.smithi153.stdout:9/837: rename d1 to d1/d10/d16/d120/d126/d12b 22 2024-04-05T03:03:05.210 INFO:tasks.workunit.client.0.smithi153.stdout:9/838: read d1/d10/d16/d22/d2c/d68/fa1 [158827,111291] 0 2024-04-05T03:03:05.210 INFO:tasks.workunit.client.0.smithi153.stdout:0/922: dread d9/db/d42/d43/f98 [0,4194304] 0 2024-04-05T03:03:05.212 INFO:tasks.workunit.client.0.smithi153.stdout:9/839: creat d1/d10/d36/d57/d5c/f12c x:0 0 0 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:8/793: sync 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:4/955: sync 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:1/991: sync 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:5/856: sync 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:2/997: sync 2024-04-05T03:03:05.216 INFO:tasks.workunit.client.0.smithi153.stdout:3/918: sync 2024-04-05T03:03:05.217 INFO:tasks.workunit.client.0.smithi153.stdout:9/840: rename d1/d10/d16/d22/d2c to d1/d10/d36/d57/d12d 0 2024-04-05T03:03:05.218 INFO:tasks.workunit.client.0.smithi153.stdout:5/857: read - d0/d25/d4d/d9c/fef zero size 2024-04-05T03:03:05.220 INFO:tasks.workunit.client.0.smithi153.stdout:3/919: dread - d1d/d32/d93/d3c/fbb zero size 2024-04-05T03:03:05.221 INFO:tasks.workunit.client.0.smithi153.stdout:0/923: write d9/db/d42/d7e/db6/f17 [1553363,130523] 0 2024-04-05T03:03:05.221 INFO:tasks.workunit.client.0.smithi153.stdout:2/998: rmdir d9/da/d37/d8a/da8/de5 39 2024-04-05T03:03:05.227 INFO:tasks.workunit.client.0.smithi153.stdout:4/956: write da/db/d20/d45/f69 [2984457,58592] 0 2024-04-05T03:03:05.230 INFO:tasks.workunit.client.0.smithi153.stdout:5/858: dread d0/d25/d4d/d62/d5f/fbf [0,4194304] 0 2024-04-05T03:03:05.231 INFO:tasks.workunit.client.0.smithi153.stdout:9/841: mknod d1/d9/dbc/d122/c12e 0 2024-04-05T03:03:05.233 INFO:tasks.workunit.client.0.smithi153.stdout:2/999: unlink d9/d68/db9/cd4 0 2024-04-05T03:03:05.234 INFO:tasks.workunit.client.0.smithi153.stdout:0/924: mkdir d9/db/d42/d7e/db6/d1d/d61/d117 0 2024-04-05T03:03:05.235 INFO:tasks.workunit.client.0.smithi153.stdout:3/920: dwrite d1d/d32/d93/d3c/d3f/d90/db6/f113 [0,4194304] 0 2024-04-05T03:03:05.237 INFO:tasks.workunit.client.0.smithi153.stdout:1/992: rmdir d5/d8/d20/d2f/d40/d128 0 2024-04-05T03:03:05.238 INFO:tasks.workunit.client.0.smithi153.stdout:5/859: symlink d0/d25/d4d/d62/l128 0 2024-04-05T03:03:05.240 INFO:tasks.workunit.client.0.smithi153.stdout:1/993: chown d5/de2/df5/d33/f6b 1 1 2024-04-05T03:03:05.240 INFO:tasks.workunit.client.0.smithi153.stdout:3/921: write d1d/d32/d93/d3c/d3f/d7c/f114 [547866,130867] 0 2024-04-05T03:03:05.246 INFO:tasks.workunit.client.0.smithi153.stdout:0/925: getdents d9/db/d42/d7e/db6/daf/dd0 0 2024-04-05T03:03:05.247 INFO:tasks.workunit.client.0.smithi153.stdout:8/794: dread d7/d13/d19/d1b/d4f/d68/dbc/fd3 [0,4194304] 0 2024-04-05T03:03:05.248 INFO:tasks.workunit.client.0.smithi153.stdout:5/860: truncate d0/d25/d4d/d62/d5f/d86/da4/ff1 9938 0 2024-04-05T03:03:05.253 INFO:tasks.workunit.client.0.smithi153.stdout:8/795: dread - d7/d13/d19/d1b/d4f/d5f/fe9 zero size 2024-04-05T03:03:05.254 INFO:tasks.workunit.client.0.smithi153.stdout:3/922: rename d1d/d32/cc6 to d1d/d32/d93/dc2/c147 0 2024-04-05T03:03:05.254 INFO:tasks.workunit.client.0.smithi153.stdout:1/994: dwrite d5/de2/df5/d48/d96/ff7 [0,4194304] 0 2024-04-05T03:03:05.255 INFO:tasks.workunit.client.0.smithi153.stdout:4/957: dwrite da/de/d56/f70 [0,4194304] 0 2024-04-05T03:03:05.258 INFO:tasks.workunit.client.0.smithi153.stdout:1/995: chown d5/d8/d20/d2f/db8/ccf 9202 1 2024-04-05T03:03:05.260 INFO:tasks.workunit.client.0.smithi153.stdout:8/796: creat d7/dd5/ffe x:0 0 0 2024-04-05T03:03:05.262 INFO:tasks.workunit.client.0.smithi153.stdout:4/958: write da/db/dbd/ffc [2519184,15835] 0 2024-04-05T03:03:05.263 INFO:tasks.workunit.client.0.smithi153.stdout:3/923: fdatasync d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd/ff7 0 2024-04-05T03:03:05.264 INFO:tasks.workunit.client.0.smithi153.stdout:5/861: dwrite d0/d25/d4d/d62/d5f/f7e [0,4194304] 0 2024-04-05T03:03:05.268 INFO:tasks.workunit.client.0.smithi153.stdout:5/862: dread - d0/d2c/d3f/d102/f111 zero size 2024-04-05T03:03:05.269 INFO:tasks.workunit.client.0.smithi153.stdout:3/924: dread - d1d/d32/d93/f10b zero size 2024-04-05T03:03:05.271 INFO:tasks.workunit.client.0.smithi153.stdout:0/926: dread d9/db/d42/d7e/db6/d1e/d45/d62/fff [0,4194304] 0 2024-04-05T03:03:05.271 INFO:tasks.workunit.client.0.smithi153.stdout:5/863: dread - d0/d25/d4d/d9c/dc4/ff3 zero size 2024-04-05T03:03:05.279 INFO:tasks.workunit.client.0.smithi153.stdout:3/925: creat d1d/d27/d2b/f148 x:0 0 0 2024-04-05T03:03:05.279 INFO:tasks.workunit.client.0.smithi153.stdout:4/959: truncate da/db/d20/d35/d38/d76/f138 587017 0 2024-04-05T03:03:05.280 INFO:tasks.workunit.client.0.smithi153.stdout:0/927: unlink d9/d14/d67/fbf 0 2024-04-05T03:03:05.282 INFO:tasks.workunit.client.0.smithi153.stdout:1/996: link d5/de2/df5/d45/d68/da4/fac d5/d8/d20/d4b/de1/d11c/f137 0 2024-04-05T03:03:05.282 INFO:tasks.workunit.client.0.smithi153.stdout:4/960: creat da/db/d20/d35/d38/df7/f14d x:0 0 0 2024-04-05T03:03:05.285 INFO:tasks.workunit.client.0.smithi153.stdout:1/997: readlink d5/dd/l47 0 2024-04-05T03:03:05.287 INFO:tasks.workunit.client.0.smithi153.stdout:4/961: truncate da/db/d20/d35/d38/df7/f143 240972 0 2024-04-05T03:03:05.287 INFO:tasks.workunit.client.0.smithi153.stdout:5/864: dread d0/d25/d4d/d62/f80 [0,4194304] 0 2024-04-05T03:03:05.288 INFO:tasks.workunit.client.0.smithi153.stdout:1/998: dread - d5/de2/df5/d2c/d62/fd1 zero size 2024-04-05T03:03:05.290 INFO:tasks.workunit.client.0.smithi153.stdout:0/928: dwrite d9/db/d42/d43/f85 [0,4194304] 0 2024-04-05T03:03:05.291 INFO:tasks.workunit.client.0.smithi153.stdout:4/962: write da/db/d8b/fa4 [452887,58194] 0 2024-04-05T03:03:05.292 INFO:tasks.workunit.client.0.smithi153.stdout:5/865: creat d0/d25/d4d/d9c/dc7/f129 x:0 0 0 2024-04-05T03:03:05.293 INFO:tasks.workunit.client.0.smithi153.stdout:1/999: stat d5/de2/df5/d48/c5e 0 2024-04-05T03:03:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: pgmap v17: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 132 MiB/s rd, 304 MiB/s wr, 677 op/s 2024-04-05T03:03:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:05 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.298 INFO:tasks.workunit.client.0.smithi153.stdout:0/929: creat d9/db/ddf/f118 x:0 0 0 2024-04-05T03:03:05.301 INFO:tasks.workunit.client.0.smithi153.stdout:4/963: mknod da/de/d56/d78/d8f/c14e 0 2024-04-05T03:03:05.302 INFO:tasks.workunit.client.0.smithi153.stdout:5/866: symlink d0/d2c/d3f/l12a 0 2024-04-05T03:03:05.304 INFO:tasks.workunit.client.0.smithi153.stdout:5/867: chown d0/d25/d4d/d62/d5f/d86/da4/fa1 4656 1 2024-04-05T03:03:05.309 INFO:tasks.workunit.client.0.smithi153.stdout:0/930: link d9/db/cbe d9/db/d42/d7e/db6/d1e/d45/d62/dde/c119 0 2024-04-05T03:03:05.311 INFO:tasks.workunit.client.0.smithi153.stdout:5/868: dread d0/d25/d4d/fec [0,4194304] 0 2024-04-05T03:03:05.319 INFO:tasks.workunit.client.0.smithi153.stdout:0/931: dwrite d9/f30 [0,4194304] 0 2024-04-05T03:03:05.319 INFO:tasks.workunit.client.0.smithi153.stdout:4/964: dwrite da/de/d66/d111/f47 [0,4194304] 0 2024-04-05T03:03:05.321 INFO:tasks.workunit.client.0.smithi153.stdout:0/932: rename d9/db/d42/d7e/db6/d1d/d61 to d9/db/d42/d7e/db6/d1d/d61/d8a/d103/d11a 22 2024-04-05T03:03:05.324 INFO:tasks.workunit.client.0.smithi153.stdout:0/933: fdatasync d9/db/d42/d7e/d8b/fe5 0 2024-04-05T03:03:05.325 INFO:tasks.workunit.client.0.smithi153.stdout:4/965: rename da/de/d66/d111/ld7 to da/de/d66/d111/l14f 0 2024-04-05T03:03:05.328 INFO:tasks.workunit.client.0.smithi153.stdout:0/934: unlink d9/db/l9c 0 2024-04-05T03:03:05.337 INFO:tasks.workunit.client.0.smithi153.stdout:5/869: dread d0/d25/d4d/d62/dbb/f10e [0,4194304] 0 2024-04-05T03:03:05.341 INFO:tasks.workunit.client.0.smithi153.stdout:5/870: getdents d0/d25/d4d/d62/d5f/d86/da4 0 2024-04-05T03:03:05.343 INFO:tasks.workunit.client.0.smithi153.stdout:5/871: unlink d0/d25/d4d/d62/dbb/c114 0 2024-04-05T03:03:05.344 INFO:tasks.workunit.client.0.smithi153.stdout:0/935: rmdir d9/db 39 2024-04-05T03:03:05.344 INFO:tasks.workunit.client.0.smithi153.stdout:4/966: write da/f1d [3317437,101329] 0 2024-04-05T03:03:05.345 INFO:tasks.workunit.client.0.smithi153.stdout:5/872: creat d0/d25/d4d/d9c/d10f/f12b x:0 0 0 2024-04-05T03:03:05.346 INFO:tasks.workunit.client.0.smithi153.stdout:5/873: chown d0/d2c/c43 1965195 1 2024-04-05T03:03:05.346 INFO:tasks.workunit.client.0.smithi153.stdout:4/967: rmdir da/db/d20/d3c/d91/dd5 39 2024-04-05T03:03:05.348 INFO:tasks.workunit.client.0.smithi153.stdout:5/874: unlink d0/d9/f21 0 2024-04-05T03:03:05.350 INFO:tasks.workunit.client.0.smithi153.stdout:5/875: fdatasync d0/d25/d4d/d62/d5f/fbf 0 2024-04-05T03:03:05.350 INFO:tasks.workunit.client.0.smithi153.stdout:5/876: dread - d0/d4e/db6/f11f zero size 2024-04-05T03:03:05.355 INFO:tasks.workunit.client.0.smithi153.stdout:4/968: write da/db/f79 [610490,80464] 0 2024-04-05T03:03:05.360 INFO:tasks.workunit.client.0.smithi153.stdout:4/969: dread - da/de/d66/ff0 zero size 2024-04-05T03:03:05.367 INFO:tasks.workunit.client.0.smithi153.stdout:4/970: link da/db/c25 da/de/d66/d111/c150 0 2024-04-05T03:03:05.368 INFO:tasks.workunit.client.0.smithi153.stdout:5/877: dread d0/f3c [0,4194304] 0 2024-04-05T03:03:05.377 INFO:tasks.workunit.client.0.smithi153.stdout:5/878: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T03:03:05.381 INFO:tasks.workunit.client.0.smithi153.stdout:5/879: truncate d0/d9/d22/f33 905883 0 2024-04-05T03:03:05.384 INFO:tasks.workunit.client.0.smithi153.stdout:4/971: dread da/db/d20/d35/d38/f59 [0,4194304] 0 2024-04-05T03:03:05.385 INFO:tasks.workunit.client.0.smithi153.stdout:4/972: chown da/db/d20/d35/d41/d67/f12b 9 1 2024-04-05T03:03:05.387 INFO:tasks.workunit.client.0.smithi153.stdout:5/880: creat d0/d25/d4d/d9c/f12c x:0 0 0 2024-04-05T03:03:05.391 INFO:tasks.workunit.client.0.smithi153.stdout:4/973: read - da/de/d56/d78/d8f/d68/f95 zero size 2024-04-05T03:03:05.396 INFO:tasks.workunit.client.0.smithi153.stdout:4/974: rename da/db/d8b to da/db/d20/d45/db1/d151 0 2024-04-05T03:03:05.398 INFO:tasks.workunit.client.0.smithi153.stdout:4/975: mknod da/de/da1/c152 0 2024-04-05T03:03:05.402 INFO:tasks.workunit.client.0.smithi153.stdout:4/976: unlink da/db/d20/d3c/d81/leb 0 2024-04-05T03:03:05.402 INFO:tasks.workunit.client.0.smithi153.stdout:5/881: dwrite d0/d25/d4d/d62/d37/f6a [0,4194304] 0 2024-04-05T03:03:05.406 INFO:tasks.workunit.client.0.smithi153.stdout:4/977: chown da/db/d20/d35/d41/da8/dc1/f9b 2646176 1 2024-04-05T03:03:05.411 INFO:tasks.workunit.client.0.smithi153.stdout:4/978: rename da/db/d20/d35/d38/d76/d90/fca to da/de/d66/f153 0 2024-04-05T03:03:05.415 INFO:tasks.workunit.client.0.smithi153.stdout:4/979: symlink da/db/d20/d35/d41/da8/l154 0 2024-04-05T03:03:05.416 INFO:tasks.workunit.client.0.smithi153.stdout:5/882: write d0/d2c/d5e/d11a/f124 [3895711,130085] 0 2024-04-05T03:03:05.416 INFO:tasks.workunit.client.0.smithi153.stdout:4/980: write f7 [6232342,40281] 0 2024-04-05T03:03:05.417 INFO:tasks.workunit.client.0.smithi153.stdout:5/883: chown d0/d9/d22/d125 5929 1 2024-04-05T03:03:05.422 INFO:tasks.workunit.client.0.smithi153.stdout:5/884: getdents d0/d25/d4d/d9c/db7 0 2024-04-05T03:03:05.429 INFO:tasks.workunit.client.0.smithi153.stdout:5/885: write d0/d25/d4d/d9c/dc7/dc9/fcc [619095,20854] 0 2024-04-05T03:03:05.432 INFO:tasks.workunit.client.0.smithi153.stdout:5/886: creat d0/d2c/d5e/d11a/f12d x:0 0 0 2024-04-05T03:03:05.434 INFO:tasks.workunit.client.0.smithi153.stdout:4/981: dread da/db/d20/d45/f69 [0,4194304] 0 2024-04-05T03:03:05.436 INFO:tasks.workunit.client.0.smithi153.stdout:4/982: read da/db/d20/d45/f69 [611733,92090] 0 2024-04-05T03:03:05.439 INFO:tasks.workunit.client.0.smithi153.stdout:5/887: dwrite d0/d2c/f110 [0,4194304] 0 2024-04-05T03:03:05.439 INFO:tasks.workunit.client.0.smithi153.stdout:4/983: fsync da/de/d66/f153 0 2024-04-05T03:03:05.443 INFO:tasks.workunit.client.0.smithi153.stdout:4/984: creat da/db/d20/d45/db1/d151/d10f/f155 x:0 0 0 2024-04-05T03:03:05.444 INFO:tasks.workunit.client.0.smithi153.stdout:5/888: fdatasync d0/d25/d4d/d62/d37/f4b 0 2024-04-05T03:03:05.454 INFO:tasks.workunit.client.0.smithi153.stdout:5/889: dwrite d0/d2c/d3f/d102/f111 [0,4194304] 0 2024-04-05T03:03:05.461 INFO:tasks.workunit.client.0.smithi153.stdout:4/985: dwrite da/de/d56/d6a/d2d/f2e [0,4194304] 0 2024-04-05T03:03:05.466 INFO:tasks.workunit.client.0.smithi153.stdout:4/986: unlink da/db/d20/d3c/d81/db6/lde 0 2024-04-05T03:03:05.466 INFO:tasks.workunit.client.0.smithi153.stdout:5/890: truncate d0/d25/d4d/d62/d5f/d86/f104 369334 0 2024-04-05T03:03:05.470 INFO:tasks.workunit.client.0.smithi153.stdout:5/891: read d0/d25/f2b [3403031,92995] 0 2024-04-05T03:03:05.479 INFO:tasks.workunit.client.0.smithi153.stdout:4/987: dwrite da/f114 [0,4194304] 0 2024-04-05T03:03:05.481 INFO:tasks.workunit.client.0.smithi153.stdout:4/988: rmdir da/db/dbd 39 2024-04-05T03:03:05.485 INFO:tasks.workunit.client.0.smithi153.stdout:4/989: chown da/db/d20/d3c/c3f 1 1 2024-04-05T03:03:05.486 INFO:tasks.workunit.client.0.smithi153.stdout:4/990: chown da/db/d20/d3c/d91/fe7 31 1 2024-04-05T03:03:05.490 INFO:tasks.workunit.client.0.smithi153.stdout:4/991: creat da/db/d20/d45/db1/d151/d10f/d11f/f156 x:0 0 0 2024-04-05T03:03:05.495 INFO:tasks.workunit.client.0.smithi153.stdout:4/992: dwrite da/f1d [0,4194304] 0 2024-04-05T03:03:05.497 INFO:tasks.workunit.client.0.smithi153.stdout:4/993: write da/db/f79 [1386300,117705] 0 2024-04-05T03:03:05.500 INFO:tasks.workunit.client.0.smithi153.stdout:4/994: write da/de/da1/fc0 [5705848,121503] 0 2024-04-05T03:03:05.507 INFO:tasks.workunit.client.0.smithi153.stdout:4/995: getdents da/db/d20/d3c/d81/db6 0 2024-04-05T03:03:05.508 INFO:tasks.workunit.client.0.smithi153.stdout:4/996: chown da/de/d66/d111/d86/f11e 0 1 2024-04-05T03:03:05.510 INFO:tasks.workunit.client.0.smithi153.stdout:4/997: symlink da/db/d20/d45/dc3/l157 0 2024-04-05T03:03:05.511 INFO:tasks.workunit.client.0.smithi153.stdout:4/998: chown da/de/d56/d78/d8f/d68/c140 28 1 2024-04-05T03:03:05.515 INFO:tasks.workunit.client.0.smithi153.stdout:4/999: rename da/db/de6 to da/db/d20/d35/d41/da8/dc1/d158 0 2024-04-05T03:03:05.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: pgmap v17: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 132 MiB/s rd, 304 MiB/s wr, 677 op/s 2024-04-05T03:03:05.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:05.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:05 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:05.600 INFO:tasks.workunit.client.0.smithi153.stdout:9/842: sync 2024-04-05T03:03:05.600 INFO:tasks.workunit.client.0.smithi153.stdout:3/926: sync 2024-04-05T03:03:05.600 INFO:tasks.workunit.client.0.smithi153.stdout:0/936: sync 2024-04-05T03:03:05.601 INFO:tasks.workunit.client.0.smithi153.stdout:8/797: sync 2024-04-05T03:03:05.602 INFO:tasks.workunit.client.0.smithi153.stdout:0/937: write d9/f30 [568486,13880] 0 2024-04-05T03:03:05.603 INFO:tasks.workunit.client.0.smithi153.stdout:9/843: symlink d1/d6c/l12f 0 2024-04-05T03:03:05.606 INFO:tasks.workunit.client.0.smithi153.stdout:0/938: readlink d9/db/d42/d7e/db6/d1e/d84/la8 0 2024-04-05T03:03:05.606 INFO:tasks.workunit.client.0.smithi153.stdout:8/798: truncate d7/d13/d19/d1b/d4f/d72/ff8 381643 0 2024-04-05T03:03:05.607 INFO:tasks.workunit.client.0.smithi153.stdout:9/844: unlink d1/d3a/d91/d96/dbd/dbe/fda 0 2024-04-05T03:03:05.608 INFO:tasks.workunit.client.0.smithi153.stdout:9/845: chown d1/d9/d1b/de2/d125/f7a 21726331 1 2024-04-05T03:03:05.609 INFO:tasks.workunit.client.0.smithi153.stdout:3/927: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/ff0 to d1d/d32/d93/d3c/d3f/d90/db6/f149 0 2024-04-05T03:03:05.610 INFO:tasks.workunit.client.0.smithi153.stdout:0/939: creat d9/d14/d67/d6f/f11b x:0 0 0 2024-04-05T03:03:05.612 INFO:tasks.workunit.client.0.smithi153.stdout:8/799: unlink d7/d13/d19/d1b/d4f/d68/cea 0 2024-04-05T03:03:05.613 INFO:tasks.workunit.client.0.smithi153.stdout:9/846: creat d1/d9/d1b/f130 x:0 0 0 2024-04-05T03:03:05.613 INFO:tasks.workunit.client.0.smithi153.stdout:3/928: dread d1d/d32/d93/d3c/d3f/d7c/f114 [0,4194304] 0 2024-04-05T03:03:05.614 INFO:tasks.workunit.client.0.smithi153.stdout:8/800: fsync d7/fa 0 2024-04-05T03:03:05.615 INFO:tasks.workunit.client.0.smithi153.stdout:0/940: symlink d9/db/d42/d7e/db6/d1e/d45/d62/dd2/l11c 0 2024-04-05T03:03:05.616 INFO:tasks.workunit.client.0.smithi153.stdout:8/801: chown d7/d13/d19/l29 330 1 2024-04-05T03:03:05.617 INFO:tasks.workunit.client.0.smithi153.stdout:9/847: truncate d1/d10/d16/fdc 930733 0 2024-04-05T03:03:05.618 INFO:tasks.workunit.client.0.smithi153.stdout:8/802: read f3 [1484992,28962] 0 2024-04-05T03:03:05.618 INFO:tasks.workunit.client.0.smithi153.stdout:0/941: dread d9/f1a [0,4194304] 0 2024-04-05T03:03:05.619 INFO:tasks.workunit.client.0.smithi153.stdout:3/929: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/dbd to d1d/d10d/d116/d14a 0 2024-04-05T03:03:05.619 INFO:tasks.workunit.client.0.smithi153.stdout:9/848: read d1/d6c/f94 [2352978,126042] 0 2024-04-05T03:03:05.621 INFO:tasks.workunit.client.0.smithi153.stdout:9/849: readlink d1/d10/d16/l59 0 2024-04-05T03:03:05.622 INFO:tasks.workunit.client.0.smithi153.stdout:3/930: creat d1d/d10d/d116/f14b x:0 0 0 2024-04-05T03:03:05.623 INFO:tasks.workunit.client.0.smithi153.stdout:0/942: mkdir d9/db/d42/d7e/db6/d1e/d84/d11d 0 2024-04-05T03:03:05.624 INFO:tasks.workunit.client.0.smithi153.stdout:0/943: readlink d9/db/d42/d7e/db6/l20 0 2024-04-05T03:03:05.625 INFO:tasks.workunit.client.0.smithi153.stdout:9/850: fdatasync d1/fcd 0 2024-04-05T03:03:05.627 INFO:tasks.workunit.client.0.smithi153.stdout:0/944: chown d9/db/d42/d7e/db6/daf/dd0/f10a 85 1 2024-04-05T03:03:05.628 INFO:tasks.workunit.client.0.smithi153.stdout:9/851: chown d1/d10/d36/d57/d97/f114 137036 1 2024-04-05T03:03:05.629 INFO:tasks.workunit.client.0.smithi153.stdout:3/931: dwrite d1d/f6a [4194304,4194304] 0 2024-04-05T03:03:05.630 INFO:tasks.workunit.client.0.smithi153.stdout:9/852: dread - d1/d9/d1b/f130 zero size 2024-04-05T03:03:05.633 INFO:tasks.workunit.client.0.smithi153.stdout:3/932: mkdir d1d/d32/d93/dc2/d14c 0 2024-04-05T03:03:05.639 INFO:tasks.workunit.client.0.smithi153.stdout:0/945: dwrite d9/db/d42/d7e/db6/d1d/d61/f2b [0,4194304] 0 2024-04-05T03:03:05.644 INFO:tasks.workunit.client.0.smithi153.stdout:9/853: write d1/d9/d1b/f75 [2562005,36354] 0 2024-04-05T03:03:05.645 INFO:tasks.workunit.client.0.smithi153.stdout:8/803: dwrite d7/d2a/f2c [0,4194304] 0 2024-04-05T03:03:05.650 INFO:tasks.workunit.client.0.smithi153.stdout:9/854: mknod d1/d10/d16/c131 0 2024-04-05T03:03:05.651 INFO:tasks.workunit.client.0.smithi153.stdout:3/933: dread d1d/d32/d93/d3c/d94/fd1 [0,4194304] 0 2024-04-05T03:03:05.651 INFO:tasks.workunit.client.0.smithi153.stdout:9/855: dread - d1/f110 zero size 2024-04-05T03:03:05.655 INFO:tasks.workunit.client.0.smithi153.stdout:3/934: creat d1d/d32/d93/d3c/d3f/d90/da4/f14d x:0 0 0 2024-04-05T03:03:05.657 INFO:tasks.workunit.client.0.smithi153.stdout:0/946: rename d9/db/d42/d7e/db6/d1e/d45/d62/dde/c119 to d9/db/d42/d7e/db6/d1d/c11e 0 2024-04-05T03:03:05.659 INFO:tasks.workunit.client.0.smithi153.stdout:0/947: write d9/db/d42/d7e/db6/d1e/d45/dae/f111 [195813,82735] 0 2024-04-05T03:03:05.661 INFO:tasks.workunit.client.0.smithi153.stdout:8/804: dread d7/fc [0,4194304] 0 2024-04-05T03:03:05.661 INFO:tasks.workunit.client.0.smithi153.stdout:9/856: rename d1/d3a/d91/f9b to d1/f132 0 2024-04-05T03:03:05.663 INFO:tasks.workunit.client.0.smithi153.stdout:8/805: write d7/fe [1322318,15909] 0 2024-04-05T03:03:05.664 INFO:tasks.workunit.client.0.smithi153.stdout:9/857: mkdir d1/d9/d1b/de2/d133 0 2024-04-05T03:03:05.666 INFO:tasks.workunit.client.0.smithi153.stdout:0/948: getdents d9/db/d42/d7e 0 2024-04-05T03:03:05.666 INFO:tasks.workunit.client.0.smithi153.stdout:8/806: readlink d7/d13/l48 0 2024-04-05T03:03:05.666 INFO:tasks.workunit.client.0.smithi153.stdout:0/949: chown d9/db/fd 326429 1 2024-04-05T03:03:05.670 INFO:tasks.workunit.client.0.smithi153.stdout:0/950: mkdir d9/db/d42/d43/d89/d11f 0 2024-04-05T03:03:05.676 INFO:tasks.workunit.client.0.smithi153.stdout:8/807: write d7/d13/d19/fb5 [413540,72580] 0 2024-04-05T03:03:05.692 INFO:tasks.workunit.client.0.smithi153.stdout:9/858: dwrite d1/d3a/d91/f9f [4194304,4194304] 0 2024-04-05T03:03:05.692 INFO:tasks.workunit.client.0.smithi153.stdout:0/951: dwrite d9/db/d42/d7e/db6/d1e/d45/fc2 [4194304,4194304] 0 2024-04-05T03:03:05.698 INFO:tasks.workunit.client.0.smithi153.stdout:8/808: dwrite d7/d46/db1/fe4 [4194304,4194304] 0 2024-04-05T03:03:05.699 INFO:tasks.workunit.client.0.smithi153.stdout:0/952: dread d9/db/d42/d43/f82 [0,4194304] 0 2024-04-05T03:03:05.718 INFO:tasks.workunit.client.0.smithi153.stdout:9/859: write d1/d6c/f94 [3884236,83223] 0 2024-04-05T03:03:05.719 INFO:tasks.workunit.client.0.smithi153.stdout:9/860: chown d1/d10 65367734 1 2024-04-05T03:03:05.721 INFO:tasks.workunit.client.0.smithi153.stdout:8/809: unlink d7/d2a/f2c 0 2024-04-05T03:03:05.722 INFO:tasks.workunit.client.0.smithi153.stdout:0/953: creat d9/db/d42/d7e/db6/d1e/d45/d62/dd2/f120 x:0 0 0 2024-04-05T03:03:05.725 INFO:tasks.workunit.client.0.smithi153.stdout:9/861: rmdir d1/d10/d36/d57/d12d/d68 39 2024-04-05T03:03:05.726 INFO:tasks.workunit.client.0.smithi153.stdout:0/954: symlink d9/db/d42/d7e/db6/d1d/d61/l121 0 2024-04-05T03:03:05.727 INFO:tasks.workunit.client.0.smithi153.stdout:9/862: mkdir d1/d10/d36/d134 0 2024-04-05T03:03:05.727 INFO:tasks.workunit.client.0.smithi153.stdout:9/863: write d1/d10/f4c [1409506,50418] 0 2024-04-05T03:03:05.728 INFO:tasks.workunit.client.0.smithi153.stdout:0/955: creat d9/db/d42/d7e/f122 x:0 0 0 2024-04-05T03:03:05.732 INFO:tasks.workunit.client.0.smithi153.stdout:9/864: truncate d1/d10/d36/d57/d12d/d68/fa7 856604 0 2024-04-05T03:03:05.734 INFO:tasks.workunit.client.0.smithi153.stdout:9/865: mknod d1/d10/d36/d57/d97/c135 0 2024-04-05T03:03:05.734 INFO:tasks.workunit.client.0.smithi153.stdout:9/866: stat d1/d3a 0 2024-04-05T03:03:05.736 INFO:tasks.workunit.client.0.smithi153.stdout:9/867: creat d1/d9/d1b/f136 x:0 0 0 2024-04-05T03:03:05.738 INFO:tasks.workunit.client.0.smithi153.stdout:9/868: rename d1/d10/d16/d120/d126/fc7 to d1/d3a/d91/d96/f137 0 2024-04-05T03:03:05.739 INFO:tasks.workunit.client.0.smithi153.stdout:9/869: readlink d1/d9/d1b/de2/d125/lec 0 2024-04-05T03:03:05.744 INFO:tasks.workunit.client.0.smithi153.stdout:9/870: dwrite d1/d6c/f104 [0,4194304] 0 2024-04-05T03:03:05.744 INFO:tasks.workunit.client.0.smithi153.stdout:9/871: fdatasync d1/d9/d1b/f136 0 2024-04-05T03:03:05.748 INFO:tasks.workunit.client.0.smithi153.stdout:9/872: mkdir d1/d9/d138 0 2024-04-05T03:03:05.755 INFO:tasks.workunit.client.0.smithi153.stdout:9/873: symlink d1/l139 0 2024-04-05T03:03:05.755 INFO:tasks.workunit.client.0.smithi153.stdout:9/874: stat d1/d1d/db2/ff2 0 2024-04-05T03:03:05.757 INFO:tasks.workunit.client.0.smithi153.stdout:9/875: creat d1/d10/d36/d57/d5c/f13a x:0 0 0 2024-04-05T03:03:05.760 INFO:tasks.workunit.client.0.smithi153.stdout:0/956: link d9/db/d42/d7e/c107 d9/db/d42/d7e/db6/daf/dd0/c123 0 2024-04-05T03:03:05.762 INFO:tasks.workunit.client.0.smithi153.stdout:9/876: getdents d1/d10/d16/d120 0 2024-04-05T03:03:05.762 INFO:tasks.workunit.client.0.smithi153.stdout:9/877: chown d1/d9/d1b/de2 139238 1 2024-04-05T03:03:05.765 INFO:tasks.workunit.client.0.smithi153.stdout:9/878: fsync d1/d10/d36/fd3 0 2024-04-05T03:03:05.766 INFO:tasks.workunit.client.0.smithi153.stdout:9/879: read d1/d3a/d91/d96/dfc/f117 [442255,78150] 0 2024-04-05T03:03:05.768 INFO:tasks.workunit.client.0.smithi153.stdout:5/892: sync 2024-04-05T03:03:05.772 INFO:tasks.workunit.client.0.smithi153.stdout:5/893: mkdir d0/d25/d4d/d12e 0 2024-04-05T03:03:05.772 INFO:tasks.workunit.client.0.smithi153.stdout:0/957: truncate d9/db/d42/d7e/db6/d1e/d45/d62/dde/fa4 1954600 0 2024-04-05T03:03:05.774 INFO:tasks.workunit.client.0.smithi153.stdout:9/880: mkdir d1/d10/d16/d13b 0 2024-04-05T03:03:05.774 INFO:tasks.workunit.client.0.smithi153.stdout:0/958: rmdir d9/db/d42/d7e/db6/d1e/d45/d62/dde 39 2024-04-05T03:03:05.775 INFO:tasks.workunit.client.0.smithi153.stdout:9/881: rmdir d1/d10/d36/d71 39 2024-04-05T03:03:05.776 INFO:tasks.workunit.client.0.smithi153.stdout:0/959: fsync d9/db/d42/d7e/db6/f39 0 2024-04-05T03:03:05.777 INFO:tasks.workunit.client.0.smithi153.stdout:9/882: readlink d1/d10/d36/d57/d5c/l70 0 2024-04-05T03:03:05.781 INFO:tasks.workunit.client.0.smithi153.stdout:0/960: creat d9/db/d42/d7e/d8b/f124 x:0 0 0 2024-04-05T03:03:05.783 INFO:tasks.workunit.client.0.smithi153.stdout:0/961: rmdir d9/db/d42/d43/de4 39 2024-04-05T03:03:05.786 INFO:tasks.workunit.client.0.smithi153.stdout:0/962: rename d9/d14/d67/d6f/f9d to d9/db/d42/d7e/db6/d1e/f125 0 2024-04-05T03:03:05.788 INFO:tasks.workunit.client.0.smithi153.stdout:0/963: symlink d9/db/d42/d7e/db6/d1d/d61/l126 0 2024-04-05T03:03:05.791 INFO:tasks.workunit.client.0.smithi153.stdout:0/964: mknod d9/db/d42/d7e/db6/d1e/d84/c127 0 2024-04-05T03:03:05.791 INFO:tasks.workunit.client.0.smithi153.stdout:9/883: dwrite d1/d10/d16/f7f [0,4194304] 0 2024-04-05T03:03:05.795 INFO:tasks.workunit.client.0.smithi153.stdout:9/884: mkdir d1/d9/d138/d13c 0 2024-04-05T03:03:05.796 INFO:tasks.workunit.client.0.smithi153.stdout:0/965: unlink d9/d14/d67/d6f/d10e/c110 0 2024-04-05T03:03:05.797 INFO:tasks.workunit.client.0.smithi153.stdout:9/885: readlink d1/d1d/lee 0 2024-04-05T03:03:05.801 INFO:tasks.workunit.client.0.smithi153.stdout:0/966: creat d9/db/d42/d7e/db6/d1e/d45/d62/de7/f128 x:0 0 0 2024-04-05T03:03:05.813 INFO:tasks.workunit.client.0.smithi153.stdout:9/886: write d1/d6c/fc0 [841443,40903] 0 2024-04-05T03:03:05.816 INFO:tasks.workunit.client.0.smithi153.stdout:9/887: rename d1/d10/d36/lb6 to d1/d10/d16/d120/l13d 0 2024-04-05T03:03:05.818 INFO:tasks.workunit.client.0.smithi153.stdout:0/967: dwrite d9/db/d42/d7e/db6/d1e/f54 [0,4194304] 0 2024-04-05T03:03:05.833 INFO:tasks.workunit.client.0.smithi153.stdout:9/888: dread d1/f4 [0,4194304] 0 2024-04-05T03:03:05.833 INFO:tasks.workunit.client.0.smithi153.stdout:9/889: stat d1/d6c/f104 0 2024-04-05T03:03:05.836 INFO:tasks.workunit.client.0.smithi153.stdout:9/890: symlink d1/d10/d36/d57/d5c/l13e 0 2024-04-05T03:03:05.836 INFO:tasks.workunit.client.0.smithi153.stdout:0/968: fdatasync d9/db/d42/d7e/db6/d1e/d45/d62/f7d 0 2024-04-05T03:03:05.837 INFO:tasks.workunit.client.0.smithi153.stdout:0/969: dread - d9/d14/d67/d6f/f11b zero size 2024-04-05T03:03:05.838 INFO:tasks.workunit.client.0.smithi153.stdout:9/891: dread - d1/d3a/d91/d96/f137 zero size 2024-04-05T03:03:05.841 INFO:tasks.workunit.client.0.smithi153.stdout:9/892: dread d1/d10/d36/d57/d12d/d68/fa7 [0,4194304] 0 2024-04-05T03:03:05.843 INFO:tasks.workunit.client.0.smithi153.stdout:9/893: stat d1/d10/d36/d57/lfb 0 2024-04-05T03:03:05.843 INFO:tasks.workunit.client.0.smithi153.stdout:9/894: dread - d1/d3a/d91/d96/dbd/fc2 zero size 2024-04-05T03:03:05.845 INFO:tasks.workunit.client.0.smithi153.stdout:0/970: rename d9/db/d42/d7e/db6/d1d/c11e to d9/db/d42/d7e/db6/d1e/d45/d62/dde/d6d/c129 0 2024-04-05T03:03:05.847 INFO:tasks.workunit.client.0.smithi153.stdout:0/971: read d9/db/f37 [3942456,50810] 0 2024-04-05T03:03:05.853 INFO:tasks.workunit.client.0.smithi153.stdout:0/972: dwrite d9/db/d42/d7e/db6/d1e/d45/fc2 [0,4194304] 0 2024-04-05T03:03:05.856 INFO:tasks.workunit.client.0.smithi153.stdout:0/973: creat d9/f12a x:0 0 0 2024-04-05T03:03:05.861 INFO:tasks.workunit.client.0.smithi153.stdout:0/974: creat d9/db/d42/f12b x:0 0 0 2024-04-05T03:03:05.863 INFO:tasks.workunit.client.0.smithi153.stdout:9/895: dread d1/d10/d36/d57/d12d/d68/fd5 [0,4194304] 0 2024-04-05T03:03:05.865 INFO:tasks.workunit.client.0.smithi153.stdout:0/975: creat d9/db/d42/d7e/db6/d1e/d45/dae/df4/f12c x:0 0 0 2024-04-05T03:03:05.867 INFO:tasks.workunit.client.0.smithi153.stdout:9/896: chown d1/d10/d36/d71/c116 546 1 2024-04-05T03:03:05.869 INFO:tasks.workunit.client.0.smithi153.stdout:0/976: symlink d9/db/d42/d7e/db6/d1d/l12d 0 2024-04-05T03:03:05.870 INFO:tasks.workunit.client.0.smithi153.stdout:9/897: symlink d1/d10/d36/l13f 0 2024-04-05T03:03:05.874 INFO:tasks.workunit.client.0.smithi153.stdout:0/977: creat d9/db/d42/d7e/db6/d1e/d45/d62/f12e x:0 0 0 2024-04-05T03:03:05.877 INFO:tasks.workunit.client.0.smithi153.stdout:9/898: dwrite d1/d10/d36/d57/d5c/f12c [0,4194304] 0 2024-04-05T03:03:05.880 INFO:tasks.workunit.client.0.smithi153.stdout:9/899: mknod d1/d6c/c140 0 2024-04-05T03:03:05.882 INFO:tasks.workunit.client.0.smithi153.stdout:0/978: dwrite d9/db/d42/d43/de4/ffc [0,4194304] 0 2024-04-05T03:03:05.890 INFO:tasks.workunit.client.0.smithi153.stdout:9/900: dwrite d1/d1d/f5a [0,4194304] 0 2024-04-05T03:03:05.893 INFO:tasks.workunit.client.0.smithi153.stdout:3/935: sync 2024-04-05T03:03:05.895 INFO:tasks.workunit.client.0.smithi153.stdout:9/901: creat d1/d3a/d91/d96/dfc/f141 x:0 0 0 2024-04-05T03:03:05.901 INFO:tasks.workunit.client.0.smithi153.stdout:3/936: write d1d/d32/d93/d3c/d3f/d90/da4/f130 [982884,24614] 0 2024-04-05T03:03:05.901 INFO:tasks.workunit.client.0.smithi153.stdout:0/979: dwrite d9/db/d42/d7e/db6/d1e/d45/d62/dde/f9b [0,4194304] 0 2024-04-05T03:03:05.903 INFO:tasks.workunit.client.0.smithi153.stdout:3/937: write d1d/d27/d2b/f71 [489537,21297] 0 2024-04-05T03:03:05.912 INFO:tasks.workunit.client.0.smithi153.stdout:0/980: truncate d9/db/d42/d7e/db6/d1e/f80 1650011 0 2024-04-05T03:03:05.918 INFO:tasks.workunit.client.0.smithi153.stdout:3/938: dwrite d1d/d32/d93/d3c/d3f/d90/d92/fac [0,4194304] 0 2024-04-05T03:03:05.923 INFO:tasks.workunit.client.0.smithi153.stdout:3/939: symlink d1d/d32/d93/d3c/d3f/d90/da4/l14e 0 2024-04-05T03:03:05.931 INFO:tasks.workunit.client.0.smithi153.stdout:3/940: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/fa6 [0,4194304] 0 2024-04-05T03:03:05.935 INFO:tasks.workunit.client.0.smithi153.stdout:3/941: creat d1d/d32/d93/f14f x:0 0 0 2024-04-05T03:03:05.939 INFO:tasks.workunit.client.0.smithi153.stdout:3/942: symlink d1d/d32/d93/d3c/dc5/l150 0 2024-04-05T03:03:05.951 INFO:tasks.workunit.client.0.smithi153.stdout:3/943: write d1d/d32/d93/d3c/f7b [2356678,76989] 0 2024-04-05T03:03:05.953 INFO:tasks.workunit.client.0.smithi153.stdout:3/944: mknod d1d/ddf/c151 0 2024-04-05T03:03:05.954 INFO:tasks.workunit.client.0.smithi153.stdout:3/945: stat d1d/d27/fa1 0 2024-04-05T03:03:05.955 INFO:tasks.workunit.client.0.smithi153.stdout:3/946: write d1d/d27/d2b/fc9 [294258,20948] 0 2024-04-05T03:03:05.957 INFO:tasks.workunit.client.0.smithi153.stdout:3/947: rmdir d1d/d32/d93/d3c/d3f/d7c 39 2024-04-05T03:03:05.960 INFO:tasks.workunit.client.0.smithi153.stdout:3/948: creat d1d/d32/df9/f152 x:0 0 0 2024-04-05T03:03:05.961 INFO:tasks.workunit.client.0.smithi153.stdout:3/949: chown d1d/d32/d93/d3c/fd9 15 1 2024-04-05T03:03:05.963 INFO:tasks.workunit.client.0.smithi153.stdout:3/950: mknod d1d/d32/d93/d3c/d3f/d9c/c153 0 2024-04-05T03:03:05.970 INFO:tasks.workunit.client.0.smithi153.stdout:3/951: write d1d/d10d/d116/d14a/ff7 [216416,62333] 0 2024-04-05T03:03:05.972 INFO:tasks.workunit.client.0.smithi153.stdout:3/952: symlink d1d/d32/d93/d3c/l154 0 2024-04-05T03:03:05.973 INFO:tasks.workunit.client.0.smithi153.stdout:3/953: creat d1d/d32/df9/d104/f155 x:0 0 0 2024-04-05T03:03:05.975 INFO:tasks.workunit.client.0.smithi153.stdout:3/954: chown d1d/ce5 126637718 1 2024-04-05T03:03:05.982 INFO:tasks.workunit.client.0.smithi153.stdout:3/955: rmdir d1d/d32/d93/d3c/d3f/d90/d12d 0 2024-04-05T03:03:05.986 INFO:tasks.workunit.client.0.smithi153.stdout:3/956: dread d1d/d32/d93/d3c/d3f/d90/d92/fac [0,4194304] 0 2024-04-05T03:03:05.987 INFO:tasks.workunit.client.0.smithi153.stdout:3/957: fsync d1d/d27/d2b/f71 0 2024-04-05T03:03:05.990 INFO:tasks.workunit.client.0.smithi153.stdout:3/958: rename d1d/d32/c37 to d1d/d32/d93/d3c/dc5/c156 0 2024-04-05T03:03:05.992 INFO:tasks.workunit.client.0.smithi153.stdout:3/959: write d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/f118 [47846,14832] 0 2024-04-05T03:03:06.009 INFO:tasks.workunit.client.0.smithi153.stdout:3/960: dwrite d1d/d32/d93/d3c/d3f/d9c/faa [0,4194304] 0 2024-04-05T03:03:06.010 INFO:tasks.workunit.client.0.smithi153.stdout:8/810: sync 2024-04-05T03:03:06.017 INFO:tasks.workunit.client.0.smithi153.stdout:3/961: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/f157 x:0 0 0 2024-04-05T03:03:06.017 INFO:tasks.workunit.client.0.smithi153.stdout:8/811: write d7/d13/f8e [1406647,60363] 0 2024-04-05T03:03:06.019 INFO:tasks.workunit.client.0.smithi153.stdout:3/962: fdatasync d1d/d12a/d138/f115 0 2024-04-05T03:03:06.024 INFO:tasks.workunit.client.0.smithi153.stdout:3/963: creat d1d/d32/d93/d3c/dc5/f158 x:0 0 0 2024-04-05T03:03:06.026 INFO:tasks.workunit.client.0.smithi153.stdout:8/812: link d7/d13/dab/fdc d7/d13/d19/d1b/d4f/d5f/fff 0 2024-04-05T03:03:06.031 INFO:tasks.workunit.client.0.smithi153.stdout:3/964: dread d1d/d32/d93/d3c/d3f/d9c/f102 [0,4194304] 0 2024-04-05T03:03:06.042 INFO:tasks.workunit.client.0.smithi153.stdout:8/813: dwrite d7/d85/f4a [4194304,4194304] 0 2024-04-05T03:03:06.047 INFO:tasks.workunit.client.0.smithi153.stdout:3/965: link d1d/l23 d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/l159 0 2024-04-05T03:03:06.047 INFO:tasks.workunit.client.0.smithi153.stdout:8/814: symlink d7/d85/l100 0 2024-04-05T03:03:06.054 INFO:tasks.workunit.client.0.smithi153.stdout:8/815: creat d7/d13/d19/d1b/d4f/d72/d73/f101 x:0 0 0 2024-04-05T03:03:06.056 INFO:tasks.workunit.client.0.smithi153.stdout:8/816: write d7/d13/f8e [1652699,82608] 0 2024-04-05T03:03:06.059 INFO:tasks.workunit.client.0.smithi153.stdout:3/966: write d1d/d10d/d116/d14a/ffd [349589,81688] 0 2024-04-05T03:03:06.068 INFO:tasks.workunit.client.0.smithi153.stdout:3/967: write d1d/d32/d93/d3c/d3f/d7c/fad [4877179,44682] 0 2024-04-05T03:03:06.068 INFO:tasks.workunit.client.0.smithi153.stdout:8/817: write d7/d13/d19/d1b/d4f/d72/d73/fd8 [4131598,98038] 0 2024-04-05T03:03:06.069 INFO:tasks.workunit.client.0.smithi153.stdout:8/818: chown d7/d13/d19/l9d 2700438 1 2024-04-05T03:03:06.070 INFO:tasks.workunit.client.0.smithi153.stdout:8/819: readlink d7/d85/l70 0 2024-04-05T03:03:06.070 INFO:tasks.workunit.client.0.smithi153.stdout:3/968: readlink d1d/d27/d2b/l65 0 2024-04-05T03:03:06.072 INFO:tasks.workunit.client.0.smithi153.stdout:8/820: rmdir d7/d13/d19 39 2024-04-05T03:03:06.073 INFO:tasks.workunit.client.0.smithi153.stdout:3/969: dread - d1d/d32/d93/d3c/f117 zero size 2024-04-05T03:03:06.074 INFO:tasks.workunit.client.0.smithi153.stdout:8/821: dread - d7/d2a/d50/fee zero size 2024-04-05T03:03:06.081 INFO:tasks.workunit.client.0.smithi153.stdout:3/970: dwrite d1d/d32/d93/f4c [0,4194304] 0 2024-04-05T03:03:06.083 INFO:tasks.workunit.client.0.smithi153.stdout:8/822: truncate d7/d13/d19/d1b/d6d/f7f 1136010 0 2024-04-05T03:03:06.084 INFO:tasks.workunit.client.0.smithi153.stdout:3/971: stat d1d/d32/d93/d3c/d3f/d7c 0 2024-04-05T03:03:06.091 INFO:tasks.workunit.client.0.smithi153.stdout:8/823: getdents d7/d13/d19/d1b/d6c 0 2024-04-05T03:03:06.095 INFO:tasks.workunit.client.0.smithi153.stdout:8/824: fdatasync d7/d13/d19/d1b/d4f/f86 0 2024-04-05T03:03:06.096 INFO:tasks.workunit.client.0.smithi153.stdout:8/825: write d7/d85/f78 [66732,52973] 0 2024-04-05T03:03:06.097 INFO:tasks.workunit.client.0.smithi153.stdout:8/826: readlink d7/d85/lf7 0 2024-04-05T03:03:06.097 INFO:tasks.workunit.client.0.smithi153.stdout:3/972: dread d1d/d32/d93/d3c/d3f/d90/db6/fb7 [0,4194304] 0 2024-04-05T03:03:06.100 INFO:tasks.workunit.client.0.smithi153.stdout:3/973: creat d1d/d32/d93/d3c/dc5/f15a x:0 0 0 2024-04-05T03:03:06.101 INFO:tasks.workunit.client.0.smithi153.stdout:8/827: write d7/d13/d19/d1b/d33/fdd [1879777,96264] 0 2024-04-05T03:03:06.103 INFO:tasks.workunit.client.0.smithi153.stdout:8/828: read d7/d13/d19/d1b/d4f/d68/dbc/fd3 [3584681,30748] 0 2024-04-05T03:03:06.107 INFO:tasks.workunit.client.0.smithi153.stdout:8/829: creat d7/d13/d19/d1b/d4f/de2/f102 x:0 0 0 2024-04-05T03:03:06.107 INFO:tasks.workunit.client.0.smithi153.stdout:8/830: readlink d7/d13/d19/l29 0 2024-04-05T03:03:06.110 INFO:tasks.workunit.client.0.smithi153.stdout:8/831: truncate d7/f74 1366726 0 2024-04-05T03:03:06.110 INFO:tasks.workunit.client.0.smithi153.stdout:8/832: dread - d7/d46/ffb zero size 2024-04-05T03:03:06.112 INFO:tasks.workunit.client.0.smithi153.stdout:5/894: sync 2024-04-05T03:03:06.112 INFO:tasks.workunit.client.0.smithi153.stdout:8/833: mkdir d7/d13/d19/d1b/d4f/d5f/df5/d103 0 2024-04-05T03:03:06.114 INFO:tasks.workunit.client.0.smithi153.stdout:5/895: mknod d0/d25/d4d/d9c/dc7/c12f 0 2024-04-05T03:03:06.115 INFO:tasks.workunit.client.0.smithi153.stdout:5/896: fsync d0/d25/d4d/d62/d37/fa6 0 2024-04-05T03:03:06.115 INFO:tasks.workunit.client.0.smithi153.stdout:8/834: creat d7/d13/d19/d1b/d33/f104 x:0 0 0 2024-04-05T03:03:06.117 INFO:tasks.workunit.client.0.smithi153.stdout:5/897: fdatasync d0/d2c/f8f 0 2024-04-05T03:03:06.119 INFO:tasks.workunit.client.0.smithi153.stdout:5/898: chown d0/d25/d68/f9b 64329590 1 2024-04-05T03:03:06.126 INFO:tasks.workunit.client.0.smithi153.stdout:8/835: getdents d7/d13/d19/d1b/db9 0 2024-04-05T03:03:06.136 INFO:tasks.workunit.client.0.smithi153.stdout:8/836: dwrite d7/d2a/f7b [0,4194304] 0 2024-04-05T03:03:06.137 INFO:tasks.workunit.client.0.smithi153.stdout:5/899: dread d0/f44 [0,4194304] 0 2024-04-05T03:03:06.138 INFO:tasks.workunit.client.0.smithi153.stdout:8/837: write d7/d85/f6a [4302275,29592] 0 2024-04-05T03:03:06.140 INFO:tasks.workunit.client.0.smithi153.stdout:5/900: fdatasync d0/d25/d4d/d62/d5f/db2/fdf 0 2024-04-05T03:03:06.143 INFO:tasks.workunit.client.0.smithi153.stdout:5/901: mkdir d0/d25/d4d/d62/d130 0 2024-04-05T03:03:06.150 INFO:tasks.workunit.client.0.smithi153.stdout:5/902: dwrite d0/d2c/f2d [0,4194304] 0 2024-04-05T03:03:06.166 INFO:tasks.workunit.client.0.smithi153.stdout:8/838: dread d7/d13/d19/f22 [0,4194304] 0 2024-04-05T03:03:06.167 INFO:tasks.workunit.client.0.smithi153.stdout:8/839: write d7/d46/ffb [764176,123140] 0 2024-04-05T03:03:06.169 INFO:tasks.workunit.client.0.smithi153.stdout:8/840: symlink d7/d13/d19/d1b/d6d/l105 0 2024-04-05T03:03:06.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/841: write d7/d13/d19/d1b/d4f/d72/d73/fd8 [420065,107668] 0 2024-04-05T03:03:06.286 INFO:tasks.workunit.client.0.smithi153.stdout:9/902: sync 2024-04-05T03:03:06.286 INFO:tasks.workunit.client.0.smithi153.stdout:0/981: sync 2024-04-05T03:03:06.288 INFO:tasks.workunit.client.0.smithi153.stdout:9/903: truncate d1/d9/f49 728150 0 2024-04-05T03:03:06.289 INFO:tasks.workunit.client.0.smithi153.stdout:9/904: truncate d1/d3a/d91/d96/de6/ff6 1518901 0 2024-04-05T03:03:06.290 INFO:tasks.workunit.client.0.smithi153.stdout:0/982: rename d9/db/d42/d7e/db6/f46 to d9/db/d42/d7e/db6/d1e/d45/d62/dde/d6d/f12f 0 2024-04-05T03:03:06.294 INFO:tasks.workunit.client.0.smithi153.stdout:0/983: getdents d9/db 0 2024-04-05T03:03:06.297 INFO:tasks.workunit.client.0.smithi153.stdout:9/905: dwrite d1/d10/d36/d71/f119 [0,4194304] 0 2024-04-05T03:03:06.298 INFO:tasks.workunit.client.0.smithi153.stdout:0/984: unlink d9/db/d42/d7e/db6/d1e/f25 0 2024-04-05T03:03:06.299 INFO:tasks.workunit.client.0.smithi153.stdout:0/985: readlink d9/db/d42/d7e/db6/d1e/d45/dae/df4/lda 0 2024-04-05T03:03:06.302 INFO:tasks.workunit.client.0.smithi153.stdout:9/906: rename d1/d9/d1b/l61 to d1/d10/d36/d134/l142 0 2024-04-05T03:03:06.304 INFO:tasks.workunit.client.0.smithi153.stdout:0/986: symlink d9/d14/d67/d6f/l130 0 2024-04-05T03:03:06.305 INFO:tasks.workunit.client.0.smithi153.stdout:9/907: mknod d1/d10/d36/d134/c143 0 2024-04-05T03:03:06.306 INFO:tasks.workunit.client.0.smithi153.stdout:0/987: read d9/db/d42/d7e/db6/d1e/d45/f6a [1153657,53399] 0 2024-04-05T03:03:06.309 INFO:tasks.workunit.client.0.smithi153.stdout:0/988: read d9/db/d42/d7e/db6/d1e/d45/dae/df4/fc6 [193015,24650] 0 2024-04-05T03:03:06.312 INFO:tasks.workunit.client.0.smithi153.stdout:9/908: dwrite d1/d9/d1b/de2/d125/fbb [4194304,4194304] 0 2024-04-05T03:03:06.313 INFO:tasks.workunit.client.0.smithi153.stdout:9/909: readlink d1/d10/d36/d57/lfb 0 2024-04-05T03:03:06.316 INFO:tasks.workunit.client.0.smithi153.stdout:9/910: write d1/d10/d36/d57/f6f [530927,33374] 0 2024-04-05T03:03:06.321 INFO:tasks.workunit.client.0.smithi153.stdout:0/989: dwrite d9/ff1 [0,4194304] 0 2024-04-05T03:03:06.330 INFO:tasks.workunit.client.0.smithi153.stdout:0/990: dwrite d9/db/d42/d7e/db6/f59 [0,4194304] 0 2024-04-05T03:03:06.334 INFO:tasks.workunit.client.0.smithi153.stdout:0/991: read - d9/fa9 zero size 2024-04-05T03:03:06.342 INFO:tasks.workunit.client.0.smithi153.stdout:9/911: dwrite d1/fcd [0,4194304] 0 2024-04-05T03:03:06.343 INFO:tasks.workunit.client.0.smithi153.stdout:9/912: creat d1/d9/dbc/f144 x:0 0 0 2024-04-05T03:03:06.344 INFO:tasks.workunit.client.0.smithi153.stdout:9/913: dread - d1/d9/d1b/f130 zero size 2024-04-05T03:03:06.344 INFO:tasks.workunit.client.0.smithi153.stdout:9/914: stat d1/d1d 0 2024-04-05T03:03:06.348 INFO:tasks.workunit.client.0.smithi153.stdout:9/915: symlink d1/d10/d36/d57/d12d/l145 0 2024-04-05T03:03:06.349 INFO:tasks.workunit.client.0.smithi153.stdout:0/992: write f7 [286098,104286] 0 2024-04-05T03:03:06.351 INFO:tasks.workunit.client.0.smithi153.stdout:9/916: fsync d1/d10/d16/d22/fcc 0 2024-04-05T03:03:06.351 INFO:tasks.workunit.client.0.smithi153.stdout:0/993: dread - d9/db/d42/d7e/d8b/f124 zero size 2024-04-05T03:03:06.353 INFO:tasks.workunit.client.0.smithi153.stdout:9/917: write d1/d6c/f104 [2474419,79594] 0 2024-04-05T03:03:06.355 INFO:tasks.workunit.client.0.smithi153.stdout:0/994: mknod d9/db/d42/d7e/db6/d1d/d61/d8a/df8/c131 0 2024-04-05T03:03:06.356 INFO:tasks.workunit.client.0.smithi153.stdout:9/918: symlink d1/d10/l146 0 2024-04-05T03:03:06.361 INFO:tasks.workunit.client.0.smithi153.stdout:9/919: creat d1/d10/d16/d13b/f147 x:0 0 0 2024-04-05T03:03:06.363 INFO:tasks.workunit.client.0.smithi153.stdout:0/995: dwrite d9/db/d42/d7e/db6/d1d/d61/d8a/fa5 [4194304,4194304] 0 2024-04-05T03:03:06.370 INFO:tasks.workunit.client.0.smithi153.stdout:0/996: rename d9/db/d42/d43/f76 to d9/db/d42/d7e/db6/d1d/f132 0 2024-04-05T03:03:06.373 INFO:tasks.workunit.client.0.smithi153.stdout:0/997: dread - d9/db/d42/dd4/f113 zero size 2024-04-05T03:03:06.375 INFO:tasks.workunit.client.0.smithi153.stdout:9/920: dwrite d1/fc4 [0,4194304] 0 2024-04-05T03:03:06.378 INFO:tasks.workunit.client.0.smithi153.stdout:0/998: symlink d9/db/d42/d43/d89/d11f/l133 0 2024-04-05T03:03:06.378 INFO:tasks.workunit.client.0.smithi153.stdout:3/974: sync 2024-04-05T03:03:06.379 INFO:tasks.workunit.client.0.smithi153.stdout:0/999: read - d9/db/d42/dd4/f113 zero size 2024-04-05T03:03:06.380 INFO:tasks.workunit.client.0.smithi153.stdout:9/921: link d1/f110 d1/d3a/d91/f148 0 2024-04-05T03:03:06.383 INFO:tasks.workunit.client.0.smithi153.stdout:9/922: mkdir d1/d3a/d91/d96/d149 0 2024-04-05T03:03:06.387 INFO:tasks.workunit.client.0.smithi153.stdout:9/923: creat d1/d3a/d91/d96/d149/f14a x:0 0 0 2024-04-05T03:03:06.388 INFO:tasks.workunit.client.0.smithi153.stdout:9/924: truncate d1/d10/d16/d120/d126/f102 788088 0 2024-04-05T03:03:06.389 INFO:tasks.workunit.client.0.smithi153.stdout:3/975: truncate d1d/d27/f74 1637780 0 2024-04-05T03:03:06.391 INFO:tasks.workunit.client.0.smithi153.stdout:3/976: creat d1d/d10d/f15b x:0 0 0 2024-04-05T03:03:06.391 INFO:tasks.workunit.client.0.smithi153.stdout:9/925: creat d1/d10/d36/d57/d97/ded/f14b x:0 0 0 2024-04-05T03:03:06.394 INFO:tasks.workunit.client.0.smithi153.stdout:9/926: unlink d1/d3a/d91/d96/d149/f14a 0 2024-04-05T03:03:06.396 INFO:tasks.workunit.client.0.smithi153.stdout:9/927: creat d1/d10/d16/d120/f14c x:0 0 0 2024-04-05T03:03:06.398 INFO:tasks.workunit.client.0.smithi153.stdout:3/977: rename d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d123/c142 to d1d/d32/d93/d3c/d94/c15c 0 2024-04-05T03:03:06.403 INFO:tasks.workunit.client.0.smithi153.stdout:9/928: dread d1/f132 [0,4194304] 0 2024-04-05T03:03:06.403 INFO:tasks.workunit.client.0.smithi153.stdout:3/978: dread d1d/d27/d2b/f71 [0,4194304] 0 2024-04-05T03:03:06.408 INFO:tasks.workunit.client.0.smithi153.stdout:3/979: mkdir d1d/d27/d2b/d15d 0 2024-04-05T03:03:06.408 INFO:tasks.workunit.client.0.smithi153.stdout:9/929: dwrite d1/f18 [0,4194304] 0 2024-04-05T03:03:06.412 INFO:tasks.workunit.client.0.smithi153.stdout:9/930: creat d1/d10/d16/d13b/f14d x:0 0 0 2024-04-05T03:03:06.414 INFO:tasks.workunit.client.0.smithi153.stdout:9/931: symlink d1/d10/d36/d71/l14e 0 2024-04-05T03:03:06.414 INFO:tasks.workunit.client.0.smithi153.stdout:3/980: read d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fa5 [3073978,6284] 0 2024-04-05T03:03:06.422 INFO:tasks.workunit.client.0.smithi153.stdout:3/981: link d1d/d27/lf3 d1d/d32/d93/d3c/d3f/l15e 0 2024-04-05T03:03:06.423 INFO:tasks.workunit.client.0.smithi153.stdout:3/982: read - d1d/d27/f135 zero size 2024-04-05T03:03:06.424 INFO:tasks.workunit.client.0.smithi153.stdout:3/983: dread - d1d/d12a/d138/ff6 zero size 2024-04-05T03:03:06.426 INFO:tasks.workunit.client.0.smithi153.stdout:3/984: rmdir d1d/d32/df9/d104 39 2024-04-05T03:03:06.427 INFO:tasks.workunit.client.0.smithi153.stdout:3/985: dread - d1d/d10d/f15b zero size 2024-04-05T03:03:06.431 INFO:tasks.workunit.client.0.smithi153.stdout:3/986: creat d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f/d112/f15f x:0 0 0 2024-04-05T03:03:06.432 INFO:tasks.workunit.client.0.smithi153.stdout:3/987: dread - d1d/d27/f135 zero size 2024-04-05T03:03:06.432 INFO:tasks.workunit.client.0.smithi153.stdout:9/932: truncate d1/d10/d36/d71/fa0 3831579 0 2024-04-05T03:03:06.433 INFO:tasks.workunit.client.0.smithi153.stdout:3/988: write d1d/d27/d2b/f62 [4497576,43164] 0 2024-04-05T03:03:06.434 INFO:tasks.workunit.client.0.smithi153.stdout:9/933: dread - d1/d9/dbc/f144 zero size 2024-04-05T03:03:06.437 INFO:tasks.workunit.client.0.smithi153.stdout:3/989: creat d1d/d32/d93/d3c/d3f/d7c/de7/f160 x:0 0 0 2024-04-05T03:03:06.439 INFO:tasks.workunit.client.0.smithi153.stdout:3/990: unlink d1d/fcf 0 2024-04-05T03:03:06.440 INFO:tasks.workunit.client.0.smithi153.stdout:3/991: write d1d/d10d/d116/f13c [807272,229] 0 2024-04-05T03:03:06.443 INFO:tasks.workunit.client.0.smithi153.stdout:3/992: truncate d1d/d27/feb 188045 0 2024-04-05T03:03:06.444 INFO:tasks.workunit.client.0.smithi153.stdout:9/934: write d1/d10/d16/d120/d126/fdf [932309,68204] 0 2024-04-05T03:03:06.449 INFO:tasks.workunit.client.0.smithi153.stdout:3/993: write d1d/d32/d93/d3c/d3f/d7c/f114 [354762,45070] 0 2024-04-05T03:03:06.455 INFO:tasks.workunit.client.0.smithi153.stdout:3/994: dwrite d1d/d32/d93/d3c/f117 [0,4194304] 0 2024-04-05T03:03:06.460 INFO:tasks.workunit.client.0.smithi153.stdout:3/995: link d1d/d32/d93/d3c/dc5/f158 d1d/d32/d93/d3c/d3f/d9c/f161 0 2024-04-05T03:03:06.466 INFO:tasks.workunit.client.0.smithi153.stdout:3/996: creat d1d/d32/d93/d3c/d3f/d90/da4/f162 x:0 0 0 2024-04-05T03:03:06.467 INFO:tasks.workunit.client.0.smithi153.stdout:3/997: truncate d1d/d10d/f15b 598729 0 2024-04-05T03:03:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi203 to 2240M 2024-04-05T03:03:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:03:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi203 to 2240M 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:03:06.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:03:06.584 INFO:tasks.workunit.client.0.smithi153.stdout:5/903: sync 2024-04-05T03:03:06.584 INFO:tasks.workunit.client.0.smithi153.stdout:8/842: sync 2024-04-05T03:03:06.588 INFO:tasks.workunit.client.0.smithi153.stdout:3/998: dwrite d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/fd6 [0,4194304] 0 2024-04-05T03:03:06.588 INFO:tasks.workunit.client.0.smithi153.stdout:8/843: chown d7/d9a/fbd 63393 1 2024-04-05T03:03:06.594 INFO:tasks.workunit.client.0.smithi153.stdout:8/844: mknod d7/d13/d19/d1b/d4f/c106 0 2024-04-05T03:03:06.596 INFO:tasks.workunit.client.0.smithi153.stdout:8/845: rmdir d7/d46/db1 39 2024-04-05T03:03:06.597 INFO:tasks.workunit.client.0.smithi153.stdout:5/904: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T03:03:06.598 INFO:tasks.workunit.client.0.smithi153.stdout:3/999: getdents d1d/d32/d93/d3c/d3f/d90/d92/d99/dd4/d6b/d9f 0 2024-04-05T03:03:06.598 INFO:tasks.workunit.client.0.smithi153.stdout:8/846: mkdir d7/d9a/d107 0 2024-04-05T03:03:06.606 INFO:tasks.workunit.client.0.smithi153.stdout:5/905: write d0/f3c [1242463,50683] 0 2024-04-05T03:03:06.611 INFO:tasks.workunit.client.0.smithi153.stdout:5/906: dread d0/d2c/f110 [0,4194304] 0 2024-04-05T03:03:06.613 INFO:tasks.workunit.client.0.smithi153.stdout:5/907: truncate d0/d25/f2b 660489 0 2024-04-05T03:03:06.615 INFO:tasks.workunit.client.0.smithi153.stdout:5/908: creat d0/d2c/d5e/d11a/f131 x:0 0 0 2024-04-05T03:03:06.616 INFO:tasks.workunit.client.0.smithi153.stdout:5/909: chown d0/d25/d4d/d62/d37/f6a 1490129 1 2024-04-05T03:03:06.620 INFO:tasks.workunit.client.0.smithi153.stdout:5/910: link d0/d25/d4d/d62/d5f/d86/dd7/cee d0/d25/d4d/d62/d130/c132 0 2024-04-05T03:03:06.622 INFO:tasks.workunit.client.0.smithi153.stdout:5/911: creat d0/d25/d4d/d62/d5f/db2/f133 x:0 0 0 2024-04-05T03:03:07.016 INFO:tasks.workunit.client.0.smithi153.stdout:9/935: sync 2024-04-05T03:03:07.017 INFO:tasks.workunit.client.0.smithi153.stdout:9/936: write d1/d10/d36/d57/d97/f114 [333913,116047] 0 2024-04-05T03:03:07.018 INFO:tasks.workunit.client.0.smithi153.stdout:9/937: write d1/d10/d36/d57/f78 [3514467,9501] 0 2024-04-05T03:03:07.021 INFO:tasks.workunit.client.0.smithi153.stdout:9/938: fsync d1/d3a/fe5 0 2024-04-05T03:03:07.028 INFO:tasks.workunit.client.0.smithi153.stdout:9/939: link d1/d10/d36/d57/lfb d1/d3a/l14f 0 2024-04-05T03:03:07.029 INFO:tasks.workunit.client.0.smithi153.stdout:9/940: dread - d1/d9/d1b/f130 zero size 2024-04-05T03:03:07.031 INFO:tasks.workunit.client.0.smithi153.stdout:9/941: symlink d1/d10/d36/d57/d12d/l150 0 2024-04-05T03:03:07.033 INFO:tasks.workunit.client.0.smithi153.stdout:9/942: dread - d1/d10/d16/d120/d126/ff0 zero size 2024-04-05T03:03:07.035 INFO:tasks.workunit.client.0.smithi153.stdout:9/943: symlink d1/d10/d36/d57/d12d/l151 0 2024-04-05T03:03:07.043 INFO:tasks.workunit.client.0.smithi153.stdout:9/944: dwrite d1/d3a/d91/fc5 [0,4194304] 0 2024-04-05T03:03:07.048 INFO:tasks.workunit.client.0.smithi153.stdout:9/945: creat d1/d9/dbc/f152 x:0 0 0 2024-04-05T03:03:07.050 INFO:tasks.workunit.client.0.smithi153.stdout:9/946: truncate d1/d1d/f105 938666 0 2024-04-05T03:03:07.050 INFO:tasks.workunit.client.0.smithi153.stdout:8/847: sync 2024-04-05T03:03:07.050 INFO:tasks.workunit.client.0.smithi153.stdout:5/912: sync 2024-04-05T03:03:07.051 INFO:tasks.workunit.client.0.smithi153.stdout:8/848: fsync d7/d13/f8e 0 2024-04-05T03:03:07.057 INFO:tasks.workunit.client.0.smithi153.stdout:8/849: rmdir d7/d13/d19/d1b 39 2024-04-05T03:03:07.060 INFO:tasks.workunit.client.0.smithi153.stdout:5/913: unlink d0/cc3 0 2024-04-05T03:03:07.061 INFO:tasks.workunit.client.0.smithi153.stdout:5/914: chown d0/d4e/db6/f11f 10 1 2024-04-05T03:03:07.070 INFO:tasks.workunit.client.0.smithi153.stdout:5/915: rename d0/d2c/f3d to d0/d2c/d3f/da8/f134 0 2024-04-05T03:03:07.075 INFO:tasks.workunit.client.0.smithi153.stdout:5/916: read d0/d4e/f9e [192763,122354] 0 2024-04-05T03:03:07.076 INFO:tasks.workunit.client.0.smithi153.stdout:8/850: link d7/c2e d7/d13/d19/d1b/d6c/c108 0 2024-04-05T03:03:07.079 INFO:tasks.workunit.client.0.smithi153.stdout:8/851: fsync d7/d13/d19/d1b/d4f/f94 0 2024-04-05T03:03:07.085 INFO:tasks.workunit.client.0.smithi153.stdout:8/852: symlink d7/d13/d19/d95/l109 0 2024-04-05T03:03:07.094 INFO:tasks.workunit.client.0.smithi153.stdout:9/947: write d1/d3a/fcb [779628,110998] 0 2024-04-05T03:03:07.098 INFO:tasks.workunit.client.0.smithi153.stdout:8/853: getdents d7/d2a 0 2024-04-05T03:03:07.099 INFO:tasks.workunit.client.0.smithi153.stdout:9/948: mkdir d1/d3a/d91/d96/de6/d153 0 2024-04-05T03:03:07.102 INFO:tasks.workunit.client.0.smithi153.stdout:8/854: mkdir d7/d13/d19/d1b/d4f/d72/d10a 0 2024-04-05T03:03:07.103 INFO:tasks.workunit.client.0.smithi153.stdout:9/949: symlink d1/d9/dbc/d122/l154 0 2024-04-05T03:03:07.106 INFO:tasks.workunit.client.0.smithi153.stdout:9/950: chown d1/d10/d36/d57/d5c/l70 831 1 2024-04-05T03:03:07.113 INFO:tasks.workunit.client.0.smithi153.stdout:8/855: link d7/d13/d19/d1b/d6c/cb8 d7/c10b 0 2024-04-05T03:03:07.118 INFO:tasks.workunit.client.0.smithi153.stdout:9/951: dwrite d1/d3a/d91/fc5 [0,4194304] 0 2024-04-05T03:03:07.120 INFO:tasks.workunit.client.0.smithi153.stdout:5/917: dread d0/d9/d22/fc2 [0,4194304] 0 2024-04-05T03:03:07.125 INFO:tasks.workunit.client.0.smithi153.stdout:8/856: dread d7/d13/d19/d1b/db9/ff6 [0,4194304] 0 2024-04-05T03:03:07.128 INFO:tasks.workunit.client.0.smithi153.stdout:9/952: rename d1/d10/d36/d57/d97/fa5 to d1/d10/d36/d57/f155 0 2024-04-05T03:03:07.130 INFO:tasks.workunit.client.0.smithi153.stdout:8/857: creat d7/d13/d19/d95/f10c x:0 0 0 2024-04-05T03:03:07.131 INFO:tasks.workunit.client.0.smithi153.stdout:8/858: chown d7/d2a/d8b 21592 1 2024-04-05T03:03:07.132 INFO:tasks.workunit.client.0.smithi153.stdout:8/859: write d7/d46/fa7 [3621105,49944] 0 2024-04-05T03:03:07.141 INFO:tasks.workunit.client.0.smithi153.stdout:8/860: dwrite d7/d13/d19/d1b/d4f/d5f/fe9 [0,4194304] 0 2024-04-05T03:03:07.147 INFO:tasks.workunit.client.0.smithi153.stdout:8/861: symlink d7/d85/l10d 0 2024-04-05T03:03:07.147 INFO:tasks.workunit.client.0.smithi153.stdout:5/918: write d0/d25/d4d/d62/d5f/d86/da4/ff1 [634698,19904] 0 2024-04-05T03:03:07.150 INFO:tasks.workunit.client.0.smithi153.stdout:5/919: symlink d0/d25/d68/l135 0 2024-04-05T03:03:07.151 INFO:tasks.workunit.client.0.smithi153.stdout:8/862: write d7/d13/d19/d1b/d6c/fac [703498,30309] 0 2024-04-05T03:03:07.152 INFO:tasks.workunit.client.0.smithi153.stdout:5/920: write d0/d25/d4d/d62/d37/f4f [2254879,116376] 0 2024-04-05T03:03:07.154 INFO:tasks.workunit.client.0.smithi153.stdout:8/863: fdatasync d7/d13/d19/d1b/d4f/d5f/fe3 0 2024-04-05T03:03:07.155 INFO:tasks.workunit.client.0.smithi153.stdout:5/921: chown d0/d25/d4d/d62/d5f/d86/f77 11445 1 2024-04-05T03:03:07.161 INFO:tasks.workunit.client.0.smithi153.stdout:5/922: rename d0/d25/d4d/d9c/ff6 to d0/d25/d4d/d62/d5f/db2/f136 0 2024-04-05T03:03:07.162 INFO:tasks.workunit.client.0.smithi153.stdout:8/864: truncate d7/d13/dab/fdc 907023 0 2024-04-05T03:03:07.162 INFO:tasks.workunit.client.0.smithi153.stdout:5/923: dread - d0/d2c/d5e/d11a/f131 zero size 2024-04-05T03:03:07.167 INFO:tasks.workunit.client.0.smithi153.stdout:8/865: symlink d7/d13/d19/d1b/d4f/d72/d73/l10e 0 2024-04-05T03:03:07.168 INFO:tasks.workunit.client.0.smithi153.stdout:5/924: mknod d0/d25/d4d/d9c/dc4/c137 0 2024-04-05T03:03:07.169 INFO:tasks.workunit.client.0.smithi153.stdout:8/866: mkdir d7/d13/d19/d95/d10f 0 2024-04-05T03:03:07.171 INFO:tasks.workunit.client.0.smithi153.stdout:8/867: chown d7/d13/d19/d1b/d4f/d68/fef 0 1 2024-04-05T03:03:07.176 INFO:tasks.workunit.client.0.smithi153.stdout:5/925: dwrite d0/d2c/fbe [0,4194304] 0 2024-04-05T03:03:07.178 INFO:tasks.workunit.client.0.smithi153.stdout:5/926: symlink d0/d25/d4d/d9c/db7/l138 0 2024-04-05T03:03:07.182 INFO:tasks.workunit.client.0.smithi153.stdout:5/927: unlink d0/d25/d4d/d62/d5f/d86/da4/fa1 0 2024-04-05T03:03:07.183 INFO:tasks.workunit.client.0.smithi153.stdout:8/868: write d7/d13/f14 [5069129,7954] 0 2024-04-05T03:03:07.186 INFO:tasks.workunit.client.0.smithi153.stdout:8/869: mknod d7/d13/dc8/c110 0 2024-04-05T03:03:07.187 INFO:tasks.workunit.client.0.smithi153.stdout:8/870: write d7/d13/d19/fb5 [960586,71300] 0 2024-04-05T03:03:07.188 INFO:tasks.workunit.client.0.smithi153.stdout:8/871: truncate d7/d13/f8e 1765586 0 2024-04-05T03:03:07.191 INFO:tasks.workunit.client.0.smithi153.stdout:8/872: chown d7/d13/fa3 1 1 2024-04-05T03:03:07.192 INFO:tasks.workunit.client.0.smithi153.stdout:8/873: dread d7/d9a/fe7 [0,4194304] 0 2024-04-05T03:03:07.193 INFO:tasks.workunit.client.0.smithi153.stdout:8/874: readlink d7/d13/dab/le8 0 2024-04-05T03:03:07.194 INFO:tasks.workunit.client.0.smithi153.stdout:8/875: write d7/d13/d19/fb5 [1582749,67879] 0 2024-04-05T03:03:07.197 INFO:tasks.workunit.client.0.smithi153.stdout:8/876: mkdir d7/d13/d19/d95/d10f/d111 0 2024-04-05T03:03:07.199 INFO:tasks.workunit.client.0.smithi153.stdout:8/877: rmdir d7/d13/d19/d1b/d6d/d8c 39 2024-04-05T03:03:07.212 INFO:tasks.workunit.client.0.smithi153.stdout:8/878: dwrite d7/d2a/fcf [0,4194304] 0 2024-04-05T03:03:07.220 INFO:tasks.workunit.client.0.smithi153.stdout:8/879: dwrite f3 [4194304,4194304] 0 2024-04-05T03:03:07.223 INFO:tasks.workunit.client.0.smithi153.stdout:8/880: unlink d7/d13/d19/d1b/d4f/c106 0 2024-04-05T03:03:07.235 INFO:tasks.workunit.client.0.smithi153.stdout:8/881: dwrite d7/d13/d19/f4e [0,4194304] 0 2024-04-05T03:03:07.247 INFO:tasks.workunit.client.0.smithi153.stdout:8/882: dread d7/d13/f8e [0,4194304] 0 2024-04-05T03:03:07.249 INFO:tasks.workunit.client.0.smithi153.stdout:8/883: stat d7/d46/db1/fe4 0 2024-04-05T03:03:07.253 INFO:tasks.workunit.client.0.smithi153.stdout:8/884: dread d7/d2a/f7b [0,4194304] 0 2024-04-05T03:03:07.255 INFO:tasks.workunit.client.0.smithi153.stdout:8/885: creat d7/d13/dc8/f112 x:0 0 0 2024-04-05T03:03:07.257 INFO:tasks.workunit.client.0.smithi153.stdout:8/886: readlink d7/d85/l49 0 2024-04-05T03:03:07.258 INFO:tasks.workunit.client.0.smithi153.stdout:8/887: stat d7/d13/d19/d95/l109 0 2024-04-05T03:03:07.260 INFO:tasks.workunit.client.0.smithi153.stdout:8/888: creat d7/f113 x:0 0 0 2024-04-05T03:03:07.263 INFO:tasks.workunit.client.0.smithi153.stdout:8/889: mknod d7/d13/d19/d1b/d4f/de2/c114 0 2024-04-05T03:03:07.265 INFO:tasks.workunit.client.0.smithi153.stdout:8/890: truncate d7/d13/d19/f8f 347016 0 2024-04-05T03:03:07.272 INFO:tasks.workunit.client.0.smithi153.stdout:8/891: write d7/d13/f62 [3134773,92695] 0 2024-04-05T03:03:07.286 INFO:tasks.workunit.client.0.smithi153.stdout:8/892: dread d7/d13/d19/d1b/d4f/f94 [0,4194304] 0 2024-04-05T03:03:07.288 INFO:tasks.workunit.client.0.smithi153.stdout:8/893: chown d7/d13/d19/d1b/d4f/d68/dbc/lc5 1673061 1 2024-04-05T03:03:07.292 INFO:tasks.workunit.client.0.smithi153.stdout:8/894: dread d7/d13/d19/d1b/d4f/d5f/fff [0,4194304] 0 2024-04-05T03:03:07.297 INFO:tasks.workunit.client.0.smithi153.stdout:8/895: write d7/f16 [860689,78263] 0 2024-04-05T03:03:07.305 INFO:tasks.workunit.client.0.smithi153.stdout:8/896: dwrite d7/f8 [0,4194304] 0 2024-04-05T03:03:07.307 INFO:tasks.workunit.client.0.smithi153.stdout:8/897: read d7/d13/d19/d1b/d33/fdd [4171240,110019] 0 2024-04-05T03:03:07.308 INFO:tasks.workunit.client.0.smithi153.stdout:8/898: readlink d7/d46/db1/lfd 0 2024-04-05T03:03:07.310 INFO:tasks.workunit.client.0.smithi153.stdout:8/899: mknod d7/d13/d19/d1b/d4f/de2/c115 0 2024-04-05T03:03:07.318 INFO:tasks.workunit.client.0.smithi153.stdout:8/900: truncate d7/fe 76084 0 2024-04-05T03:03:07.321 INFO:tasks.workunit.client.0.smithi153.stdout:8/901: unlink d7/d13/d19/d1b/d4f/d68/dbc/lfa 0 2024-04-05T03:03:07.323 INFO:tasks.workunit.client.0.smithi153.stdout:8/902: mkdir d7/d13/d19/d1b/d116 0 2024-04-05T03:03:07.326 INFO:tasks.workunit.client.0.smithi153.stdout:8/903: rename d7/d13/d19/d1b/d4f/d5f/d54/c79 to d7/d13/d19/d95/d10f/d111/c117 0 2024-04-05T03:03:07.328 INFO:tasks.workunit.client.0.smithi153.stdout:8/904: mkdir d7/d13/d19/d1b/d6c/d118 0 2024-04-05T03:03:07.329 INFO:tasks.workunit.client.0.smithi153.stdout:8/905: dread - d7/fa9 zero size 2024-04-05T03:03:07.330 INFO:tasks.workunit.client.0.smithi153.stdout:8/906: stat d7/d13/d19/d1b/d4f/de2/c114 0 2024-04-05T03:03:07.331 INFO:tasks.workunit.client.0.smithi153.stdout:8/907: truncate d7/ff 1755987 0 2024-04-05T03:03:07.333 INFO:tasks.workunit.client.0.smithi153.stdout:8/908: creat d7/d85/f119 x:0 0 0 2024-04-05T03:03:07.335 INFO:tasks.workunit.client.0.smithi153.stdout:8/909: mknod d7/d2a/d8b/c11a 0 2024-04-05T03:03:07.338 INFO:tasks.workunit.client.0.smithi153.stdout:8/910: getdents d7/d13/dc8 0 2024-04-05T03:03:07.347 INFO:tasks.workunit.client.0.smithi153.stdout:8/911: truncate d7/d13/dc8/fed 363443 0 2024-04-05T03:03:07.349 INFO:tasks.workunit.client.0.smithi153.stdout:8/912: symlink d7/d13/d19/d1b/d4f/d5f/df5/l11b 0 2024-04-05T03:03:07.361 INFO:tasks.workunit.client.0.smithi153.stdout:8/913: dwrite d7/d13/dc8/fcc [4194304,4194304] 0 2024-04-05T03:03:07.366 INFO:tasks.workunit.client.0.smithi153.stdout:8/914: dwrite d7/d13/d19/d1b/d4f/d68/f84 [0,4194304] 0 2024-04-05T03:03:07.369 INFO:tasks.workunit.client.0.smithi153.stdout:8/915: mknod d7/d13/d19/d95/c11c 0 2024-04-05T03:03:07.370 INFO:tasks.workunit.client.0.smithi153.stdout:8/916: readlink d7/d13/d19/d1b/d21/l43 0 2024-04-05T03:03:07.389 INFO:tasks.workunit.client.0.smithi153.stdout:8/917: write d7/d13/d19/d1b/d6c/fb0 [3501382,24257] 0 2024-04-05T03:03:07.391 INFO:tasks.workunit.client.0.smithi153.stdout:8/918: creat d7/d13/d19/d95/d10f/f11d x:0 0 0 2024-04-05T03:03:07.392 INFO:tasks.workunit.client.0.smithi153.stdout:8/919: chown d7/d13/d19/d1b/d4f/d72/d73/f101 1865 1 2024-04-05T03:03:07.394 INFO:tasks.workunit.client.0.smithi153.stdout:8/920: dread - d7/d13/d19/d1b/d4f/d68/dbc/fbe zero size 2024-04-05T03:03:07.396 INFO:tasks.workunit.client.0.smithi153.stdout:8/921: mkdir d7/d46/d11e 0 2024-04-05T03:03:07.400 INFO:tasks.workunit.client.0.smithi153.stdout:8/922: dread d7/d13/d19/d1b/d4f/d68/dbc/fd3 [0,4194304] 0 2024-04-05T03:03:07.401 INFO:tasks.workunit.client.0.smithi153.stdout:8/923: truncate d7/dd5/ffe 322061 0 2024-04-05T03:03:07.401 INFO:tasks.workunit.client.0.smithi153.stdout:8/924: write d7/d13/d19/d1b/db9/fd7 [315629,17128] 0 2024-04-05T03:03:07.406 INFO:tasks.workunit.client.0.smithi153.stdout:9/953: sync 2024-04-05T03:03:07.406 INFO:tasks.workunit.client.0.smithi153.stdout:8/925: creat d7/d13/d19/d1b/d4f/d5f/d54/f11f x:0 0 0 2024-04-05T03:03:07.408 INFO:tasks.workunit.client.0.smithi153.stdout:9/954: mkdir d1/d9/d1b/de9/d156 0 2024-04-05T03:03:07.409 INFO:tasks.workunit.client.0.smithi153.stdout:8/926: creat d7/d9a/d107/f120 x:0 0 0 2024-04-05T03:03:07.413 INFO:tasks.workunit.client.0.smithi153.stdout:9/955: link d1/d10/d36/d57/d5c/ld8 d1/d10/d36/d57/d5c/l157 0 2024-04-05T03:03:07.413 INFO:tasks.workunit.client.0.smithi153.stdout:9/956: fdatasync d1/d6c/f82 0 2024-04-05T03:03:07.414 INFO:tasks.workunit.client.0.smithi153.stdout:9/957: chown d1/d10/d36/d57/d12d/d68/l72 11347453 1 2024-04-05T03:03:07.419 INFO:tasks.workunit.client.0.smithi153.stdout:8/927: dwrite d7/d13/d19/f22 [0,4194304] 0 2024-04-05T03:03:07.419 INFO:tasks.workunit.client.0.smithi153.stdout:9/958: truncate d1/fcd 366893 0 2024-04-05T03:03:07.422 INFO:tasks.workunit.client.0.smithi153.stdout:8/928: mknod d7/d13/d19/d1b/d4f/d72/c121 0 2024-04-05T03:03:07.424 INFO:tasks.workunit.client.0.smithi153.stdout:9/959: read d1/d10/d16/f111 [478963,84887] 0 2024-04-05T03:03:07.427 INFO:tasks.workunit.client.0.smithi153.stdout:9/960: fdatasync d1/d10/f67 0 2024-04-05T03:03:07.429 INFO:tasks.workunit.client.0.smithi153.stdout:9/961: fsync d1/d10/d36/d57/d97/ded/f10c 0 2024-04-05T03:03:07.429 INFO:tasks.workunit.client.0.smithi153.stdout:8/929: dwrite d7/d13/d19/d1b/d4f/d5f/fec [0,4194304] 0 2024-04-05T03:03:07.431 INFO:tasks.workunit.client.0.smithi153.stdout:9/962: readlink d1/d10/d36/d57/d12d/l145 0 2024-04-05T03:03:07.434 INFO:tasks.workunit.client.0.smithi153.stdout:9/963: rename d1/d1d/c128 to d1/d1d/c158 0 2024-04-05T03:03:07.438 INFO:tasks.workunit.client.0.smithi153.stdout:8/930: dwrite d7/d85/f119 [0,4194304] 0 2024-04-05T03:03:07.438 INFO:tasks.workunit.client.0.smithi153.stdout:9/964: creat d1/d9/dbc/d122/f159 x:0 0 0 2024-04-05T03:03:07.442 INFO:tasks.workunit.client.0.smithi153.stdout:9/965: mknod d1/d3a/d91/d96/de6/c15a 0 2024-04-05T03:03:07.444 INFO:tasks.workunit.client.0.smithi153.stdout:9/966: chown d1/d10/d16/d13b 181 1 2024-04-05T03:03:07.445 INFO:tasks.workunit.client.0.smithi153.stdout:8/931: dread d7/d13/d19/d1b/d4f/f94 [0,4194304] 0 2024-04-05T03:03:07.447 INFO:tasks.workunit.client.0.smithi153.stdout:8/932: fdatasync d7/d13/d19/d1b/d33/f51 0 2024-04-05T03:03:07.449 INFO:tasks.workunit.client.0.smithi153.stdout:5/928: sync 2024-04-05T03:03:07.449 INFO:tasks.workunit.client.0.smithi153.stdout:9/967: creat d1/d10/d36/d57/d97/f15b x:0 0 0 2024-04-05T03:03:07.451 INFO:tasks.workunit.client.0.smithi153.stdout:8/933: chown d7/d13/d19/d1b/d4f/d5f/d54/l7c 602145 1 2024-04-05T03:03:07.457 INFO:tasks.workunit.client.0.smithi153.stdout:5/929: creat d0/d25/d4d/d9c/dc4/f139 x:0 0 0 2024-04-05T03:03:07.459 INFO:tasks.workunit.client.0.smithi153.stdout:9/968: rename d1/d10/d16/d120/d126/fd4 to d1/d10/f15c 0 2024-04-05T03:03:07.460 INFO:tasks.workunit.client.0.smithi153.stdout:5/930: symlink d0/d25/d4d/d9c/dc4/dcb/d109/l13a 0 2024-04-05T03:03:07.461 INFO:tasks.workunit.client.0.smithi153.stdout:5/931: readlink d0/d25/d4d/d9c/dc7/lea 0 2024-04-05T03:03:07.461 INFO:tasks.workunit.client.0.smithi153.stdout:9/969: mkdir d1/d10/d16/d15d 0 2024-04-05T03:03:07.464 INFO:tasks.workunit.client.0.smithi153.stdout:5/932: creat d0/d9/d22/d125/f13b x:0 0 0 2024-04-05T03:03:07.465 INFO:tasks.workunit.client.0.smithi153.stdout:9/970: mknod d1/d3a/d91/d96/d149/c15e 0 2024-04-05T03:03:07.468 INFO:tasks.workunit.client.0.smithi153.stdout:8/934: write d7/d2a/f2d [1803864,93403] 0 2024-04-05T03:03:07.469 INFO:tasks.workunit.client.0.smithi153.stdout:8/935: chown d7/d2a/f37 320 1 2024-04-05T03:03:07.470 INFO:tasks.workunit.client.0.smithi153.stdout:9/971: unlink d1/d10/d36/d134/l142 0 2024-04-05T03:03:07.472 INFO:tasks.workunit.client.0.smithi153.stdout:5/933: rmdir d0/d2c/d117 0 2024-04-05T03:03:07.473 INFO:tasks.workunit.client.0.smithi153.stdout:9/972: mkdir d1/d10/d16/d120/d15f 0 2024-04-05T03:03:07.476 INFO:tasks.workunit.client.0.smithi153.stdout:8/936: dwrite d7/d2a/d50/f6b [0,4194304] 0 2024-04-05T03:03:07.480 INFO:tasks.workunit.client.0.smithi153.stdout:9/973: creat d1/d10/d36/d57/d12d/f160 x:0 0 0 2024-04-05T03:03:07.484 INFO:tasks.workunit.client.0.smithi153.stdout:9/974: mkdir d1/d3a/d91/d96/d161 0 2024-04-05T03:03:07.487 INFO:tasks.workunit.client.0.smithi153.stdout:8/937: dwrite d7/d13/d19/d1b/d4f/de2/f102 [0,4194304] 0 2024-04-05T03:03:07.497 INFO:tasks.workunit.client.0.smithi153.stdout:5/934: dwrite d0/f75 [0,4194304] 0 2024-04-05T03:03:07.497 INFO:tasks.workunit.client.0.smithi153.stdout:8/938: dread d7/d13/dc8/fcc [4194304,4194304] 0 2024-04-05T03:03:07.499 INFO:tasks.workunit.client.0.smithi153.stdout:8/939: read - d7/d13/d19/d95/f10c zero size 2024-04-05T03:03:07.501 INFO:tasks.workunit.client.0.smithi153.stdout:5/935: fsync d0/d4e/f9f 0 2024-04-05T03:03:07.506 INFO:tasks.workunit.client.0.smithi153.stdout:8/940: mknod d7/d13/d19/d1b/d4f/d5f/d54/c122 0 2024-04-05T03:03:07.507 INFO:tasks.workunit.client.0.smithi153.stdout:5/936: mkdir d0/d2c/d3f/d13c 0 2024-04-05T03:03:07.508 INFO:tasks.workunit.client.0.smithi153.stdout:8/941: rmdir d7/d13/d19/d1b/d4f/d72 39 2024-04-05T03:03:07.510 INFO:tasks.workunit.client.0.smithi153.stdout:8/942: dread - d7/d46/faa zero size 2024-04-05T03:03:07.511 INFO:tasks.workunit.client.0.smithi153.stdout:5/937: write d0/d25/d4d/d62/d5f/fbf [3170862,79283] 0 2024-04-05T03:03:07.513 INFO:tasks.workunit.client.0.smithi153.stdout:9/975: dwrite d1/d10/d36/fd3 [0,4194304] 0 2024-04-05T03:03:07.518 INFO:tasks.workunit.client.0.smithi153.stdout:9/976: creat d1/d10/d16/d22/f162 x:0 0 0 2024-04-05T03:03:07.520 INFO:tasks.workunit.client.0.smithi153.stdout:5/938: write d0/d25/d4d/d62/dbb/ff5 [273211,106274] 0 2024-04-05T03:03:07.524 INFO:tasks.workunit.client.0.smithi153.stdout:5/939: symlink d0/d25/d4d/d9c/d10f/l13d 0 2024-04-05T03:03:07.526 INFO:tasks.workunit.client.0.smithi153.stdout:5/940: mknod d0/d25/d4d/d9c/dc4/c13e 0 2024-04-05T03:03:07.531 INFO:tasks.workunit.client.0.smithi153.stdout:8/943: dread d7/d85/f41 [0,4194304] 0 2024-04-05T03:03:07.532 INFO:tasks.workunit.client.0.smithi153.stdout:5/941: write d0/d25/d4d/d9c/db7/fd6 [736097,3682] 0 2024-04-05T03:03:07.536 INFO:tasks.workunit.client.0.smithi153.stdout:8/944: creat d7/d13/d19/d1b/d6d/d8c/f123 x:0 0 0 2024-04-05T03:03:07.540 INFO:tasks.workunit.client.0.smithi153.stdout:8/945: write d7/d13/d19/d1b/d4f/d5f/fb2 [1570605,2344] 0 2024-04-05T03:03:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:07 smithi203 ceph-mon[39911]: pgmap v18: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 83 MiB/s rd, 188 MiB/s wr, 441 op/s 2024-04-05T03:03:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:07 smithi203 ceph-mon[39911]: Adjusting osd_memory_target on smithi153 to 1898M 2024-04-05T03:03:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:07 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:03:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:07 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:03:07.547 INFO:tasks.workunit.client.0.smithi153.stdout:8/946: dwrite d7/d13/d19/d1b/d33/f104 [0,4194304] 0 2024-04-05T03:03:07.552 INFO:tasks.workunit.client.0.smithi153.stdout:5/942: dread d0/d25/d4d/d62/d5f/f7e [0,4194304] 0 2024-04-05T03:03:07.555 INFO:tasks.workunit.client.0.smithi153.stdout:5/943: mknod d0/d25/d4d/d62/c13f 0 2024-04-05T03:03:07.558 INFO:tasks.workunit.client.0.smithi153.stdout:5/944: creat d0/d9/d22/d125/f140 x:0 0 0 2024-04-05T03:03:07.559 INFO:tasks.workunit.client.0.smithi153.stdout:8/947: dwrite d7/d9a/fe5 [0,4194304] 0 2024-04-05T03:03:07.566 INFO:tasks.workunit.client.0.smithi153.stdout:8/948: symlink d7/d13/d19/d1b/d4f/d72/l124 0 2024-04-05T03:03:07.566 INFO:tasks.workunit.client.0.smithi153.stdout:5/945: dread d0/d4e/fa5 [0,4194304] 0 2024-04-05T03:03:07.568 INFO:tasks.workunit.client.0.smithi153.stdout:5/946: write d0/d25/f127 [67938,82970] 0 2024-04-05T03:03:07.570 INFO:tasks.workunit.client.0.smithi153.stdout:5/947: chown d0/d25/d4d/d62/d5f/d86/fff 3176 1 2024-04-05T03:03:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:07 smithi153 ceph-mon[29381]: pgmap v18: 65 pgs: 65 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 83 MiB/s rd, 188 MiB/s wr, 441 op/s 2024-04-05T03:03:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:07 smithi153 ceph-mon[29381]: Adjusting osd_memory_target on smithi153 to 1898M 2024-04-05T03:03:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:07 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:03:07.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:07 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:03:07.572 INFO:tasks.workunit.client.0.smithi153.stdout:5/948: symlink d0/d4e/db6/l141 0 2024-04-05T03:03:07.574 INFO:tasks.workunit.client.0.smithi153.stdout:8/949: dwrite d7/d2a/f77 [0,4194304] 0 2024-04-05T03:03:07.581 INFO:tasks.workunit.client.0.smithi153.stdout:5/949: link d0/d25/le6 d0/d25/d4d/d62/d5f/d86/da4/l142 0 2024-04-05T03:03:07.583 INFO:tasks.workunit.client.0.smithi153.stdout:5/950: dread d0/d25/f36 [0,4194304] 0 2024-04-05T03:03:07.588 INFO:tasks.workunit.client.0.smithi153.stdout:8/950: dwrite d7/d13/d19/d1b/d6d/fae [4194304,4194304] 0 2024-04-05T03:03:07.588 INFO:tasks.workunit.client.0.smithi153.stdout:5/951: getdents d0/d2c/d5e 0 2024-04-05T03:03:07.592 INFO:tasks.workunit.client.0.smithi153.stdout:5/952: creat d0/d25/d4d/d9c/dc7/dc9/f143 x:0 0 0 2024-04-05T03:03:07.594 INFO:tasks.workunit.client.0.smithi153.stdout:8/951: rename f6 to d7/d13/f125 0 2024-04-05T03:03:07.600 INFO:tasks.workunit.client.0.smithi153.stdout:8/952: dwrite d7/d13/d19/d1b/d4f/d68/f75 [0,4194304] 0 2024-04-05T03:03:07.600 INFO:tasks.workunit.client.0.smithi153.stdout:5/953: write d0/d25/d4d/d62/d37/d38/f87 [749069,65439] 0 2024-04-05T03:03:07.603 INFO:tasks.workunit.client.0.smithi153.stdout:8/953: rmdir d7/d13/d19/d1b/d33 39 2024-04-05T03:03:07.606 INFO:tasks.workunit.client.0.smithi153.stdout:5/954: creat d0/d25/d4d/d62/d130/f144 x:0 0 0 2024-04-05T03:03:07.608 INFO:tasks.workunit.client.0.smithi153.stdout:8/954: mknod d7/d13/d19/d1b/d4f/d5f/df5/d103/c126 0 2024-04-05T03:03:07.609 INFO:tasks.workunit.client.0.smithi153.stdout:8/955: dread - d7/d9a/d107/f120 zero size 2024-04-05T03:03:07.612 INFO:tasks.workunit.client.0.smithi153.stdout:8/956: creat d7/d13/d19/d1b/d21/f127 x:0 0 0 2024-04-05T03:03:07.613 INFO:tasks.workunit.client.0.smithi153.stdout:5/955: write d0/d25/d4d/d62/f1a [3597318,126669] 0 2024-04-05T03:03:07.615 INFO:tasks.workunit.client.0.smithi153.stdout:8/957: rmdir d7/d13/d19/d95/d10f 39 2024-04-05T03:03:07.616 INFO:tasks.workunit.client.0.smithi153.stdout:8/958: write d7/d13/dab/fc0 [248698,72008] 0 2024-04-05T03:03:07.619 INFO:tasks.workunit.client.0.smithi153.stdout:5/956: getdents d0 0 2024-04-05T03:03:07.620 INFO:tasks.workunit.client.0.smithi153.stdout:8/959: dread d7/f28 [0,4194304] 0 2024-04-05T03:03:07.622 INFO:tasks.workunit.client.0.smithi153.stdout:8/960: symlink d7/d13/d19/d95/l128 0 2024-04-05T03:03:07.622 INFO:tasks.workunit.client.0.smithi153.stdout:8/961: fdatasync d7/dd5/ffe 0 2024-04-05T03:03:07.624 INFO:tasks.workunit.client.0.smithi153.stdout:8/962: creat d7/d13/dc8/f129 x:0 0 0 2024-04-05T03:03:07.626 INFO:tasks.workunit.client.0.smithi153.stdout:8/963: chown d7/d13/dc8/fed 12336846 1 2024-04-05T03:03:07.628 INFO:tasks.workunit.client.0.smithi153.stdout:8/964: truncate d7/d13/d19/d1b/d4f/d68/dbc/fbe 379898 0 2024-04-05T03:03:07.631 INFO:tasks.workunit.client.0.smithi153.stdout:8/965: rename d7/d13/dab/fc0 to d7/d13/d19/d1b/d6d/d8c/f12a 0 2024-04-05T03:03:07.633 INFO:tasks.workunit.client.0.smithi153.stdout:8/966: symlink d7/d85/l12b 0 2024-04-05T03:03:07.633 INFO:tasks.workunit.client.0.smithi153.stdout:8/967: fdatasync d7/d2a/fcf 0 2024-04-05T03:03:07.636 INFO:tasks.workunit.client.0.smithi153.stdout:8/968: mknod d7/d13/d19/d1b/d4f/d5f/c12c 0 2024-04-05T03:03:07.640 INFO:tasks.workunit.client.0.smithi153.stdout:8/969: rename d7/d13/d19/d1b/d4f/d68/f75 to d7/d13/d19/d1b/d6d/d8c/f12d 0 2024-04-05T03:03:07.648 INFO:tasks.workunit.client.0.smithi153.stdout:8/970: dwrite d7/d13/d19/d95/d10f/f11d [0,4194304] 0 2024-04-05T03:03:07.653 INFO:tasks.workunit.client.0.smithi153.stdout:8/971: creat d7/d46/f12e x:0 0 0 2024-04-05T03:03:07.656 INFO:tasks.workunit.client.0.smithi153.stdout:8/972: mknod d7/d13/d19/d1b/d6c/d118/c12f 0 2024-04-05T03:03:07.656 INFO:tasks.workunit.client.0.smithi153.stdout:8/973: readlink d7/d13/l48 0 2024-04-05T03:03:07.658 INFO:tasks.workunit.client.0.smithi153.stdout:8/974: creat d7/d2a/f130 x:0 0 0 2024-04-05T03:03:07.660 INFO:tasks.workunit.client.0.smithi153.stdout:8/975: truncate d7/d13/d19/d1b/db9/ff6 5238389 0 2024-04-05T03:03:07.662 INFO:tasks.workunit.client.0.smithi153.stdout:8/976: creat d7/d13/d19/d1b/d6d/d8c/f131 x:0 0 0 2024-04-05T03:03:07.664 INFO:tasks.workunit.client.0.smithi153.stdout:8/977: creat d7/d9a/f132 x:0 0 0 2024-04-05T03:03:07.665 INFO:tasks.workunit.client.0.smithi153.stdout:8/978: write d7/d13/d19/d1b/d4f/d5f/fb2 [2410996,65842] 0 2024-04-05T03:03:07.670 INFO:tasks.workunit.client.0.smithi153.stdout:8/979: dread d7/d13/d19/d1b/d4f/d5f/d54/fc3 [0,4194304] 0 2024-04-05T03:03:07.672 INFO:tasks.workunit.client.0.smithi153.stdout:8/980: creat d7/d2a/d50/f133 x:0 0 0 2024-04-05T03:03:07.676 INFO:tasks.workunit.client.0.smithi153.stdout:8/981: rename d7/d13/dab/fdc to d7/d13/d19/d1b/d6d/f134 0 2024-04-05T03:03:07.678 INFO:tasks.workunit.client.0.smithi153.stdout:8/982: mkdir d7/d2a/d135 0 2024-04-05T03:03:07.679 INFO:tasks.workunit.client.0.smithi153.stdout:8/983: write d7/d13/d19/f22 [1414286,98257] 0 2024-04-05T03:03:07.679 INFO:tasks.workunit.client.0.smithi153.stdout:8/984: readlink d7/d85/l49 0 2024-04-05T03:03:07.685 INFO:tasks.workunit.client.0.smithi153.stdout:8/985: dwrite d7/fa [0,4194304] 0 2024-04-05T03:03:07.689 INFO:tasks.workunit.client.0.smithi153.stdout:8/986: creat d7/d13/d19/d1b/d4f/d5f/df5/d103/f136 x:0 0 0 2024-04-05T03:03:07.690 INFO:tasks.workunit.client.0.smithi153.stdout:8/987: write d7/f8 [3331445,56828] 0 2024-04-05T03:03:07.696 INFO:tasks.workunit.client.0.smithi153.stdout:8/988: link d7/d9a/fe5 d7/d13/d19/d1b/d4f/d5f/df5/f137 0 2024-04-05T03:03:07.920 INFO:tasks.workunit.client.0.smithi153.stdout:9/977: sync 2024-04-05T03:03:07.923 INFO:tasks.workunit.client.0.smithi153.stdout:9/978: creat d1/d9/d1b/de9/d156/f163 x:0 0 0 2024-04-05T03:03:07.924 INFO:tasks.workunit.client.0.smithi153.stdout:9/979: write d1/d10/d36/d71/f119 [3527528,2567] 0 2024-04-05T03:03:07.946 INFO:tasks.workunit.client.0.smithi153.stdout:9/980: dwrite d1/fcd [0,4194304] 0 2024-04-05T03:03:07.949 INFO:tasks.workunit.client.0.smithi153.stdout:9/981: symlink d1/d9/d1b/de9/l164 0 2024-04-05T03:03:07.950 INFO:tasks.workunit.client.0.smithi153.stdout:9/982: write d1/d3a/d91/d96/dfc/f141 [889023,79021] 0 2024-04-05T03:03:07.951 INFO:tasks.workunit.client.0.smithi153.stdout:9/983: dread - d1/f110 zero size 2024-04-05T03:03:07.952 INFO:tasks.workunit.client.0.smithi153.stdout:9/984: fdatasync d1/d1d/f5a 0 2024-04-05T03:03:07.957 INFO:tasks.workunit.client.0.smithi153.stdout:9/985: dread d1/fcd [0,4194304] 0 2024-04-05T03:03:07.961 INFO:tasks.workunit.client.0.smithi153.stdout:9/986: truncate d1/d9/d1b/de2/d125/f76 1860996 0 2024-04-05T03:03:07.971 INFO:tasks.workunit.client.0.smithi153.stdout:9/987: dwrite d1/d9/d1b/de2/d125/fd1 [0,4194304] 0 2024-04-05T03:03:07.973 INFO:tasks.workunit.client.0.smithi153.stdout:9/988: chown d1/d10/d36/d57/d5c/l10d 9674 1 2024-04-05T03:03:07.974 INFO:tasks.workunit.client.0.smithi153.stdout:9/989: truncate d1/d3a/f83 758329 0 2024-04-05T03:03:07.977 INFO:tasks.workunit.client.0.smithi153.stdout:9/990: symlink d1/d10/d36/d57/d97/l165 0 2024-04-05T03:03:07.985 INFO:tasks.workunit.client.0.smithi153.stdout:9/991: dwrite d1/d9/d1b/f75 [0,4194304] 0 2024-04-05T03:03:07.987 INFO:tasks.workunit.client.0.smithi153.stdout:9/992: write d1/d10/d16/f23 [3975978,67105] 0 2024-04-05T03:03:07.991 INFO:tasks.workunit.client.0.smithi153.stdout:9/993: creat d1/d3a/d91/d96/dbd/dbe/f166 x:0 0 0 2024-04-05T03:03:07.994 INFO:tasks.workunit.client.0.smithi153.stdout:9/994: truncate d1/d9/d1b/f73 553243 0 2024-04-05T03:03:07.999 INFO:tasks.workunit.client.0.smithi153.stdout:5/957: sync 2024-04-05T03:03:08.000 INFO:tasks.workunit.client.0.smithi153.stdout:5/958: chown d0/d25/d4d/d62/d37 2502 1 2024-04-05T03:03:08.004 INFO:tasks.workunit.client.0.smithi153.stdout:9/995: dwrite d1/d10/d16/d13b/f147 [0,4194304] 0 2024-04-05T03:03:08.004 INFO:tasks.workunit.client.0.smithi153.stdout:5/959: creat d0/d25/d4d/d62/d5f/f145 x:0 0 0 2024-04-05T03:03:08.005 INFO:tasks.workunit.client.0.smithi153.stdout:5/960: fsync d0/f14 0 2024-04-05T03:03:08.009 INFO:tasks.workunit.client.0.smithi153.stdout:5/961: creat d0/d25/d4d/d62/d5f/db2/f146 x:0 0 0 2024-04-05T03:03:08.019 INFO:tasks.workunit.client.0.smithi153.stdout:5/962: rename d0/d25/d4d/d9c/dc4/dcb/d109/l11e to d0/d25/d4d/d9c/d10f/l147 0 2024-04-05T03:03:08.019 INFO:tasks.workunit.client.0.smithi153.stdout:9/996: dwrite d1/d3a/d91/d96/dbd/dbe/f106 [0,4194304] 0 2024-04-05T03:03:08.024 INFO:tasks.workunit.client.0.smithi153.stdout:5/963: mkdir d0/d2c/d148 0 2024-04-05T03:03:08.028 INFO:tasks.workunit.client.0.smithi153.stdout:9/997: rename d1/d6c/f82 to d1/d10/d16/d22/f167 0 2024-04-05T03:03:08.030 INFO:tasks.workunit.client.0.smithi153.stdout:5/964: symlink d0/d25/d4d/d9c/dc4/dcb/d109/l149 0 2024-04-05T03:03:08.037 INFO:tasks.workunit.client.0.smithi153.stdout:9/998: dwrite d1/d6c/f94 [0,4194304] 0 2024-04-05T03:03:08.056 INFO:tasks.workunit.client.0.smithi153.stdout:9/999: dwrite d1/d10/d36/d57/f155 [0,4194304] 0 2024-04-05T03:03:08.064 INFO:tasks.workunit.client.0.smithi153.stdout:8/989: sync 2024-04-05T03:03:08.067 INFO:tasks.workunit.client.0.smithi153.stdout:8/990: readlink d7/d13/d19/d1b/l5a 0 2024-04-05T03:03:08.070 INFO:tasks.workunit.client.0.smithi153.stdout:5/965: dread d0/d2c/d3f/d102/f111 [0,4194304] 0 2024-04-05T03:03:08.071 INFO:tasks.workunit.client.0.smithi153.stdout:8/991: symlink d7/d9a/l138 0 2024-04-05T03:03:08.072 INFO:tasks.workunit.client.0.smithi153.stdout:5/966: dread - d0/d25/d4d/d9c/d10f/f119 zero size 2024-04-05T03:03:08.074 INFO:tasks.workunit.client.0.smithi153.stdout:8/992: write d7/d13/d19/d1b/d4f/d5f/fe3 [2898692,81165] 0 2024-04-05T03:03:08.091 INFO:tasks.workunit.client.0.smithi153.stdout:5/967: dwrite d0/d25/fb1 [0,4194304] 0 2024-04-05T03:03:08.096 INFO:tasks.workunit.client.0.smithi153.stdout:5/968: symlink d0/d2c/d3f/d102/l14a 0 2024-04-05T03:03:08.099 INFO:tasks.workunit.client.0.smithi153.stdout:5/969: dread d0/d4e/fa5 [0,4194304] 0 2024-04-05T03:03:08.100 INFO:tasks.workunit.client.0.smithi153.stdout:5/970: dread - d0/d25/d4d/d62/d5f/db2/fdf zero size 2024-04-05T03:03:08.106 INFO:tasks.workunit.client.0.smithi153.stdout:8/993: dread d7/d13/d19/d1b/d4f/d5f/fe3 [0,4194304] 0 2024-04-05T03:03:08.108 INFO:tasks.workunit.client.0.smithi153.stdout:8/994: write d7/d13/d19/d1b/d4f/d5f/fe9 [1362934,92083] 0 2024-04-05T03:03:08.109 INFO:tasks.workunit.client.0.smithi153.stdout:8/995: fsync d7/d13/d19/d1b/d4f/de2/f102 0 2024-04-05T03:03:08.111 INFO:tasks.workunit.client.0.smithi153.stdout:8/996: chown d7/d13/d19/d1b/d6c/ccd 1341798 1 2024-04-05T03:03:08.114 INFO:tasks.workunit.client.0.smithi153.stdout:8/997: unlink d7/d85/l49 0 2024-04-05T03:03:08.116 INFO:tasks.workunit.client.0.smithi153.stdout:8/998: chown d7/d13/d19/d1b/d4f/d72/f99 318 1 2024-04-05T03:03:08.121 INFO:tasks.workunit.client.0.smithi153.stdout:5/971: dwrite d0/d25/d4d/d9c/dc4/dcb/fdc [0,4194304] 0 2024-04-05T03:03:08.125 INFO:tasks.workunit.client.0.smithi153.stdout:5/972: mknod d0/d4e/db6/c14b 0 2024-04-05T03:03:08.126 INFO:tasks.workunit.client.0.smithi153.stdout:8/999: getdents d7/d13/d19/d1b/d6c 0 2024-04-05T03:03:08.129 INFO:tasks.workunit.client.0.smithi153.stdout:5/973: mknod d0/d25/d4d/d62/d130/c14c 0 2024-04-05T03:03:08.132 INFO:tasks.workunit.client.0.smithi153.stdout:5/974: mknod d0/d25/d4d/d9c/dc7/dc9/c14d 0 2024-04-05T03:03:08.135 INFO:tasks.workunit.client.0.smithi153.stdout:5/975: creat d0/d25/d68/f14e x:0 0 0 2024-04-05T03:03:08.136 INFO:tasks.workunit.client.0.smithi153.stdout:5/976: fdatasync d0/d2c/d3f/fd9 0 2024-04-05T03:03:08.138 INFO:tasks.workunit.client.0.smithi153.stdout:5/977: creat d0/d25/d68/f14f x:0 0 0 2024-04-05T03:03:08.144 INFO:tasks.workunit.client.0.smithi153.stdout:5/978: dwrite d0/d2c/d5e/d11a/f131 [0,4194304] 0 2024-04-05T03:03:08.146 INFO:tasks.workunit.client.0.smithi153.stdout:5/979: chown d0/d25/d4d/d62/d37/fa6 49 1 2024-04-05T03:03:08.166 INFO:tasks.workunit.client.0.smithi153.stdout:5/980: dread d0/d25/f70 [0,4194304] 0 2024-04-05T03:03:08.170 INFO:tasks.workunit.client.0.smithi153.stdout:5/981: write d0/d25/d4d/d9c/faa [834036,44379] 0 2024-04-05T03:03:08.172 INFO:tasks.workunit.client.0.smithi153.stdout:5/982: creat d0/d25/d4d/d62/d130/f150 x:0 0 0 2024-04-05T03:03:08.174 INFO:tasks.workunit.client.0.smithi153.stdout:5/983: mkdir d0/d25/d4d/d9c/dc4/dcb/d109/d151 0 2024-04-05T03:03:08.179 INFO:tasks.workunit.client.0.smithi153.stdout:5/984: write d0/d25/d4d/d62/d5f/d86/f104 [815867,109961] 0 2024-04-05T03:03:08.183 INFO:tasks.workunit.client.0.smithi153.stdout:5/985: getdents d0 0 2024-04-05T03:03:08.184 INFO:tasks.workunit.client.0.smithi153.stdout:5/986: chown d0/d25/d4d/d9c/dc7/dc9/cd4 178558 1 2024-04-05T03:03:08.185 INFO:tasks.workunit.client.0.smithi153.stdout:5/987: write d0/d2c/d5e/d11a/f131 [728860,90506] 0 2024-04-05T03:03:08.189 INFO:tasks.workunit.client.0.smithi153.stdout:5/988: link d0/d25/d4d/d9c/dc7/dc9/c14d d0/d25/d4d/d9c/dc7/c152 0 2024-04-05T03:03:08.192 INFO:tasks.workunit.client.0.smithi153.stdout:5/989: creat d0/d25/d4d/d62/d5f/d86/dd7/f153 x:0 0 0 2024-04-05T03:03:08.193 INFO:tasks.workunit.client.0.smithi153.stdout:5/990: chown d0/d25/d4d/d9c/ce8 67708043 1 2024-04-05T03:03:08.194 INFO:tasks.workunit.client.0.smithi153.stdout:5/991: mknod d0/d25/d4d/d62/d5f/db2/c154 0 2024-04-05T03:03:08.197 INFO:tasks.workunit.client.0.smithi153.stdout:5/992: creat d0/d25/d4d/d9c/dc7/dc9/f155 x:0 0 0 2024-04-05T03:03:08.197 INFO:tasks.workunit.client.0.smithi153.stdout:5/993: stat d0/d25/d4d/d62/d130 0 2024-04-05T03:03:08.203 INFO:tasks.workunit.client.0.smithi153.stdout:5/994: dwrite d0/d25/d4d/d62/d37/fa6 [0,4194304] 0 2024-04-05T03:03:08.207 INFO:tasks.workunit.client.0.smithi153.stdout:5/995: dread - d0/d25/d68/fce zero size 2024-04-05T03:03:08.210 INFO:tasks.workunit.client.0.smithi153.stdout:5/996: fsync d0/d2c/d5e/d11a/f12d 0 2024-04-05T03:03:08.212 INFO:tasks.workunit.client.0.smithi153.stdout:5/997: mknod d0/d25/d4d/d62/c156 0 2024-04-05T03:03:08.214 INFO:tasks.workunit.client.0.smithi153.stdout:5/998: mknod d0/d2c/d148/c157 0 2024-04-05T03:03:08.221 INFO:tasks.workunit.client.0.smithi153.stdout:5/999: dwrite d0/d25/d4d/d62/dbb/ff5 [0,4194304] 0 2024-04-05T03:03:08.228 INFO:tasks.workunit.client.0.smithi153.stderr:+ rm -rf -- ./tmp.poLuWgRt5g 2024-04-05T03:03:08.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:03:08.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:03:08.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:03:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:03:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:03:08.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:08.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:09.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:09 smithi203 ceph-mon[39911]: Reconfiguring node-exporter.smithi153 (dependencies changed)... 2024-04-05T03:03:09.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:09 smithi203 ceph-mon[39911]: Reconfiguring daemon node-exporter.smithi153 on smithi153 2024-04-05T03:03:09.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:09 smithi203 ceph-mon[39911]: pgmap v19: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 104 MiB/s rd, 234 MiB/s wr, 598 op/s 2024-04-05T03:03:09.553 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:09 smithi153 ceph-mon[29381]: Reconfiguring node-exporter.smithi153 (dependencies changed)... 2024-04-05T03:03:09.553 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:09 smithi153 ceph-mon[29381]: Reconfiguring daemon node-exporter.smithi153 on smithi153 2024-04-05T03:03:09.553 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:09 smithi153 ceph-mon[29381]: pgmap v19: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 104 MiB/s rd, 234 MiB/s wr, 598 op/s 2024-04-05T03:03:11.498 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-05T03:03:11.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-05T03:03:11.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:11 smithi153 ceph-mon[29381]: pgmap v20: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 53 MiB/s rd, 112 MiB/s wr, 341 op/s 2024-04-05T03:03:11.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:11 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:11.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:11 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:11.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:11 smithi153 ceph-mon[29381]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-04-05T03:03:11.524 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:11 smithi203 ceph-mon[39911]: pgmap v20: 65 pgs: 65 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 53 MiB/s rd, 112 MiB/s wr, 341 op/s 2024-04-05T03:03:11.524 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:11 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:11.524 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:11 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:11.524 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:11 smithi203 ceph-mon[39911]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-04-05T03:03:12.257 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:12 smithi153 ceph-mon[29381]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-04-05T03:03:12.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:12 smithi203 ceph-mon[39911]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-04-05T03:03:13.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:13 smithi203 ceph-mon[39911]: pgmap v21: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 54 MiB/s rd, 118 MiB/s wr, 391 op/s 2024-04-05T03:03:13.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:13 smithi153 ceph-mon[29381]: pgmap v21: 65 pgs: 65 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 54 MiB/s rd, 118 MiB/s wr, 391 op/s 2024-04-05T03:03:15.462 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:15 smithi153 ceph-mon[29381]: pgmap v22: 65 pgs: 65 active+clean; 2.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 46 MiB/s rd, 103 MiB/s wr, 370 op/s 2024-04-05T03:03:15.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:15 smithi203 ceph-mon[39911]: pgmap v22: 65 pgs: 65 active+clean; 2.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 46 MiB/s rd, 103 MiB/s wr, 370 op/s 2024-04-05T03:03:17.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:17 smithi203 ceph-mon[39911]: pgmap v23: 65 pgs: 65 active+clean; 2.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 24 MiB/s rd, 59 MiB/s wr, 267 op/s 2024-04-05T03:03:17.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:17 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:17.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:17 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:17.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:17 smithi203 ceph-mon[39911]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-04-05T03:03:17.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:17 smithi203 ceph-mon[39911]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-04-05T03:03:17.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:17 smithi153 ceph-mon[29381]: pgmap v23: 65 pgs: 65 active+clean; 2.4 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 24 MiB/s rd, 59 MiB/s wr, 267 op/s 2024-04-05T03:03:17.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:17 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:17.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:17 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:17.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:17 smithi153 ceph-mon[29381]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-04-05T03:03:17.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:17 smithi153 ceph-mon[29381]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-04-05T03:03:19.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:19 smithi203 ceph-mon[39911]: pgmap v24: 65 pgs: 65 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 24 MiB/s rd, 60 MiB/s wr, 427 op/s 2024-04-05T03:03:19.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:19 smithi153 ceph-mon[29381]: pgmap v24: 65 pgs: 65 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 24 MiB/s rd, 60 MiB/s wr, 427 op/s 2024-04-05T03:03:19.724 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-05T03:03:19.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-05T03:03:21.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:21 smithi203 ceph-mon[39911]: pgmap v25: 65 pgs: 65 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 270 op/s 2024-04-05T03:03:21.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:21 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:21 smithi153 ceph-mon[29381]: pgmap v25: 65 pgs: 65 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 270 op/s 2024-04-05T03:03:21.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:21 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:23.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:23 smithi203 ceph-mon[39911]: pgmap v26: 65 pgs: 65 active+clean; 789 MiB data, 3.9 GiB used, 532 GiB / 536 GiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 338 op/s 2024-04-05T03:03:23.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:23 smithi153 ceph-mon[29381]: pgmap v26: 65 pgs: 65 active+clean; 789 MiB data, 3.9 GiB used, 532 GiB / 536 GiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 338 op/s 2024-04-05T03:03:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:25 smithi153 ceph-mon[29381]: pgmap v27: 65 pgs: 65 active+clean; 290 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 395 op/s 2024-04-05T03:03:25.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:25 smithi203 ceph-mon[39911]: pgmap v27: 65 pgs: 65 active+clean; 290 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 395 op/s 2024-04-05T03:03:27.290 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:27 smithi153 ceph-mon[29381]: pgmap v28: 65 pgs: 65 active+clean; 290 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-05T03:03:27.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:27 smithi203 ceph-mon[39911]: pgmap v28: 65 pgs: 65 active+clean; 290 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 -- 172.21.15.153:0/2356134202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fcf80 msgr2=0x7fcb600fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 --2- 172.21.15.153:0/2356134202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fcf80 0x7fcb600fd3f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb54004940 tx=0x7fcb54009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 -- 172.21.15.153:0/2356134202 shutdown_connections 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 --2- 172.21.15.153:0/2356134202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fcf80 0x7fcb600fd3f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 --2- 172.21.15.153:0/2356134202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fbcd0 0x7fcb600fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 -- 172.21.15.153:0/2356134202 >> 172.21.15.153:0/2356134202 conn(0x7fcb600f7430 msgr2=0x7fcb600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 -- 172.21.15.153:0/2356134202 shutdown_connections 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.518+0000 7fcb65b07700 1 -- 172.21.15.153:0/2356134202 wait complete. 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 Processor -- start 2024-04-05T03:03:28.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 -- start start 2024-04-05T03:03:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb601106c0 con 0x7fcb600fcf80 2024-04-05T03:03:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.519+0000 7fcb65b07700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60110f10 con 0x7fcb600fbcd0 2024-04-05T03:03:28.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39808/0 (socket says 172.21.15.153:39808) 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:43170/0 (socket says 172.21.15.153:43170) 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5effd700 1 -- 172.21.15.153:0/1176198803 learned_addr learned my addr 172.21.15.153:0/1176198803 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 -- 172.21.15.153:0/1176198803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 msgr2=0x7fcb601100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 -- 172.21.15.153:0/1176198803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54005040 con 0x7fcb600fbcd0 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5effd700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb5f7fe700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ba80 tx=0x7fcb5000d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:28.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb50010040 con 0x7fcb600fbcd0 2024-04-05T03:03:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb5000ee10 con 0x7fcb600fbcd0 2024-04-05T03:03:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.520+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb60111190 con 0x7fcb600fbcd0 2024-04-05T03:03:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.521+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb50014400 con 0x7fcb600fbcd0 2024-04-05T03:03:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.521+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb601116d0 con 0x7fcb600fbcd0 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.522+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6005f5f0 con 0x7fcb600fbcd0 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.523+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7fcb5000e940 con 0x7fcb600fbcd0 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.523+0000 7fcb47fff700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 0x7fcb4806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.523+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fcb50089d00 con 0x7fcb600fbcd0 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.523+0000 7fcb5effd700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 0x7fcb4806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.524+0000 7fcb5effd700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 0x7fcb4806d9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009fd0 tx=0x7fcb54009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:28.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.528+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fcb5004fb80 con 0x7fcb600fbcd0 2024-04-05T03:03:28.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.827+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb60111ab0 con 0x7fcb4806b540 2024-04-05T03:03:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.829+0000 7fcb47fff700 1 -- 172.21.15.153:0/1176198803 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fcb60111ab0 con 0x7fcb4806b540 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 msgr2=0x7fcb4806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 0x7fcb4806d9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb54009fd0 tx=0x7fcb54009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 msgr2=0x7fcb6010fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000ba80 tx=0x7fcb5000d040 comp rx=0 tx=0).stop 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 shutdown_connections 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fcb4806b540 0x7fcb4806d9f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb600fbcd0 0x7fcb6010fb80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 --2- 172.21.15.153:0/1176198803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb600fcf80 0x7fcb601100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 >> 172.21.15.153:0/1176198803 conn(0x7fcb600f7430 msgr2=0x7fcb600f8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 shutdown_connections 2024-04-05T03:03:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.831+0000 7fcb65b07700 1 -- 172.21.15.153:0/1176198803 wait complete. 2024-04-05T03:03:28.845 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:03:28.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/687617104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fcf80 msgr2=0x7fdd380fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/687617104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fcf80 0x7fdd380fd3f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c004940 tx=0x7fdd2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:28.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/687617104 shutdown_connections 2024-04-05T03:03:28.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/687617104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fcf80 0x7fdd380fd3f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/687617104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fbcd0 0x7fdd380fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/687617104 >> 172.21.15.153:0/687617104 conn(0x7fdd380f7430 msgr2=0x7fdd380f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:28.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/687617104 shutdown_connections 2024-04-05T03:03:28.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.968+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/687617104 wait complete. 2024-04-05T03:03:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.969+0000 7fdd3d2b6700 1 Processor -- start 2024-04-05T03:03:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.969+0000 7fdd3d2b6700 1 -- start start 2024-04-05T03:03:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd3d2b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 0x7fdd3810c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd3d2b6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 0x7fdd3810cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd3d2b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3810d3c0 con 0x7fdd380fbcd0 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 0x7fdd3810c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd367fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 0x7fdd3810cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd3d2b6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3810dc10 con 0x7fdd380fcf80 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 0x7fdd3810c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34000/0 (socket says 172.21.15.153:34000) 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd36ffd700 1 -- 172.21.15.153:0/2320690755 learned_addr learned my addr 172.21.15.153:0/2320690755 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd367fc700 1 -- 172.21.15.153:0/2320690755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 msgr2=0x7fdd3810c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd367fc700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 0x7fdd3810c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.970+0000 7fdd367fc700 1 -- 172.21.15.153:0/2320690755 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c005040 con 0x7fdd380fcf80 2024-04-05T03:03:28.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd367fc700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 0x7fdd3810cdc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c000c00 tx=0x7fdd2c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c00e030 con 0x7fdd380fcf80 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd2c013de0 con 0x7fdd380fcf80 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c01ce50 con 0x7fdd380fcf80 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3810de90 con 0x7fdd380fcf80 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.971+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3810e2c0 con 0x7fdd380fcf80 2024-04-05T03:03:28.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.972+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3805f5f0 con 0x7fdd380fcf80 2024-04-05T03:03:28.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.973+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7fdd2c00b4a0 con 0x7fdd380fcf80 2024-04-05T03:03:28.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.974+0000 7fdd1f7fe700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 0x7fdd2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:28.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.974+0000 7fdd36ffd700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 0x7fdd2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:28.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.974+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fdd2c08b9f0 con 0x7fdd380fcf80 2024-04-05T03:03:28.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.975+0000 7fdd36ffd700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 0x7fdd2006d990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd28008fd0 tx=0x7fdd2800b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:28.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:28.979+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fdd2c0593d0 con 0x7fdd380fcf80 2024-04-05T03:03:29.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.320+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd3810e630 con 0x7fdd2006b4e0 2024-04-05T03:03:29.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.322+0000 7fdd1f7fe700 1 -- 172.21.15.153:0/2320690755 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fdd3810e630 con 0x7fdd2006b4e0 2024-04-05T03:03:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.326+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 msgr2=0x7fdd2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.326+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 0x7fdd2006d990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdd28008fd0 tx=0x7fdd2800b040 comp rx=0 tx=0).stop 2024-04-05T03:03:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.326+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 msgr2=0x7fdd3810cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.326+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 0x7fdd3810cdc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c000c00 tx=0x7fdd2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 shutdown_connections 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd380fbcd0 0x7fdd3810c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdd2006b4e0 0x7fdd2006d990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 --2- 172.21.15.153:0/2320690755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380fcf80 0x7fdd3810cdc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 >> 172.21.15.153:0/2320690755 conn(0x7fdd380f7430 msgr2=0x7fdd380f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 shutdown_connections 2024-04-05T03:03:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.327+0000 7fdd3d2b6700 1 -- 172.21.15.153:0/2320690755 wait complete. 2024-04-05T03:03:29.470 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:29 smithi153 ceph-mon[29381]: pgmap v29: 65 pgs: 65 active+clean; 283 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 573 op/s 2024-04-05T03:03:29.471 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:29 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:29.471 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:29 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 -- 172.21.15.153:0/2745443749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fcfa0 msgr2=0x7fdf500fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 --2- 172.21.15.153:0/2745443749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fcfa0 0x7fdf500fd410 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdf40004940 tx=0x7fdf40009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 -- 172.21.15.153:0/2745443749 shutdown_connections 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 --2- 172.21.15.153:0/2745443749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fcfa0 0x7fdf500fd410 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 --2- 172.21.15.153:0/2745443749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fbcf0 0x7fdf500fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 -- 172.21.15.153:0/2745443749 >> 172.21.15.153:0/2745443749 conn(0x7fdf500f7450 msgr2=0x7fdf500f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 -- 172.21.15.153:0/2745443749 shutdown_connections 2024-04-05T03:03:29.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.471+0000 7fdf582d3700 1 -- 172.21.15.153:0/2745443749 wait complete. 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 Processor -- start 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 -- start start 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5010c300 con 0x7fdf500fbcf0 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf582d3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5010cb50 con 0x7fdf500fcfa0 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf5606f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf5586e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf5606f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34022/0 (socket says 172.21.15.153:34022) 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf5586e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:37864/0 (socket says 172.21.15.153:37864) 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.472+0000 7fdf5606f700 1 -- 172.21.15.153:0/4194057959 learned_addr learned my addr 172.21.15.153:0/4194057959 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf5606f700 1 -- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 msgr2=0x7fdf5010bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf5606f700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf5606f700 1 -- 172.21.15.153:0/4194057959 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf40005040 con 0x7fdf500fbcf0 2024-04-05T03:03:29.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf5586e700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:03:29.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf5606f700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00eb70 tx=0x7fdf4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.473+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf4c007cc0 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.474+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf4c00f040 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.474+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5010cdd0 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.474+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf4c009640 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.474+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5010d230 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.475+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5005f5f0 con 0x7fdf500fbcf0 2024-04-05T03:03:29.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.475+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7fdf4c0097a0 con 0x7fdf500fbcf0 2024-04-05T03:03:29.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.476+0000 7fdf46ffd700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 0x7fdf3c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.476+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fdf4c089d20 con 0x7fdf500fbcf0 2024-04-05T03:03:29.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.476+0000 7fdf5586e700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 0x7fdf3c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.477+0000 7fdf5586e700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 0x7fdf3c06d9e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009b30 tx=0x7fdf40009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:29.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.483+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fdf4c057570 con 0x7fdf500fbcf0 2024-04-05T03:03:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:29 smithi203 ceph-mon[39911]: pgmap v29: 65 pgs: 65 active+clean; 283 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 573 op/s 2024-04-05T03:03:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:29 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:29 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:29.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.697+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf5010d650 con 0x7fdf3c06b530 2024-04-05T03:03:29.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.711+0000 7fdf46ffd700 1 -- 172.21.15.153:0/4194057959 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fdf5010d650 con 0x7fdf3c06b530 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 starting - - - - 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (10m) 25s ago 10m 6660k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (8m) 30s ago 8m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 starting - - - - 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (3m) 25s ago 3m 124M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (3m) 25s ago 3m 69.9M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (3m) 30s ago 3m 72.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (3m) 30s ago 3m 117M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:03:29.711 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:9283 running (11m) 25s ago 11m 285M - 17.2.7-805-g6fda41ff a49113eacc58 18dc5839d05a 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (89s) 30s ago 7m 489M - 19.0.0-2013-g0028525c d8c35eaf3c25 f4c2fc4aeae0 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (11m) 25s ago 11m 49.2M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (7m) 30s ago 7m 41.8M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 starting - - - - 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (8m) 30s ago 8m 15.7M - 1.5.0 0da6a335fe13 b5787d29758b 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (7m) 25s ago 7m 193M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 25s ago 6m 191M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 25s ago 6m 147M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (6m) 30s ago 5m 228M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (5m) 30s ago 5m 193M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (5m) 30s ago 5m 236M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:03:29.712 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (7m) 30s ago 7m 61.0M - 2.43.0 a07b618ecd1d 350bb8305d38 2024-04-05T03:03:29.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 msgr2=0x7fdf3c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 0x7fdf3c06d9e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009b30 tx=0x7fdf40009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 msgr2=0x7fdf5010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00eb70 tx=0x7fdf4c00b040 comp rx=0 tx=0).stop 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 shutdown_connections 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf500fbcf0 0x7fdf5010b7c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fdf3c06b530 0x7fdf3c06d9e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 --2- 172.21.15.153:0/4194057959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf500fcfa0 0x7fdf5010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.713+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 >> 172.21.15.153:0/4194057959 conn(0x7fdf500f7450 msgr2=0x7fdf500f8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.714+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 shutdown_connections 2024-04-05T03:03:29.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.714+0000 7fdf582d3700 1 -- 172.21.15.153:0/4194057959 wait complete. 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 -- 172.21.15.153:0/4018286471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 msgr2=0x7fe5840fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 --2- 172.21.15.153:0/4018286471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe5840fd3f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe574004940 tx=0x7fe574009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 -- 172.21.15.153:0/4018286471 shutdown_connections 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 --2- 172.21.15.153:0/4018286471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe5840fd3f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 --2- 172.21.15.153:0/4018286471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe5840fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 -- 172.21.15.153:0/4018286471 >> 172.21.15.153:0/4018286471 conn(0x7fe5840f7430 msgr2=0x7fe5840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 -- 172.21.15.153:0/4018286471 shutdown_connections 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.841+0000 7fe58baad700 1 -- 172.21.15.153:0/4018286471 wait complete. 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 Processor -- start 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 -- start start 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe58410fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe584110110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584110710 con 0x7fe5840fcf80 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe58baad700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584110f10 con 0x7fe5840fbcd0 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe589048700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe584110110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe589048700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe584110110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34032/0 (socket says 172.21.15.153:34032) 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe589849700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe58410fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.842+0000 7fe589048700 1 -- 172.21.15.153:0/698108439 learned_addr learned my addr 172.21.15.153:0/698108439 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe589849700 1 -- 172.21.15.153:0/698108439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 msgr2=0x7fe584110110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe589849700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe584110110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe589849700 1 -- 172.21.15.153:0/698108439 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574005040 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe589849700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe58410fbd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe580009fd0 tx=0x7fe58000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe580004e90 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe584111190 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe58000f040 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5841116d0 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.843+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58000ca50 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.844+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5680052e0 con 0x7fe5840fbcd0 2024-04-05T03:03:29.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.845+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7fe58000cbb0 con 0x7fe5840fbcd0 2024-04-05T03:03:29.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.846+0000 7fe57a7fc700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 0x7fe57006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:29.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.846+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fe580089a40 con 0x7fe5840fbcd0 2024-04-05T03:03:29.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.846+0000 7fe589048700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 0x7fe57006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:29.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.846+0000 7fe589048700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 0x7fe57006d9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe574009b30 tx=0x7fe574009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:29.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:29.853+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fe58004fb80 con 0x7fe5840fbcd0 2024-04-05T03:03:30.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.287+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe568005860 con 0x7fe5840fbcd0 2024-04-05T03:03:30.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.288+0000 7fe57a7fc700 1 -- 172.21.15.153:0/698108439 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7fe580019360 con 0x7fe5840fbcd0 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:03:30.288 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:03:30.289 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 13, 2024-04-05T03:03:30.289 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:03:30.289 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:03:30.289 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 msgr2=0x7fe57006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 0x7fe57006d9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe574009b30 tx=0x7fe574009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 msgr2=0x7fe58410fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe58410fbd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe580009fd0 tx=0x7fe58000b040 comp rx=0 tx=0).stop 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 shutdown_connections 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fe57006b540 0x7fe57006d9f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5840fbcd0 0x7fe58410fbd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 --2- 172.21.15.153:0/698108439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5840fcf80 0x7fe584110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 >> 172.21.15.153:0/698108439 conn(0x7fe5840f7430 msgr2=0x7fe584065ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 shutdown_connections 2024-04-05T03:03:30.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.290+0000 7fe58baad700 1 -- 172.21.15.153:0/698108439 wait complete. 2024-04-05T03:03:30.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 -- 172.21.15.153:0/109060350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fe040 msgr2=0x7fbf300fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 --2- 172.21.15.153:0/109060350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fe040 0x7fbf300fe4b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c004940 tx=0x7fbf1c009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:30.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 -- 172.21.15.153:0/109060350 shutdown_connections 2024-04-05T03:03:30.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:30 smithi153 ceph-mon[29381]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:30.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:30 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:03:30.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:30 smithi153 ceph-mon[29381]: Reconfiguring node-exporter.smithi203 (dependencies changed)... 2024-04-05T03:03:30.400 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:30 smithi153 ceph-mon[29381]: Reconfiguring daemon node-exporter.smithi203 on smithi203 2024-04-05T03:03:30.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 --2- 172.21.15.153:0/109060350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fe040 0x7fbf300fe4b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 --2- 172.21.15.153:0/109060350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fce40 0x7fbf300fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 -- 172.21.15.153:0/109060350 >> 172.21.15.153:0/109060350 conn(0x7fbf300f8510 msgr2=0x7fbf300fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:30.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 -- 172.21.15.153:0/109060350 shutdown_connections 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.399+0000 7fbf3705c700 1 -- 172.21.15.153:0/109060350 wait complete. 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 Processor -- start 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 -- start start 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fe040 0x7fbf3010bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3010c2e0 con 0x7fbf300fce40 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf3705c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3010cac0 con 0x7fbf300fe040 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf34df8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf34df8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34060/0 (socket says 172.21.15.153:34060) 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.400+0000 7fbf34df8700 1 -- 172.21.15.153:0/787742445 learned_addr learned my addr 172.21.15.153:0/787742445 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:03:30.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf2ffff700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fe040 0x7fbf3010bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf34df8700 1 -- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fe040 msgr2=0x7fbf3010bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf34df8700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fe040 0x7fbf3010bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf34df8700 1 -- 172.21.15.153:0/787742445 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c005040 con 0x7fbf300fce40 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf34df8700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400eb70 tx=0x7fbf2400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf24007cc0 con 0x7fbf300fce40 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3010cd40 con 0x7fbf300fce40 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf2400f040 con 0x7fbf300fce40 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3010d280 con 0x7fbf300fce40 2024-04-05T03:03:30.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.401+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf2400cb20 con 0x7fbf300fce40 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.403+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3005f5f0 con 0x7fbf300fce40 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.403+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7fbf2400cc80 con 0x7fbf300fce40 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.403+0000 7fbf2d7fa700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 0x7fbf20071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.404+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fbf24089b70 con 0x7fbf300fce40 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.404+0000 7fbf2ffff700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 0x7fbf20071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:03:30.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.404+0000 7fbf2ffff700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 0x7fbf20071df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b30 tx=0x7fbf1c009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:03:30.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.411+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fbf2404fb80 con 0x7fbf300fce40 2024-04-05T03:03:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:30 smithi203 ceph-mon[39911]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:30 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:03:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:30 smithi203 ceph-mon[39911]: Reconfiguring node-exporter.smithi203 (dependencies changed)... 2024-04-05T03:03:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:30 smithi203 ceph-mon[39911]: Reconfiguring daemon node-exporter.smithi203 on smithi203 2024-04-05T03:03:30.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.715+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf3010d5b0 con 0x7fbf2006f940 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.716+0000 7fbf2d7fa700 1 -- 172.21.15.153:0/787742445 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fbf3010d5b0 con 0x7fbf2006f940 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "", 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:03:30.716 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:03:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 msgr2=0x7fbf20071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 0x7fbf20071df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b30 tx=0x7fbf1c009e30 comp rx=0 tx=0).stop 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 msgr2=0x7fbf3010b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf2400eb70 tx=0x7fbf2400b040 comp rx=0 tx=0).stop 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 shutdown_connections 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf300fce40 0x7fbf3010b7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7fbf2006f940 0x7fbf20071df0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.718+0000 7fbf3705c700 1 --2- 172.21.15.153:0/787742445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf300fe040 0x7fbf3010bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.719+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 >> 172.21.15.153:0/787742445 conn(0x7fbf300f8510 msgr2=0x7fbf300f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.719+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 shutdown_connections 2024-04-05T03:03:30.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:03:30.719+0000 7fbf3705c700 1 -- 172.21.15.153:0/787742445 wait complete. 2024-04-05T03:03:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:31 smithi203 ceph-mon[39911]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:31 smithi203 ceph-mon[39911]: pgmap v30: 65 pgs: 65 active+clean; 283 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 413 op/s 2024-04-05T03:03:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:31 smithi203 ceph-mon[39911]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:31 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/698108439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:03:31.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:31 smithi153 ceph-mon[29381]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:31.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:31 smithi153 ceph-mon[29381]: pgmap v30: 65 pgs: 65 active+clean; 283 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 413 op/s 2024-04-05T03:03:31.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:31 smithi153 ceph-mon[29381]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:31.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:31 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/698108439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:03:32.453 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:32 smithi203 ceph-mon[39911]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:32.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:32 smithi153 ceph-mon[29381]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:03:33.510 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:33 smithi153 ceph-mon[29381]: pgmap v31: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 481 op/s 2024-04-05T03:03:33.510 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:33 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:33.510 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:33 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:33.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:33 smithi203 ceph-mon[39911]: pgmap v31: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 481 op/s 2024-04-05T03:03:33.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:33 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:33.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:33 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:34.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:34 smithi203 ceph-mon[39911]: Reconfiguring prometheus.smithi203 (dependencies changed)... 2024-04-05T03:03:34.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:34 smithi153 ceph-mon[29381]: Reconfiguring prometheus.smithi203 (dependencies changed)... 2024-04-05T03:03:35.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:35 smithi203 ceph-mon[39911]: Reconfiguring daemon prometheus.smithi203 on smithi203 2024-04-05T03:03:35.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:35 smithi203 ceph-mon[39911]: pgmap v32: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 541 op/s 2024-04-05T03:03:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:35 smithi153 ceph-mon[29381]: Reconfiguring daemon prometheus.smithi203 on smithi203 2024-04-05T03:03:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:35 smithi153 ceph-mon[29381]: pgmap v32: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 541 op/s 2024-04-05T03:03:36.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:36 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:36.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:36 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:37.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:37 smithi203 ceph-mon[39911]: pgmap v33: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 434 op/s 2024-04-05T03:03:37.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:37 smithi153 ceph-mon[29381]: pgmap v33: 65 pgs: 65 active+clean; 280 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 434 op/s 2024-04-05T03:03:39.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:39 smithi203 ceph-mon[39911]: pgmap v34: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 647 op/s 2024-04-05T03:03:39.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:39 smithi153 ceph-mon[29381]: pgmap v34: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 647 op/s 2024-04-05T03:03:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: pgmap v35: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 409 op/s 2024-04-05T03:03:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:41.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: Upgrade: Need to upgrade myself (mgr.smithi203.iakodz) 2024-04-05T03:03:41.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:40 smithi153 ceph-mon[29381]: Upgrade: Need to upgrade myself (mgr.smithi203.iakodz) 2024-04-05T03:03:41.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: pgmap v35: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 409 op/s 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: Upgrade: Need to upgrade myself (mgr.smithi203.iakodz) 2024-04-05T03:03:41.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:40 smithi203 ceph-mon[39911]: Upgrade: Need to upgrade myself (mgr.smithi203.iakodz) 2024-04-05T03:03:43.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:42 smithi153 ceph-mon[29381]: pgmap v36: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 539 op/s 2024-04-05T03:03:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:42 smithi203 ceph-mon[39911]: pgmap v36: 65 pgs: 65 active+clean; 278 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 539 op/s 2024-04-05T03:03:45.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:44 smithi203 ceph-mon[39911]: pgmap v37: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 575 op/s 2024-04-05T03:03:45.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:44 smithi153 ceph-mon[29381]: pgmap v37: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 575 op/s 2024-04-05T03:03:47.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:46 smithi203 ceph-mon[39911]: pgmap v38: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 446 op/s 2024-04-05T03:03:47.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:46 smithi153 ceph-mon[29381]: pgmap v38: 65 pgs: 65 active+clean; 273 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 446 op/s 2024-04-05T03:03:49.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: pgmap v39: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 670 op/s 2024-04-05T03:03:49.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:49.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:03:49.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:03:49.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:03:49.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:48 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: pgmap v39: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 670 op/s 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:03:49.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:48 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:03:50.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:49 smithi153 ceph-mon[29381]: Upgrade: Updating mgr.smithi153.npzyqy 2024-04-05T03:03:50.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:49 smithi153 ceph-mon[29381]: Deploying daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T03:03:50.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:49 smithi203 ceph-mon[39911]: Upgrade: Updating mgr.smithi153.npzyqy 2024-04-05T03:03:50.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:49 smithi203 ceph-mon[39911]: Deploying daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T03:03:51.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:50 smithi153 ceph-mon[29381]: pgmap v40: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-04-05T03:03:51.289 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:50 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:51.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:50 smithi203 ceph-mon[39911]: pgmap v40: 65 pgs: 65 active+clean; 268 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 457 op/s 2024-04-05T03:03:51.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:50 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:03:53.285 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:52 smithi153 ceph-mon[29381]: pgmap v41: 65 pgs: 65 active+clean; 265 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 556 op/s 2024-04-05T03:03:53.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:52 smithi203 ceph-mon[39911]: pgmap v41: 65 pgs: 65 active+clean; 265 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 556 op/s 2024-04-05T03:03:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:54 smithi203 ceph-mon[39911]: pgmap v42: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 533 op/s 2024-04-05T03:03:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:54 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:54 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:54 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:03:55.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:54 smithi153 ceph-mon[29381]: pgmap v42: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 533 op/s 2024-04-05T03:03:55.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:54 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:55.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:54 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:55.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:54 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:03:56.499 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:56 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:56.499 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:56 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:56.499 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:56 smithi153 ceph-mon[29381]: pgmap v43: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 429 op/s 2024-04-05T03:03:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:56 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:56 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:03:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:56 smithi203 ceph-mon[39911]: pgmap v43: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 429 op/s 2024-04-05T03:03:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:03:58 smithi203 ceph-mon[39911]: pgmap v44: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 614 op/s 2024-04-05T03:03:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:03:58 smithi153 ceph-mon[29381]: pgmap v44: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 614 op/s 2024-04-05T03:04:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 -- 172.21.15.153:0/1935608435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa40fbb40 msgr2=0x7f0aa40fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/1935608435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa40fbb40 0x7f0aa40fbf50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004940 tx=0x7f0a98009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:00.880 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:00 smithi153 ceph-mon[29381]: pgmap v45: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 390 op/s 2024-04-05T03:04:00.880 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:00 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:00.880 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:00 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 -- 172.21.15.153:0/1935608435 shutdown_connections 2024-04-05T03:04:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/1935608435 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa40fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/1935608435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa40fbb40 0x7f0aa40fbf50 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004940 tx=0x7f0a98009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.879+0000 7f0aa9b63700 1 -- 172.21.15.153:0/1935608435 >> 172.21.15.153:0/1935608435 conn(0x7f0aa40f72a0 msgr2=0x7f0aa40f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.880+0000 7f0aa9b63700 1 -- 172.21.15.153:0/1935608435 shutdown_connections 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.880+0000 7f0aa9b63700 1 -- 172.21.15.153:0/1935608435 wait complete. 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 Processor -- start 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 -- start start 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa410bc70 0x7f0aa410cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410c1f0 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.881+0000 7f0aa9b63700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa410c360 con 0x7f0aa410bc70 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41626/0 (socket says 172.21.15.153:41626) 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 -- 172.21.15.153:0/2992673996 learned_addr learned my addr 172.21.15.153:0/2992673996 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa3fff700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa410bc70 0x7f0aa410cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 -- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa410bc70 msgr2=0x7f0aa410cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa410bc70 0x7f0aa410cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 -- 172.21.15.153:0/2992673996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98005040 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.882+0000 7f0aa8b61700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0a98000c00 tx=0x7f0a98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.883+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a9800e070 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.883+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a98009430 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.883+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa410d0f0 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.884+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a9800b410 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.884+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa410d650 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.885+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa4101780 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.886+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f0a9801c750 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.887+0000 7f0aa17fa700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 0x7f0a8c06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:00.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.887+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f0a9808b670 con 0x7f0aa40fcdf0 2024-04-05T03:04:00.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.888+0000 7f0aa3fff700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 0x7f0a8c06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:00.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.888+0000 7f0aa3fff700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 0x7f0a8c06dbd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a94004600 tx=0x7f0a9400b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:00.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:00.892+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f0a98055410 con 0x7f0aa40fcdf0 2024-04-05T03:04:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:00 smithi203 ceph-mon[39911]: pgmap v45: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 390 op/s 2024-04-05T03:04:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:00 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:00 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:01.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.189+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa4104710 con 0x7f0a8c06b720 2024-04-05T03:04:01.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.190+0000 7f0aa17fa700 1 -- 172.21.15.153:0/2992673996 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f0aa4104710 con 0x7f0a8c06b720 2024-04-05T03:04:01.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 msgr2=0x7f0a8c06dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 0x7f0a8c06dbd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a94004600 tx=0x7f0a9400b040 comp rx=0 tx=0).stop 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 msgr2=0x7f0aa410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0a98000c00 tx=0x7f0a98009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 shutdown_connections 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aa40fcdf0 0x7f0aa410b730 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f0a8c06b720 0x7f0a8c06dbd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 --2- 172.21.15.153:0/2992673996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aa410bc70 0x7f0aa410cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.193+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 >> 172.21.15.153:0/2992673996 conn(0x7f0aa40f72a0 msgr2=0x7f0aa4066720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.194+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 shutdown_connections 2024-04-05T03:04:01.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.194+0000 7f0aa9b63700 1 -- 172.21.15.153:0/2992673996 wait complete. 2024-04-05T03:04:01.211 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:04:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1332901373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fcf70 msgr2=0x7efe180fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1332901373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fcf70 0x7efe180fd3e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efe04004940 tx=0x7efe04009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1332901373 shutdown_connections 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1332901373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fcf70 0x7efe180fd3e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1332901373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fbe00 0x7efe180fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.340+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1332901373 >> 172.21.15.153:0/1332901373 conn(0x7efe180f74e0 msgr2=0x7efe180f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.341+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1332901373 shutdown_connections 2024-04-05T03:04:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.341+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1332901373 wait complete. 2024-04-05T03:04:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.341+0000 7efe1ebfb700 1 Processor -- start 2024-04-05T03:04:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.341+0000 7efe1ebfb700 1 -- start start 2024-04-05T03:04:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1ebfb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 0x7efe18113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1ebfb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1ebfb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe18114b10 con 0x7efe180fbe00 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1ebfb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe18115360 con 0x7efe180fcf70 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1c997700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 0x7efe18113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:40366/0 (socket says 172.21.15.153:40366) 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe1c997700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 0x7efe18113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41644/0 (socket says 172.21.15.153:41644) 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 -- 172.21.15.153:0/1287243078 learned_addr learned my addr 172.21.15.153:0/1287243078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 -- 172.21.15.153:0/1287243078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 msgr2=0x7efe18113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 0x7efe18113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.342+0000 7efe17fff700 1 -- 172.21.15.153:0/1287243078 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04005040 con 0x7efe180fcf70 2024-04-05T03:04:01.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe17fff700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe04013870 tx=0x7efe04009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe04021070 con 0x7efe180fcf70 2024-04-05T03:04:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe04009430 con 0x7efe180fcf70 2024-04-05T03:04:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe181155e0 con 0x7efe180fcf70 2024-04-05T03:04:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe18115b20 con 0x7efe180fcf70 2024-04-05T03:04:01.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.343+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe0400e070 con 0x7efe180fcf70 2024-04-05T03:04:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.344+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe18101900 con 0x7efe180fcf70 2024-04-05T03:04:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.345+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7efe04013af0 con 0x7efe180fcf70 2024-04-05T03:04:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.345+0000 7efe157fa700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 0x7efe0806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.345+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7efe0408b260 con 0x7efe180fcf70 2024-04-05T03:04:01.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.345+0000 7efe1c997700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 0x7efe0806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.346+0000 7efe1c997700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 0x7efe0806d9f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009b80 tx=0x7efe0c00b040 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:01.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.351+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7efe04058c50 con 0x7efe180fcf70 2024-04-05T03:04:01.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.718+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe18104960 con 0x7efe0806b540 2024-04-05T03:04:01.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.719+0000 7efe157fa700 1 -- 172.21.15.153:0/1287243078 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7efe18104960 con 0x7efe0806b540 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 msgr2=0x7efe0806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 0x7efe0806d9f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009b80 tx=0x7efe0c00b040 comp rx=0 tx=0).stop 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 msgr2=0x7efe18114510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe04013870 tx=0x7efe04009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 shutdown_connections 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fbe00 0x7efe18113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7efe0806b540 0x7efe0806d9f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 --2- 172.21.15.153:0/1287243078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fcf70 0x7efe18114510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 >> 172.21.15.153:0/1287243078 conn(0x7efe180f74e0 msgr2=0x7efe180f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 shutdown_connections 2024-04-05T03:04:01.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.722+0000 7efe1ebfb700 1 -- 172.21.15.153:0/1287243078 wait complete. 2024-04-05T03:04:01.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 -- 172.21.15.153:0/1401335358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 msgr2=0x7ff7180fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 --2- 172.21.15.153:0/1401335358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff7180fd3f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff70c004940 tx=0x7ff70c009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 -- 172.21.15.153:0/1401335358 shutdown_connections 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 --2- 172.21.15.153:0/1401335358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff7180fd3f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 --2- 172.21.15.153:0/1401335358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff7180fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 -- 172.21.15.153:0/1401335358 >> 172.21.15.153:0/1401335358 conn(0x7ff7180f7430 msgr2=0x7ff7180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 -- 172.21.15.153:0/1401335358 shutdown_connections 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.876+0000 7ff71e832700 1 -- 172.21.15.153:0/1401335358 wait complete. 2024-04-05T03:04:01.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.877+0000 7ff71e832700 1 Processor -- start 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.877+0000 7ff71e832700 1 -- start start 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.877+0000 7ff71e832700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff71810d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.877+0000 7ff71e832700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff71e832700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71810e4c0 con 0x7ff7180fcf80 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff71e832700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71810ed10 con 0x7ff7180fbcd0 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff7177fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff7177fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41660/0 (socket says 172.21.15.153:41660) 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff7177fe700 1 -- 172.21.15.153:0/2948975423 learned_addr learned my addr 172.21.15.153:0/2948975423 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff717fff700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff71810d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff717fff700 1 -- 172.21.15.153:0/2948975423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 msgr2=0x7ff71810dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff717fff700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff717fff700 1 -- 172.21.15.153:0/2948975423 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c005040 con 0x7ff7180fbcd0 2024-04-05T03:04:01.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff7177fe700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff717fff700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff71810d980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff708002a50 tx=0x7ff708004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.878+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff708007cc0 con 0x7ff7180fbcd0 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.879+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70800f040 con 0x7ff7180fbcd0 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.879+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7080094b0 con 0x7ff7180fbcd0 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.879+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71810eff0 con 0x7ff7180fbcd0 2024-04-05T03:04:01.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.879+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71810f530 con 0x7ff7180fbcd0 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.880+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff718101b10 con 0x7ff7180fbcd0 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.881+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7ff708009720 con 0x7ff7180fbcd0 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.881+0000 7ff714ff9700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 0x7ff70006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.881+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ff708089e50 con 0x7ff7180fbcd0 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.881+0000 7ff7177fe700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 0x7ff70006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:01.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.882+0000 7ff7177fe700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 0x7ff70006d9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff70c004f90 tx=0x7ff70c009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:01.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:01.886+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7ff708057720 con 0x7ff7180fbcd0 2024-04-05T03:04:02.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:01 smithi203 ceph-mon[39911]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:02.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:01 smithi153 ceph-mon[29381]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:02.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.244+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff71810f7a0 con 0x7ff70006b540 2024-04-05T03:04:02.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.256+0000 7ff714ff9700 1 -- 172.21.15.153:0/2948975423 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff71810f7a0 con 0x7ff70006b540 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (46s) 1s ago 9m 15.2M - 0.25.0 c8568f914cd2 080c3d763694 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (10m) 1s ago 10m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (8m) 63s ago 8m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 unknown 1s ago 9m - - 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (4m) 1s ago 4m 135M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (4m) 1s ago 4m 65.3M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (4m) 63s ago 4m 72.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (4m) 63s ago 4m 117M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (8s) 1s ago 12m 95.5M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283 running (2m) 63s ago 8m 489M - 19.0.0-2013-g0028525c d8c35eaf3c25 f4c2fc4aeae0 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (12m) 1s ago 12m 51.5M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (8m) 63s ago 8m 41.8M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:04:02.256 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (52s) 1s ago 9m 7763k - 1.5.0 0da6a335fe13 d6609d1634cc 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 starting - - - - 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (7m) 1s ago 7m 212M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (7m) 1s ago 7m 223M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 1s ago 6m 184M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (6m) 63s ago 6m 228M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (6m) 63s ago 6m 193M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (5m) 63s ago 5m 236M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:04:02.257 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 starting - - - - 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 msgr2=0x7ff70006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 0x7ff70006d9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff70c004f90 tx=0x7ff70c009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 msgr2=0x7ff71810d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff71810d980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff708002a50 tx=0x7ff708004600 comp rx=0 tx=0).stop 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 shutdown_connections 2024-04-05T03:04:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7ff70006b540 0x7ff70006d9f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7180fbcd0 0x7ff71810d980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 --2- 172.21.15.153:0/2948975423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7180fcf80 0x7ff71810dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.259+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 >> 172.21.15.153:0/2948975423 conn(0x7ff7180f7430 msgr2=0x7ff7180f8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:02.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.260+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 shutdown_connections 2024-04-05T03:04:02.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.260+0000 7ff71e832700 1 -- 172.21.15.153:0/2948975423 wait complete. 2024-04-05T03:04:02.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.408+0000 7f4794d14700 1 -- 172.21.15.153:0/144448707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 msgr2=0x7f47880a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.408+0000 7f4794d14700 1 --2- 172.21.15.153:0/144448707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880a54e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4784004880 tx=0x7f4784009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.410+0000 7f4794d14700 1 -- 172.21.15.153:0/144448707 shutdown_connections 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.410+0000 7f4794d14700 1 --2- 172.21.15.153:0/144448707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880a54e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.410+0000 7f4794d14700 1 --2- 172.21.15.153:0/144448707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880a4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.410+0000 7f4794d14700 1 -- 172.21.15.153:0/144448707 >> 172.21.15.153:0/144448707 conn(0x7f478809f560 msgr2=0x7f47880a19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.411+0000 7f4794d14700 1 -- 172.21.15.153:0/144448707 shutdown_connections 2024-04-05T03:04:02.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.411+0000 7f4794d14700 1 -- 172.21.15.153:0/144448707 wait complete. 2024-04-05T03:04:02.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.411+0000 7f4794d14700 1 Processor -- start 2024-04-05T03:04:02.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f4794d14700 1 -- start start 2024-04-05T03:04:02.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f4794d14700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:02.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f4794d14700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:02.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f4794d14700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880b7260 con 0x7f47880a3e70 2024-04-05T03:04:02.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f4794d14700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880b73d0 con 0x7f47880a5070 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f478effd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f478effd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:40408/0 (socket says 172.21.15.153:40408) 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f478effd700 1 -- 172.21.15.153:0/2132328981 learned_addr learned my addr 172.21.15.153:0/2132328981 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.412+0000 7f478f7fe700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.413+0000 7f478f7fe700 1 -- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 msgr2=0x7f47880b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.413+0000 7f478f7fe700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.413+0000 7f478f7fe700 1 -- 172.21.15.153:0/2132328981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784005040 con 0x7f47880a3e70 2024-04-05T03:04:02.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.413+0000 7f478effd700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:04:02.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.413+0000 7f478f7fe700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880b6720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47880a4be0 tx=0x7f477c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:02.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.414+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f477c00e070 con 0x7f47880a3e70 2024-04-05T03:04:02.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.414+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f477c00bdd0 con 0x7f47880a3e70 2024-04-05T03:04:02.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.414+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f477c0098b0 con 0x7f47880a3e70 2024-04-05T03:04:02.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.414+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47880a74d0 con 0x7f47880a3e70 2024-04-05T03:04:02.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.415+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47880a7a90 con 0x7f47880a3e70 2024-04-05T03:04:02.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.416+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4788004c80 con 0x7f47880a3e70 2024-04-05T03:04:02.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.417+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f477c0253b0 con 0x7f47880a3e70 2024-04-05T03:04:02.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.418+0000 7f477bfff700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 0x7f478006db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:02.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.418+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f477c091b60 con 0x7f47880a3e70 2024-04-05T03:04:02.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.419+0000 7f478effd700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 0x7f478006db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:02.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.420+0000 7f478effd700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 0x7f478006db80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4784009fd0 tx=0x7f4784009e30 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:02.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.427+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f477c057b80 con 0x7f47880a3e70 2024-04-05T03:04:02.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.892+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f47880b0900 con 0x7f47880a3e70 2024-04-05T03:04:02.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.894+0000 7f477bfff700 1 -- 172.21.15.153:0/2132328981 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f477c021360 con 0x7f47880a3e70 2024-04-05T03:04:02.896 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:04:02.896 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:04:02.896 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:04:02.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:02.896 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 13, 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:04:02.897 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:04:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 msgr2=0x7f478006db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 0x7f478006db80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4784009fd0 tx=0x7f4784009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:02.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 msgr2=0x7f47880b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880b6720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f47880a4be0 tx=0x7f477c00b040 comp rx=0 tx=0).stop 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 shutdown_connections 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47880a3e70 0x7f47880b6720 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f478006b6d0 0x7f478006db80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 --2- 172.21.15.153:0/2132328981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47880a5070 0x7f47880b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.898+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 >> 172.21.15.153:0/2132328981 conn(0x7f478809f560 msgr2=0x7f47880a0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.899+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 shutdown_connections 2024-04-05T03:04:02.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:02.899+0000 7f4794d14700 1 -- 172.21.15.153:0/2132328981 wait complete. 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 -- 172.21.15.153:0/2519952829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 msgr2=0x7f1fd40fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2519952829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd40fd300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004940 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 -- 172.21.15.153:0/2519952829 shutdown_connections 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2519952829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 0x7f1fd40fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2519952829 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd40fd300 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.050+0000 7f1fdab57700 1 -- 172.21.15.153:0/2519952829 >> 172.21.15.153:0/2519952829 conn(0x7f1fd40f85b0 msgr2=0x7f1fd40fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 -- 172.21.15.153:0/2519952829 shutdown_connections 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 -- 172.21.15.153:0/2519952829 wait complete. 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 Processor -- start 2024-04-05T03:04:03.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 -- start start 2024-04-05T03:04:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 0x7f1fd410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd410e5c0 con 0x7f1fd40fcef0 2024-04-05T03:04:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.051+0000 7f1fdab57700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd410edc0 con 0x7f1fd40fe060 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41694/0 (socket says 172.21.15.153:41694) 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 -- 172.21.15.153:0/2112250359 learned_addr learned my addr 172.21.15.153:0/2112250359 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd3fff700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 0x7f1fd410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 -- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 msgr2=0x7f1fd410dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 0x7f1fd410dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 -- 172.21.15.153:0/2112250359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8005040 con 0x7f1fd40fcef0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd88f3700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800b3b0 tx=0x7f1fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc800e040 con 0x7f1fd40fcef0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd410f040 con 0x7f1fd40fcef0 2024-04-05T03:04:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fc800bbf0 con 0x7f1fd40fcef0 2024-04-05T03:04:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fc801c630 con 0x7f1fd40fcef0 2024-04-05T03:04:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.052+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd410f580 con 0x7f1fd40fcef0 2024-04-05T03:04:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.053+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd405f5f0 con 0x7f1fd40fcef0 2024-04-05T03:04:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.054+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) v1 ==== 89201+0+0 (secure 0 0 0) 0x7f1fc800bd60 con 0x7f1fd40fcef0 2024-04-05T03:04:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.054+0000 7f1fd17fa700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 0x7f1fc406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.054+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f1fc808ad40 con 0x7f1fd40fcef0 2024-04-05T03:04:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.054+0000 7f1fd3fff700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 0x7f1fc406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.055+0000 7f1fd3fff700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 0x7f1fc406d9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00041d0 tx=0x7f1fc00043c0 comp rx=0 tx=0).ready entity=mgr.24361 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.059+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f1fc808e030 con 0x7f1fd40fcef0 2024-04-05T03:04:03.210 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: pgmap v46: 65 pgs: 65 active+clean; 265 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 475 op/s 2024-04-05T03:04:03.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:03.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:03.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:03.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:03.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:03 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/2132328981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:03.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.372+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 --> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fd410f860 con 0x7f1fc406b540 2024-04-05T03:04:03.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.374+0000 7f1fd17fa700 1 -- 172.21.15.153:0/2112250359 <== mgr.24361 v2:172.21.15.203:6828/2106885355 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f1fd410f860 con 0x7f1fc406b540 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:04:03.374 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 msgr2=0x7f1fc406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 0x7f1fc406d9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00041d0 tx=0x7f1fc00043c0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 msgr2=0x7f1fd410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800b3b0 tx=0x7f1fc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 shutdown_connections 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1fd40fcef0 0x7f1fd410da80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:6828/2106885355,v1:172.21.15.203:6829/2106885355] conn(0x7f1fc406b540 0x7f1fc406d9f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 --2- 172.21.15.153:0/2112250359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fd40fe060 0x7f1fd410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 >> 172.21.15.153:0/2112250359 conn(0x7f1fd40f85b0 msgr2=0x7f1fd40f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 shutdown_connections 2024-04-05T03:04:03.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:03.377+0000 7f1fdab57700 1 -- 172.21.15.153:0/2112250359 wait complete. 2024-04-05T03:04:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: pgmap v46: 65 pgs: 65 active+clean; 265 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 475 op/s 2024-04-05T03:04:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:03.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:03.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:03 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/2132328981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: Standby manager daemon smithi153.npzyqy restarted 2024-04-05T03:04:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: Standby manager daemon smithi153.npzyqy started 2024-04-05T03:04:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/crt"}]: dispatch 2024-04-05T03:04:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/key"}]: dispatch 2024-04-05T03:04:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:04 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: Standby manager daemon smithi153.npzyqy restarted 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: Standby manager daemon smithi153.npzyqy started 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/crt"}]: dispatch 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi153.npzyqy/key"}]: dispatch 2024-04-05T03:04:04.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:04 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:04.625 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-05T03:04:04.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-05T03:04:05.031 DEBUG:teuthology.parallel:result is None 2024-04-05T03:04:05.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:05 smithi153 ceph-mon[29381]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:05.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:05 smithi153 ceph-mon[29381]: pgmap v47: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 461 op/s 2024-04-05T03:04:05.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:05 smithi153 ceph-mon[29381]: mgrmap e26: smithi203.iakodz(active, since 88s), standbys: smithi153.npzyqy 2024-04-05T03:04:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:05 smithi203 ceph-mon[39911]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:05 smithi203 ceph-mon[39911]: pgmap v47: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 461 op/s 2024-04-05T03:04:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:05 smithi203 ceph-mon[39911]: mgrmap e26: smithi203.iakodz(active, since 88s), standbys: smithi153.npzyqy 2024-04-05T03:04:06.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:06 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:06.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:06 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:07.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:07 smithi153 ceph-mon[29381]: pgmap v48: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-04-05T03:04:07.434 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:07 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:07.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:07 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:07 smithi203 ceph-mon[39911]: pgmap v48: 65 pgs: 65 active+clean; 267 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 354 op/s 2024-04-05T03:04:07.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:07 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:07.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:07 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:08.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd='[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]': finished 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: mgrmap e27: smithi153.npzyqy(active, starting, since 0.00610683s) 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:04:08.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: Manager daemon smithi153.npzyqy is now available 2024-04-05T03:04:08.800 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:08.801 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:08.801 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:04:08.801 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T03:04:08.801 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:08 smithi203 ceph-mon[39911]: [05/Apr/2024:03:04:08] ENGINE Bus STARTING 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 172.21.15.203:0/2960279838' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T03:04:08.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.24361 ' entity='mgr.smithi203.iakodz' cmd='[{"prefix": "mgr fail", "who": "smithi203.iakodz"}]': finished 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: mgrmap e27: smithi153.npzyqy(active, starting, since 0.00610683s) 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:04:08.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: Manager daemon smithi153.npzyqy is now available 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T03:04:08.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:08 smithi153 ceph-mon[29381]: [05/Apr/2024:03:04:08] ENGINE Bus STARTING 2024-04-05T03:04:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-05T03:04:09.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: [05/Apr/2024:03:04:08] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T03:04:09.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: [05/Apr/2024:03:04:08] ENGINE Serving on http://172.21.15.153:8765 2024-04-05T03:04:09.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: [05/Apr/2024:03:04:08] ENGINE Bus STARTED 2024-04-05T03:04:09.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: mgrmap e28: smithi153.npzyqy(active, since 1.00833s) 2024-04-05T03:04:09.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:09 smithi203 ceph-mon[39911]: pgmap v3: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: [05/Apr/2024:03:04:08] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: [05/Apr/2024:03:04:08] ENGINE Serving on http://172.21.15.153:8765 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: [05/Apr/2024:03:04:08] ENGINE Bus STARTED 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: mgrmap e28: smithi153.npzyqy(active, since 1.00833s) 2024-04-05T03:04:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:09 smithi153 ceph-mon[29381]: pgmap v3: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:10.726 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:10 smithi203 ceph-mon[39911]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-05T03:04:10.726 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:10 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T03:04:10.726 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:10 smithi203 ceph-mon[39911]: pgmap v4: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:10.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:10 smithi153 ceph-mon[29381]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-05T03:04:10.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:10 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T03:04:10.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:10 smithi153 ceph-mon[29381]: pgmap v4: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:12.067 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:11 smithi153 ceph-mon[29381]: mgrmap e29: smithi153.npzyqy(active, since 3s) 2024-04-05T03:04:12.088 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:11 smithi203 ceph-mon[39911]: mgrmap e29: smithi153.npzyqy(active, since 3s) 2024-04-05T03:04:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:12 smithi153 ceph-mon[29381]: pgmap v5: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:12 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:12 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:13.210 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:12 smithi203 ceph-mon[39911]: pgmap v5: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:04:13.210 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:12 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:13.210 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:12 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:13.779 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-05T03:04:13.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-05T03:04:14.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:13 smithi153 ceph-mon[29381]: mgrmap e30: smithi153.npzyqy(active, since 5s) 2024-04-05T03:04:14.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:13 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:14.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:13 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:14.095 DEBUG:teuthology.parallel:result is None 2024-04-05T03:04:14.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-05T03:04:14.103 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:13 smithi203 ceph-mon[39911]: mgrmap e30: smithi153.npzyqy(active, since 5s) 2024-04-05T03:04:14.103 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:13 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:14.103 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:13 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:14.139 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-05T03:04:14.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-05T03:04:14.177 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-05T03:04:14.177 DEBUG:teuthology.parallel:result is None 2024-04-05T03:04:15.256 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:14 smithi203 ceph-mon[39911]: pgmap v6: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 891 KiB/s rd, 948 KiB/s wr, 92 op/s 2024-04-05T03:04:15.256 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:14 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.256 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:14 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.256 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:14 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.256 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:14 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:14 smithi153 ceph-mon[29381]: pgmap v6: 65 pgs: 65 active+clean; 261 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 891 KiB/s rd, 948 KiB/s wr, 92 op/s 2024-04-05T03:04:15.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:14 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:14 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:14 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:15.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:14 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: pgmap v7: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 727 KiB/s rd, 696 KiB/s wr, 203 op/s 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:04:17.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:16 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: pgmap v7: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 727 KiB/s rd, 696 KiB/s wr, 203 op/s 2024-04-05T03:04:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:04:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:04:17.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:17.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:04:17.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:16 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3637925556' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:18.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:17 smithi203 ceph-mon[39911]: mgrmap e31: smithi153.npzyqy(active, since 9s), standbys: smithi203.iakodz 2024-04-05T03:04:18.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:17 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:18.306 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:17 smithi153 ceph-mon[29381]: mgrmap e31: smithi153.npzyqy(active, since 9s), standbys: smithi203.iakodz 2024-04-05T03:04:18.306 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:17 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:04:19.211 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:18 smithi153 ceph-mon[29381]: pgmap v8: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 565 KiB/s rd, 541 KiB/s wr, 158 op/s 2024-04-05T03:04:19.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:18 smithi203 ceph-mon[39911]: pgmap v8: 65 pgs: 65 active+clean; 257 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 565 KiB/s rd, 541 KiB/s wr, 158 op/s 2024-04-05T03:04:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:20 smithi153 ceph-mon[29381]: pgmap v9: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 530 KiB/s rd, 463 KiB/s wr, 176 op/s 2024-04-05T03:04:21.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:20 smithi203 ceph-mon[39911]: pgmap v9: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 530 KiB/s rd, 463 KiB/s wr, 176 op/s 2024-04-05T03:04:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:22 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:04:22.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:22.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:22 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:23.351 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: pgmap v10: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 485 KiB/s rd, 424 KiB/s wr, 161 op/s 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: pgmap v11: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 560 KiB/s rd, 490 KiB/s wr, 186 op/s 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: Reconfiguring prometheus.smithi203 (dependencies changed)... 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:23.352 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:23 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:23.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:04:23.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:04:23.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: pgmap v10: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 485 KiB/s rd, 424 KiB/s wr, 161 op/s 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: pgmap v11: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 560 KiB/s rd, 490 KiB/s wr, 186 op/s 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: Reconfiguring prometheus.smithi203 (dependencies changed)... 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:23.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:23 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:24.476 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:24 smithi203 ceph-mon[39911]: Reconfiguring daemon prometheus.smithi203 on smithi203 2024-04-05T03:04:24.506 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:24 smithi153 ceph-mon[29381]: Reconfiguring daemon prometheus.smithi203 on smithi203 2024-04-05T03:04:25.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:25 smithi203 ceph-mon[39911]: pgmap v12: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 147 KiB/s rd, 49 KiB/s wr, 145 op/s 2024-04-05T03:04:25.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:25 smithi153 ceph-mon[29381]: pgmap v12: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 147 KiB/s rd, 49 KiB/s wr, 145 op/s 2024-04-05T03:04:27.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:27 smithi203 ceph-mon[39911]: pgmap v13: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 86 KiB/s rd, 36 KiB/s wr, 52 op/s 2024-04-05T03:04:27.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:27 smithi153 ceph-mon[29381]: pgmap v13: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 86 KiB/s rd, 36 KiB/s wr, 52 op/s 2024-04-05T03:04:28.842 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:28.842 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:28.842 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:04:28.843 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:04:28.843 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:28.843 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:28 smithi203 ceph-mon[39911]: pgmap v14: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 86 KiB/s rd, 36 KiB/s wr, 52 op/s 2024-04-05T03:04:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:04:29.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:04:29.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:29.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:28 smithi153 ceph-mon[29381]: pgmap v14: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 86 KiB/s rd, 36 KiB/s wr, 52 op/s 2024-04-05T03:04:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:29 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:04:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:29 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:04:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:29 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:30.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:29 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:04:30.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:29 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:04:30.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:29 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:30 smithi203 ceph-mon[39911]: Upgrade: Updating mgr.smithi203.iakodz 2024-04-05T03:04:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:30 smithi203 ceph-mon[39911]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:04:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:30 smithi203 ceph-mon[39911]: pgmap v15: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-05T03:04:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:30 smithi153 ceph-mon[29381]: Upgrade: Updating mgr.smithi203.iakodz 2024-04-05T03:04:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:30 smithi153 ceph-mon[29381]: Deploying daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:04:31.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:30 smithi153 ceph-mon[29381]: pgmap v15: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-05T03:04:33.380 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:33 smithi203 ceph-mon[39911]: pgmap v16: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-05T03:04:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 -- 172.21.15.153:0/173664115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fdc10 msgr2=0x7f76740fc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 --2- 172.21.15.153:0/173664115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fdc10 0x7f76740fc020 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7660004940 tx=0x7f7660009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 -- 172.21.15.153:0/173664115 shutdown_connections 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 --2- 172.21.15.153:0/173664115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fdc10 0x7f76740fc020 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 --2- 172.21.15.153:0/173664115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fd2c0 0x7f76740fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 -- 172.21.15.153:0/173664115 >> 172.21.15.153:0/173664115 conn(0x7f76740f6ba0 msgr2=0x7f76740f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 -- 172.21.15.153:0/173664115 shutdown_connections 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.486+0000 7f767af8e700 1 -- 172.21.15.153:0/173664115 wait complete. 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 Processor -- start 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 -- start start 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 0x7f76740fe370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76740fe900 con 0x7f76740fd2c0 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f767af8e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76740fea40 con 0x7f76740fdc10 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f7678d2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f7678d2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47276/0 (socket says 172.21.15.153:47276) 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.487+0000 7f7678d2a700 1 -- 172.21.15.153:0/3748263315 learned_addr learned my addr 172.21.15.153:0/3748263315 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7673fff700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 0x7f76740fe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7678d2a700 1 -- 172.21.15.153:0/3748263315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 msgr2=0x7f76740fe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7678d2a700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 0x7f76740fe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7678d2a700 1 -- 172.21.15.153:0/3748263315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7660005040 con 0x7f76740fd2c0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7673fff700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 0x7f76740fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f7678d2a700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7668002a50 tx=0x7f76680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7668007cc0 con 0x7f76740fd2c0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76740fed20 con 0x7f76740fd2c0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f766800f040 con 0x7f76740fd2c0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f766800da30 con 0x7f76740fd2c0 2024-04-05T03:04:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.488+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7674115ad0 con 0x7f76740fd2c0 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.490+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7674101bc0 con 0x7f76740fd2c0 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.490+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f766800db90 con 0x7f76740fd2c0 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.490+0000 7f76717fa700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 0x7f766406fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.490+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f766808cbb0 con 0x7f76740fd2c0 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.490+0000 7f7673fff700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 0x7f766406fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.491+0000 7f7673fff700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 0x7f766406fb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7660004890 tx=0x7f7660009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.495+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f7668093080 con 0x7f76740fd2c0 2024-04-05T03:04:33.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:33 smithi153 ceph-mon[29381]: pgmap v16: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-05T03:04:33.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.720+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f767405e400 con 0x7f766406d6c0 2024-04-05T03:04:33.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.722+0000 7f76717fa700 1 -- 172.21.15.153:0/3748263315 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f767405e400 con 0x7f766406d6c0 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 msgr2=0x7f766406fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 0x7f766406fb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7660004890 tx=0x7f7660009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 msgr2=0x7f76740ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7668002a50 tx=0x7f76680046b0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 shutdown_connections 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f766406d6c0 0x7f766406fb70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76740fd2c0 0x7f76740ffe50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 --2- 172.21.15.153:0/3748263315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76740fdc10 0x7f76740fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 >> 172.21.15.153:0/3748263315 conn(0x7f76740f6ba0 msgr2=0x7f76740fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 shutdown_connections 2024-04-05T03:04:33.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.724+0000 7f767af8e700 1 -- 172.21.15.153:0/3748263315 wait complete. 2024-04-05T03:04:33.742 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:04:33.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 -- 172.21.15.153:0/3647406556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 msgr2=0x7f14080fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 --2- 172.21.15.153:0/3647406556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f14080fd3f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f13fc004940 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 -- 172.21.15.153:0/3647406556 shutdown_connections 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 --2- 172.21.15.153:0/3647406556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f14080fd3f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 --2- 172.21.15.153:0/3647406556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 0x7f14080fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.841+0000 7f140cb28700 1 -- 172.21.15.153:0/3647406556 >> 172.21.15.153:0/3647406556 conn(0x7f14080f7430 msgr2=0x7f14080f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 -- 172.21.15.153:0/3647406556 shutdown_connections 2024-04-05T03:04:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 -- 172.21.15.153:0/3647406556 wait complete. 2024-04-05T03:04:33.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 Processor -- start 2024-04-05T03:04:33.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 -- start start 2024-04-05T03:04:33.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 0x7f140810c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.842+0000 7f140cb28700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f140cb28700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140810d370 con 0x7f14080fcf80 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f140cb28700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140810d4e0 con 0x7f14080fbcd0 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f140659c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 0x7f140810c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47286/0 (socket says 172.21.15.153:47286) 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 -- 172.21.15.153:0/4080724053 learned_addr learned my addr 172.21.15.153:0/4080724053 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 -- 172.21.15.153:0/4080724053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 msgr2=0x7f140810c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 0x7f140810c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 -- 172.21.15.153:0/4080724053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc005040 con 0x7f14080fcf80 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f1405d9b700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009fd0 tx=0x7f13fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc00e040 con 0x7f14080fcf80 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.843+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13fc00be00 con 0x7f14080fcf80 2024-04-05T03:04:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.844+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1408066210 con 0x7f14080fcf80 2024-04-05T03:04:33.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.844+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14080667d0 con 0x7f14080fcf80 2024-04-05T03:04:33.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.845+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13fc0137f0 con 0x7f14080fcf80 2024-04-05T03:04:33.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.845+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140804c710 con 0x7f14080fcf80 2024-04-05T03:04:33.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.846+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f13fc013950 con 0x7f14080fcf80 2024-04-05T03:04:33.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.846+0000 7f13f6ffd700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 0x7f13f006f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:33.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.846+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f13fc0905f0 con 0x7f14080fcf80 2024-04-05T03:04:33.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.846+0000 7f140659c700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 0x7f13f006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:33.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.847+0000 7f140659c700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 0x7f13f006f970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f13f8004f40 tx=0x7f13f800b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:33.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:33.855+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f13fc05de60 con 0x7f14080fcf80 2024-04-05T03:04:34.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.098+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1408066bc0 con 0x7f13f006d4c0 2024-04-05T03:04:34.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.099+0000 7f13f6ffd700 1 -- 172.21.15.153:0/4080724053 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f1408066bc0 con 0x7f13f006d4c0 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.101+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 msgr2=0x7f13f006f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.101+0000 7f140cb28700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 0x7f13f006f970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f13f8004f40 tx=0x7f13f800b040 comp rx=0 tx=0).stop 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.101+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 msgr2=0x7f140810cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.101+0000 7f140cb28700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009fd0 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 shutdown_connections 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14080fbcd0 0x7f140810c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f13f006d4c0 0x7f13f006f970 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 --2- 172.21.15.153:0/4080724053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14080fcf80 0x7f140810cd70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 >> 172.21.15.153:0/4080724053 conn(0x7f14080f7430 msgr2=0x7f14080f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 shutdown_connections 2024-04-05T03:04:34.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.102+0000 7f140cb28700 1 -- 172.21.15.153:0/4080724053 wait complete. 2024-04-05T03:04:34.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- 172.21.15.153:0/1088687483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 msgr2=0x7fe6740fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 --2- 172.21.15.153:0/1088687483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe6740fe860 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe670004940 tx=0x7fe670009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- 172.21.15.153:0/1088687483 shutdown_connections 2024-04-05T03:04:34.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 --2- 172.21.15.153:0/1088687483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe6740faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 --2- 172.21.15.153:0/1088687483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe6740fe860 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- 172.21.15.153:0/1088687483 >> 172.21.15.153:0/1088687483 conn(0x7fe6740f6c10 msgr2=0x7fe6740f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- 172.21.15.153:0/1088687483 shutdown_connections 2024-04-05T03:04:34.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- 172.21.15.153:0/1088687483 wait complete. 2024-04-05T03:04:34.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 Processor -- start 2024-04-05T03:04:34.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.203+0000 7fe67b252700 1 -- start start 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe67b252700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe67b252700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe67410ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe67b252700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67410e3d0 con 0x7fe6740fe450 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe67b252700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67410e540 con 0x7fe6740feda0 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47300/0 (socket says 172.21.15.153:47300) 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 -- 172.21.15.153:0/3878058136 learned_addr learned my addr 172.21.15.153:0/3878058136 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe66bfff700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe67410ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 -- 172.21.15.153:0/3878058136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 msgr2=0x7fe67410ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe67410ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 -- 172.21.15.153:0/3878058136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670005040 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe66bfff700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe67410ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.204+0000 7fe678fee700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe67000b3b0 tx=0x7fe670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.205+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe67000e040 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.205+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe670004290 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.205+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674112f40 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.205+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe67001c630 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.205+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe674113470 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.206+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67404c710 con 0x7fe6740fe450 2024-04-05T03:04:34.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.206+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 91677+0+0 (secure 0 0 0) 0x7fe67000bbf0 con 0x7fe6740fe450 2024-04-05T03:04:34.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.206+0000 7fe6697fa700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 0x7fe65c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.207+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe67008e290 con 0x7fe6740fe450 2024-04-05T03:04:34.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.207+0000 7fe66bfff700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 0x7fe65c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.207+0000 7fe66bfff700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 0x7fe65c06fb80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe664007fd0 tx=0x7fe664009f60 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.211+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fe67005ba50 con 0x7fe6740fe450 2024-04-05T03:04:34.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.413+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe674113690 con 0x7fe65c06d6d0 2024-04-05T03:04:34.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.423+0000 7fe6697fa700 1 -- 172.21.15.153:0/3878058136 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe674113690 con 0x7fe65c06d6d0 2024-04-05T03:04:34.423 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:04:34.423 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (79s) 21s ago 10m 15.2M - 0.25.0 c8568f914cd2 080c3d763694 2024-04-05T03:04:34.423 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (11m) 21s ago 11m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:04:34.423 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (9m) 21s ago 9m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 unknown 21s ago 10m - - 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (4m) 21s ago 4m 118M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (4m) 21s ago 4m 64.7M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (4m) 21s ago 4m 66.3M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (4m) 21s ago 4m 128M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (40s) 21s ago 12m 474M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 starting - - - - 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (12m) 21s ago 12m 53.5M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (9m) 21s ago 8m 45.5M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (84s) 21s ago 10m 8568k - 1.5.0 0da6a335fe13 d6609d1634cc 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (61s) 21s ago 9m 7880k - 1.5.0 0da6a335fe13 4ce97f4c9803 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (8m) 21s ago 8m 217M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (7m) 21s ago 7m 229M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (7m) 21s ago 7m 194M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (7m) 21s ago 7m 260M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:04:34.424 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (6m) 21s ago 6m 228M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:04:34.425 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (6m) 21s ago 6m 229M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:04:34.425 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 starting - - - - 2024-04-05T03:04:34.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 msgr2=0x7fe65c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 0x7fe65c06fb80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe664007fd0 tx=0x7fe664009f60 comp rx=0 tx=0).stop 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 msgr2=0x7fe67410d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe67000b3b0 tx=0x7fe670009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 shutdown_connections 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe65c06d6d0 0x7fe65c06fb80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6740fe450 0x7fe67410d890 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 --2- 172.21.15.153:0/3878058136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6740feda0 0x7fe67410ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 >> 172.21.15.153:0/3878058136 conn(0x7fe6740f6c10 msgr2=0x7fe6740f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 shutdown_connections 2024-04-05T03:04:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.426+0000 7fe67b252700 1 -- 172.21.15.153:0/3878058136 wait complete. 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.526+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/3208312187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180fff00 msgr2=0x7f5b18102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.526+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/3208312187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180fff00 0x7f5b18102320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5b04004940 tx=0x7f5b04009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/3208312187 shutdown_connections 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/3208312187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180fff00 0x7f5b18102320 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/3208312187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180662e0 0x7f5b180ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/3208312187 >> 172.21.15.153:0/3208312187 conn(0x7f5b180f7430 msgr2=0x7f5b180f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/3208312187 shutdown_connections 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/3208312187 wait complete. 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 Processor -- start 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- start start 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180fff00 0x7f5b1810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810e5a0 con 0x7f5b180662e0 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.527+0000 7f5b1ec7b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810edf0 con 0x7f5b180fff00 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47330/0 (socket says 172.21.15.153:47330) 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 -- 172.21.15.153:0/915487013 learned_addr learned my addr 172.21.15.153:0/915487013 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b17fff700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180fff00 0x7f5b1810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 -- 172.21.15.153:0/915487013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180fff00 msgr2=0x7f5b1810dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180fff00 0x7f5b1810dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 -- 172.21.15.153:0/915487013 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04005040 con 0x7f5b180662e0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ca17700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c002a50 tx=0x7f5b0c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b0c007cc0 con 0x7f5b180662e0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1810f0d0 con 0x7f5b180662e0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b0c00f040 con 0x7f5b180662e0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b0c00d9c0 con 0x7f5b180662e0 2024-04-05T03:04:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.528+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1810f610 con 0x7f5b180662e0 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.529+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1805f5f0 con 0x7f5b180662e0 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.530+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f5b0c009730 con 0x7f5b180662e0 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.530+0000 7f5b157fa700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 0x7f5b0806fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.530+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5b0c08cb70 con 0x7f5b180662e0 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.530+0000 7f5b17fff700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 0x7f5b0806fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.531+0000 7f5b17fff700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 0x7f5b0806fb70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400b970 tx=0x7f5b04009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.535+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f5b0c05a2b0 con 0x7f5b180662e0 2024-04-05T03:04:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:34.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: pgmap v17: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-04-05T03:04:34.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:34 smithi203 ceph-mon[39911]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.811+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5b181058d0 con 0x7f5b180662e0 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.812+0000 7f5b157fa700 1 -- 172.21.15.153:0/915487013 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f5b0c059e40 con 0x7f5b180662e0 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:34.812 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 12, 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:04:34.813 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:04:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.814+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 msgr2=0x7f5b0806fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.814+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 0x7f5b0806fb70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b0400b970 tx=0x7f5b04009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.814+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 msgr2=0x7f5b1810da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.814+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c002a50 tx=0x7f5b0c004600 comp rx=0 tx=0).stop 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 shutdown_connections 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f5b0806d6c0 0x7f5b0806fb70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b180662e0 0x7f5b1810da60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 --2- 172.21.15.153:0/915487013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b180fff00 0x7f5b1810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 >> 172.21.15.153:0/915487013 conn(0x7f5b180f7430 msgr2=0x7f5b180f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 shutdown_connections 2024-04-05T03:04:34.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.815+0000 7f5b1ec7b700 1 -- 172.21.15.153:0/915487013 wait complete. 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 -- 172.21.15.153:0/1421898552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fcf90 msgr2=0x7f712c0fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 --2- 172.21.15.153:0/1421898552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fcf90 0x7f712c0fd400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7128004940 tx=0x7f7128009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 -- 172.21.15.153:0/1421898552 shutdown_connections 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 --2- 172.21.15.153:0/1421898552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fcf90 0x7f712c0fd400 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 --2- 172.21.15.153:0/1421898552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fbce0 0x7f712c0fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 -- 172.21.15.153:0/1421898552 >> 172.21.15.153:0/1421898552 conn(0x7f712c0f7440 msgr2=0x7f712c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 -- 172.21.15.153:0/1421898552 shutdown_connections 2024-04-05T03:04:34.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.917+0000 7f7133537700 1 -- 172.21.15.153:0/1421898552 wait complete. 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: pgmap v17: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-04-05T03:04:34.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:34 smithi153 ceph-mon[29381]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 Processor -- start 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 -- start start 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fcf90 0x7f712c112ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c066be0 con 0x7f712c0fbce0 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7133537700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c066d20 con 0x7f712c0fcf90 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47354/0 (socket says 172.21.15.153:47354) 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 -- 172.21.15.153:0/527841271 learned_addr learned my addr 172.21.15.153:0/527841271 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f7130ad2700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fcf90 0x7f712c112ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 -- 172.21.15.153:0/527841271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fcf90 msgr2=0x7f712c112ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fcf90 0x7f712c112ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.918+0000 7f71312d3700 1 -- 172.21.15.153:0/527841271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128005040 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f71312d3700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f711c002a50 tx=0x7f711c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f711c007cc0 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f711c00f040 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c113240 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f711c00da30 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.919+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1137b0 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.920+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c05f5f0 con 0x7f712c0fbce0 2024-04-05T03:04:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.920+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 31) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f711c00db90 con 0x7f712c0fbce0 2024-04-05T03:04:34.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.921+0000 7f7121ffb700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 0x7f711806fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:04:34.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.921+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f711c08cbc0 con 0x7f712c0fbce0 2024-04-05T03:04:34.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.921+0000 7f7130ad2700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 0x7f711806fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:04:34.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.921+0000 7f7130ad2700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 0x7f711806fb70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f712800b970 tx=0x7f7128009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:04:34.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:34.925+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f711c0567d0 con 0x7f712c0fbce0 2024-04-05T03:04:35.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.137+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f712c1012d0 con 0x7f711806d6c0 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.138+0000 7f7121ffb700 1 -- 172.21.15.153:0/527841271 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f712c1012d0 con 0x7f711806d6c0 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "", 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T03:04:35.138 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:04:35.139 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:04:35.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 msgr2=0x7f711806fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:35.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 0x7f711806fb70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f712800b970 tx=0x7f7128009e30 comp rx=0 tx=0).stop 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 msgr2=0x7f712c110750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f711c002a50 tx=0x7f711c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 shutdown_connections 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f711806d6c0 0x7f711806fb70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f712c0fbce0 0x7f712c110750 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 --2- 172.21.15.153:0/527841271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f712c0fcf90 0x7f712c112ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.140+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 >> 172.21.15.153:0/527841271 conn(0x7f712c0f7440 msgr2=0x7f712c0f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.141+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 shutdown_connections 2024-04-05T03:04:35.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:04:35.141+0000 7f7133537700 1 -- 172.21.15.153:0/527841271 wait complete. 2024-04-05T03:04:36.035 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:35 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/915487013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:36.036 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:35 smithi203 ceph-mon[39911]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:35 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/915487013' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:35 smithi153 ceph-mon[29381]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:04:37.020 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:36 smithi203 ceph-mon[39911]: pgmap v18: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:36 smithi153 ceph-mon[29381]: pgmap v18: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:37 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:38.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:37 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:39.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:38 smithi203 ceph-mon[39911]: pgmap v19: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:39.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:38 smithi153 ceph-mon[29381]: pgmap v19: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:41.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: pgmap v20: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:04:41.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:41.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:04:41.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:41 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:41.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: pgmap v20: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:04:41.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:04:41.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:41 smithi153 ceph-mon[29381]: from='mgr.? 172.21.15.203:0/3064676880' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:04:42.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:42 smithi203 ceph-mon[39911]: mgrmap e32: smithi153.npzyqy(active, since 33s), standbys: smithi203.iakodz 2024-04-05T03:04:42.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:42 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:42.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:42 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:42.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:42 smithi153 ceph-mon[29381]: mgrmap e32: smithi153.npzyqy(active, since 33s), standbys: smithi203.iakodz 2024-04-05T03:04:42.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:42 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:42.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:42 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:43.457 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:43 smithi203 ceph-mon[39911]: pgmap v21: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:43.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:43 smithi153 ceph-mon[29381]: pgmap v21: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: pgmap v22: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:45.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.npzyqy"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.npzyqy"}]': finished 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi203.iakodz"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi203.iakodz"}]': finished 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:45 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: pgmap v22: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.npzyqy"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi153.npzyqy"}]': finished 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi203.iakodz"}]: dispatch 2024-04-05T03:04:45.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi203.iakodz"}]': finished 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:45.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:04:45.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:45 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:46.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: pgmap v23: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: pgmap v24: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all mgr 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all rgw 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all iscsi 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all nfs 2024-04-05T03:04:46.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:46 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all nvmeof 2024-04-05T03:04:46.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: pgmap v23: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:04:46.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: pgmap v24: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all mgr 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all rgw 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all iscsi 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all nfs 2024-04-05T03:04:46.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:46 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all nvmeof 2024-04-05T03:04:48.590 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:48 smithi153 ceph-mon[29381]: Upgrade: Updating node-exporter.smithi153 (1/2) 2024-04-05T03:04:48.590 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:48 smithi153 ceph-mon[29381]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-05T03:04:48.590 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:48 smithi153 ceph-mon[29381]: pgmap v25: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:48.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:48 smithi203 ceph-mon[39911]: Upgrade: Updating node-exporter.smithi153 (1/2) 2024-04-05T03:04:48.660 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:48 smithi203 ceph-mon[39911]: Deploying daemon node-exporter.smithi153 on smithi153 2024-04-05T03:04:48.660 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:48 smithi203 ceph-mon[39911]: pgmap v25: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:50 smithi203 ceph-mon[39911]: pgmap v26: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:50 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:50 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:50 smithi153 ceph-mon[29381]: pgmap v26: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:50 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:50 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:52.433 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:52 smithi203 ceph-mon[39911]: pgmap v27: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:52.433 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:52 smithi203 ceph-mon[39911]: Upgrade: Updating node-exporter.smithi203 (2/2) 2024-04-05T03:04:52.434 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:52 smithi203 ceph-mon[39911]: Deploying daemon node-exporter.smithi203 on smithi203 2024-04-05T03:04:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:52 smithi153 ceph-mon[29381]: pgmap v27: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:04:52.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:52 smithi153 ceph-mon[29381]: Upgrade: Updating node-exporter.smithi203 (2/2) 2024-04-05T03:04:52.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:52 smithi153 ceph-mon[29381]: Deploying daemon node-exporter.smithi203 on smithi203 2024-04-05T03:04:53.489 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:53 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:53.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:53 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:04:54.727 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:54 smithi153 ceph-mon[29381]: pgmap v28: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:04:54.727 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:54 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:54.727 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:54 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:54.727 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:54 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:54.732 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:54 smithi203 ceph-mon[39911]: pgmap v28: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:04:54.732 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:54 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:54.732 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:54 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:54.733 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:54 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:04:56.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:56 smithi153 ceph-mon[29381]: pgmap v29: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:04:56.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:56 smithi203 ceph-mon[39911]: pgmap v29: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:04:58.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:58 smithi203 ceph-mon[39911]: pgmap v30: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:58.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:58 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:58.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:58 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:58 smithi153 ceph-mon[29381]: pgmap v30: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:04:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:58 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:58 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:59 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:59 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:59 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:59 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:04:59 smithi203 ceph-mon[39911]: pgmap v31: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:04:59.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:59 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:59 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:59 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:59 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:04:59.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:04:59 smithi153 ceph-mon[29381]: pgmap v31: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:00 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:00 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:00 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:00 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:02.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:01 smithi203 ceph-mon[39911]: pgmap v32: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:02.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:01 smithi153 ceph-mon[29381]: pgmap v32: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:03.309 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:03 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:03.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:03 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:03 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:03.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:03 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: pgmap v33: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: pgmap v34: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: pgmap v35: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:04 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: pgmap v33: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: pgmap v34: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: pgmap v35: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:04.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:04 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 -- 172.21.15.153:0/986709844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc071f50 msgr2=0x7f9ecc070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/986709844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc071f50 0x7f9ecc070460 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc004940 tx=0x7f9ebc009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 -- 172.21.15.153:0/986709844 shutdown_connections 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/986709844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/986709844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc071f50 0x7f9ecc070460 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 -- 172.21.15.153:0/986709844 >> 172.21.15.153:0/986709844 conn(0x7f9ecc0f8560 msgr2=0x7f9ecc0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 -- 172.21.15.153:0/986709844 shutdown_connections 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 -- 172.21.15.153:0/986709844 wait complete. 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.249+0000 7f9ed18b1700 1 Processor -- start 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ed18b1700 1 -- start start 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ed18b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ed18b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc071f50 0x7f9ecc110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ed18b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc110870 con 0x7f9ecc0709a0 2024-04-05T03:05:05.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ed18b1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc110fb0 con 0x7f9ecc071f50 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48526/0 (socket says 172.21.15.153:48526) 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 -- 172.21.15.153:0/1395301702 learned_addr learned my addr 172.21.15.153:0/1395301702 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9eca7fc700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc071f50 0x7f9ecc110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 -- 172.21.15.153:0/1395301702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc071f50 msgr2=0x7f9ecc110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc071f50 0x7f9ecc110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 -- 172.21.15.153:0/1395301702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc005040 con 0x7f9ecc0709a0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.250+0000 7f9ecaffd700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.251+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ebc00e040 con 0x7f9ecc0709a0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.251+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc111230 con 0x7f9ecc0709a0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.251+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ebc00bbf0 con 0x7f9ecc0709a0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.251+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ebc01c630 con 0x7f9ecc0709a0 2024-04-05T03:05:05.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.251+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc1117f0 con 0x7f9ecc0709a0 2024-04-05T03:05:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.252+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc05f5f0 con 0x7f9ecc0709a0 2024-04-05T03:05:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.252+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f9ebc00bd60 con 0x7f9ecc0709a0 2024-04-05T03:05:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.253+0000 7f9eb37fe700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 0x7f9eb406fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.253+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9ebc08e130 con 0x7f9ecc0709a0 2024-04-05T03:05:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.253+0000 7f9eca7fc700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 0x7f9eb406fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.253+0000 7f9eca7fc700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 0x7f9eb406fb70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0007fd0 tx=0x7f9ec0009040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.261+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f9ebc05b8f0 con 0x7f9ecc0709a0 2024-04-05T03:05:05.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.506+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ecc111b40 con 0x7f9eb406d6c0 2024-04-05T03:05:05.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.507+0000 7f9eb37fe700 1 -- 172.21.15.153:0/1395301702 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f9ecc111b40 con 0x7f9eb406d6c0 2024-04-05T03:05:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 msgr2=0x7f9eb406fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 0x7f9eb406fb70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0007fd0 tx=0x7f9ec0009040 comp rx=0 tx=0).stop 2024-04-05T03:05:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 msgr2=0x7f9ecc10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:05.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 shutdown_connections 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9eb406d6c0 0x7f9eb406fb70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ecc0709a0 0x7f9ecc10fd30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 --2- 172.21.15.153:0/1395301702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ecc071f50 0x7f9ecc110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 >> 172.21.15.153:0/1395301702 conn(0x7f9ecc0f8560 msgr2=0x7f9ecc0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 shutdown_connections 2024-04-05T03:05:05.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.509+0000 7f9ed18b1700 1 -- 172.21.15.153:0/1395301702 wait complete. 2024-04-05T03:05:05.520 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 -- 172.21.15.153:0/3034076484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80662e0 msgr2=0x7f4bc80ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 --2- 172.21.15.153:0/3034076484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80662e0 0x7f4bc80ff9c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8004940 tx=0x7f4bb8009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 -- 172.21.15.153:0/3034076484 shutdown_connections 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 --2- 172.21.15.153:0/3034076484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80fff00 0x7f4bc8102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 --2- 172.21.15.153:0/3034076484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80662e0 0x7f4bc80ff9c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 -- 172.21.15.153:0/3034076484 >> 172.21.15.153:0/3034076484 conn(0x7f4bc80f7430 msgr2=0x7f4bc80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:05.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 -- 172.21.15.153:0/3034076484 shutdown_connections 2024-04-05T03:05:05.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 -- 172.21.15.153:0/3034076484 wait complete. 2024-04-05T03:05:05.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.612+0000 7f4bce591700 1 Processor -- start 2024-04-05T03:05:05.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bce591700 1 -- start start 2024-04-05T03:05:05.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bce591700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 0x7f4bc810da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bce591700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bce591700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc810e540 con 0x7f4bc80fff00 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bce591700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc810ed40 con 0x7f4bc80662e0 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 0x7f4bc810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48548/0 (socket says 172.21.15.153:48548) 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 -- 172.21.15.153:0/2651653302 learned_addr learned my addr 172.21.15.153:0/2651653302 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 -- 172.21.15.153:0/2651653302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 msgr2=0x7f4bc810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 0x7f4bc810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 -- 172.21.15.153:0/2651653302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8005040 con 0x7f4bc80fff00 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc7fff700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 0x7f4bc810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.613+0000 7f4bc77fe700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00eb40 tx=0x7f4bbc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.614+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc007cc0 con 0x7f4bc80fff00 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.614+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bbc00f040 con 0x7f4bc80fff00 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.614+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc810efc0 con 0x7f4bc80fff00 2024-04-05T03:05:05.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.614+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc00ca10 con 0x7f4bc80fff00 2024-04-05T03:05:05.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.614+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc810f500 con 0x7f4bc80fff00 2024-04-05T03:05:05.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.615+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc805f5f0 con 0x7f4bc80fff00 2024-04-05T03:05:05.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.615+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f4bbc00cb70 con 0x7f4bc80fff00 2024-04-05T03:05:05.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.616+0000 7f4bc4ff9700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 0x7f4bb006fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.616+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4bbc08c8e0 con 0x7f4bc80fff00 2024-04-05T03:05:05.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.616+0000 7f4bc7fff700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 0x7f4bb006fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.616+0000 7f4bc7fff700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 0x7f4bb006fb70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8004890 tx=0x7f4bb8009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.620+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f4bbc05a150 con 0x7f4bc80fff00 2024-04-05T03:05:05.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.831+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bc810f830 con 0x7f4bb006d6c0 2024-04-05T03:05:05.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.832+0000 7f4bc4ff9700 1 -- 172.21.15.153:0/2651653302 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f4bc810f830 con 0x7f4bb006d6c0 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.834+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 msgr2=0x7f4bb006fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.834+0000 7f4bce591700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 0x7f4bb006fb70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8004890 tx=0x7f4bb8009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.834+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 msgr2=0x7f4bc810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.834+0000 7f4bce591700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00eb40 tx=0x7f4bbc00b040 comp rx=0 tx=0).stop 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 shutdown_connections 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc80662e0 0x7f4bc810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f4bb006d6c0 0x7f4bb006fb70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 --2- 172.21.15.153:0/2651653302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bc80fff00 0x7f4bc810df40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 >> 172.21.15.153:0/2651653302 conn(0x7f4bc80f7430 msgr2=0x7f4bc80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 shutdown_connections 2024-04-05T03:05:05.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.835+0000 7f4bce591700 1 -- 172.21.15.153:0/2651653302 wait complete. 2024-04-05T03:05:05.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/3424454542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 msgr2=0x7f9f180faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/3424454542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f180faec0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c004940 tx=0x7f9f0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:05.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/3424454542 shutdown_connections 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/3424454542 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f180faec0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/3424454542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 0x7f9f180fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/3424454542 >> 172.21.15.153:0/3424454542 conn(0x7f9f180f6bd0 msgr2=0x7f9f180f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/3424454542 shutdown_connections 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.936+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/3424454542 wait complete. 2024-04-05T03:05:05.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 Processor -- start 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 -- start start 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 0x7f9f1810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18110680 con 0x7f9f180fed80 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f1e0b1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181107f0 con 0x7f9f180fe430 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f16ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f16ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48580/0 (socket says 172.21.15.153:48580) 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f16ffd700 1 -- 172.21.15.153:0/4282300847 learned_addr learned my addr 172.21.15.153:0/4282300847 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.937+0000 7f9f177fe700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 0x7f9f1810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f16ffd700 1 -- 172.21.15.153:0/4282300847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 msgr2=0x7f9f1810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f16ffd700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 0x7f9f1810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f16ffd700 1 -- 172.21.15.153:0/4282300847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c005040 con 0x7f9f180fed80 2024-04-05T03:05:05.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f16ffd700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00b3b0 tx=0x7f9f0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c00e040 con 0x7f9f180fed80 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f0c00bbf0 con 0x7f9f180fed80 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f181151f0 con 0x7f9f180fed80 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0c01c630 con 0x7f9f180fed80 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.938+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f181157b0 con 0x7f9f180fed80 2024-04-05T03:05:05.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.939+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f18101b70 con 0x7f9f180fed80 2024-04-05T03:05:05.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.940+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f9f0c00bd60 con 0x7f9f180fed80 2024-04-05T03:05:05.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.940+0000 7f9efffff700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 0x7f9f0006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:05.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.940+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9f0c08d930 con 0x7f9f180fed80 2024-04-05T03:05:05.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.940+0000 7f9f177fe700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 0x7f9f0006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:05.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.941+0000 7f9f177fe700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 0x7f9f0006fb80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f08004f40 tx=0x7f9f0800b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:05.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:05.945+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f9f0c05b190 con 0x7f9f180fed80 2024-04-05T03:05:06.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.149+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f1802b3b0 con 0x7f9f0006d6d0 2024-04-05T03:05:06.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.160+0000 7f9efffff700 1 -- 172.21.15.153:0/4282300847 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9f1802b3b0 con 0x7f9f0006d6d0 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (110s) 7s ago 10m 15.4M - 0.25.0 c8568f914cd2 080c3d763694 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (11m) 7s ago 11m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (9m) 8s ago 9m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 unknown 7s ago 10m - - 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (5m) 7s ago 5m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (5m) 7s ago 5m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (5m) 8s ago 5m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (5m) 8s ago 5m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (72s) 7s ago 13m 497M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (32s) 8s ago 9m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (13m) 7s ago 13m 53.2M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (9m) 8s ago 9m 46.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:05:06.160 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (17s) 7s ago 10m 5431k - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (12s) 8s ago 9m 3657k - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (9m) 7s ago 8m 222M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (8m) 7s ago 8m 228M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (8m) 7s ago 7m 196M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (7m) 8s ago 7m 264M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (7m) 8s ago 7m 229M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (6m) 8s ago 6m 231M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:05:06.161 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (38s) 8s ago 9m 46.7M - 2.43.0 a07b618ecd1d 651d19e37cd4 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 msgr2=0x7f9f0006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 0x7f9f0006fb80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f08004f40 tx=0x7f9f0800b040 comp rx=0 tx=0).stop 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 msgr2=0x7f9f18110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00b3b0 tx=0x7f9f0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 shutdown_connections 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f180fe430 0x7f9f1810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9f0006d6d0 0x7f9f0006fb80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 --2- 172.21.15.153:0/4282300847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f180fed80 0x7f9f18110080 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 >> 172.21.15.153:0/4282300847 conn(0x7f9f180f6bd0 msgr2=0x7f9f180f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 shutdown_connections 2024-04-05T03:05:06.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.162+0000 7f9f1e0b1700 1 -- 172.21.15.153:0/4282300847 wait complete. 2024-04-05T03:05:06.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 -- 172.21.15.153:0/928276779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 msgr2=0x7f99000fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 --2- 172.21.15.153:0/928276779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f99000fe350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f98f4004940 tx=0x7f98f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:06.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 -- 172.21.15.153:0/928276779 shutdown_connections 2024-04-05T03:05:06.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 --2- 172.21.15.153:0/928276779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f99000fe350 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 --2- 172.21.15.153:0/928276779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 0x7f99000fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 -- 172.21.15.153:0/928276779 >> 172.21.15.153:0/928276779 conn(0x7f99000f8390 msgr2=0x7f99000fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:06.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 -- 172.21.15.153:0/928276779 shutdown_connections 2024-04-05T03:05:06.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.263+0000 7f9904803700 1 -- 172.21.15.153:0/928276779 wait complete. 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 Processor -- start 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 -- start start 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 0x7f990010d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990010e3e0 con 0x7f99000fdee0 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f9904803700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990010ebe0 con 0x7f99000fcc30 2024-04-05T03:05:06.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48610/0 (socket says 172.21.15.153:48610) 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 -- 172.21.15.153:0/271462194 learned_addr learned my addr 172.21.15.153:0/271462194 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fed9d700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 0x7f990010d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 -- 172.21.15.153:0/271462194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 msgr2=0x7f990010d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 0x7f990010d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.264+0000 7f98fe59c700 1 -- 172.21.15.153:0/271462194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f4005040 con 0x7f99000fdee0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f98fe59c700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f98f400b3b0 tx=0x7f98f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f400e040 con 0x7f99000fdee0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990010ee60 con 0x7f99000fdee0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f400bbf0 con 0x7f99000fdee0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f401c630 con 0x7f99000fdee0 2024-04-05T03:05:06.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.265+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990010f3a0 con 0x7f99000fdee0 2024-04-05T03:05:06.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.266+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990005f5f0 con 0x7f99000fdee0 2024-04-05T03:05:06.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.266+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f98f400bd60 con 0x7f99000fdee0 2024-04-05T03:05:06.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.267+0000 7f98e77fe700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 0x7f98e806fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.267+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f98f408da20 con 0x7f99000fdee0 2024-04-05T03:05:06.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.267+0000 7f98fed9d700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 0x7f98e806fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.267+0000 7f98fed9d700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 0x7f98e806fb70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98f0004f40 tx=0x7f98f000b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:06.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.271+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f98f405b1e0 con 0x7f99000fdee0 2024-04-05T03:05:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:06 smithi203 ceph-mon[39911]: Upgrade: Updating prometheus.smithi203 2024-04-05T03:05:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:06 smithi203 ceph-mon[39911]: Deploying daemon prometheus.smithi203 on smithi203 2024-04-05T03:05:06.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:06 smithi203 ceph-mon[39911]: pgmap v36: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:06.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.549+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f990004bed0 con 0x7f99000fdee0 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.549+0000 7f98e77fe700 1 -- 172.21.15.153:0/271462194 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f98f405ad70 con 0x7f99000fdee0 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:05:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 12, 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:05:06.550 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:05:06.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.551+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 msgr2=0x7f98e806fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.551+0000 7f9904803700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 0x7f98e806fb70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98f0004f40 tx=0x7f98f000b040 comp rx=0 tx=0).stop 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.551+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 msgr2=0x7f990010dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.551+0000 7f9904803700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f98f400b3b0 tx=0x7f98f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 shutdown_connections 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99000fcc30 0x7f990010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f98e806d6c0 0x7f98e806fb70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 --2- 172.21.15.153:0/271462194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f99000fdee0 0x7f990010dde0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 >> 172.21.15.153:0/271462194 conn(0x7f99000f8390 msgr2=0x7f99000f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 shutdown_connections 2024-04-05T03:05:06.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.552+0000 7f9904803700 1 -- 172.21.15.153:0/271462194 wait complete. 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.654+0000 7fc88b014700 1 -- 172.21.15.153:0/3799879724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 msgr2=0x7fc8840fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.654+0000 7fc88b014700 1 --2- 172.21.15.153:0/3799879724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc8840fc0e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc878004940 tx=0x7fc878009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.654+0000 7fc88b014700 1 -- 172.21.15.153:0/3799879724 shutdown_connections 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.654+0000 7fc88b014700 1 --2- 172.21.15.153:0/3799879724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 0x7fc8840fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 --2- 172.21.15.153:0/3799879724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc8840fc0e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- 172.21.15.153:0/3799879724 >> 172.21.15.153:0/3799879724 conn(0x7fc8840f7430 msgr2=0x7fc8840f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- 172.21.15.153:0/3799879724 shutdown_connections 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- 172.21.15.153:0/3799879724 wait complete. 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 Processor -- start 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- start start 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 0x7fc88410df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88410e540 con 0x7fc8840fbcd0 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.655+0000 7fc88b014700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88410e6b0 con 0x7fc8840fcf80 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48634/0 (socket says 172.21.15.153:48634) 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 -- 172.21.15.153:0/285556593 learned_addr learned my addr 172.21.15.153:0/285556593 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc883fff700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 0x7fc88410df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 -- 172.21.15.153:0/285556593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 msgr2=0x7fc88410df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 0x7fc88410df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 -- 172.21.15.153:0/285556593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878005040 con 0x7fc8840fbcd0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc888db0700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc87800b970 tx=0x7fc878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc87800e040 con 0x7fc8840fbcd0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc87800bbf0 con 0x7fc8840fbcd0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc87801c630 con 0x7fc8840fbcd0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc884113010 con 0x7fc8840fbcd0 2024-04-05T03:05:06.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.656+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc884113580 con 0x7fc8840fbcd0 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.657+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc884101b10 con 0x7fc8840fbcd0 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.658+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7fc87800bd60 con 0x7fc8840fbcd0 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.658+0000 7fc8817fa700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 0x7fc87406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.658+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fc87808e160 con 0x7fc8840fbcd0 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.658+0000 7fc883fff700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 0x7fc87406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:06.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.659+0000 7fc883fff700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 0x7fc87406fb80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc870004f40 tx=0x7fc87000b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:06.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.663+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fc87805b920 con 0x7fc8840fbcd0 2024-04-05T03:05:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:06 smithi153 ceph-mon[29381]: Upgrade: Updating prometheus.smithi203 2024-04-05T03:05:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:06 smithi153 ceph-mon[29381]: Deploying daemon prometheus.smithi203 on smithi203 2024-04-05T03:05:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:06 smithi153 ceph-mon[29381]: pgmap v36: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:06.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.874+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc88402b3b0 con 0x7fc87406d6d0 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.875+0000 7fc8817fa700 1 -- 172.21.15.153:0/285556593 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fc88402b3b0 con 0x7fc87406d6d0 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:05:06.875 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:05:06.876 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:05:06.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 msgr2=0x7fc87406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 0x7fc87406fb80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc870004f40 tx=0x7fc87000b040 comp rx=0 tx=0).stop 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 msgr2=0x7fc88410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc87800b970 tx=0x7fc878009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 shutdown_connections 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fc87406d6d0 0x7fc87406fb80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8840fbcd0 0x7fc88410da00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 --2- 172.21.15.153:0/285556593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8840fcf80 0x7fc88410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 >> 172.21.15.153:0/285556593 conn(0x7fc8840f7430 msgr2=0x7fc884066880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 shutdown_connections 2024-04-05T03:05:06.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:06.878+0000 7fc88b014700 1 -- 172.21.15.153:0/285556593 wait complete. 2024-04-05T03:05:07.712 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:07 smithi203 ceph-mon[39911]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.712 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:07 smithi203 ceph-mon[39911]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.712 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:07 smithi203 ceph-mon[39911]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.712 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:07 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/271462194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:07 smithi153 ceph-mon[29381]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:07 smithi153 ceph-mon[29381]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:07 smithi153 ceph-mon[29381]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:07 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/271462194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:08.768 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:08 smithi203 ceph-mon[39911]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:08.768 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:08 smithi203 ceph-mon[39911]: pgmap v37: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:05:08.769 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:08.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:08 smithi153 ceph-mon[29381]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:08.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:08 smithi153 ceph-mon[29381]: pgmap v37: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:05:08.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:09.481 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:09 smithi203 ceph-mon[39911]: pgmap v38: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:09 smithi153 ceph-mon[29381]: pgmap v38: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:11.984 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:11 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:11.984 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:11 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:11.984 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:11 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:11.984 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:11 smithi203 ceph-mon[39911]: pgmap v39: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:12.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:11 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:12.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:11 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:12.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:11 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:12.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:11 smithi153 ceph-mon[29381]: pgmap v39: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:14 smithi203 ceph-mon[39911]: pgmap v40: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:05:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:14 smithi153 ceph-mon[29381]: pgmap v40: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:05:16.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:15 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:16.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:15 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:16.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:15 smithi203 ceph-mon[39911]: pgmap v41: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:16.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:15 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:16.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:15 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:16.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:15 smithi153 ceph-mon[29381]: pgmap v41: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:17.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:16 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:17.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:16 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:16 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:16 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:18.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:17 smithi203 ceph-mon[39911]: pgmap v42: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:17 smithi153 ceph-mon[29381]: pgmap v42: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:20.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: pgmap v43: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: pgmap v44: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:20 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: pgmap v43: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: pgmap v44: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:20.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:20 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:21.459 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:21 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:05:21.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:21 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T03:05:22.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:22 smithi153 ceph-mon[29381]: Upgrade: Updating alertmanager.smithi153 2024-04-05T03:05:22.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:22 smithi153 ceph-mon[29381]: Deploying daemon alertmanager.smithi153 on smithi153 2024-04-05T03:05:22.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:22 smithi153 ceph-mon[29381]: pgmap v45: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:22.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:22 smithi203 ceph-mon[39911]: Upgrade: Updating alertmanager.smithi153 2024-04-05T03:05:22.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:22 smithi203 ceph-mon[39911]: Deploying daemon alertmanager.smithi153 on smithi153 2024-04-05T03:05:22.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:22 smithi203 ceph-mon[39911]: pgmap v45: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:23.497 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:23 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:23 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:24.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:24 smithi153 ceph-mon[29381]: pgmap v46: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:24 smithi203 ceph-mon[39911]: pgmap v46: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:26.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:26 smithi153 ceph-mon[29381]: pgmap v47: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:26 smithi203 ceph-mon[39911]: pgmap v47: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:28.658 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:28 smithi203 ceph-mon[39911]: pgmap v48: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:28.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:28.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:28.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:28.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:28 smithi153 ceph-mon[29381]: pgmap v48: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-05T03:05:28.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:28.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:28.732 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:29 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:29 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:29 smithi203 ceph-mon[39911]: pgmap v49: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:05:29.789 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:29 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:29.789 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:29 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:29.790 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:29 smithi153 ceph-mon[29381]: pgmap v49: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:05:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:32 smithi203 ceph-mon[39911]: pgmap v50: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:32 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:32 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:32 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:32 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:32 smithi153 ceph-mon[29381]: pgmap v50: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:32 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:32 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:32 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:32 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:33 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:33 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:33 smithi203 ceph-mon[39911]: pgmap v51: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:34.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:33 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:34.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:33 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:34.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:33 smithi153 ceph-mon[29381]: pgmap v51: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: pgmap v52: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:05:36.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:36 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.750 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: pgmap v52: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-05T03:05:36.750 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.750 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.750 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.751 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:36 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 -- 172.21.15.153:0/2744759633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fbcd0 msgr2=0x7ff4480fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 --2- 172.21.15.153:0/2744759633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fbcd0 0x7ff4480fc0e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff438009ce0 tx=0x7ff438004d10 comp rx=0 tx=0).stop 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 -- 172.21.15.153:0/2744759633 shutdown_connections 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 --2- 172.21.15.153:0/2744759633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fcf80 0x7ff4480fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 --2- 172.21.15.153:0/2744759633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fbcd0 0x7ff4480fc0e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 -- 172.21.15.153:0/2744759633 >> 172.21.15.153:0/2744759633 conn(0x7ff4480f7450 msgr2=0x7ff4480f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 -- 172.21.15.153:0/2744759633 shutdown_connections 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 -- 172.21.15.153:0/2744759633 wait complete. 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.983+0000 7ff44cbde700 1 Processor -- start 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44cbde700 1 -- start start 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44cbde700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fbcd0 0x7ff448066d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:36.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44cbde700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44cbde700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448112dd0 con 0x7ff4480fcf80 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44cbde700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448112f10 con 0x7ff4480fbcd0 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51848/0 (socket says 172.21.15.153:51848) 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 -- 172.21.15.153:0/659823427 learned_addr learned my addr 172.21.15.153:0/659823427 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff44659c700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fbcd0 0x7ff448066d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 -- 172.21.15.153:0/659823427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fbcd0 msgr2=0x7ff448066d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fbcd0 0x7ff448066d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 -- 172.21.15.153:0/659823427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff438009940 con 0x7ff4480fcf80 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43dd9b700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff430002a50 tx=0x7ff4300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.984+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff430007cc0 con 0x7ff4480fcf80 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.985+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff43000f040 con 0x7ff4480fcf80 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.985+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4481131f0 con 0x7ff4480fcf80 2024-04-05T03:05:36.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.985+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff43000da30 con 0x7ff4480fcf80 2024-04-05T03:05:36.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.985+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481137b0 con 0x7ff4480fcf80 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.986+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44804c710 con 0x7ff4480fcf80 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.986+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7ff43000db90 con 0x7ff4480fcf80 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.987+0000 7ff43f7fe700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 0x7ff43406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.987+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff43008cbc0 con 0x7ff4480fcf80 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.987+0000 7ff44659c700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 0x7ff43406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:36.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.987+0000 7ff44659c700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 0x7ff43406fb80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff438015040 tx=0x7ff438004d10 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:36.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:36.991+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7ff43005a300 con 0x7ff4480fcf80 2024-04-05T03:05:37.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.205+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff448101360 con 0x7ff43406d6d0 2024-04-05T03:05:37.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.206+0000 7ff43f7fe700 1 -- 172.21.15.153:0/659823427 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7ff448101360 con 0x7ff43406d6d0 2024-04-05T03:05:37.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.208+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 msgr2=0x7ff43406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.208+0000 7ff44cbde700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 0x7ff43406fb80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff438015040 tx=0x7ff438004d10 comp rx=0 tx=0).stop 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.208+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 msgr2=0x7ff448112890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.208+0000 7ff44cbde700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff430002a50 tx=0x7ff4300046b0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.208+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 shutdown_connections 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4480fbcd0 0x7ff448066d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7ff43406d6d0 0x7ff43406fb80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 --2- 172.21.15.153:0/659823427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff4480fcf80 0x7ff448112890 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 >> 172.21.15.153:0/659823427 conn(0x7ff4480f7450 msgr2=0x7ff4480f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 shutdown_connections 2024-04-05T03:05:37.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.209+0000 7ff44cbde700 1 -- 172.21.15.153:0/659823427 wait complete. 2024-04-05T03:05:37.219 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:05:37.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- 172.21.15.153:0/3706446470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 msgr2=0x7f2c780fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 --2- 172.21.15.153:0/3706446470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c780fe450 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c004940 tx=0x7f2c6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- 172.21.15.153:0/3706446470 shutdown_connections 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 --2- 172.21.15.153:0/3706446470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c780fe450 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 --2- 172.21.15.153:0/3706446470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 0x7f2c780fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- 172.21.15.153:0/3706446470 >> 172.21.15.153:0/3706446470 conn(0x7f2c780f8550 msgr2=0x7f2c780fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- 172.21.15.153:0/3706446470 shutdown_connections 2024-04-05T03:05:37.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- 172.21.15.153:0/3706446470 wait complete. 2024-04-05T03:05:37.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 Processor -- start 2024-04-05T03:05:37.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.312+0000 7f2c7de92700 1 -- start start 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c7de92700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 0x7f2c78071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c7de92700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c7de92700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78072260 con 0x7f2c780fdfe0 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c7de92700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78070850 con 0x7f2c780fce70 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51854/0 (socket says 172.21.15.153:51854) 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 -- 172.21.15.153:0/1399693160 learned_addr learned my addr 172.21.15.153:0/1399693160 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c777fe700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 0x7f2c78071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 -- 172.21.15.153:0/1399693160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 msgr2=0x7f2c78071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 0x7f2c78071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 -- 172.21.15.153:0/1399693160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c005040 con 0x7f2c780fdfe0 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c76ffd700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00b3b0 tx=0x7f2c6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:37.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.313+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c00e040 con 0x7f2c780fdfe0 2024-04-05T03:05:37.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.314+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c6c00bbf0 con 0x7f2c780fdfe0 2024-04-05T03:05:37.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.314+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c78070aa0 con 0x7f2c780fdfe0 2024-04-05T03:05:37.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.314+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6c01c630 con 0x7f2c780fdfe0 2024-04-05T03:05:37.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.314+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c78071060 con 0x7f2c780fdfe0 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.315+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7804c710 con 0x7f2c780fdfe0 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.315+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f2c6c00bd60 con 0x7f2c780fdfe0 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.316+0000 7f2c5ffff700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 0x7f2c6006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.316+0000 7f2c777fe700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 0x7f2c6006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.316+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2c6c08d910 con 0x7f2c780fdfe0 2024-04-05T03:05:37.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.316+0000 7f2c777fe700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 0x7f2c6006fb80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c68004f60 tx=0x7f2c6800b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:37.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.320+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f2c6c05b170 con 0x7f2c780fdfe0 2024-04-05T03:05:37.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.536+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c78102300 con 0x7f2c6006d6d0 2024-04-05T03:05:37.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.538+0000 7f2c5ffff700 1 -- 172.21.15.153:0/1399693160 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f2c78102300 con 0x7f2c6006d6d0 2024-04-05T03:05:37.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 msgr2=0x7f2c6006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 0x7f2c6006fb80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2c68004f60 tx=0x7f2c6800b040 comp rx=0 tx=0).stop 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 msgr2=0x7f2c78070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c00b3b0 tx=0x7f2c6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 shutdown_connections 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c780fce70 0x7f2c78071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f2c6006d6d0 0x7f2c6006fb80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 --2- 172.21.15.153:0/1399693160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c780fdfe0 0x7f2c78070310 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 >> 172.21.15.153:0/1399693160 conn(0x7f2c780f8550 msgr2=0x7f2c780f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 shutdown_connections 2024-04-05T03:05:37.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.540+0000 7f2c7de92700 1 -- 172.21.15.153:0/1399693160 wait complete. 2024-04-05T03:05:37.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:37 smithi153 ceph-mon[29381]: pgmap v53: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-05T03:05:37.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:37 smithi153 ceph-mon[29381]: pgmap v54: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:37.567 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:37 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:05:37.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 -- 172.21.15.153:0/1376632229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2800fff90 msgr2=0x7fa2801023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1376632229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2800fff90 0x7fa2801023b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa270004940 tx=0x7fa270009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:37.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 -- 172.21.15.153:0/1376632229 shutdown_connections 2024-04-05T03:05:37.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1376632229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2800fff90 0x7fa2801023b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1376632229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280066b70 0x7fa2800ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 -- 172.21.15.153:0/1376632229 >> 172.21.15.153:0/1376632229 conn(0x7fa2800f74e0 msgr2=0x7fa2800f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:37.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 -- 172.21.15.153:0/1376632229 shutdown_connections 2024-04-05T03:05:37.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.648+0000 7fa2862f6700 1 -- 172.21.15.153:0/1376632229 wait complete. 2024-04-05T03:05:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 Processor -- start 2024-04-05T03:05:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 -- start start 2024-04-05T03:05:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 0x7fa2800fb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800fbe90 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.649+0000 7fa2862f6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800fbfd0 con 0x7fa2800fff90 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 0x7fa2800fb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51884/0 (socket says 172.21.15.153:51884) 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 -- 172.21.15.153:0/1538558424 learned_addr learned my addr 172.21.15.153:0/1538558424 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 -- 172.21.15.153:0/1538558424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 msgr2=0x7fa2800fb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 0x7fa2800fb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 -- 172.21.15.153:0/1538558424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa270005040 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27f7fe700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 0x7fa2800fb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27ffff700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa27400eb40 tx=0x7fa27400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.650+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa274007cc0 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.651+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2800fc250 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.651+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2800fc790 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.651+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa27400f040 con 0x7fa280066b70 2024-04-05T03:05:37.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.651+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa27400c9d0 con 0x7fa280066b70 2024-04-05T03:05:37.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.652+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28005f5f0 con 0x7fa280066b70 2024-04-05T03:05:37.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.653+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7fa27400cb30 con 0x7fa280066b70 2024-04-05T03:05:37.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.653+0000 7fa27cff9700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 0x7fa26806fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:37.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.654+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa27408c900 con 0x7fa280066b70 2024-04-05T03:05:37.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.654+0000 7fa27f7fe700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 0x7fa26806fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:37.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.654+0000 7fa27f7fe700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 0x7fa26806fb70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa270009b30 tx=0x7fa270009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:37.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.658+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fa2740566e0 con 0x7fa280066b70 2024-04-05T03:05:37.771 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:37 smithi203 ceph-mon[39911]: pgmap v53: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-05T03:05:37.771 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:37 smithi203 ceph-mon[39911]: pgmap v54: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-05T03:05:37.771 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:37 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T03:05:37.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.893+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2800f8fd0 con 0x7fa26806d6c0 2024-04-05T03:05:37.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.908+0000 7fa27cff9700 1 -- 172.21.15.153:0/1538558424 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fa2800f8fd0 con 0x7fa26806d6c0 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (10s) 6s ago 11m 14.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (12m) 6s ago 12m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (10m) 5s ago 10m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 unknown 6s ago 11m - - 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (5m) 6s ago 5m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (5m) 6s ago 5m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (5m) 5s ago 5m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (5m) 5s ago 5m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (104s) 6s ago 13m 500M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (64s) 5s ago 10m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (13m) 6s ago 13m 54.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (10m) 5s ago 10m 46.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:05:37.908 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (48s) 6s ago 11m 8368k - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (44s) 5s ago 10m 14.8M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (9m) 6s ago 9m 222M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (8m) 6s ago 8m 228M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (8m) 6s ago 8m 197M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (8m) 5s ago 8m 263M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (7m) 5s ago 7m 230M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (7m) 5s ago 7m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:05:37.909 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (27s) 5s ago 9m 50.9M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 msgr2=0x7fa26806fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 0x7fa26806fb70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa270009b30 tx=0x7fa270009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 msgr2=0x7fa2800fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa27400eb40 tx=0x7fa27400b040 comp rx=0 tx=0).stop 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 shutdown_connections 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa26806d6c0 0x7fa26806fb70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa280066b70 0x7fa2800fd3e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 --2- 172.21.15.153:0/1538558424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2800fff90 0x7fa2800fb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:37.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 >> 172.21.15.153:0/1538558424 conn(0x7fa2800f74e0 msgr2=0x7fa2800f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:37.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 shutdown_connections 2024-04-05T03:05:37.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:37.910+0000 7fa2862f6700 1 -- 172.21.15.153:0/1538558424 wait complete. 2024-04-05T03:05:38.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 -- 172.21.15.153:0/1426420754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 msgr2=0x7f6b3c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/1426420754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fd420 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009ce0 tx=0x7f6b30004d10 comp rx=0 tx=0).stop 2024-04-05T03:05:38.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 -- 172.21.15.153:0/1426420754 shutdown_connections 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/1426420754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fd420 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/1426420754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 0x7f6b3c0fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 -- 172.21.15.153:0/1426420754 >> 172.21.15.153:0/1426420754 conn(0x7f6b3c0f7480 msgr2=0x7f6b3c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 -- 172.21.15.153:0/1426420754 shutdown_connections 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.019+0000 7f6b40dfe700 1 -- 172.21.15.153:0/1426420754 wait complete. 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 Processor -- start 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 -- start start 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 0x7f6b3c0ffc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c100240 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b40dfe700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0fe830 con 0x7f6b3c0fbd00 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51904/0 (socket says 172.21.15.153:51904) 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 -- 172.21.15.153:0/3937148680 learned_addr learned my addr 172.21.15.153:0/3937148680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b3a59c700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 0x7f6b3c0ffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 -- 172.21.15.153:0/3937148680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 msgr2=0x7f6b3c0ffc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 0x7f6b3c0ffc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 -- 172.21.15.153:0/3937148680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30009940 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.020+0000 7f6b39d9b700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009380 tx=0x7f6b30004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3000e040 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c0fea80 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b300128e0 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3001c930 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c0ff040 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.021+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c101b40 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.022+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7f6b30012a50 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.022+0000 7f6b2affd700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 0x7f6b2406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.023+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f6b3008e970 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.023+0000 7f6b3a59c700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 0x7f6b2406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.023+0000 7f6b3a59c700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 0x7f6b2406fb80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b34004f40 tx=0x7f6b3400b040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:38.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.027+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f6b3005c0b0 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.314+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6b3c04c710 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.315+0000 7f6b2affd700 1 -- 172.21.15.153:0/3937148680 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f6b30012d00 con 0x7f6b3c0fcfb0 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:38.315 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 12, 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:05:38.316 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:05:38.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 msgr2=0x7f6b2406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 0x7f6b2406fb80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b34004f40 tx=0x7f6b3400b040 comp rx=0 tx=0).stop 2024-04-05T03:05:38.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 msgr2=0x7f6b3c0fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009380 tx=0x7f6b30004d10 comp rx=0 tx=0).stop 2024-04-05T03:05:38.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 shutdown_connections 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c0fbd00 0x7f6b3c0ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f6b2406d6d0 0x7f6b2406fb80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 --2- 172.21.15.153:0/3937148680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b3c0fcfb0 0x7f6b3c0fe2f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 >> 172.21.15.153:0/3937148680 conn(0x7f6b3c0f7480 msgr2=0x7f6b3c0667d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 shutdown_connections 2024-04-05T03:05:38.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.317+0000 7f6b40dfe700 1 -- 172.21.15.153:0/3937148680 wait complete. 2024-04-05T03:05:38.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 -- 172.21.15.153:0/3511632999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fcf80 msgr2=0x7fb45c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 --2- 172.21.15.153:0/3511632999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fcf80 0x7fb45c0fd3f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb444004940 tx=0x7fb444009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 -- 172.21.15.153:0/3511632999 shutdown_connections 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 --2- 172.21.15.153:0/3511632999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fcf80 0x7fb45c0fd3f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 --2- 172.21.15.153:0/3511632999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 -- 172.21.15.153:0/3511632999 >> 172.21.15.153:0/3511632999 conn(0x7fb45c0f7450 msgr2=0x7fb45c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 -- 172.21.15.153:0/3511632999 shutdown_connections 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.420+0000 7fb460ec9700 1 -- 172.21.15.153:0/3511632999 wait complete. 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 Processor -- start 2024-04-05T03:05:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 -- start start 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fcf80 0x7fb45c10ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c10d2f0 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb460ec9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c10d460 con 0x7fb45c0fcf80 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb45a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb45a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51924/0 (socket says 172.21.15.153:51924) 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb45a59c700 1 -- 172.21.15.153:0/1278318034 learned_addr learned my addr 172.21.15.153:0/1278318034 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.421+0000 7fb459d9b700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fcf80 0x7fb45c10ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb45a59c700 1 -- 172.21.15.153:0/1278318034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fcf80 msgr2=0x7fb45c10ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb45a59c700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fcf80 0x7fb45c10ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb45a59c700 1 -- 172.21.15.153:0/1278318034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb444005040 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb45a59c700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb45400ec90 tx=0x7fb45400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb454007b30 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c111ec0 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb45400f040 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb45400cc80 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.422+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c112370 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.423+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c04c710 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.423+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 32) v1 ==== 91677+0+0 (secure 0 0 0) 0x7fb454007c90 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.424+0000 7fb452ffd700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 0x7fb44806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:05:38.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.424+0000 7fb459d9b700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 0x7fb44806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:05:38.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.424+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb45408d250 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.424+0000 7fb459d9b700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 0x7fb44806fb80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb444009b30 tx=0x7fb444009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:05:38.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.429+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fb45405a990 con 0x7fb45c0fbcd0 2024-04-05T03:05:38.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:38 smithi153 ceph-mon[29381]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:38.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:38 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:38.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:38 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3937148680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.643+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb45c112560 con 0x7fb44806d6d0 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.644+0000 7fb452ffd700 1 -- 172.21.15.153:0/1278318034 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fb45c112560 con 0x7fb44806d6d0 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T03:05:38.644 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading grafana daemons", 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:05:38.645 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:05:38.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 msgr2=0x7fb44806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 0x7fb44806fb80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb444009b30 tx=0x7fb444009e30 comp rx=0 tx=0).stop 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 msgr2=0x7fb45c10c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb45400ec90 tx=0x7fb45400b040 comp rx=0 tx=0).stop 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 shutdown_connections 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb44806d6d0 0x7fb44806fb80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.646+0000 7fb460ec9700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb45c0fbcd0 0x7fb45c10c7b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.647+0000 7fb460ec9700 1 --2- 172.21.15.153:0/1278318034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb45c0fcf80 0x7fb45c10ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.647+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 >> 172.21.15.153:0/1278318034 conn(0x7fb45c0f7450 msgr2=0x7fb45c0f8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.647+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 shutdown_connections 2024-04-05T03:05:38.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:05:38.647+0000 7fb460ec9700 1 -- 172.21.15.153:0/1278318034 wait complete. 2024-04-05T03:05:38.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:38 smithi203 ceph-mon[39911]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:38.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:38 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:38.659 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:38 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3937148680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:39.679 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:39 smithi153 ceph-mon[29381]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:39.680 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:39 smithi153 ceph-mon[29381]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:39.680 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:39 smithi153 ceph-mon[29381]: pgmap v55: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-05T03:05:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:39 smithi203 ceph-mon[39911]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:39 smithi203 ceph-mon[39911]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:39 smithi203 ceph-mon[39911]: pgmap v55: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-05T03:05:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:40 smithi153 ceph-mon[29381]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:40 smithi153 ceph-mon[29381]: Upgrade: Updating grafana.smithi153 2024-04-05T03:05:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:40 smithi153 ceph-mon[29381]: Deploying daemon grafana.smithi153 on smithi153 2024-04-05T03:05:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:40 smithi203 ceph-mon[39911]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:05:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:40 smithi203 ceph-mon[39911]: Upgrade: Updating grafana.smithi153 2024-04-05T03:05:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:40 smithi203 ceph-mon[39911]: Deploying daemon grafana.smithi153 on smithi153 2024-04-05T03:05:41.559 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:41 smithi153 ceph-mon[29381]: pgmap v56: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 6 op/s 2024-04-05T03:05:41.559 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:41 smithi153 ceph-mon[29381]: mgrmap e33: smithi153.npzyqy(active, since 92s), standbys: smithi203.iakodz 2024-04-05T03:05:41.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:41 smithi203 ceph-mon[39911]: pgmap v56: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 6 op/s 2024-04-05T03:05:41.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:41 smithi203 ceph-mon[39911]: mgrmap e33: smithi153.npzyqy(active, since 92s), standbys: smithi203.iakodz 2024-04-05T03:05:43.668 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:43 smithi153 ceph-mon[29381]: pgmap v57: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 6 op/s 2024-04-05T03:05:43.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:43.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:43.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:43 smithi203 ceph-mon[39911]: pgmap v57: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 6 op/s 2024-04-05T03:05:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:43.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:45.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:45 smithi153 ceph-mon[29381]: pgmap v58: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 5 op/s 2024-04-05T03:05:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:45 smithi203 ceph-mon[39911]: pgmap v58: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 692 B/s wr, 5 op/s 2024-04-05T03:05:47.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:47 smithi203 ceph-mon[39911]: pgmap v59: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 614 B/s wr, 5 op/s 2024-04-05T03:05:47.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:47 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:47.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:47 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:47 smithi153 ceph-mon[29381]: pgmap v59: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 614 B/s wr, 5 op/s 2024-04-05T03:05:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:47 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:47 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:49.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:49 smithi153 ceph-mon[29381]: pgmap v60: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:05:49.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:49 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:49.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:49 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:49 smithi203 ceph-mon[39911]: pgmap v60: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T03:05:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:49 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:49 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:51.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:51 smithi153 ceph-mon[29381]: pgmap v61: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T03:05:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:51 smithi203 ceph-mon[39911]: pgmap v61: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: pgmap v62: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: Upgrade: Finalizing container_image settings 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T03:05:53.049 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: Upgrade: Complete! 2024-04-05T03:05:53.050 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.051 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:52 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: pgmap v62: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: Upgrade: Finalizing container_image settings 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T03:05:53.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T03:05:53.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: Upgrade: Complete! 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.076 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.077 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:53.077 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:05:53.077 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:05:53.077 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:05:53.077 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:52 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:53 smithi203 ceph-mon[39911]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-05T03:05:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:53 smithi203 ceph-mon[39911]: Cluster is now healthy 2024-04-05T03:05:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:53 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:53 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:54.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:53 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:53 smithi153 ceph-mon[29381]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-05T03:05:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:53 smithi153 ceph-mon[29381]: Cluster is now healthy 2024-04-05T03:05:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:53 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:53 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:05:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:53 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:05:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:54 smithi203 ceph-mon[39911]: pgmap v63: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:54 smithi153 ceph-mon[29381]: pgmap v63: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:56 smithi153 ceph-mon[29381]: pgmap v64: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:57.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:56 smithi203 ceph-mon[39911]: pgmap v64: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:05:58 smithi203 ceph-mon[39911]: pgmap v65: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:05:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:05:58 smithi153 ceph-mon[29381]: pgmap v65: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:00 smithi153 ceph-mon[29381]: pgmap v66: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-05T03:06:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:00 smithi203 ceph-mon[39911]: pgmap v66: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-05T03:06:03.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:02 smithi153 ceph-mon[29381]: pgmap v67: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:02 smithi203 ceph-mon[39911]: pgmap v67: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:05.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:04 smithi153 ceph-mon[29381]: pgmap v68: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:04 smithi203 ceph-mon[39911]: pgmap v68: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:07.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:06 smithi153 ceph-mon[29381]: pgmap v69: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:07.098 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:06 smithi203 ceph-mon[39911]: pgmap v69: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:08.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- 172.21.15.153:0/2654636819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fbcd0 msgr2=0x7f0e600fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 --2- 172.21.15.153:0/2654636819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fbcd0 0x7f0e600fc0e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c004940 tx=0x7f0e5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- 172.21.15.153:0/2654636819 shutdown_connections 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 --2- 172.21.15.153:0/2654636819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fcf80 0x7f0e600fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 --2- 172.21.15.153:0/2654636819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fbcd0 0x7f0e600fc0e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- 172.21.15.153:0/2654636819 >> 172.21.15.153:0/2654636819 conn(0x7f0e600f7430 msgr2=0x7f0e600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- 172.21.15.153:0/2654636819 shutdown_connections 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- 172.21.15.153:0/2654636819 wait complete. 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 Processor -- start 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.763+0000 7f0e685f8700 1 -- start start 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e685f8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fbcd0 0x7f0e60066cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e685f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e685f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60112c00 con 0x7f0e600fcf80 2024-04-05T03:06:08.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e685f8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60112d40 con 0x7f0e600fbcd0 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58878/0 (socket says 172.21.15.153:58878) 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e66394700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fbcd0 0x7f0e60066cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 -- 172.21.15.153:0/853970874 learned_addr learned my addr 172.21.15.153:0/853970874 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 -- 172.21.15.153:0/853970874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fbcd0 msgr2=0x7f0e60066cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fbcd0 0x7f0e60066cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 -- 172.21.15.153:0/853970874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c005040 con 0x7f0e600fcf80 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e65b93700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000eb40 tx=0x7f0e5000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.764+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e50007cc0 con 0x7f0e600fcf80 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.765+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e5000f040 con 0x7f0e600fcf80 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.765+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e60112f40 con 0x7f0e600fcf80 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.765+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e5000ca10 con 0x7f0e600fcf80 2024-04-05T03:06:08.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.765+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e60113450 con 0x7f0e600fcf80 2024-04-05T03:06:08.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.766+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6004c710 con 0x7f0e600fcf80 2024-04-05T03:06:08.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.766+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f0e5000cc80 con 0x7f0e600fcf80 2024-04-05T03:06:08.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.766+0000 7f0e56ffd700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 0x7f0e4c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:08.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.767+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0e5008c770 con 0x7f0e600fcf80 2024-04-05T03:06:08.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.767+0000 7f0e66394700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 0x7f0e4c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:08.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.767+0000 7f0e66394700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 0x7f0e4c06fa50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009b30 tx=0x7f0e5c009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:08.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.771+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f0e5005a180 con 0x7f0e600fcf80 2024-04-05T03:06:08.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.987+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e60101220 con 0x7f0e4c06d5a0 2024-04-05T03:06:08.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.988+0000 7f0e56ffd700 1 -- 172.21.15.153:0/853970874 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0e60101220 con 0x7f0e4c06d5a0 2024-04-05T03:06:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 msgr2=0x7f0e4c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 0x7f0e4c06fa50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c009b30 tx=0x7f0e5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 msgr2=0x7f0e601126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000eb40 tx=0x7f0e5000b040 comp rx=0 tx=0).stop 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 shutdown_connections 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600fbcd0 0x7f0e60066cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f0e4c06d5a0 0x7f0e4c06fa50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 --2- 172.21.15.153:0/853970874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e600fcf80 0x7f0e601126c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 >> 172.21.15.153:0/853970874 conn(0x7f0e600f7430 msgr2=0x7f0e600f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 shutdown_connections 2024-04-05T03:06:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:08.991+0000 7f0e685f8700 1 -- 172.21.15.153:0/853970874 wait complete. 2024-04-05T03:06:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:08 smithi203 ceph-mon[39911]: pgmap v70: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:08 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:08 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:09.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:08 smithi153 ceph-mon[29381]: pgmap v70: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:09.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:08 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:09.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:08 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:09.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-05T03:06:09.849 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:10.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:09 smithi153 ceph-mon[29381]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:10.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:09 smithi203 ceph-mon[39911]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:11.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:10 smithi153 ceph-mon[29381]: pgmap v71: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 -- 172.21.15.153:0/1156100974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18102710 msgr2=0x7f1d180fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 --2- 172.21.15.153:0/1156100974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18102710 0x7f1d180fbd80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1d08004940 tx=0x7f1d08009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 -- 172.21.15.153:0/1156100974 shutdown_connections 2024-04-05T03:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 --2- 172.21.15.153:0/1156100974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18102710 0x7f1d180fbd80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 --2- 172.21.15.153:0/1156100974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18101d70 0x7f1d18102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 -- 172.21.15.153:0/1156100974 >> 172.21.15.153:0/1156100974 conn(0x7f1d18072bc0 msgr2=0x7f1d18072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 -- 172.21.15.153:0/1156100974 shutdown_connections 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.274+0000 7f1d2064f700 1 -- 172.21.15.153:0/1156100974 wait complete. 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 Processor -- start 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 -- start start 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18102710 0x7f1d1810f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1810f720 con 0x7f1d18101d70 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d2064f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1810f860 con 0x7f1d18102710 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58904/0 (socket says 172.21.15.153:58904) 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 -- 172.21.15.153:0/940707552 learned_addr learned my addr 172.21.15.153:0/940707552 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1dbea700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18102710 0x7f1d1810f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 -- 172.21.15.153:0/940707552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18102710 msgr2=0x7f1d1810f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18102710 0x7f1d1810f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 -- 172.21.15.153:0/940707552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08005040 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.275+0000 7f1d1e3eb700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400c830 tx=0x7f1d14004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.276+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d140106f0 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.276+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d18117b50 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.276+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d14010d20 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.276+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d140113a0 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.276+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d181180e0 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.277+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18104240 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.277+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f1d14012400 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.278+0000 7f1d0effd700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 0x7f1d0406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.278+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1d1408cb00 con 0x7f1d18101d70 2024-04-05T03:06:11.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.278+0000 7f1d1dbea700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 0x7f1d0406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:11.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.278+0000 7f1d1dbea700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 0x7f1d0406fa50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d08004890 tx=0x7f1d08009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:11.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.282+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f1d140569b0 con 0x7f1d18101d70 2024-04-05T03:06:11.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:10 smithi203 ceph-mon[39911]: pgmap v71: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:11.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.555+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d1804bed0 con 0x7f1d18101d70 2024-04-05T03:06:11.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.556+0000 7f1d0effd700 1 -- 172.21.15.153:0/940707552 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f1d1405a070 con 0x7f1d18101d70 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 msgr2=0x7f1d0406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 0x7f1d0406fa50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d08004890 tx=0x7f1d08009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 msgr2=0x7f1d1810eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400c830 tx=0x7f1d14004470 comp rx=0 tx=0).stop 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 shutdown_connections 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f1d0406d5a0 0x7f1d0406fa50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d18101d70 0x7f1d1810eca0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 --2- 172.21.15.153:0/940707552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d18102710 0x7f1d1810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 >> 172.21.15.153:0/940707552 conn(0x7f1d18072bc0 msgr2=0x7f1d180fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 shutdown_connections 2024-04-05T03:06:11.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:11.558+0000 7f1d2064f700 1 -- 172.21.15.153:0/940707552 wait complete. 2024-04-05T03:06:11.568 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:06:12.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:11 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/940707552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:12.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-05T03:06:12.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:11 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/940707552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:12.523 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:12 smithi153 ceph-mon[29381]: pgmap v72: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:13.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:12 smithi203 ceph-mon[39911]: pgmap v72: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- 172.21.15.153:0/283967242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7494101d30 msgr2=0x7f7494102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 --2- 172.21.15.153:0/283967242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7494101d30 0x7f7494102100 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7484004940 tx=0x7f7484009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:14.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- 172.21.15.153:0/283967242 shutdown_connections 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 --2- 172.21.15.153:0/283967242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74941026d0 0x7f74940fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 --2- 172.21.15.153:0/283967242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7494101d30 0x7f7494102100 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- 172.21.15.153:0/283967242 >> 172.21.15.153:0/283967242 conn(0x7f74940f78e0 msgr2=0x7f74940f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- 172.21.15.153:0/283967242 shutdown_connections 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- 172.21.15.153:0/283967242 wait complete. 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 Processor -- start 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.172+0000 7f749a697700 1 -- start start 2024-04-05T03:06:14.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f749a697700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7494101d30 0x7f7494102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f749a697700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f749a697700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494104ae0 con 0x7f74941026d0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f749a697700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494103610 con 0x7f7494101d30 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58920/0 (socket says 172.21.15.153:58920) 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 -- 172.21.15.153:0/1874826956 learned_addr learned my addr 172.21.15.153:0/1874826956 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f7493fff700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7494101d30 0x7f7494102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 -- 172.21.15.153:0/1874826956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7494101d30 msgr2=0x7f7494102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7494101d30 0x7f7494102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 -- 172.21.15.153:0/1874826956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484005040 con 0x7f74941026d0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f74937fe700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f748800c800 tx=0x7f7488004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f748800a760 con 0x7f74941026d0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74941038f0 con 0x7f74941026d0 2024-04-05T03:06:14.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f748800ad90 con 0x7f74941026d0 2024-04-05T03:06:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.173+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74880113a0 con 0x7f74941026d0 2024-04-05T03:06:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.174+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749411a150 con 0x7f74941026d0 2024-04-05T03:06:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.174+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74940fd480 con 0x7f74941026d0 2024-04-05T03:06:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.175+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f7488012400 con 0x7f74941026d0 2024-04-05T03:06:14.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.175+0000 7f7490ff9700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 0x7f747c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:14.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.176+0000 7f7493fff700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 0x7f747c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:14.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.176+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f748808cbd0 con 0x7f74941026d0 2024-04-05T03:06:14.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.176+0000 7f7493fff700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 0x7f747c06fa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7484004f90 tx=0x7f7484009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:14.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.180+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f7488056a80 con 0x7f74941026d0 2024-04-05T03:06:14.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.455+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f749411a3a0 con 0x7f74941026d0 2024-04-05T03:06:14.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.456+0000 7f7490ff9700 1 -- 172.21.15.153:0/1874826956 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f748805a140 con 0x7f74941026d0 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 msgr2=0x7f747c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 0x7f747c06fa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7484004f90 tx=0x7f7484009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 msgr2=0x7f74941030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f748800c800 tx=0x7f7488004470 comp rx=0 tx=0).stop 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 shutdown_connections 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7494101d30 0x7f7494102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f747c06d5a0 0x7f747c06fa50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 --2- 172.21.15.153:0/1874826956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74941026d0 0x7f74941030d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 >> 172.21.15.153:0/1874826956 conn(0x7f74940f78e0 msgr2=0x7f74940f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 shutdown_connections 2024-04-05T03:06:14.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:14.458+0000 7f749a697700 1 -- 172.21.15.153:0/1874826956 wait complete. 2024-04-05T03:06:14.468 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)" 2024-04-05T03:06:15.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:14 smithi153 ceph-mon[29381]: pgmap v73: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:15.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:14 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1874826956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:15.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-05T03:06:15.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:14 smithi203 ceph-mon[39911]: pgmap v73: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:15.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:14 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1874826956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:15.439 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:16.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.930+0000 7f47cec8e700 1 -- 172.21.15.153:0/3295357951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8102710 msgr2=0x7f47c80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:16.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.930+0000 7f47cec8e700 1 --2- 172.21.15.153:0/3295357951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8102710 0x7f47c80fbd80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47b4004940 tx=0x7f47b4009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:16.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- 172.21.15.153:0/3295357951 shutdown_connections 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 --2- 172.21.15.153:0/3295357951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8102710 0x7f47c80fbd80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 --2- 172.21.15.153:0/3295357951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8101d70 0x7f47c8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- 172.21.15.153:0/3295357951 >> 172.21.15.153:0/3295357951 conn(0x7f47c8072bc0 msgr2=0x7f47c8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- 172.21.15.153:0/3295357951 shutdown_connections 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- 172.21.15.153:0/3295357951 wait complete. 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 Processor -- start 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- start start 2024-04-05T03:06:16.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 0x7f47c810f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c810f760 con 0x7f47c8101d70 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.931+0000 7f47cec8e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c810f8a0 con 0x7f47c8102710 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58928/0 (socket says 172.21.15.153:58928) 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 -- 172.21.15.153:0/1626216990 learned_addr learned my addr 172.21.15.153:0/1626216990 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47c7fff700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 0x7f47c810f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 -- 172.21.15.153:0/1626216990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 msgr2=0x7f47c810f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 0x7f47c810f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 -- 172.21.15.153:0/1626216990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4005040 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47c7fff700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 0x7f47c810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cca2a700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00c8c0 tx=0x7f47bc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47bc00a980 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47bc00ce60 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c8117b90 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47bc0115e0 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.932+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c8118120 con 0x7f47c8101d70 2024-04-05T03:06:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.933+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c8104240 con 0x7f47c8101d70 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.934+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f47bc012400 con 0x7f47c8101d70 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.934+0000 7f47c57fa700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 0x7f47b806fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.934+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f47bc08cd00 con 0x7f47c8101d70 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.934+0000 7f47c7fff700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 0x7f47b806fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.935+0000 7f47c7fff700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 0x7f47b806fa60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47c8066020 tx=0x7f47b4009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:16.939+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f47bc090030 con 0x7f47c8101d70 2024-04-05T03:06:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:16 smithi153 ceph-mon[29381]: pgmap v74: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:17.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.214+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f47c8108e80 con 0x7f47c8101d70 2024-04-05T03:06:17.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.214+0000 7f47c57fa700 1 -- 172.21.15.153:0/1626216990 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f47bc056bb0 con 0x7f47c8101d70 2024-04-05T03:06:17.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.216+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 msgr2=0x7f47b806fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.216+0000 7f47cec8e700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 0x7f47b806fa60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47c8066020 tx=0x7f47b4009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 msgr2=0x7f47c810ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00c8c0 tx=0x7f47bc0044c0 comp rx=0 tx=0).stop 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 shutdown_connections 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f47b806d5b0 0x7f47b806fa60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8101d70 0x7f47c810ece0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 --2- 172.21.15.153:0/1626216990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47c8102710 0x7f47c810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 >> 172.21.15.153:0/1626216990 conn(0x7f47c8072bc0 msgr2=0x7f47c80fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 shutdown_connections 2024-04-05T03:06:17.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:17.217+0000 7f47cec8e700 1 -- 172.21.15.153:0/1626216990 wait complete. 2024-04-05T03:06:17.226 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:06:17.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:16 smithi203 ceph-mon[39911]: pgmap v74: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:17.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-05T03:06:18.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:17 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1626216990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:18.128 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:18.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:17 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1626216990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:18 smithi203 ceph-mon[39911]: pgmap v75: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:19.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:18 smithi153 ceph-mon[29381]: pgmap v75: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:19.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- 172.21.15.153:0/1804457191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 msgr2=0x7f90b4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/1804457191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4106ec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f90a4004940 tx=0x7f90a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- 172.21.15.153:0/1804457191 shutdown_connections 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/1804457191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4106ec0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/1804457191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 0x7f90b40fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- 172.21.15.153:0/1804457191 >> 172.21.15.153:0/1804457191 conn(0x7f90b40f7870 msgr2=0x7f90b40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:19.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- 172.21.15.153:0/1804457191 shutdown_connections 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- 172.21.15.153:0/1804457191 wait complete. 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 Processor -- start 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.756+0000 7f90bbf8e700 1 -- start start 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90bbf8e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 0x7f90b4113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90bbf8e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90bbf8e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b41145a0 con 0x7f90b40fe7f0 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90bbf8e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b410db30 con 0x7f90b40fde50 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36030/0 (socket says 172.21.15.153:36030) 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 -- 172.21.15.153:0/3562820723 learned_addr learned my addr 172.21.15.153:0/3562820723 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9d2a700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 0x7f90b4113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:19.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 -- 172.21.15.153:0/3562820723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 msgr2=0x7f90b4113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 0x7f90b4113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 -- 172.21.15.153:0/3562820723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4005040 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90b9529700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f90a400b3b0 tx=0x7f90a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.757+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a400e040 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.758+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b410ddb0 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.758+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90a400bca0 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.758+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a4013660 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.758+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b410e2f0 con 0x7f90b40fe7f0 2024-04-05T03:06:19.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.758+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b4066a90 con 0x7f90b40fe7f0 2024-04-05T03:06:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.759+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f90a4022020 con 0x7f90b40fe7f0 2024-04-05T03:06:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.759+0000 7f90aa7fc700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 0x7f90a006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.760+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f90a4091b80 con 0x7f90b40fe7f0 2024-04-05T03:06:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.760+0000 7f90b9d2a700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 0x7f90a006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.760+0000 7f90b9d2a700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 0x7f90a006fa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009bf0 tx=0x7f90b00091a0 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:19.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.764+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f90a405b950 con 0x7f90b40fe7f0 2024-04-05T03:06:19.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:19.980+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}) v1 -- 0x7f90b4109fb0 con 0x7f90a006d5a0 2024-04-05T03:06:21.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:20 smithi153 ceph-mon[29381]: pgmap v76: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:21.151 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:20 smithi153 ceph-mon[29381]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:21.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:20 smithi203 ceph-mon[39911]: pgmap v76: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:21.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:20 smithi203 ceph-mon[39911]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:22.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.150+0000 7f90aa7fc700 1 -- 172.21.15.153:0/3562820723 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5092 (secure 0 0 0) 0x7f90b4109fb0 con 0x7f90a006d5a0 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 msgr2=0x7f90a006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 0x7f90a006fa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009bf0 tx=0x7f90b00091a0 comp rx=0 tx=0).stop 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 msgr2=0x7f90b4113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f90a400b3b0 tx=0x7f90a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 shutdown_connections 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b40fde50 0x7f90b4113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f90a006d5a0 0x7f90a006fa50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 --2- 172.21.15.153:0/3562820723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90b40fe7f0 0x7f90b4113ec0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 >> 172.21.15.153:0/3562820723 conn(0x7f90b40f7870 msgr2=0x7f90b4103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 shutdown_connections 2024-04-05T03:06:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:22.152+0000 7f90bbf8e700 1 -- 172.21.15.153:0/3562820723 wait complete. 2024-04-05T03:06:22.163 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:06:22.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch ps' 2024-04-05T03:06:22.981 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:22 smithi153 ceph-mon[29381]: pgmap v77: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:22 smithi203 ceph-mon[39911]: pgmap v77: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:24.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:23 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:23 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:24.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.561+0000 7fe72e0c6700 1 -- 172.21.15.153:0/218015409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 msgr2=0x7fe728106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.561+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/218015409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728106e50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe71c004940 tx=0x7fe71c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 -- 172.21.15.153:0/218015409 shutdown_connections 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/218015409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728106e50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/218015409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe7280fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 -- 172.21.15.153:0/218015409 >> 172.21.15.153:0/218015409 conn(0x7fe7280f7880 msgr2=0x7fe7280f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 -- 172.21.15.153:0/218015409 shutdown_connections 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 -- 172.21.15.153:0/218015409 wait complete. 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 Processor -- start 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.562+0000 7fe72e0c6700 1 -- start start 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe728113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7281144a0 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72810daa0 con 0x7fe7280fde70 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36050/0 (socket says 172.21.15.153:36050) 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 -- 172.21.15.153:0/1824751874 learned_addr learned my addr 172.21.15.153:0/1824751874 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe7277fe700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe728113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 -- 172.21.15.153:0/1824751874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 msgr2=0x7fe728113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe728113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 -- 172.21.15.153:0/1824751874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c005040 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe7277fe700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe728113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe726ffd700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009380 tx=0x7fe71c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71c00e040 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72810dd80 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe71c009af0 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71c01c630 con 0x7fe7280fe810 2024-04-05T03:06:24.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.563+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72810e2c0 con 0x7fe7280fe810 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.565+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe728066b30 con 0x7fe7280fe810 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.565+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fe71c01c8a0 con 0x7fe7280fe810 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.565+0000 7fe70ffff700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 0x7fe71006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.565+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fe71c08dee0 con 0x7fe7280fe810 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.565+0000 7fe7277fe700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 0x7fe71006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.566+0000 7fe7277fe700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 0x7fe71006fa50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe718007fd0 tx=0x7fe718009040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:24.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.570+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fe71c05ba10 con 0x7fe7280fe810 2024-04-05T03:06:24.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.774+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe728109e30 con 0x7fe71006d5a0 2024-04-05T03:06:24.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.784+0000 7fe70ffff700 1 -- 172.21.15.153:0/1824751874 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe728109e30 con 0x7fe71006d5a0 2024-04-05T03:06:24.786 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:06:24.786 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (57s) 37s ago 12m 14.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (13m) 37s ago 13m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (11m) 52s ago 11m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (42s) 37s ago 11m 56.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (6m) 37s ago 6m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (6m) 37s ago 6m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (6m) 52s ago 6m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (6m) 52s ago 6m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (2m) 37s ago 14m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (111s) 52s ago 10m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (14m) 37s ago 14m 55.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (10m) 52s ago 10m 46.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (95s) 37s ago 12m 8530k - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:06:24.787 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (91s) 52s ago 10m 14.8M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (10m) 37s ago 10m 222M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (9m) 37s ago 9m 228M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (9m) 37s ago 9m 197M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (8m) 52s ago 8m 263M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (8m) 52s ago 8m 230M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (8m) 52s ago 8m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:06:24.788 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (74s) 52s ago 10m 50.9M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 msgr2=0x7fe71006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 0x7fe71006fa50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe718007fd0 tx=0x7fe718009040 comp rx=0 tx=0).stop 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 msgr2=0x7fe728113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009380 tx=0x7fe71c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 shutdown_connections 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7280fde70 0x7fe728113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fe71006d5a0 0x7fe71006fa50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 --2- 172.21.15.153:0/1824751874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe7280fe810 0x7fe728113e50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 >> 172.21.15.153:0/1824751874 conn(0x7fe7280f7880 msgr2=0x7fe7280f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 shutdown_connections 2024-04-05T03:06:24.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:24.786+0000 7fe72e0c6700 1 -- 172.21.15.153:0/1824751874 wait complete. 2024-04-05T03:06:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:24 smithi153 ceph-mon[29381]: pgmap v78: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:25.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:24 smithi203 ceph-mon[39911]: pgmap v78: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:25.383 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T03:06:25.384 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:06:25.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-05T03:06:25.646 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:25 smithi153 ceph-mon[29381]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:26.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:25 smithi153 ceph-mon[29381]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:26.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:25 smithi203 ceph-mon[39911]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:26.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:25 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:27.004 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:26 smithi153 ceph-mon[29381]: pgmap v79: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:27.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- 172.21.15.153:0/6616522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 msgr2=0x7fcc701020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:27.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/6616522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc701020f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c004940 tx=0x7fcc6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- 172.21.15.153:0/6616522 shutdown_connections 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/6616522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 0x7fcc700fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/6616522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc701020f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- 172.21.15.153:0/6616522 >> 172.21.15.153:0/6616522 conn(0x7fcc700f78b0 msgr2=0x7fcc700f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- 172.21.15.153:0/6616522 shutdown_connections 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- 172.21.15.153:0/6616522 wait complete. 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 Processor -- start 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- start start 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:27.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 0x7fcc7010ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700671b0 con 0x7fcc70101d20 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.231+0000 7fcc77bbb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7010d3c0 con 0x7fcc701026c0 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36066/0 (socket says 172.21.15.153:36066) 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 -- 172.21.15.153:0/1478489916 learned_addr learned my addr 172.21.15.153:0/1478489916 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75156700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 0x7fcc7010ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 -- 172.21.15.153:0/1478489916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 msgr2=0x7fcc7010ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:27.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 0x7fcc7010ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 -- 172.21.15.153:0/1478489916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c005040 con 0x7fcc70101d20 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc75957700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00b3b0 tx=0x7fcc6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6c00e040 con 0x7fcc70101d20 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc6c00bbc0 con 0x7fcc70101d20 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7010d640 con 0x7fcc70101d20 2024-04-05T03:06:27.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6c01c3f0 con 0x7fcc70101d20 2024-04-05T03:06:27.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.232+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7011a1f0 con 0x7fcc70101d20 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.233+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc700fd470 con 0x7fcc70101d20 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.234+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fcc6c00bd30 con 0x7fcc70101d20 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.234+0000 7fcc667fc700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 0x7fcc5c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.234+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fcc6c08dd70 con 0x7fcc70101d20 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.234+0000 7fcc75156700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 0x7fcc5c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.235+0000 7fcc75156700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 0x7fcc5c06fa50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc60006fd0 tx=0x7fcc60008040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:27.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.239+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fcc6c05b750 con 0x7fcc70101d20 2024-04-05T03:06:27.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:26 smithi203 ceph-mon[39911]: pgmap v79: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:27.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.442+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fcc7004c710 con 0x7fcc70101d20 2024-04-05T03:06:27.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.446+0000 7fcc667fc700 1 -- 172.21.15.153:0/1478489916 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v46)=0 v46) v1 ==== 125+0+0 (secure 0 0 0) 0x7fcc6c05b2e0 con 0x7fcc70101d20 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 msgr2=0x7fcc5c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 0x7fcc5c06fa50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc60006fd0 tx=0x7fcc60008040 comp rx=0 tx=0).stop 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 msgr2=0x7fcc7010c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00b3b0 tx=0x7fcc6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 shutdown_connections 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fcc5c06d5a0 0x7fcc5c06fa50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc70101d20 0x7fcc7010c940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 --2- 172.21.15.153:0/1478489916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc701026c0 0x7fcc7010ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 >> 172.21.15.153:0/1478489916 conn(0x7fcc700f78b0 msgr2=0x7fcc700f87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 shutdown_connections 2024-04-05T03:06:27.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:27.448+0000 7fcc77bbb700 1 -- 172.21.15.153:0/1478489916 wait complete. 2024-04-05T03:06:28.133 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T03:06:28.133 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:06:28.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-05T03:06:28.389 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:28 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1478489916' entity='client.admin' 2024-04-05T03:06:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:06:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:28 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:28 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1478489916' entity='client.admin' 2024-04-05T03:06:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:06:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:28 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:29.620 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:29 smithi153 ceph-mon[29381]: pgmap v80: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:29 smithi203 ceph-mon[39911]: pgmap v80: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 -- 172.21.15.153:0/3264722837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 msgr2=0x7fd8600fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 --2- 172.21.15.153:0/3264722837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd8600fce10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd84c004940 tx=0x7fd84c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 -- 172.21.15.153:0/3264722837 shutdown_connections 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 --2- 172.21.15.153:0/3264722837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd8600fce10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 --2- 172.21.15.153:0/3264722837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8601031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.127+0000 7fd86721b700 1 -- 172.21.15.153:0/3264722837 >> 172.21.15.153:0/3264722837 conn(0x7fd8600f89b0 msgr2=0x7fd8600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 -- 172.21.15.153:0/3264722837 shutdown_connections 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 -- 172.21.15.153:0/3264722837 wait complete. 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 Processor -- start 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 -- start start 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600722a0 con 0x7fd8601037a0 2024-04-05T03:06:30.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.128+0000 7fd86721b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860070d40 con 0x7fd860102e00 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56784/0 (socket says 172.21.15.153:56784) 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 -- 172.21.15.153:0/4132767914 learned_addr learned my addr 172.21.15.153:0/4132767914 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd864fb7700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 -- 172.21.15.153:0/4132767914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 msgr2=0x7fd8600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 -- 172.21.15.153:0/4132767914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c005040 con 0x7fd8601037a0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd864fb7700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85ffff700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009100 tx=0x7fd84c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd84c00e040 con 0x7fd8601037a0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd84c013420 con 0x7fd8601037a0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd84c01c610 con 0x7fd8601037a0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd860070fc0 con 0x7fd8601037a0 2024-04-05T03:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.129+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86011a170 con 0x7fd8601037a0 2024-04-05T03:06:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.130+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860100110 con 0x7fd8601037a0 2024-04-05T03:06:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.131+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fd84c013590 con 0x7fd8601037a0 2024-04-05T03:06:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.131+0000 7fd85d7fa700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 0x7fd85006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.131+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd84c08de50 con 0x7fd8601037a0 2024-04-05T03:06:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.131+0000 7fd864fb7700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 0x7fd85006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:30.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.131+0000 7fd864fb7700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 0x7fd85006fa50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd854009a80 tx=0x7fd854008040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.136+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fd84c057d00 con 0x7fd8601037a0 2024-04-05T03:06:30.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.330+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fd8600fe550 con 0x7fd8601037a0 2024-04-05T03:06:30.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.330+0000 7fd85d7fa700 1 -- 172.21.15.153:0/4132767914 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v46)=0 v46) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd84c05b3c0 con 0x7fd8601037a0 2024-04-05T03:06:30.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.332+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 msgr2=0x7fd85006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:30.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.332+0000 7fd86721b700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 0x7fd85006fa50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd854009a80 tx=0x7fd854008040 comp rx=0 tx=0).stop 2024-04-05T03:06:30.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.332+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 msgr2=0x7fd860070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:30.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.332+0000 7fd86721b700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009100 tx=0x7fd84c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:30.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 shutdown_connections 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd860102e00 0x7fd8600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fd85006d5a0 0x7fd85006fa50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 --2- 172.21.15.153:0/4132767914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8601037a0 0x7fd860070800 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 >> 172.21.15.153:0/4132767914 conn(0x7fd8600f89b0 msgr2=0x7fd8600fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 shutdown_connections 2024-04-05T03:06:30.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:30.333+0000 7fd86721b700 1 -- 172.21.15.153:0/4132767914 wait complete. 2024-04-05T03:06:31.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-05T03:06:31.314 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:31.756 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:31 smithi153 ceph-mon[29381]: pgmap v81: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:31.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:31 smithi203 ceph-mon[39911]: pgmap v81: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 -- 172.21.15.153:0/3678683851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 msgr2=0x7f65580fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 --2- 172.21.15.153:0/3678683851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f65580fbd80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6540004940 tx=0x7f6540009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 -- 172.21.15.153:0/3678683851 shutdown_connections 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 --2- 172.21.15.153:0/3678683851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f65580fbd80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 --2- 172.21.15.153:0/3678683851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 0x7f6558102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 -- 172.21.15.153:0/3678683851 >> 172.21.15.153:0/3678683851 conn(0x7f6558072bc0 msgr2=0x7f6558072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 -- 172.21.15.153:0/3678683851 shutdown_connections 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.911+0000 7f655dcf2700 1 -- 172.21.15.153:0/3678683851 wait complete. 2024-04-05T03:06:32.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 Processor -- start 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 -- start start 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 0x7f655810ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655810f740 con 0x7f6558102710 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f655dcf2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655810f880 con 0x7f6558101d70 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56796/0 (socket says 172.21.15.153:56796) 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 -- 172.21.15.153:0/2963930022 learned_addr learned my addr 172.21.15.153:0/2963930022 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f65577fe700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 0x7f655810ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 -- 172.21.15.153:0/2963930022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 msgr2=0x7f655810ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 0x7f655810ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.912+0000 7f654ffff700 1 -- 172.21.15.153:0/2963930022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540005040 con 0x7f6558102710 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f654ffff700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6540009180 tx=0x7f6540009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:32.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f654000e040 con 0x7f6558102710 2024-04-05T03:06:32.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6558117b70 con 0x7f6558102710 2024-04-05T03:06:32.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f654000bbc0 con 0x7f6558102710 2024-04-05T03:06:32.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f654001c3f0 con 0x7f6558102710 2024-04-05T03:06:32.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.913+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65581180d0 con 0x7f6558102710 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.914+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6558104240 con 0x7f6558102710 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.914+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f654000bd30 con 0x7f6558102710 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.915+0000 7f6554ff9700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 0x7f654406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.915+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f654008dec0 con 0x7f6558102710 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.915+0000 7f65577fe700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 0x7f654406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.915+0000 7f65577fe700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 0x7f654406fa50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6548007fd0 tx=0x7f6548009040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:32.919+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f6540057d70 con 0x7f6558102710 2024-04-05T03:06:33.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.116+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f6558065e10 con 0x7f6558102710 2024-04-05T03:06:33.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.116+0000 7f6554ff9700 1 -- 172.21.15.153:0/2963930022 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v46)=0 v46) v1 ==== 163+0+0 (secure 0 0 0) 0x7f654005b430 con 0x7f6558102710 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 msgr2=0x7f654406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 0x7f654406fa50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6548007fd0 tx=0x7f6548009040 comp rx=0 tx=0).stop 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 msgr2=0x7f655810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6540009180 tx=0x7f6540009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 shutdown_connections 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6558101d70 0x7f655810ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f654406d5a0 0x7f654406fa50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:33.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 --2- 172.21.15.153:0/2963930022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6558102710 0x7f655810f200 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:33.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 >> 172.21.15.153:0/2963930022 conn(0x7f6558072bc0 msgr2=0x7f65580fb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:33.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 shutdown_connections 2024-04-05T03:06:33.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:33.118+0000 7f655dcf2700 1 -- 172.21.15.153:0/2963930022 wait complete. 2024-04-05T03:06:33.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:33 smithi153 ceph-mon[29381]: pgmap v82: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:33.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-05T03:06:33.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:33 smithi203 ceph-mon[39911]: pgmap v82: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 -- 172.21.15.153:0/3906859931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 msgr2=0x7f9cb01020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 --2- 172.21.15.153:0/3906859931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb01020c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0004940 tx=0x7f9ca0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 -- 172.21.15.153:0/3906859931 shutdown_connections 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 --2- 172.21.15.153:0/3906859931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 0x7f9cb00fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 --2- 172.21.15.153:0/3906859931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb01020c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.606+0000 7f9cb512a700 1 -- 172.21.15.153:0/3906859931 >> 172.21.15.153:0/3906859931 conn(0x7f9cb00f7880 msgr2=0x7f9cb00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 -- 172.21.15.153:0/3906859931 shutdown_connections 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 -- 172.21.15.153:0/3906859931 wait complete. 2024-04-05T03:06:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 Processor -- start 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 -- start start 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 0x7f9cb0103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0104af0 con 0x7f9cb0101cf0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9cb512a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb01035d0 con 0x7f9cb0102690 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.607+0000 7f9caed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56814/0 (socket says 172.21.15.153:56814) 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 -- 172.21.15.153:0/2151695325 learned_addr learned my addr 172.21.15.153:0/2151695325 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9cae59c700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 0x7f9cb0103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 -- 172.21.15.153:0/2151695325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 msgr2=0x7f9cb0103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 0x7f9cb0103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 -- 172.21.15.153:0/2151695325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca0005040 con 0x7f9cb0101cf0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9caed9d700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009150 tx=0x7f9ca0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca000e040 con 0x7f9cb0101cf0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb0103820 con 0x7f9cb0101cf0 2024-04-05T03:06:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ca0013420 con 0x7f9cb0101cf0 2024-04-05T03:06:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca001c630 con 0x7f9cb0101cf0 2024-04-05T03:06:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.608+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb011a060 con 0x7f9cb0101cf0 2024-04-05T03:06:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.609+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb00fd440 con 0x7f9cb0101cf0 2024-04-05T03:06:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.610+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f9ca0013590 con 0x7f9cb0101cf0 2024-04-05T03:06:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.610+0000 7f9c977fe700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 0x7f9c9806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.610+0000 7f9cae59c700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 0x7f9c9806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.610+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f9ca008de50 con 0x7f9cb0101cf0 2024-04-05T03:06:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.611+0000 7f9cae59c700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 0x7f9c9806fa50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0104280 tx=0x7f9ca4009ee0 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:35.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.615+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f9ca0057d00 con 0x7f9cb0101cf0 2024-04-05T03:06:35.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:35 smithi203 ceph-mon[39911]: pgmap v83: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:35.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.810+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f9cb004c710 con 0x7f9cb0101cf0 2024-04-05T03:06:35.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:35 smithi153 ceph-mon[29381]: pgmap v83: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:35.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.810+0000 7f9c977fe700 1 -- 172.21.15.153:0/2151695325 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v46)=0 v46) v1 ==== 135+0+0 (secure 0 0 0) 0x7f9ca005b3c0 con 0x7f9cb0101cf0 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 msgr2=0x7f9c9806fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 0x7f9c9806fa50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0104280 tx=0x7f9ca4009ee0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 msgr2=0x7f9cb0102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009150 tx=0x7f9ca0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 shutdown_connections 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f9c9806d5a0 0x7f9c9806fa50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cb0101cf0 0x7f9cb0102b50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 --2- 172.21.15.153:0/2151695325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cb0102690 0x7f9cb0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:35.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 >> 172.21.15.153:0/2151695325 conn(0x7f9cb00f7880 msgr2=0x7f9cb0066310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:35.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 shutdown_connections 2024-04-05T03:06:35.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:35.812+0000 7f9cb512a700 1 -- 172.21.15.153:0/2151695325 wait complete. 2024-04-05T03:06:36.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-05T03:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:37.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:37 smithi203 ceph-mon[39911]: pgmap v84: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:37 smithi153 ceph-mon[29381]: pgmap v84: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:38.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2184691553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a4101d10 msgr2=0x7f55a41020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:38.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2184691553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a4101d10 0x7f55a41020e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5594004940 tx=0x7f5594009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:38.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2184691553 shutdown_connections 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2184691553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a41026b0 0x7f55a40fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2184691553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a4101d10 0x7f55a41020e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2184691553 >> 172.21.15.153:0/2184691553 conn(0x7f55a40f7880 msgr2=0x7f55a40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2184691553 shutdown_connections 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.326+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2184691553 wait complete. 2024-04-05T03:06:38.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 Processor -- start 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 -- start start 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 0x7f55a410efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a410fa20 con 0x7f55a41026b0 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55aa1e6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a410fb90 con 0x7f55a4101d10 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55a2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55a2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56828/0 (socket says 172.21.15.153:56828) 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.327+0000 7f55a2ffd700 1 -- 172.21.15.153:0/2722366272 learned_addr learned my addr 172.21.15.153:0/2722366272 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a37fe700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 0x7f55a410efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a2ffd700 1 -- 172.21.15.153:0/2722366272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 msgr2=0x7f55a410efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a2ffd700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 0x7f55a410efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a2ffd700 1 -- 172.21.15.153:0/2722366272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594005040 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a37fe700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 0x7f55a410efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55a2ffd700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f559800c800 tx=0x7f5598004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f559800a760 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f559800ad90 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a4117f10 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55980113a0 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.328+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a4118420 con 0x7f55a41026b0 2024-04-05T03:06:38.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.329+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a40fd460 con 0x7f55a41026b0 2024-04-05T03:06:38.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.329+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f559800a8c0 con 0x7f55a41026b0 2024-04-05T03:06:38.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.330+0000 7f558bfff700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 0x7f558c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:38.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.330+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f559808cae0 con 0x7f55a41026b0 2024-04-05T03:06:38.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.330+0000 7f55a37fe700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 0x7f558c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:38.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.330+0000 7f55a37fe700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 0x7f558c06fa50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5594004560 tx=0x7f5594009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:38.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.335+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f5598056a30 con 0x7f55a41026b0 2024-04-05T03:06:38.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.552+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}) v1 -- 0x7f55a404bed0 con 0x7f558c06d5a0 2024-04-05T03:06:38.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.557+0000 7f558bfff700 1 -- 172.21.15.153:0/2722366272 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f55a404bed0 con 0x7f558c06d5a0 2024-04-05T03:06:38.557 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:06:38.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 msgr2=0x7f558c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:38.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 0x7f558c06fa50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5594004560 tx=0x7f5594009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:38.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 msgr2=0x7f55a410f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:38.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f559800c800 tx=0x7f5598004470 comp rx=0 tx=0).stop 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 shutdown_connections 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101d10 0x7f55a410efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f558c06d5a0 0x7f558c06fa50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 --2- 172.21.15.153:0/2722366272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55a41026b0 0x7f55a410f4e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 >> 172.21.15.153:0/2722366272 conn(0x7f55a40f7880 msgr2=0x7f55a40f88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 shutdown_connections 2024-04-05T03:06:38.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:38.560+0000 7f55aa1e6700 1 -- 172.21.15.153:0/2722366272 wait complete. 2024-04-05T03:06:38.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:38 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:38 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:39.195 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T03:06:39.195 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:06:39.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-05T03:06:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:06:39.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: pgmap v85: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:39.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:39 smithi153 ceph-mon[29381]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:06:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: pgmap v85: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:39.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:39.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:39 smithi203 ceph-mon[39911]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0028525c15e19309c20289c609e4fb9d0b12dd06 2024-04-05T03:06:40.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:40 smithi153 ceph-mon[29381]: pgmap v86: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:40 smithi203 ceph-mon[39911]: pgmap v86: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:42.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 -- 172.21.15.153:0/3866797278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 msgr2=0x7fbf5c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3866797278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c0fce20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c004940 tx=0x7fbf4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:42.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 -- 172.21.15.153:0/3866797278 shutdown_connections 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3866797278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c0fce20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3866797278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 0x7fbf5c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 -- 172.21.15.153:0/3866797278 >> 172.21.15.153:0/3866797278 conn(0x7fbf5c0f8980 msgr2=0x7fbf5c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 -- 172.21.15.153:0/3866797278 shutdown_connections 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.313+0000 7fbf63acf700 1 -- 172.21.15.153:0/3866797278 wait complete. 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 Processor -- start 2024-04-05T03:06:42.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 -- start start 2024-04-05T03:06:42.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 0x7fbf5c113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c114740 con 0x7fbf5c1037b0 2024-04-05T03:06:42.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf63acf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c10db80 con 0x7fbf5c102e10 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf6106a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf6106a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55844/0 (socket says 172.21.15.153:55844) 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf6106a700 1 -- 172.21.15.153:0/3328630261 learned_addr learned my addr 172.21.15.153:0/3328630261 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.314+0000 7fbf6186b700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 0x7fbf5c113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf6106a700 1 -- 172.21.15.153:0/3328630261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 msgr2=0x7fbf5c113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf6106a700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 0x7fbf5c113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf6106a700 1 -- 172.21.15.153:0/3328630261 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c005040 con 0x7fbf5c1037b0 2024-04-05T03:06:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf6106a700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009100 tx=0x7fbf4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4c00e040 con 0x7fbf5c1037b0 2024-04-05T03:06:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf4c013420 con 0x7fbf5c1037b0 2024-04-05T03:06:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c10de00 con 0x7fbf5c1037b0 2024-04-05T03:06:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4c01c610 con 0x7fbf5c1037b0 2024-04-05T03:06:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.315+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c10e3c0 con 0x7fbf5c1037b0 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.316+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c1052e0 con 0x7fbf5c1037b0 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.317+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fbf4c013590 con 0x7fbf5c1037b0 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.318+0000 7fbf527fc700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 0x7fbf4806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.318+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbf4c08e830 con 0x7fbf5c1037b0 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.318+0000 7fbf6186b700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 0x7fbf4806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.319+0000 7fbf6186b700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 0x7fbf4806fa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009fd0 tx=0x7fbf58009e50 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.322+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fbf4c05c2b0 con 0x7fbf5c1037b0 2024-04-05T03:06:42.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.546+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf5c113880 con 0x7fbf4806d5a0 2024-04-05T03:06:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.548+0000 7fbf527fc700 1 -- 172.21.15.153:0/3328630261 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fbf5c113880 con 0x7fbf4806d5a0 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 msgr2=0x7fbf4806fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 0x7fbf4806fa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbf58009fd0 tx=0x7fbf58009e50 comp rx=0 tx=0).stop 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 msgr2=0x7fbf5c1140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009100 tx=0x7fbf4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 shutdown_connections 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf5c102e10 0x7fbf5c113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fbf4806d5a0 0x7fbf4806fa50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 --2- 172.21.15.153:0/3328630261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf5c1037b0 0x7fbf5c1140f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 >> 172.21.15.153:0/3328630261 conn(0x7fbf5c0f8980 msgr2=0x7fbf5c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:42.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 shutdown_connections 2024-04-05T03:06:42.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.550+0000 7fbf63acf700 1 -- 172.21.15.153:0/3328630261 wait complete. 2024-04-05T03:06:42.561 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:06:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 -- 172.21.15.153:0/1414403198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 msgr2=0x7f58dc106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 --2- 172.21.15.153:0/1414403198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc106f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f58d0004940 tx=0x7f58d0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 -- 172.21.15.153:0/1414403198 shutdown_connections 2024-04-05T03:06:42.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 --2- 172.21.15.153:0/1414403198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc106f40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 --2- 172.21.15.153:0/1414403198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 0x7f58dc0fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 -- 172.21.15.153:0/1414403198 >> 172.21.15.153:0/1414403198 conn(0x7f58dc072bc0 msgr2=0x7f58dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 -- 172.21.15.153:0/1414403198 shutdown_connections 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.665+0000 7f58e27cf700 1 -- 172.21.15.153:0/1414403198 wait complete. 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.666+0000 7f58e27cf700 1 Processor -- start 2024-04-05T03:06:42.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58e27cf700 1 -- start start 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58e27cf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 0x7f58dc113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58e27cf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58e27cf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1147a0 con 0x7f58dc0fe870 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58e27cf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc10db00 con 0x7f58dc0fded0 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58db7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.667+0000 7f58db7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55862/0 (socket says 172.21.15.153:55862) 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58db7fe700 1 -- 172.21.15.153:0/3923059288 learned_addr learned my addr 172.21.15.153:0/3923059288 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58dbfff700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 0x7f58dc113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58db7fe700 1 -- 172.21.15.153:0/3923059288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 msgr2=0x7f58dc113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58db7fe700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 0x7f58dc113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58db7fe700 1 -- 172.21.15.153:0/3923059288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d0005040 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58db7fe700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009380 tx=0x7f58d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d000e040 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58d0009af0 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc10dde0 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d001c3f0 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.668+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc10e2a0 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.669+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc065df0 con 0x7f58dc0fe870 2024-04-05T03:06:42.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.670+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f58d0013420 con 0x7f58dc0fe870 2024-04-05T03:06:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.670+0000 7f58d8ff9700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 0x7f58c406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.670+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f58d008ded0 con 0x7f58dc0fe870 2024-04-05T03:06:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.671+0000 7f58dbfff700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 0x7f58c406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:42.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.671+0000 7f58dbfff700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 0x7f58c406fa50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009bf0 tx=0x7f58cc0091a0 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:42.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.679+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f58d005b830 con 0x7f58dc0fe870 2024-04-05T03:06:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.900+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58dc109f80 con 0x7f58c406d5a0 2024-04-05T03:06:42.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.901+0000 7f58d8ff9700 1 -- 172.21.15.153:0/3923059288 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f58dc109f80 con 0x7f58c406d5a0 2024-04-05T03:06:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.903+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 msgr2=0x7f58c406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.903+0000 7f58e27cf700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 0x7f58c406fa50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009bf0 tx=0x7f58cc0091a0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 msgr2=0x7f58dc1140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009380 tx=0x7f58d0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 shutdown_connections 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58dc0fded0 0x7f58dc113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f58c406d5a0 0x7f58c406fa50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 --2- 172.21.15.153:0/3923059288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58dc0fe870 0x7f58dc1140c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 >> 172.21.15.153:0/3923059288 conn(0x7f58dc072bc0 msgr2=0x7f58dc103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 shutdown_connections 2024-04-05T03:06:42.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:42.904+0000 7f58e27cf700 1 -- 172.21.15.153:0/3923059288 wait complete. 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 -- 172.21.15.153:0/2212576940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fe830 msgr2=0x7fee40106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 --2- 172.21.15.153:0/2212576940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fe830 0x7fee40106f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fee30004940 tx=0x7fee30009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 -- 172.21.15.153:0/2212576940 shutdown_connections 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 --2- 172.21.15.153:0/2212576940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fe830 0x7fee40106f00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 --2- 172.21.15.153:0/2212576940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fde90 0x7fee400fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.011+0000 7fee47fee700 1 -- 172.21.15.153:0/2212576940 >> 172.21.15.153:0/2212576940 conn(0x7fee40072bc0 msgr2=0x7fee40072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 -- 172.21.15.153:0/2212576940 shutdown_connections 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 -- 172.21.15.153:0/2212576940 wait complete. 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 Processor -- start 2024-04-05T03:06:43.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 -- start start 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fe830 0x7fee400fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400fc820 con 0x7fee400fde90 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee47fee700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400fc960 con 0x7fee400fe830 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee45d8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee45d8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55876/0 (socket says 172.21.15.153:55876) 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.012+0000 7fee45d8a700 1 -- 172.21.15.153:0/2831118234 learned_addr learned my addr 172.21.15.153:0/2831118234 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee45589700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fe830 0x7fee400fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee45d8a700 1 -- 172.21.15.153:0/2831118234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fe830 msgr2=0x7fee400fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee45d8a700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fe830 0x7fee400fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee45d8a700 1 -- 172.21.15.153:0/2831118234 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee30005040 con 0x7fee400fde90 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee45d8a700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fee3c002a80 tx=0x7fee3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee3c007cc0 con 0x7fee400fde90 2024-04-05T03:06:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee40119e20 con 0x7fee400fde90 2024-04-05T03:06:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee3c00f040 con 0x7fee400fde90 2024-04-05T03:06:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee3c00dac0 con 0x7fee400fde90 2024-04-05T03:06:43.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.013+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4011a220 con 0x7fee400fde90 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.014+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee40065e30 con 0x7fee400fde90 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.015+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fee3c00dc20 con 0x7fee400fde90 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.015+0000 7fee367fc700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 0x7fee2c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.015+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fee3c08c7e0 con 0x7fee400fde90 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.015+0000 7fee45589700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 0x7fee2c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.016+0000 7fee45589700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 0x7fee2c06fa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee400fd690 tx=0x7fee30009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.020+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fee3c056610 con 0x7fee400fde90 2024-04-05T03:06:43.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.234+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fee400fd440 con 0x7fee2c06d5a0 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: pgmap v87: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: Upgrade: Target is version 19.0.0-2013-g0028525c (squid) 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca'] 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:43.234 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:43.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: Upgrade: Setting container_image for all mgr 2024-04-05T03:06:43.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:43.235 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:43 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T03:06:43.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.245+0000 7fee367fc700 1 -- 172.21.15.153:0/2831118234 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fee400fd440 con 0x7fee2c06d5a0 2024-04-05T03:06:43.245 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:06:43.245 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (75s) 56s ago 12m 14.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:06:43.245 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (13m) 56s ago 13m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:06:43.245 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (11m) 71s ago 11m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (61s) 56s ago 12m 56.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (6m) 56s ago 6m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (6m) 56s ago 6m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (6m) 71s ago 6m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (6m) 71s ago 6m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (2m) 56s ago 14m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (2m) 71s ago 11m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (14m) 56s ago 14m 55.7M 2048M 17.2.7-805-g6fda41ff a49113eacc58 0aed2ef4426f 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (11m) 71s ago 11m 46.4M 2048M 17.2.7-805-g6fda41ff a49113eacc58 ed5baecccb76 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (114s) 56s ago 12m 8530k - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (109s) 71s ago 11m 14.8M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (10m) 56s ago 10m 222M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (10m) 56s ago 10m 228M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (9m) 56s ago 9m 197M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:06:43.246 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (9m) 71s ago 9m 263M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:06:43.247 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (8m) 71s ago 8m 230M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:06:43.247 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (8m) 71s ago 8m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:06:43.247 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (92s) 71s ago 10m 50.9M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:06:43.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 msgr2=0x7fee2c06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 0x7fee2c06fa50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee400fd690 tx=0x7fee30009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 msgr2=0x7fee400fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fee3c002a80 tx=0x7fee3c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 shutdown_connections 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fee2c06d5a0 0x7fee2c06fa50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee400fde90 0x7fee400fbd50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 --2- 172.21.15.153:0/2831118234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee400fe830 0x7fee400fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 >> 172.21.15.153:0/2831118234 conn(0x7fee40072bc0 msgr2=0x7fee40103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 shutdown_connections 2024-04-05T03:06:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.248+0000 7fee47fee700 1 -- 172.21.15.153:0/2831118234 wait complete. 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 -- 172.21.15.153:0/4082075000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 msgr2=0x7f54640fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 --2- 172.21.15.153:0/4082075000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54640fbd80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5454004940 tx=0x7f5454009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 -- 172.21.15.153:0/4082075000 shutdown_connections 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 --2- 172.21.15.153:0/4082075000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54640fbd80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 --2- 172.21.15.153:0/4082075000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 -- 172.21.15.153:0/4082075000 >> 172.21.15.153:0/4082075000 conn(0x7f5464072bc0 msgr2=0x7f5464072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 -- 172.21.15.153:0/4082075000 shutdown_connections 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.350+0000 7f546beae700 1 -- 172.21.15.153:0/4082075000 wait complete. 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 Processor -- start 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 -- start start 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464114730 con 0x7f5464102710 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f546beae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546410db20 con 0x7f5464101d70 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55904/0 (socket says 172.21.15.153:55904) 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 -- 172.21.15.153:0/1190667343 learned_addr learned my addr 172.21.15.153:0/1190667343 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469c4a700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 -- 172.21.15.153:0/1190667343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 msgr2=0x7f5464113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469449700 1 -- 172.21.15.153:0/1190667343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454005040 con 0x7f5464102710 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.351+0000 7f5469c4a700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f5469449700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5454009100 tx=0x7f5454009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f545400e040 con 0x7f5464102710 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546410dda0 con 0x7f5464102710 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5454013420 con 0x7f5464102710 2024-04-05T03:06:43.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f545401c610 con 0x7f5464102710 2024-04-05T03:06:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.352+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546410e360 con 0x7f5464102710 2024-04-05T03:06:43.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.353+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54640fd4c0 con 0x7f5464102710 2024-04-05T03:06:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.353+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f5454013590 con 0x7f5464102710 2024-04-05T03:06:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.354+0000 7f545a7fc700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 0x7f545006fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.354+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f545408dd70 con 0x7f5464102710 2024-04-05T03:06:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.354+0000 7f5469c4a700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 0x7f545006fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.354+0000 7f5469c4a700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 0x7f545006fa50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54600041d0 tx=0x7f5460009040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.358+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f545405b7f0 con 0x7f5464102710 2024-04-05T03:06:43.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: pgmap v87: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:43.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: Upgrade: Target is version 19.0.0-2013-g0028525c (squid) 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca'] 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: Upgrade: Setting container_image for all mgr 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:43.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:43 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T03:06:43.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.654+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5464108e50 con 0x7f5464102710 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.655+0000 7f545a7fc700 1 -- 172.21.15.153:0/1190667343 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f545405b380 con 0x7f5464102710 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:06:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 12, 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:06:43.656 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 msgr2=0x7f545006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 0x7f545006fa50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54600041d0 tx=0x7f5460009040 comp rx=0 tx=0).stop 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 msgr2=0x7f54641140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5454009100 tx=0x7f5454009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 shutdown_connections 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5464101d70 0x7f5464113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f545006d5a0 0x7f545006fa50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 --2- 172.21.15.153:0/1190667343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5464102710 0x7f54641140e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 >> 172.21.15.153:0/1190667343 conn(0x7f5464072bc0 msgr2=0x7f5464100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:43.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 shutdown_connections 2024-04-05T03:06:43.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.657+0000 7f546beae700 1 -- 172.21.15.153:0/1190667343 wait complete. 2024-04-05T03:06:43.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 -- 172.21.15.153:0/3093618763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 msgr2=0x7fb6ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 --2- 172.21.15.153:0/3093618763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec0fbd80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0004940 tx=0x7fb6e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 -- 172.21.15.153:0/3093618763 shutdown_connections 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 --2- 172.21.15.153:0/3093618763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec0fbd80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 --2- 172.21.15.153:0/3093618763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.758+0000 7fb6f2950700 1 -- 172.21.15.153:0/3093618763 >> 172.21.15.153:0/3093618763 conn(0x7fb6ec072bc0 msgr2=0x7fb6ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 -- 172.21.15.153:0/3093618763 shutdown_connections 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 -- 172.21.15.153:0/3093618763 wait complete. 2024-04-05T03:06:43.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 Processor -- start 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 -- start start 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec10ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10f7a0 con 0x7fb6ec102710 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.759+0000 7fb6f2950700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec10f8e0 con 0x7fb6ec101d70 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55922/0 (socket says 172.21.15.153:55922) 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 -- 172.21.15.153:0/1784849480 learned_addr learned my addr 172.21.15.153:0/1784849480 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6ebfff700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec10ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 -- 172.21.15.153:0/1784849480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 msgr2=0x7fb6ec10ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec10ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 -- 172.21.15.153:0/1784849480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e0005040 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6ebfff700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec10ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6eb7fe700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009100 tx=0x7fb6e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e000e040 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec117ba0 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6e0004290 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6e001c3f0 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.760+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec118130 con 0x7fb6ec102710 2024-04-05T03:06:43.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.761+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ec0ff080 con 0x7fb6ec102710 2024-04-05T03:06:43.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.762+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fb6e0013790 con 0x7fb6ec102710 2024-04-05T03:06:43.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.762+0000 7fb6e8ff9700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 0x7fb6d406fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:43.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.762+0000 7fb6ebfff700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 0x7fb6d406fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:43.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.762+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb6e008de90 con 0x7fb6ec102710 2024-04-05T03:06:43.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.763+0000 7fb6ebfff700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 0x7fb6d406fa60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009b10 tx=0x7fb6dc008040 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:43.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:43.767+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fb6e0057d40 con 0x7fb6ec102710 2024-04-05T03:06:44.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.016+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb6ec108ec0 con 0x7fb6ec102710 2024-04-05T03:06:44.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.016+0000 7fb6e8ff9700 1 -- 172.21.15.153:0/1784849480 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1836 (secure 0 0 0) 0x7fb6e0013420 con 0x7fb6ec102710 2024-04-05T03:06:44.018 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:06:44.018 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:06:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:44.021 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:44.021 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:44.021 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:06:44.021 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:06:44.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 msgr2=0x7fb6d406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 0x7fb6d406fa60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009b10 tx=0x7fb6dc008040 comp rx=0 tx=0).stop 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 msgr2=0x7fb6ec10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009100 tx=0x7fb6e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 shutdown_connections 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6ec101d70 0x7fb6ec10ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fb6d406d5b0 0x7fb6d406fa60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 --2- 172.21.15.153:0/1784849480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6ec102710 0x7fb6ec10f1d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 >> 172.21.15.153:0/1784849480 conn(0x7fb6ec072bc0 msgr2=0x7fb6ec0faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 shutdown_connections 2024-04-05T03:06:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.019+0000 7fb6f2950700 1 -- 172.21.15.153:0/1784849480 wait complete. 2024-04-05T03:06:44.023 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:06:44.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- 172.21.15.153:0/4088219937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fd1d0 msgr2=0x7fa1cc0fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 --2- 172.21.15.153:0/4088219937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0fd5a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc004940 tx=0x7fa1bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:44.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- 172.21.15.153:0/4088219937 shutdown_connections 2024-04-05T03:06:44.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 --2- 172.21.15.153:0/4088219937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc0fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 --2- 172.21.15.153:0/4088219937 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0fd5a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- 172.21.15.153:0/4088219937 >> 172.21.15.153:0/4088219937 conn(0x7fa1cc0f89b0 msgr2=0x7fa1cc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- 172.21.15.153:0/4088219937 shutdown_connections 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- 172.21.15.153:0/4088219937 wait complete. 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 Processor -- start 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.126+0000 7fa1d2338700 1 -- start start 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1d2338700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1d2338700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1d2338700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc072210 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1d2338700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc070d40 con 0x7fa1cc0fd1d0 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55930/0 (socket says 172.21.15.153:55930) 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 -- 172.21.15.153:0/3460365577 learned_addr learned my addr 172.21.15.153:0/3460365577 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:44.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cbfff700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 -- 172.21.15.153:0/3460365577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fd1d0 msgr2=0x7fa1cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 -- 172.21.15.153:0/3460365577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc005040 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1cb7fe700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000c8c0 tx=0x7fa1c00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c000a760 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1c000ad90 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc071020 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.127+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c00113a0 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.128+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc11a0f0 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.128+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc0fe570 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.129+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7fa1c0012400 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.129+0000 7fa1c8ff9700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 0x7fa1b406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.130+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa1c008cbd0 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.130+0000 7fa1cbfff700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 0x7fa1b406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.130+0000 7fa1cbfff700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 0x7fa1b406fa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc004f90 tx=0x7fa1bc009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:44.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.134+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7fa1c0056a80 con 0x7fa1cc0fe9d0 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1190667343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:06:44.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:44.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:44 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/1784849480' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:06:44.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.359+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 --> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1cc05e400 con 0x7fa1b406d5a0 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.360+0000 7fa1c8ff9700 1 -- 172.21.15.153:0/3460365577 <== mgr.14706 v2:172.21.15.153:6800/2121051366 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fa1cc05e400 con 0x7fa1b406d5a0 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:06:44.360 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:06:44.361 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "2/21 daemons upgraded", 2024-04-05T03:06:44.361 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mon daemons", 2024-04-05T03:06:44.361 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:06:44.361 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.362+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 msgr2=0x7fa1b406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.362+0000 7fa1d2338700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 0x7fa1b406fa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc004f90 tx=0x7fa1bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.362+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 msgr2=0x7fa1cc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.362+0000 7fa1d2338700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000c8c0 tx=0x7fa1c00044c0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 shutdown_connections 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1cc0fd1d0 0x7fa1cc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7fa1b406d5a0 0x7fa1b406fa50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 --2- 172.21.15.153:0/3460365577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1cc0fe9d0 0x7fa1cc070800 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 >> 172.21.15.153:0/3460365577 conn(0x7fa1cc0f89b0 msgr2=0x7fa1cc0f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 shutdown_connections 2024-04-05T03:06:44.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.363+0000 7fa1d2338700 1 -- 172.21.15.153:0/3460365577 wait complete. 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 -- 172.21.15.153:0/1195265639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 msgr2=0x7f680c0fc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 --2- 172.21.15.153:0/1195265639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c0fc930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6800004940 tx=0x7f6800009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 -- 172.21.15.153:0/1195265639 shutdown_connections 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 --2- 172.21.15.153:0/1195265639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c0fc930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 --2- 172.21.15.153:0/1195265639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 -- 172.21.15.153:0/1195265639 >> 172.21.15.153:0/1195265639 conn(0x7f680c072310 msgr2=0x7f680c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 -- 172.21.15.153:0/1195265639 shutdown_connections 2024-04-05T03:06:44.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.465+0000 7f681100a700 1 -- 172.21.15.153:0/1195265639 wait complete. 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 Processor -- start 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 -- start start 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c10f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10f620 con 0x7f680c0fc4c0 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f681100a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10f760 con 0x7f680c101db0 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55960/0 (socket says 172.21.15.153:55960) 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 -- 172.21.15.153:0/3097483360 learned_addr learned my addr 172.21.15.153:0/3097483360 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680a59c700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c10f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 -- 172.21.15.153:0/3097483360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 msgr2=0x7f680c10f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c10f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 -- 172.21.15.153:0/3097483360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800005040 con 0x7f680c0fc4c0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680a59c700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f680ad9d700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c8c0 tx=0x7f67fc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.466+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67fc00a980 con 0x7f680c0fc4c0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.467+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67fc00ce60 con 0x7f680c0fc4c0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.467+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c117c10 con 0x7f680c0fc4c0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.467+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67fc0115e0 con 0x7f680c0fc4c0 2024-04-05T03:06:44.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.467+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c117f60 con 0x7f680c0fc4c0 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.468+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c0ff090 con 0x7f680c0fc4c0 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.468+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 33) v1 ==== 91575+0+0 (secure 0 0 0) 0x7f67fc012400 con 0x7f680c0fc4c0 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.468+0000 7f67f37fe700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 0x7f67f406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.469+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f67fc08cc80 con 0x7f680c0fc4c0 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.469+0000 7f680a59c700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 0x7f67f406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:06:44.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.469+0000 7f680a59c700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 0x7f67f406fa50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6800004f90 tx=0x7f6800009e30 comp rx=0 tx=0).ready entity=mgr.14706 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:06:44.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.473+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183140 (secure 0 0 0) 0x7f67fc056b30 con 0x7f680c0fc4c0 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1190667343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' 2024-04-05T03:06:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:06:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:06:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:06:44.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:44 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/1784849480' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:06:44.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.754+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f680c10f920 con 0x7f680c0fc4c0 2024-04-05T03:06:44.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.755+0000 7f67f37fe700 1 -- 172.21.15.153:0/3097483360 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f67fc0126b0 con 0x7f680c0fc4c0 2024-04-05T03:06:44.755 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 msgr2=0x7f67f406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 0x7f67f406fa50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6800004f90 tx=0x7f6800009e30 comp rx=0 tx=0).stop 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 msgr2=0x7f680c10eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c8c0 tx=0x7f67fc0044c0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 shutdown_connections 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:6800/2121051366,v1:172.21.15.153:6801/2121051366] conn(0x7f67f406d5a0 0x7f67f406fa50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f680c0fc4c0 0x7f680c10eba0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 --2- 172.21.15.153:0/3097483360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f680c101db0 0x7f680c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 >> 172.21.15.153:0/3097483360 conn(0x7f680c072310 msgr2=0x7f680c1062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 shutdown_connections 2024-04-05T03:06:44.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:06:44.757+0000 7f681100a700 1 -- 172.21.15.153:0/3097483360 wait complete. 2024-04-05T03:06:45.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-mon[29381]: pgmap v88: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:45.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-mon[29381]: Upgrade: Updating mon.smithi153 2024-04-05T03:06:45.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-mon[29381]: Deploying daemon mon.smithi153 on smithi153 2024-04-05T03:06:45.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-mon[29381]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:45.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-mon[29381]: from='client.? 172.21.15.153:0/3097483360' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:06:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:45 smithi203 ceph-mon[39911]: pgmap v88: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:45 smithi203 ceph-mon[39911]: Upgrade: Updating mon.smithi153 2024-04-05T03:06:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:45 smithi203 ceph-mon[39911]: Deploying daemon mon.smithi153 on smithi153 2024-04-05T03:06:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:45 smithi203 ceph-mon[39911]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:06:45.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:45 smithi203 ceph-mon[39911]: from='client.? 172.21.15.153:0/3097483360' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:06:45.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:06:46.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[29377]: 2024-04-05T03:06:45.968+0000 7f531e8d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:06:46.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:45 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[29377]: 2024-04-05T03:06:45.968+0000 7f531e8d2700 -1 mon.smithi153@0(leader) e2 *** Got Signal Terminated *** 2024-04-05T03:06:46.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 podman[105926]: 2024-04-05 03:06:46.000168517 +0000 UTC m=+0.154437521 container died 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.license=GPLv2, RELEASE=quincy-6fda41f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-05T03:06:46.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 podman[105926]: 2024-04-05 03:06:46.360661265 +0000 UTC m=+0.514930258 container cleanup 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-6fda41f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-05T03:06:46.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 bash[105926]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153 2024-04-05T03:06:46.982 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 podman[105940]: 2024-04-05 03:06:46.62033918 +0000 UTC m=+0.624231049 container remove 0aed2ef4426f8a693fbc7c679c5fa3145c3eb2e8323a4dbbc6eb9c8744aba78a (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6fda41f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:06:46.983 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service: Deactivated successfully. 2024-04-05T03:06:46.983 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:06:46.983 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service: Consumed 13.018s CPU time. 2024-04-05T03:06:47.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:46 smithi153 systemd[1]: Starting Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:06:47.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 podman[106083]: 2024-04-05 03:06:47.102770924 +0000 UTC m=+0.018066868 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:06:47.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 podman[106083]: 2024-04-05 03:06:47.229962597 +0000 UTC m=+0.145258541 container create 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-05T03:06:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 podman[106083]: 2024-04-05 03:06:47.449144441 +0000 UTC m=+0.364440387 container init 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-0028525) 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 podman[106083]: 2024-04-05 03:06:47.453076319 +0000 UTC m=+0.368372272 container start 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True) 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev), process ceph-mon, pid 2 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: pidfile_write: ignore empty --pid-file 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: load: jerasure load: lrc 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: RocksDB version: 7.9.2 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Git sha 0 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Compile date 2024-04-04 02:01:22 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: DB SUMMARY 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: DB Session ID: AQ5CR6UHEAM98XI0KSZI 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: CURRENT file: CURRENT 2024-04-05T03:06:47.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000033.sst 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000031.log size: 2839243 ; 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.error_if_exists: 0 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.create_if_missing: 0 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.paranoid_checks: 1 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.env: 0x55eb94817c60 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.info_log: 0x55eb95f179a0 2024-04-05T03:06:47.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.statistics: (nil) 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.use_fsync: 0 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_log_file_size: 0 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T03:06:47.826 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_fallocate: 1 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.use_direct_reads: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.db_log_dir: 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.wal_dir: 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T03:06:47.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.write_buffer_manager: 0x55eb95eea1e0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.unordered_write: 0 2024-04-05T03:06:47.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.row_cache: None 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.wal_filter: None 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.two_write_queues: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.wal_compression: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.atomic_flush: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T03:06:47.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.log_readahead_size: 0 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_background_jobs: 2 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_background_compactions: -1 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_subcompactions: 1 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T03:06:47.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_open_files: -1 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_background_flushes: -1 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Compression algorithms supported: 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kZSTD supported: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kXpressCompression supported: 0 2024-04-05T03:06:47.831 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kZlibCompression supported: 1 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kSnappyCompression supported: 1 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kLZ4Compression supported: 1 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: kBZip2Compression supported: 0 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000015 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.merge_operator: 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_filter: None 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T03:06:47.832 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb95f17aa0) 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 4 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x55eb95f1b090 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-04-05T03:06:47.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-04-05T03:06:47.834 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: bloomfilter 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 5 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_auto_readahead_size: 262144 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: prepopulate_block_cache: 0 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: initial_auto_readahead_size: 8192 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T03:06:47.835 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression: NoCompression 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.num_levels: 7 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T03:06:47.836 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T03:06:47.837 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T03:06:47.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T03:06:47.839 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.table_properties_collectors: 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.inplace_update_support: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.bloom_locality: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.max_successive_merges: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.ttl: 2592000 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T03:06:47.840 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enable_blob_files: false 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.min_blob_size: 0 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11917, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4b3f8851-23b6-48a7-8cef-115510c9f836 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286407483108, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-05T03:06:47.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286407493724, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 2497653, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11918, "largest_seqno": 14024, "table_properties": {"data_size": 2488333, "index_size": 5932, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2373, "raw_key_size": 22540, "raw_average_key_size": 23, "raw_value_size": 2468329, "raw_average_value_size": 2620, "num_data_blocks": 270, "num_entries": 942, "num_filter_entries": 942, "num_deletions": 9, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712286407, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4b3f8851-23b6-48a7-8cef-115510c9f836", "db_session_id": "AQ5CR6UHEAM98XI0KSZI", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286407493850, "job": 1, "event": "recovery_finished"} 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb9600c000 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: DB pointer 0x55eb95ff6000 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T03:06:47.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 1/0 2.38 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L6 1/0 6.47 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 8.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T03:06:47.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 593.0 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 155.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 155.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache BinnedLRUCache@0x55eb95f1b090#2 capacity: 512.00 MB usage: 8.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-05T03:06:47.844 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: starting mon.smithi153 rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???) e2 preinit fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).mds e25 new map 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).mds e25 print_map 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e25 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: 1 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Filesystem 'cephfs' (1) 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: fs_name cephfs 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: epoch 25 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:06:47.845 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: tableserver 0 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: root 0 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: session_timeout 60 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: session_autoclose 300 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_file_size 1099511627776 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_xattr_size 65536 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: required_client_features {} 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: last_failure 0 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: last_failure_osd_epoch 45 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: max_mds 2 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: in 0,1 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: up {0=14506,1=24261} 2024-04-05T03:06:47.846 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: failed 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: damaged 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: stopped 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_pools [3] 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_pool 2 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: inline_data disabled 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: balancer 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: bal_rank_mask -1 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: standby_count_wanted 1 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: qdb_cluster leader: 0 members: 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: [mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).osd e47 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-05T03:06:47.847 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:06:47.848 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:06:47.848 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).osd e47 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:06:47.848 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 ceph-mon[106102]: mon.smithi153@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-05T03:06:47.848 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 bash[106083]: 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 2024-04-05T03:06:47.848 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:47 smithi153 systemd[1]: Started Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: pgmap v90: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: pgmap v91: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: mon.smithi153 calling monitor election 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: monmap epoch 2 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: last_changed 2024-04-05T02:55:34.525318+0000 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: created 2024-04-05T02:51:49.024022+0000 2024-04-05T03:06:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: min_mon_release 17 (quincy) 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: election_strategy: 1 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.smithi203 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: mgrmap e33: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: overall HEALTH_OK 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: mgrmap e34: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:50.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:50 smithi153 ceph-mon[106102]: from='mgr.14706 ' entity='' 2024-04-05T03:06:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:06:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: from='mgr.14706 172.21.15.153:0/1100368724' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: pgmap v90: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: pgmap v91: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: mon.smithi153 calling monitor election 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: monmap epoch 2 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: last_changed 2024-04-05T02:55:34.525318+0000 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: created 2024-04-05T02:51:49.024022+0000 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: min_mon_release 17 (quincy) 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: election_strategy: 1 2024-04-05T03:06:51.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.smithi203 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: mgrmap e33: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: overall HEALTH_OK 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: mgrmap e34: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:51.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:50 smithi203 ceph-mon[39911]: from='mgr.14706 ' entity='' 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203[39881]: 2024-04-05T03:06:57.414+0000 7fa49c8d3700 -1 mon.smithi203@1(peon).osd e48 update_from_paxos full map CRC mismatch, resetting to canonical 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:06:57.604 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:57 smithi203 ceph-mon[39911]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: Standby manager daemon smithi203.iakodz restarted 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: Standby manager daemon smithi203.iakodz started 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/crt"}]: dispatch 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi203.iakodz/key"}]: dispatch 2024-04-05T03:06:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:57 smithi153 ceph-mon[106102]: from='mgr.? 172.21.15.203:0/4184558094' entity='mgr.smithi203.iakodz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: Activating manager daemon smithi153.npzyqy 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: mgrmap e35: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: failed to encode map e48 with expected crc 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:06:58.682 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: mgrmap e36: smithi153.npzyqy(active, starting, since 0.00822293s), standbys: smithi203.iakodz 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:06:58.683 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: Manager daemon smithi153.npzyqy is now available 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T03:06:58.684 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:58 smithi203 ceph-mon[39911]: failed to encode map e48 with expected crc 2024-04-05T03:06:58.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: Active manager daemon smithi153.npzyqy restarted 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: Activating manager daemon smithi153.npzyqy 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: mgrmap e35: smithi153.npzyqy(active, since 2m), standbys: smithi203.iakodz 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: failed to encode map e48 with expected crc 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: mgrmap e36: smithi153.npzyqy(active, starting, since 0.00822293s), standbys: smithi203.iakodz 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:06:58.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi153.npzyqy", "id": "smithi153.npzyqy"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr metadata", "who": "smithi203.iakodz", "id": "smithi203.iakodz"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: Manager daemon smithi153.npzyqy is now available 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:06:58.687 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/mirror_snapshot_schedule"}]: dispatch 2024-04-05T03:06:58.688 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.npzyqy/trash_purge_schedule"}]: dispatch 2024-04-05T03:06:58.688 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:58 smithi153 ceph-mon[106102]: failed to encode map e48 with expected crc 2024-04-05T03:06:59.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: [05/Apr/2024:03:06:57] ENGINE Bus STARTING 2024-04-05T03:06:59.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: [05/Apr/2024:03:06:57] ENGINE Serving on http://172.21.15.153:8765 2024-04-05T03:06:59.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: [05/Apr/2024:03:06:58] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T03:06:59.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: [05/Apr/2024:03:06:58] ENGINE Bus STARTED 2024-04-05T03:06:59.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: mgrmap e37: smithi153.npzyqy(active, since 1.01487s), standbys: smithi203.iakodz 2024-04-05T03:06:59.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:06:59 smithi203 ceph-mon[39911]: pgmap v3: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: [05/Apr/2024:03:06:57] ENGINE Bus STARTING 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: [05/Apr/2024:03:06:57] ENGINE Serving on http://172.21.15.153:8765 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: [05/Apr/2024:03:06:58] ENGINE Serving on https://172.21.15.153:7150 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: [05/Apr/2024:03:06:58] ENGINE Bus STARTED 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: mgrmap e37: smithi153.npzyqy(active, since 1.01487s), standbys: smithi203.iakodz 2024-04-05T03:06:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:06:59 smithi153 ceph-mon[106102]: pgmap v3: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:00 smithi153 ceph-mon[106102]: pgmap v4: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:00.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:00 smithi153 ceph-mon[106102]: mgrmap e38: smithi153.npzyqy(active, since 2s), standbys: smithi203.iakodz 2024-04-05T03:07:00.740 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:00 smithi203 ceph-mon[39911]: pgmap v4: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:00.740 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:00 smithi203 ceph-mon[39911]: mgrmap e38: smithi153.npzyqy(active, since 2s), standbys: smithi203.iakodz 2024-04-05T03:07:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: pgmap v5: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: mgrmap e39: smithi153.npzyqy(active, since 4s), standbys: smithi203.iakodz 2024-04-05T03:07:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:02 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: pgmap v5: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: mgrmap e39: smithi153.npzyqy(active, since 4s), standbys: smithi203.iakodz 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:02 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:03 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:03 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:03 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:03 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:03 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:03 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:03 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:03 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:04.763 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:04 smithi153 ceph-mon[106102]: pgmap v6: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:04 smithi203 ceph-mon[39911]: pgmap v6: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:07:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:06 smithi203 ceph-mon[39911]: pgmap v7: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-05T03:07:06.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:06 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:06.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:06 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:06.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:06 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:06.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:06 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:07:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:06 smithi153 ceph-mon[106102]: pgmap v7: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-05T03:07:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:07:07.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: Detected new or changed devices on smithi153 2024-04-05T03:07:07.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:07.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:07 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: Detected new or changed devices on smithi153 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:07.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:07.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:07.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:07 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T03:07:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:07:08.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:08 smithi203 ceph-mon[39911]: pgmap v8: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: Updating smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T03:07:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:08 smithi153 ceph-mon[106102]: pgmap v8: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-05T03:07:10.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: Upgrade: Updating mon.smithi203 2024-04-05T03:07:10.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:10.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:10.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:10.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:10.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: Deploying daemon mon.smithi203 on smithi203 2024-04-05T03:07:10.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 ceph-mon[39911]: pgmap v9: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-05T03:07:10.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: Upgrade: Updating mon.smithi203 2024-04-05T03:07:10.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:10.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:10.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:10.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:10.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: Deploying daemon mon.smithi203 on smithi203 2024-04-05T03:07:10.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:10 smithi153 ceph-mon[106102]: pgmap v9: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-05T03:07:11.231 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:10 smithi203 systemd[1]: Stopping Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:07:11.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203[39881]: 2024-04-05T03:07:11.230+0000 7fa4a20de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi203 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:07:11.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203[39881]: 2024-04-05T03:07:11.230+0000 7fa4a20de700 -1 mon.smithi203@1(peon) e2 *** Got Signal Terminated *** 2024-04-05T03:07:11.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:11 smithi203 podman[95823]: 2024-04-05 03:07:11.445553371 +0000 UTC m=+0.325509940 container died ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, GIT_CLEAN=True, RELEASE=quincy-6fda41f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-05T03:07:12.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:11 smithi203 podman[95823]: 2024-04-05 03:07:11.823210426 +0000 UTC m=+0.703167003 container cleanup ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-6fda41f, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-05T03:07:12.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:11 smithi203 bash[95823]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203 2024-04-05T03:07:12.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 podman[95837]: 2024-04-05 03:07:12.241811912 +0000 UTC m=+0.793652516 container remove ed5baecccb76d7cf7b5ff1d2692cc5430a7fcfc04ccb8dfedd46a37c32d34cd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, RELEASE=quincy-6fda41f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-05T03:07:12.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi203.service: Deactivated successfully. 2024-04-05T03:07:12.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 systemd[1]: Stopped Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:07:12.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi203.service: Consumed 6.942s CPU time. 2024-04-05T03:07:12.834 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 systemd[1]: Starting Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:07:12.834 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 podman[95969]: 2024-04-05 03:07:12.70029118 +0000 UTC m=+0.018406136 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:07:12.834 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:12 smithi203 podman[95969]: 2024-04-05 03:07:12.835207613 +0000 UTC m=+0.153322615 container create 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-0028525, CEPH_POINT_RELEASE=) 2024-04-05T03:07:13.089 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 podman[95969]: 2024-04-05 03:07:13.063213861 +0000 UTC m=+0.381328794 container init 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-05T03:07:13.089 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 podman[95969]: 2024-04-05 03:07:13.066846577 +0000 UTC m=+0.384961518 container start 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-0028525, GIT_CLEAN=True) 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev), process ceph-mon, pid 2 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: pidfile_write: ignore empty --pid-file 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: load: jerasure load: lrc 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: RocksDB version: 7.9.2 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Git sha 0 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Compile date 2024-04-04 02:01:22 2024-04-05T03:07:13.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: DB SUMMARY 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: DB Session ID: YZNRC8G8PVNEQ2UHAY3I 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: CURRENT file: CURRENT 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi203/store.db dir, Total Num: 1, files: 000027.sst 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi203/store.db: 000025.log size: 7107616 ; 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.error_if_exists: 0 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.create_if_missing: 0 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.paranoid_checks: 1 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.env: 0x562f7e7d4c60 2024-04-05T03:07:13.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.info_log: 0x562f7f6cf9a0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.statistics: (nil) 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.use_fsync: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_log_file_size: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_fallocate: 1 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.use_direct_reads: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T03:07:13.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.db_log_dir: 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.wal_dir: 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.write_buffer_manager: 0x562f7f6a21e0 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T03:07:13.550 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.unordered_write: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.row_cache: None 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.wal_filter: None 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.two_write_queues: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T03:07:13.551 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.wal_compression: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.atomic_flush: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.log_readahead_size: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T03:07:13.552 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_background_jobs: 2 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_background_compactions: -1 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_subcompactions: 1 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T03:07:13.554 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_open_files: -1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_background_flushes: -1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Compression algorithms supported: 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kZSTD supported: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kXpressCompression supported: 0 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kZlibCompression supported: 1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kSnappyCompression supported: 1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kLZ4Compression supported: 1 2024-04-05T03:07:13.555 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: kBZip2Compression supported: 0 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi203/store.db/MANIFEST-000009 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.merge_operator: 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_filter: None 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f7f6cfaa0) 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: cache_index_and_filter_blocks: 1 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T03:07:13.556 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: pin_top_level_index_and_filter: 1 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_type: 0 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: data_block_index_type: 0 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_shortening: 1 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: checksum: 4 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: no_block_cache: 0 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache: 0x562f7f6d3090 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_name: BinnedLRUCache 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_options: 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: capacity : 536870912 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: num_shard_bits : 4 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: strict_capacity_limit : 0 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: high_pri_pool_ratio: 0.000 2024-04-05T03:07:13.557 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_cache_compressed: (nil) 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: persistent_cache: (nil) 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_size: 4096 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_size_deviation: 10 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_restart_interval: 16 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: index_block_restart_interval: 1 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: metadata_block_size: 4096 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: partition_filters: 0 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: use_delta_encoding: 1 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: filter_policy: bloomfilter 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: whole_key_filtering: 1 2024-04-05T03:07:13.558 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: verify_compression: 0 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: read_amp_bytes_per_bit: 0 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: format_version: 5 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: enable_index_compression: 1 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: block_align: 0 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: max_auto_readahead_size: 262144 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: prepopulate_block_cache: 0 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: initial_auto_readahead_size: 8192 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression: NoCompression 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T03:07:13.559 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.num_levels: 7 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T03:07:13.560 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T03:07:13.561 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T03:07:13.562 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.table_properties_collectors: 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.inplace_update_support: 0 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T03:07:13.563 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.bloom_locality: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.max_successive_merges: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.ttl: 2592000 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enable_blob_files: false 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.min_blob_size: 0 2024-04-05T03:07:13.564 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi203/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 11855, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4b8f1a6e-e6dd-47f6-9779-a4d981dc2f51 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286433097633, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-05T03:07:13.565 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286433115574, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4297735, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11860, "largest_seqno": 14927, "table_properties": {"data_size": 4285740, "index_size": 7454, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3525, "raw_key_size": 31778, "raw_average_key_size": 22, "raw_value_size": 4257938, "raw_average_value_size": 3034, "num_data_blocks": 336, "num_entries": 1403, "num_filter_entries": 1403, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712286433, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4b8f1a6e-e6dd-47f6-9779-a4d981dc2f51", "db_session_id": "YZNRC8G8PVNEQ2UHAY3I", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712286433115705, "job": 1, "event": "recovery_finished"} 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi203/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f7f7c4000 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: DB pointer 0x562f7f7ae000 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** DB Stats ** 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T03:07:13.566 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: L0 1/0 4.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 680.2 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: L6 1/0 6.47 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Sum 2/0 10.57 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 680.2 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 680.2 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** Compaction Stats [default] ** 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T03:07:13.567 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 680.2 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Cumulative compaction: 0.00 GB write, 179.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Interval compaction: 0.00 GB write, 179.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Block cache BinnedLRUCache@0x562f7f6d3090#2 capacity: 512.00 MB usage: 12.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.568 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: starting mon.smithi203 rank 1 at public addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] at bind addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi203 fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???) e2 preinit fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).mds e25 new map 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).mds e25 print_map 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: e25 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: legacy client fscid: 1 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: Filesystem 'cephfs' (1) 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: fs_name cephfs 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: epoch 25 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:07:13.569 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: tableserver 0 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: root 0 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: session_timeout 60 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: session_autoclose 300 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: max_file_size 1099511627776 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: max_xattr_size 65536 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: required_client_features {} 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: last_failure 0 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: last_failure_osd_epoch 45 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: max_mds 2 2024-04-05T03:07:13.570 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: in 0,1 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: up {0=14506,1=24261} 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: failed 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: damaged 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: stopped 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: data_pools [3] 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: metadata_pool 2 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: inline_data disabled 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: balancer 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: bal_rank_mask -1 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: standby_count_wanted 1 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: qdb_cluster leader: 0 members: 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: [mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: [mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: [mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: [mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:13.571 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout: 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 ceph-mon[95988]: mon.smithi203@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 bash[95969]: 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 2024-04-05T03:07:13.572 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:13 smithi203 systemd[1]: Started Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 -- 172.21.15.153:0/785436740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c101d70 msgr2=0x7f425c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 --2- 172.21.15.153:0/785436740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c101d70 0x7f425c102140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f424c004940 tx=0x7f424c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 -- 172.21.15.153:0/785436740 shutdown_connections 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 --2- 172.21.15.153:0/785436740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c102710 0x7f425c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 --2- 172.21.15.153:0/785436740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c101d70 0x7f425c102140 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:14.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 -- 172.21.15.153:0/785436740 >> 172.21.15.153:0/785436740 conn(0x7f425c072bc0 msgr2=0x7f425c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 -- 172.21.15.153:0/785436740 shutdown_connections 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.871+0000 7f42618d1700 1 -- 172.21.15.153:0/785436740 wait complete. 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 Processor -- start 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 -- start start 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c10f740 con 0x7f425c102710 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f42618d1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c10f8b0 con 0x7f425c101d70 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425affd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425affd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:58922/0 (socket says 172.21.15.153:58922) 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60542/0 (socket says 172.21.15.153:60542) 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425affd700 1 -- 172.21.15.153:0/2509651095 learned_addr learned my addr 172.21.15.153:0/2509651095 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.872+0000 7f425affd700 1 -- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 msgr2=0x7f425c10ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T03:07:14.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425affd700 1 -- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 msgr2=0x7f425c10ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425affd700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425affd700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425a7fc700 1 -- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 msgr2=0x7f425c10ecc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425a7fc700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425a7fc700 1 -- 172.21.15.153:0/2509651095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c005040 con 0x7f425c102710 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f425a7fc700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4250009ce0 tx=0x7f4250004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4250007cc0 con 0x7f425c102710 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c117bd0 con 0x7f425c102710 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f425000f040 con 0x7f425c102710 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425000dae0 con 0x7f425c102710 2024-04-05T03:07:14.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.873+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c1180e0 con 0x7f425c102710 2024-04-05T03:07:14.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.874+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c0ff080 con 0x7f425c102710 2024-04-05T03:07:14.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.875+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f425000dd50 con 0x7f425c102710 2024-04-05T03:07:14.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.875+0000 7f42437fe700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 0x7f4244070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:14.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.875+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f425008cdf0 con 0x7f425c102710 2024-04-05T03:07:14.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.875+0000 7f425affd700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 0x7f4244070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:14.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.876+0000 7f425affd700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 0x7f4244070270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f424c000c00 tx=0x7f424c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:14.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:14.880+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4250056200 con 0x7f425c102710 2024-04-05T03:07:15.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.100+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f425c108e90 con 0x7f424406ddc0 2024-04-05T03:07:15.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.101+0000 7f42437fe700 1 -- 172.21.15.153:0/2509651095 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f425c108e90 con 0x7f424406ddc0 2024-04-05T03:07:15.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 msgr2=0x7f4244070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 0x7f4244070270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f424c000c00 tx=0x7f424c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 msgr2=0x7f425c10f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4250009ce0 tx=0x7f4250004600 comp rx=0 tx=0).stop 2024-04-05T03:07:15.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 shutdown_connections 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f424406ddc0 0x7f4244070270 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f425c101d70 0x7f425c10ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 --2- 172.21.15.153:0/2509651095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f425c102710 0x7f425c10f200 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.103+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 >> 172.21.15.153:0/2509651095 conn(0x7f425c072bc0 msgr2=0x7f425c0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.104+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 shutdown_connections 2024-04-05T03:07:15.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.104+0000 7f42618d1700 1 -- 172.21.15.153:0/2509651095 wait complete. 2024-04-05T03:07:15.114 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 -- 172.21.15.153:0/3776607764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c01026a0 msgr2=0x7ff3c00fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 --2- 172.21.15.153:0/3776607764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c01026a0 0x7ff3c00fbd10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4004940 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 -- 172.21.15.153:0/3776607764 shutdown_connections 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 --2- 172.21.15.153:0/3776607764 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c01026a0 0x7ff3c00fbd10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 --2- 172.21.15.153:0/3776607764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0101d00 0x7ff3c01020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.209+0000 7ff3c5377700 1 -- 172.21.15.153:0/3776607764 >> 172.21.15.153:0/3776607764 conn(0x7ff3c00f7850 msgr2=0x7ff3c00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.210+0000 7ff3c5377700 1 -- 172.21.15.153:0/3776607764 shutdown_connections 2024-04-05T03:07:15.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.210+0000 7ff3c5377700 1 -- 172.21.15.153:0/3776607764 wait complete. 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.210+0000 7ff3c5377700 1 Processor -- start 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.210+0000 7ff3c5377700 1 -- start start 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3c5377700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3c5377700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3c5377700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0113420 con 0x7ff3c0101d00 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3c5377700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c010ca20 con 0x7ff3c01026a0 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60568/0 (socket says 172.21.15.153:60568) 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 -- 172.21.15.153:0/1044233012 learned_addr learned my addr 172.21.15.153:0/1044233012 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3be7fc700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3be7fc700 1 -- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 msgr2=0x7ff3c0112dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3be7fc700 1 -- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 msgr2=0x7ff3c0112dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:15.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3be7fc700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3be7fc700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 -- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 msgr2=0x7ff3c0112dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 -- 172.21.15.153:0/1044233012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4005040 con 0x7ff3c0101d00 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3beffd700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0002a50 tx=0x7ff3b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b0007cc0 con 0x7ff3c0101d00 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3b000f040 con 0x7ff3c0101d00 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c010cd00 con 0x7ff3c0101d00 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.211+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b000db70 con 0x7ff3c0101d00 2024-04-05T03:07:15.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.212+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c010d2c0 con 0x7ff3c0101d00 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.213+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c00fd450 con 0x7ff3c0101d00 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.213+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7ff3b000dd10 con 0x7ff3c0101d00 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.213+0000 7ff3a77fe700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 0x7ff3a8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.214+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff3b008cd80 con 0x7ff3c0101d00 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.214+0000 7ff3be7fc700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 0x7ff3a8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.214+0000 7ff3be7fc700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 0x7ff3a8070280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff3c010dbe0 tx=0x7ff3b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.218+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff3b0059d40 con 0x7ff3c0101d00 2024-04-05T03:07:15.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.443+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c0108e00 con 0x7ff3a806ddd0 2024-04-05T03:07:15.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.444+0000 7ff3a77fe700 1 -- 172.21.15.153:0/1044233012 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ff3c0108e00 con 0x7ff3a806ddd0 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 msgr2=0x7ff3a8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 0x7ff3a8070280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff3c010dbe0 tx=0x7ff3b4009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 msgr2=0x7ff3c0112890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0002a50 tx=0x7ff3b00046b0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 shutdown_connections 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3a806ddd0 0x7ff3a8070280 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c0101d00 0x7ff3c0112890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 --2- 172.21.15.153:0/1044233012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c01026a0 0x7ff3c0112dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 >> 172.21.15.153:0/1044233012 conn(0x7ff3c00f7850 msgr2=0x7ff3c00f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 shutdown_connections 2024-04-05T03:07:15.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.446+0000 7ff3c5377700 1 -- 172.21.15.153:0/1044233012 wait complete. 2024-04-05T03:07:15.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 -- 172.21.15.153:0/2948269075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 msgr2=0x7fb0b81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2948269075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b81031f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac004940 tx=0x7fb0ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 -- 172.21.15.153:0/2948269075 shutdown_connections 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2948269075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2948269075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b81031f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 -- 172.21.15.153:0/2948269075 >> 172.21.15.153:0/2948269075 conn(0x7fb0b80f89d0 msgr2=0x7fb0b80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 -- 172.21.15.153:0/2948269075 shutdown_connections 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.555+0000 7fb0bf118700 1 -- 172.21.15.153:0/2948269075 wait complete. 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 Processor -- start 2024-04-05T03:07:15.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 -- start start 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8114720 con 0x7fb0b8102e20 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bf118700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b810db30 con 0x7fb0b81037c0 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bceb4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bceb4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60582/0 (socket says 172.21.15.153:60582) 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.556+0000 7fb0bceb4700 1 -- 172.21.15.153:0/2671411643 learned_addr learned my addr 172.21.15.153:0/2671411643 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:15.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b7fff700 1 -- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 msgr2=0x7fb0b81140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b7fff700 1 -- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 msgr2=0x7fb0b81140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bceb4700 1 -- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 msgr2=0x7fb0b81140d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bceb4700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bceb4700 1 -- 172.21.15.153:0/2671411643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac005040 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bceb4700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009150 tx=0x7fb0ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0ac00e040 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0ac013420 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b810ddb0 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.557+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b810e370 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.558+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0ac01c630 con 0x7fb0b8102e20 2024-04-05T03:07:15.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.558+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b8100130 con 0x7fb0b8102e20 2024-04-05T03:07:15.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.559+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fb0ac013590 con 0x7fb0b8102e20 2024-04-05T03:07:15.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.559+0000 7fb0b57fa700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 0x7fb0a8070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.560+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb0ac08e9c0 con 0x7fb0b8102e20 2024-04-05T03:07:15.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.560+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 0x7fb0a8070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.560+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 0x7fb0a8070270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0b810f430 tx=0x7fb0a4009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.568+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb0ac057e50 con 0x7fb0b8102e20 2024-04-05T03:07:15.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.779+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0b810f250 con 0x7fb0a806ddc0 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: mgrmap e39: smithi153.npzyqy(active, since 17s), standbys: smithi203.iakodz 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: overall HEALTH_OK 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: mon.smithi203 calling monitor election 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: mon.smithi153 calling monitor election 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: monmap epoch 3 2024-04-05T03:07:15.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: last_changed 2024-04-05T03:07:14.496851+0000 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: created 2024-04-05T02:51:49.024022+0000 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: min_mon_release 19 (squid) 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: election_strategy: 1 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.smithi203 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: mgrmap e39: smithi153.npzyqy(active, since 17s), standbys: smithi203.iakodz 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: overall HEALTH_OK 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:15 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.789+0000 7fb0b57fa700 1 -- 172.21.15.153:0/2671411643 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fb0b810f250 con 0x7fb0a806ddc0 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (108s) 13s ago 13m 26.6M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (14m) 13s ago 14m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (11m) 14s ago 11m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (93s) 13s ago 12m 74.5M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (7m) 13s ago 7m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (7m) 13s ago 7m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (7m) 14s ago 7m 61.1M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (7m) 14s ago 7m 112M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (3m) 13s ago 15m 473M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (2m) 14s ago 11m 446M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:07:15.789 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (28s) 13s ago 15m 27.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 starting - - - 2048M 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (2m) 13s ago 13m 16.9M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (2m) 14s ago 11m 17.2M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (11m) 13s ago 11m 224M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (10m) 13s ago 10m 229M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (10m) 13s ago 10m 199M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (9m) 14s ago 9m 264M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (9m) 14s ago 9m 231M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (8m) 14s ago 8m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:07:15.790 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (2m) 14s ago 11m 57.7M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:07:15.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 msgr2=0x7fb0a8070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 0x7fb0a8070270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0b810f430 tx=0x7fb0a4009f70 comp rx=0 tx=0).stop 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 msgr2=0x7fb0b8113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009150 tx=0x7fb0ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 shutdown_connections 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb0a806ddc0 0x7fb0a8070270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0b8102e20 0x7fb0b8113b90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 --2- 172.21.15.153:0/2671411643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0b81037c0 0x7fb0b81140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 >> 172.21.15.153:0/2671411643 conn(0x7fb0b80f89d0 msgr2=0x7fb0b8107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 shutdown_connections 2024-04-05T03:07:15.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.792+0000 7fb0bf118700 1 -- 172.21.15.153:0/2671411643 wait complete. 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: mgrmap e39: smithi153.npzyqy(active, since 17s), standbys: smithi203.iakodz 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: overall HEALTH_OK 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-04-05T03:07:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mon metadata", "id": "smithi203"}]: dispatch 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: mon.smithi203 calling monitor election 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: mon.smithi153 calling monitor election 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: mon.smithi153 is new leader, mons smithi153,smithi203 in quorum (ranks 0,1) 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: monmap epoch 3 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: last_changed 2024-04-05T03:07:14.496851+0000 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: created 2024-04-05T02:51:49.024022+0000 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: min_mon_release 19 (squid) 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: election_strategy: 1 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.smithi153 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.smithi203 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2 up:standby-replay 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: mgrmap e39: smithi153.npzyqy(active, since 17s), standbys: smithi203.iakodz 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: overall HEALTH_OK 2024-04-05T03:07:15.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:15.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:15.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:15 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 -- 172.21.15.153:0/3381155631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 msgr2=0x7ff878106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 --2- 172.21.15.153:0/3381155631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878106c60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff86c004940 tx=0x7ff86c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 -- 172.21.15.153:0/3381155631 shutdown_connections 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 --2- 172.21.15.153:0/3381155631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878106c60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 --2- 172.21.15.153:0/3381155631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff8780fdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 -- 172.21.15.153:0/3381155631 >> 172.21.15.153:0/3381155631 conn(0x7ff8780f7690 msgr2=0x7ff8780f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 -- 172.21.15.153:0/3381155631 shutdown_connections 2024-04-05T03:07:15.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.894+0000 7ff87c803700 1 -- 172.21.15.153:0/3381155631 wait complete. 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 Processor -- start 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 -- start start 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff878114510 con 0x7ff8780fe500 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87c803700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87810d900 con 0x7ff8780fdbf0 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60606/0 (socket says 172.21.15.153:60606) 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87659c700 1 -- 172.21.15.153:0/1130289215 learned_addr learned my addr 172.21.15.153:0/1130289215 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff876d9d700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff876d9d700 1 -- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 msgr2=0x7ff878113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff876d9d700 1 -- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 msgr2=0x7ff878113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff876d9d700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff876d9d700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.895+0000 7ff87659c700 1 -- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 msgr2=0x7ff878113980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff87659c700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff87659c700 1 -- 172.21.15.153:0/1130289215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c005040 con 0x7ff8780fe500 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff87659c700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009380 tx=0x7ff86c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff86c00e040 con 0x7ff8780fe500 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87810db80 con 0x7ff8780fe500 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff86c009af0 con 0x7ff8780fe500 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff86c01c630 con 0x7ff8780fe500 2024-04-05T03:07:15.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.896+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87810e0c0 con 0x7ff8780fe500 2024-04-05T03:07:15.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.897+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7ff86c00bbc0 con 0x7ff8780fe500 2024-04-05T03:07:15.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.897+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8780666d0 con 0x7ff8780fe500 2024-04-05T03:07:15.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.898+0000 7ff85f7fe700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 0x7ff860070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:15.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.898+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff86c08ea70 con 0x7ff8780fe500 2024-04-05T03:07:15.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.898+0000 7ff876d9d700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 0x7ff860070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:15.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.898+0000 7ff876d9d700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 0x7ff860070270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff868009e50 tx=0x7ff868009f10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:15.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:15.903+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff86c05ba30 con 0x7ff8780fe500 2024-04-05T03:07:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.209+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff878109c60 con 0x7ff8780fe500 2024-04-05T03:07:16.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.210+0000 7ff85f7fe700 1 -- 172.21.15.153:0/1130289215 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7ff86c05b5c0 con 0x7ff8780fe500 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:16.210 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:07:16.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 10, 2024-04-05T03:07:16.211 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:07:16.211 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:07:16.211 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 msgr2=0x7ff860070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 0x7ff860070270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff868009e50 tx=0x7ff868009f10 comp rx=0 tx=0).stop 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 msgr2=0x7ff878113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009380 tx=0x7ff86c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 shutdown_connections 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff86006ddc0 0x7ff860070270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8780fdbf0 0x7ff878113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 --2- 172.21.15.153:0/1130289215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8780fe500 0x7ff878113ec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 >> 172.21.15.153:0/1130289215 conn(0x7ff8780f7690 msgr2=0x7ff8780f8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 shutdown_connections 2024-04-05T03:07:16.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.212+0000 7ff87c803700 1 -- 172.21.15.153:0/1130289215 wait complete. 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2596103566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c101530 msgr2=0x7f8d2c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2596103566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c101530 0x7f8d2c101900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c004940 tx=0x7f8d1c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2596103566 shutdown_connections 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2596103566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c0fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2596103566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c101530 0x7f8d2c101900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2596103566 >> 172.21.15.153:0/2596103566 conn(0x7f8d2c0714c0 msgr2=0x7f8d2c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2596103566 shutdown_connections 2024-04-05T03:07:16.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.313+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2596103566 wait complete. 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 Processor -- start 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 -- start start 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 0x7f8d2c113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c114500 con 0x7f8d2c0fb570 2024-04-05T03:07:16.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.314+0000 7f8d31b9d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c10db20 con 0x7f8d2c101530 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60620/0 (socket says 172.21.15.153:60620) 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 -- 172.21.15.153:0/2946005781 learned_addr learned my addr 172.21.15.153:0/2946005781 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2affd700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 0x7f8d2c113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 -- 172.21.15.153:0/2946005781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 msgr2=0x7f8d2c113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 0x7f8d2c113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 -- 172.21.15.153:0/2946005781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c005040 con 0x7f8d2c0fb570 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2affd700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 0x7f8d2c113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d2b7fe700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00b3b0 tx=0x7f8d1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1c00e040 con 0x7f8d2c0fb570 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d1c00bbc0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c10dda0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1c01c3f0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.315+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c10e360 con 0x7f8d2c0fb570 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.316+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c1039e0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.317+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f8d1c00bd30 con 0x7f8d2c0fb570 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.317+0000 7f8d13fff700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 0x7f8d14070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.317+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8d1c08ea00 con 0x7f8d2c0fb570 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.317+0000 7f8d2affd700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 0x7f8d14070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.318+0000 7f8d2affd700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 0x7f8d14070280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000afd0 tx=0x7f8d20009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:16.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.322+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8d1c057e10 con 0x7f8d2c0fb570 2024-04-05T03:07:16.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.563+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8d2c1039e0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.563+0000 7f8d13fff700 1 -- 172.21.15.153:0/2946005781 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7f8d1c05b4d0 con 0x7f8d2c0fb570 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:07:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:07:16.564 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 msgr2=0x7f8d14070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 0x7f8d14070280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000afd0 tx=0x7f8d20009040 comp rx=0 tx=0).stop 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 msgr2=0x7f8d2c113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00b3b0 tx=0x7f8d1c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 shutdown_connections 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8d1406ddd0 0x7f8d14070280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8d2c0fb570 0x7f8d2c113970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 --2- 172.21.15.153:0/2946005781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d2c101530 0x7f8d2c113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 >> 172.21.15.153:0/2946005781 conn(0x7f8d2c0714c0 msgr2=0x7f8d2c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 shutdown_connections 2024-04-05T03:07:16.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.565+0000 7f8d31b9d700 1 -- 172.21.15.153:0/2946005781 wait complete. 2024-04-05T03:07:16.567 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- 172.21.15.153:0/1270287141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 msgr2=0x7f635c0fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 --2- 172.21.15.153:0/1270287141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c0fbcf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6350004940 tx=0x7f6350009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- 172.21.15.153:0/1270287141 shutdown_connections 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 --2- 172.21.15.153:0/1270287141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c0fbcf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 --2- 172.21.15.153:0/1270287141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- 172.21.15.153:0/1270287141 >> 172.21.15.153:0/1270287141 conn(0x7f635c0f7870 msgr2=0x7f635c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- 172.21.15.153:0/1270287141 shutdown_connections 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- 172.21.15.153:0/1270287141 wait complete. 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 Processor -- start 2024-04-05T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.667+0000 7f6362223700 1 -- start start 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f6362223700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f6362223700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f6362223700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c10f550 con 0x7f635c102680 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f6362223700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c10f690 con 0x7f635c101ce0 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60642/0 (socket says 172.21.15.153:60642) 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 -- 172.21.15.153:0/1883424969 learned_addr learned my addr 172.21.15.153:0/1883424969 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635b7fe700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635b7fe700 1 -- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 msgr2=0x7f635c10ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635b7fe700 1 -- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 msgr2=0x7f635c10ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:16.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635b7fe700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635b7fe700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 -- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 msgr2=0x7f635c10ead0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 -- 172.21.15.153:0/1883424969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350005040 con 0x7f635c102680 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.668+0000 7f635affd700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6350009100 tx=0x7f6350009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.669+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635000e040 con 0x7f635c102680 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.669+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6350013420 con 0x7f635c102680 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.669+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c10f910 con 0x7f635c102680 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.669+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635001c610 con 0x7f635c102680 2024-04-05T03:07:16.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.669+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c117e90 con 0x7f635c102680 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.670+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c0fd430 con 0x7f635c102680 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.670+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f635000baf0 con 0x7f635c102680 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.670+0000 7f6343fff700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 0x7f6344070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.671+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f635008e9d0 con 0x7f635c102680 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.671+0000 7f635b7fe700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 0x7f6344070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.671+0000 7f635b7fe700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 0x7f6344070280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f634c007fd0 tx=0x7f634c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:16.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.675+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6350057e60 con 0x7f635c102680 2024-04-05T03:07:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:16 smithi203 ceph-mon[95988]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:16 smithi203 ceph-mon[95988]: pgmap v12: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-05T03:07:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:16 smithi203 ceph-mon[95988]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:16 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1130289215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:16 smithi153 ceph-mon[106102]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:16 smithi153 ceph-mon[106102]: pgmap v12: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-05T03:07:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:16 smithi153 ceph-mon[106102]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:16 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1130289215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:16.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.890+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f635c108e30 con 0x7f634406ddd0 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.891+0000 7f6343fff700 1 -- 172.21.15.153:0/1883424969 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f635c108e30 con 0x7f634406ddd0 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:07:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:07:16.892 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "3/21 daemons upgraded", 2024-04-05T03:07:16.892 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mon daemons", 2024-04-05T03:07:16.892 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:07:16.892 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 msgr2=0x7f6344070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 0x7f6344070280 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f634c007fd0 tx=0x7f634c009040 comp rx=0 tx=0).stop 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 msgr2=0x7f635c10f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6350009100 tx=0x7f6350009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 shutdown_connections 2024-04-05T03:07:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634406ddd0 0x7f6344070280 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f635c101ce0 0x7f635c10ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 --2- 172.21.15.153:0/1883424969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f635c102680 0x7f635c10f010 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.893+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 >> 172.21.15.153:0/1883424969 conn(0x7f635c0f7870 msgr2=0x7f635c0f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.894+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 shutdown_connections 2024-04-05T03:07:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.894+0000 7f6362223700 1 -- 172.21.15.153:0/1883424969 wait complete. 2024-04-05T03:07:16.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- 172.21.15.153:0/1334545500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 msgr2=0x7f47300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 --2- 172.21.15.153:0/1334545500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47300fce10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4718004940 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- 172.21.15.153:0/1334545500 shutdown_connections 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 --2- 172.21.15.153:0/1334545500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47300fce10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 --2- 172.21.15.153:0/1334545500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f47301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- 172.21.15.153:0/1334545500 >> 172.21.15.153:0/1334545500 conn(0x7f47300f89b0 msgr2=0x7f47300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- 172.21.15.153:0/1334545500 shutdown_connections 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- 172.21.15.153:0/1334545500 wait complete. 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 Processor -- start 2024-04-05T03:07:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.995+0000 7f47359a1700 1 -- start start 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4730114710 con 0x7f47301037a0 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473010db70 con 0x7f4730102e00 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60664/0 (socket says 172.21.15.153:60664) 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 -- 172.21.15.153:0/2051659701 learned_addr learned my addr 172.21.15.153:0/2051659701 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f472effd700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f472effd700 1 -- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 msgr2=0x7f4730113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f472effd700 1 -- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 msgr2=0x7f4730113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f472effd700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f472effd700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 -- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 msgr2=0x7f4730113b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:16.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 -- 172.21.15.153:0/2051659701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4718005040 con 0x7f47301037a0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47267fc700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4718009100 tx=0x7f4718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f471800e040 con 0x7f47301037a0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4718013420 con 0x7f47301037a0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473010ddf0 con 0x7f47301037a0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f471801c610 con 0x7f47301037a0 2024-04-05T03:07:16.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.996+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473010e3b0 con 0x7f47301037a0 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.997+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47301052d0 con 0x7f47301037a0 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.998+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f4718013590 con 0x7f47301037a0 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.998+0000 7f4727fff700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 0x7f471c070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.999+0000 7f472effd700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 0x7f471c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.999+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f471808ea00 con 0x7f47301037a0 2024-04-05T03:07:17.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:16.999+0000 7f472effd700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 0x7f471c070270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4720009e50 tx=0x7f4720009e80 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:17.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.003+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4718057e90 con 0x7f47301037a0 2024-04-05T03:07:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.279+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f47301052d0 con 0x7f47301037a0 2024-04-05T03:07:17.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.279+0000 7f4727fff700 1 -- 172.21.15.153:0/2051659701 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f471805b550 con 0x7f47301037a0 2024-04-05T03:07:17.279 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:07:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 msgr2=0x7f471c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 0x7f471c070270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4720009e50 tx=0x7f4720009e80 comp rx=0 tx=0).stop 2024-04-05T03:07:17.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 msgr2=0x7f47301140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4718009100 tx=0x7f4718009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 shutdown_connections 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471c06ddc0 0x7f471c070270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730102e00 0x7f4730113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 --2- 172.21.15.153:0/2051659701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47301037a0 0x7f47301140c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 >> 172.21.15.153:0/2051659701 conn(0x7f47300f89b0 msgr2=0x7f4730102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 shutdown_connections 2024-04-05T03:07:17.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:17.281+0000 7f47359a1700 1 -- 172.21.15.153:0/2051659701 wait complete. 2024-04-05T03:07:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:17 smithi203 ceph-mon[95988]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:17 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2946005781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:07:17.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:17 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2051659701' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:07:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:17 smithi153 ceph-mon[106102]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:17 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2946005781' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:07:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:17 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2051659701' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:07:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:18 smithi203 ceph-mon[95988]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:18 smithi203 ceph-mon[95988]: pgmap v13: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:18 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:18 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:18 smithi153 ceph-mon[106102]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:18 smithi153 ceph-mon[106102]: pgmap v13: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:18 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:18 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:20 smithi203 ceph-mon[95988]: pgmap v14: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:20 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:20.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:20 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:20 smithi153 ceph-mon[106102]: pgmap v14: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:20 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:20 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:22.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:22 smithi203 ceph-mon[95988]: pgmap v15: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:22 smithi153 ceph-mon[106102]: pgmap v15: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: Detected new or changed devices on smithi203 2024-04-05T03:07:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-05T03:07:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:24 smithi203 ceph-mon[95988]: pgmap v16: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: Detected new or changed devices on smithi203 2024-04-05T03:07:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: Reconfiguring mon.smithi153 (monmap changed)... 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: Reconfiguring daemon mon.smithi153 on smithi153 2024-04-05T03:07:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:24 smithi153 ceph-mon[106102]: pgmap v16: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: Reconfiguring mgr.smithi153.npzyqy (monmap changed)... 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:07:25.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: Reconfiguring daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T03:07:25.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:25 smithi153 ceph-mon[106102]: pgmap v17: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: Reconfiguring mgr.smithi153.npzyqy (monmap changed)... 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.npzyqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: Reconfiguring daemon mgr.smithi153.npzyqy on smithi153 2024-04-05T03:07:26.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:25 smithi203 ceph-mon[95988]: pgmap v17: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:27.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:27.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:27.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-05T03:07:27.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:27.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:27.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:27 smithi153 ceph-mon[106102]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-05T03:07:27.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:27.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: Reconfiguring crash.smithi153 (monmap changed)... 2024-04-05T03:07:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:27.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:27 smithi203 ceph-mon[95988]: Reconfiguring daemon crash.smithi153 on smithi153 2024-04-05T03:07:28.506 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: pgmap v18: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:28.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:28.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:28.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:28.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T03:07:28.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: pgmap v18: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:28.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T03:07:28.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:29.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:29 smithi153 ceph-mon[106102]: Reconfiguring osd.0 (monmap changed)... 2024-04-05T03:07:29.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:29 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.0 on smithi153 2024-04-05T03:07:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:29 smithi203 ceph-mon[95988]: Reconfiguring osd.0 (monmap changed)... 2024-04-05T03:07:29.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:29 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.0 on smithi153 2024-04-05T03:07:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: pgmap v19: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: Reconfiguring osd.1 (monmap changed)... 2024-04-05T03:07:30.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T03:07:30.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:30.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:30 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.1 on smithi153 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: pgmap v19: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: Reconfiguring osd.1 (monmap changed)... 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:30.812 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:30 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.1 on smithi153 2024-04-05T03:07:32.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:32.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:32.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: Reconfiguring osd.2 (monmap changed)... 2024-04-05T03:07:32.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T03:07:32.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:32.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.2 on smithi153 2024-04-05T03:07:32.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:32 smithi203 ceph-mon[95988]: pgmap v20: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: Reconfiguring osd.2 (monmap changed)... 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.2 on smithi153 2024-04-05T03:07:32.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:32 smithi153 ceph-mon[106102]: pgmap v20: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:34.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:34.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:34.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: Reconfiguring mds.cephfs.smithi153.vicaky (monmap changed)... 2024-04-05T03:07:34.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:34.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:34.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: Reconfiguring daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T03:07:34.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:33 smithi153 ceph-mon[106102]: pgmap v21: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:34.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:34.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:34.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: Reconfiguring mds.cephfs.smithi153.vicaky (monmap changed)... 2024-04-05T03:07:34.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:34.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:34.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: Reconfiguring daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T03:07:34.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:33 smithi203 ceph-mon[95988]: pgmap v21: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: Reconfiguring mds.cephfs.smithi153.fmpmvd (monmap changed)... 2024-04-05T03:07:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:35.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:35.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:35 smithi153 ceph-mon[106102]: Reconfiguring daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T03:07:35.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:35.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:35.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: Reconfiguring mds.cephfs.smithi153.fmpmvd (monmap changed)... 2024-04-05T03:07:35.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:35.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:35.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:35 smithi203 ceph-mon[95988]: Reconfiguring daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T03:07:36.703 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:36 smithi203 ceph-mon[95988]: pgmap v22: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:36.703 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:36 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:36.703 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:36 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:36.703 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:36 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:36.703 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:36 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:36 smithi153 ceph-mon[106102]: pgmap v22: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:36 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:36 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:36 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:36 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:37.671 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:37 smithi203 ceph-mon[95988]: Reconfiguring crash.smithi203 (monmap changed)... 2024-04-05T03:07:37.671 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:37 smithi203 ceph-mon[95988]: Reconfiguring daemon crash.smithi203 on smithi203 2024-04-05T03:07:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:37 smithi153 ceph-mon[106102]: Reconfiguring crash.smithi203 (monmap changed)... 2024-04-05T03:07:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:37 smithi153 ceph-mon[106102]: Reconfiguring daemon crash.smithi203 on smithi203 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: pgmap v23: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:07:38.579 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: pgmap v23: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi203.iakodz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T03:07:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:39.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: Reconfiguring mgr.smithi203.iakodz (monmap changed)... 2024-04-05T03:07:39.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: Reconfiguring daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:07:39.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:39.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:39.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:39.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:39.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: Reconfiguring mgr.smithi203.iakodz (monmap changed)... 2024-04-05T03:07:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: Reconfiguring daemon mgr.smithi203.iakodz on smithi203 2024-04-05T03:07:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:39.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T03:07:39.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T03:07:39.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:40.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:40 smithi203 ceph-mon[95988]: Reconfiguring mon.smithi203 (monmap changed)... 2024-04-05T03:07:40.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:40 smithi203 ceph-mon[95988]: Reconfiguring daemon mon.smithi203 on smithi203 2024-04-05T03:07:40.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:40 smithi203 ceph-mon[95988]: pgmap v24: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:40 smithi153 ceph-mon[106102]: Reconfiguring mon.smithi203 (monmap changed)... 2024-04-05T03:07:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:40 smithi153 ceph-mon[106102]: Reconfiguring daemon mon.smithi203 on smithi203 2024-04-05T03:07:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:40 smithi153 ceph-mon[106102]: pgmap v24: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:42.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: Reconfiguring osd.3 (monmap changed)... 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.3 on smithi203 2024-04-05T03:07:42.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:41 smithi203 ceph-mon[95988]: pgmap v25: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:42.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:42.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:42.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: Reconfiguring osd.3 (monmap changed)... 2024-04-05T03:07:42.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T03:07:42.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:42.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.3 on smithi203 2024-04-05T03:07:42.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:41 smithi153 ceph-mon[106102]: pgmap v25: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T03:07:43.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T03:07:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:44.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:43 smithi153 ceph-mon[106102]: Reconfiguring osd.4 (monmap changed)... 2024-04-05T03:07:44.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:43 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.4 on smithi203 2024-04-05T03:07:44.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:43 smithi153 ceph-mon[106102]: pgmap v26: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:44.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:43 smithi203 ceph-mon[95988]: Reconfiguring osd.4 (monmap changed)... 2024-04-05T03:07:44.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:43 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.4 on smithi203 2024-04-05T03:07:44.279 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:43 smithi203 ceph-mon[95988]: pgmap v26: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:45.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:45.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:45.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: Reconfiguring osd.5 (monmap changed)... 2024-04-05T03:07:45.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T03:07:45.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:45.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:45 smithi153 ceph-mon[106102]: Reconfiguring daemon osd.5 on smithi203 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: Reconfiguring osd.5 (monmap changed)... 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:45.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:45 smithi203 ceph-mon[95988]: Reconfiguring daemon osd.5 on smithi203 2024-04-05T03:07:46.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:46 smithi203 ceph-mon[95988]: pgmap v27: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:46.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:46.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:46.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:46.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:46 smithi153 ceph-mon[106102]: pgmap v27: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 -- 172.21.15.153:0/552316018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0774102e00 msgr2=0x7f07741031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 --2- 172.21.15.153:0/552316018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0774102e00 0x7f07741031d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ce0 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 -- 172.21.15.153:0/552316018 shutdown_connections 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 --2- 172.21.15.153:0/552316018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07741037a0 0x7f07740fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 --2- 172.21.15.153:0/552316018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0774102e00 0x7f07741031d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 -- 172.21.15.153:0/552316018 >> 172.21.15.153:0/552316018 conn(0x7f07740f89b0 msgr2=0x7f07740fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 -- 172.21.15.153:0/552316018 shutdown_connections 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.395+0000 7f077baac700 1 -- 172.21.15.153:0/552316018 wait complete. 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 Processor -- start 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 -- start start 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0774102e00 0x7f0774113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741148f0 con 0x7f07741037a0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f077baac700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077410dcc0 con 0x7f0774102e00 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42600/0 (socket says 172.21.15.153:42600) 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779848700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0774102e00 0x7f0774113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 -- 172.21.15.153:0/2005577659 learned_addr learned my addr 172.21.15.153:0/2005577659 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 -- 172.21.15.153:0/2005577659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0774102e00 msgr2=0x7f0774113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0774102e00 0x7f0774113d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.396+0000 7f0779047700 1 -- 172.21.15.153:0/2005577659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770009940 con 0x7f07741037a0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f0779047700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f076400c800 tx=0x7f0764004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07640106f0 con 0x7f07741037a0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077410dfa0 con 0x7f07741037a0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0764010d20 con 0x7f07741037a0 2024-04-05T03:07:47.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07640113a0 con 0x7f07741037a0 2024-04-05T03:07:47.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.397+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077410e560 con 0x7f07741037a0 2024-04-05T03:07:47.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.398+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0774100110 con 0x7f07741037a0 2024-04-05T03:07:47.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.398+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f0764012400 con 0x7f07741037a0 2024-04-05T03:07:47.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.399+0000 7f076a7fc700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 0x7f0760070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.399+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f076408d560 con 0x7f07741037a0 2024-04-05T03:07:47.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.399+0000 7f0779848700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 0x7f0760070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.399+0000 7f0779848700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 0x7f0760070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f077000b740 tx=0x7f0770004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.404+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f076405a630 con 0x7f07741037a0 2024-04-05T03:07:47.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.618+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f077410f3e0 con 0x7f076006ddc0 2024-04-05T03:07:47.645 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:47 smithi203 ceph-mon[95988]: Reconfiguring mds.cephfs.smithi203.xpteyc (monmap changed)... 2024-04-05T03:07:47.646 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:47 smithi203 ceph-mon[95988]: Reconfiguring daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T03:07:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.619+0000 7f076a7fc700 1 -- 172.21.15.153:0/2005577659 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f077410f3e0 con 0x7f076006ddc0 2024-04-05T03:07:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 msgr2=0x7f0760070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 0x7f0760070270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f077000b740 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-04-05T03:07:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 msgr2=0x7f07741142a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f076400c800 tx=0x7f0764004470 comp rx=0 tx=0).stop 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 shutdown_connections 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f076006ddc0 0x7f0760070270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0774102e00 0x7f0774113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 --2- 172.21.15.153:0/2005577659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07741037a0 0x7f07741142a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 >> 172.21.15.153:0/2005577659 conn(0x7f07740f89b0 msgr2=0x7f07741073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 shutdown_connections 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.621+0000 7f077baac700 1 -- 172.21.15.153:0/2005577659 wait complete. 2024-04-05T03:07:47.647 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:07:47.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:47 smithi153 ceph-mon[106102]: Reconfiguring mds.cephfs.smithi203.xpteyc (monmap changed)... 2024-04-05T03:07:47.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:47 smithi153 ceph-mon[106102]: Reconfiguring daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T03:07:47.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 -- 172.21.15.153:0/590710642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 msgr2=0x7ff5440fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 --2- 172.21.15.153:0/590710642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff5440fbd80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff538004940 tx=0x7ff538009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 -- 172.21.15.153:0/590710642 shutdown_connections 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 --2- 172.21.15.153:0/590710642 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff5440fbd80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 --2- 172.21.15.153:0/590710642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 0x7ff544102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 -- 172.21.15.153:0/590710642 >> 172.21.15.153:0/590710642 conn(0x7ff544072bc0 msgr2=0x7ff544072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 -- 172.21.15.153:0/590710642 shutdown_connections 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.725+0000 7ff54a04b700 1 -- 172.21.15.153:0/590710642 wait complete. 2024-04-05T03:07:47.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.726+0000 7ff54a04b700 1 Processor -- start 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.726+0000 7ff54a04b700 1 -- start start 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff54a04b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 0x7ff544112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff54a04b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff54a04b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544113630 con 0x7ff544102710 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff54a04b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54410ca20 con 0x7ff544101d70 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff5437fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 0x7ff544112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42612/0 (socket says 172.21.15.153:42612) 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 -- 172.21.15.153:0/942343743 learned_addr learned my addr 172.21.15.153:0/942343743 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 -- 172.21.15.153:0/942343743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 msgr2=0x7ff544112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 0x7ff544112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 -- 172.21.15.153:0/942343743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538005040 con 0x7ff544102710 2024-04-05T03:07:47.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff542ffd700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff538009100 tx=0x7ff538009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:47.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.727+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff53800e040 con 0x7ff544102710 2024-04-05T03:07:47.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.728+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54410cca0 con 0x7ff544102710 2024-04-05T03:07:47.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.728+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff538009af0 con 0x7ff544102710 2024-04-05T03:07:47.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.728+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff53801c3f0 con 0x7ff544102710 2024-04-05T03:07:47.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.728+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54410d260 con 0x7ff544102710 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.729+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544104240 con 0x7ff544102710 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.729+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7ff53800baf0 con 0x7ff544102710 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.730+0000 7ff52bfff700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 0x7ff52c070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.730+0000 7ff5437fe700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 0x7ff52c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.730+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff53808e990 con 0x7ff544102710 2024-04-05T03:07:47.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.730+0000 7ff5437fe700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 0x7ff52c070270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff534004eb0 tx=0x7ff53400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:47.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.734+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff53805b950 con 0x7ff544102710 2024-04-05T03:07:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.950+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff544112770 con 0x7ff52c06ddc0 2024-04-05T03:07:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.952+0000 7ff52bfff700 1 -- 172.21.15.153:0/942343743 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7ff544112770 con 0x7ff52c06ddc0 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 msgr2=0x7ff52c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 0x7ff52c070270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff534004eb0 tx=0x7ff53400b040 comp rx=0 tx=0).stop 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 msgr2=0x7ff544112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff538009100 tx=0x7ff538009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 shutdown_connections 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff52c06ddc0 0x7ff52c070270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff544101d70 0x7ff544112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 --2- 172.21.15.153:0/942343743 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff544102710 0x7ff544112fe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 >> 172.21.15.153:0/942343743 conn(0x7ff544072bc0 msgr2=0x7ff5440fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 shutdown_connections 2024-04-05T03:07:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:47.954+0000 7ff54a04b700 1 -- 172.21.15.153:0/942343743 wait complete. 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3043412880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa18101d20 msgr2=0x7ffa181020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3043412880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa18101d20 0x7ffa181020f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c004940 tx=0x7ffa0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3043412880 shutdown_connections 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3043412880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa181026c0 0x7ffa180fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3043412880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa18101d20 0x7ffa181020f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3043412880 >> 172.21.15.153:0/3043412880 conn(0x7ffa180f7870 msgr2=0x7ffa180f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3043412880 shutdown_connections 2024-04-05T03:07:48.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.054+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3043412880 wait complete. 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 Processor -- start 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 -- start start 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 0x7ffa1810c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18067100 con 0x7ffa181026c0 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa1ef91700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1810d3f0 con 0x7ffa18101d20 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.055+0000 7ffa17fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42644/0 (socket says 172.21.15.153:42644) 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 -- 172.21.15.153:0/3925030355 learned_addr learned my addr 172.21.15.153:0/3925030355 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa1cd2d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 0x7ffa1810c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 -- 172.21.15.153:0/3925030355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 msgr2=0x7ffa1810c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 0x7ffa1810c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 -- 172.21.15.153:0/3925030355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c005040 con 0x7ffa181026c0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa1cd2d700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 0x7ffa1810c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa17fff700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa04002a50 tx=0x7ffa04004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa04007cc0 con 0x7ffa181026c0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1810d6d0 con 0x7ffa181026c0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa0400f040 con 0x7ffa181026c0 2024-04-05T03:07:48.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa0400d9c0 con 0x7ffa181026c0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.056+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1811a220 con 0x7ffa181026c0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.057+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa180fd470 con 0x7ffa181026c0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.058+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7ffa04009730 con 0x7ffa181026c0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.058+0000 7ffa157fa700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 0x7ffa08070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.058+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ffa0408d410 con 0x7ffa181026c0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.058+0000 7ffa1cd2d700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 0x7ffa08070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.059+0000 7ffa1cd2d700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 0x7ffa08070280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c004560 tx=0x7ffa0c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.063+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffa0405a350 con 0x7ffa181026c0 2024-04-05T03:07:48.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.265+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa18108ec0 con 0x7ffa0806ddd0 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.276+0000 7ffa157fa700 1 -- 172.21.15.153:0/3925030355 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ffa0c00b650 con 0x7ffa0806ddd0 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (2m) 46s ago 13m 26.6M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (14m) 46s ago 14m 6665k - 17.2.7-805-g6fda41ff a49113eacc58 d25613b4d85f 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (12m) 29s ago 12m 6656k - 17.2.7-805-g6fda41ff a49113eacc58 25c0b8bd96ea 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (2m) 46s ago 13m 74.5M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:07:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (7m) 46s ago 7m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (8m) 46s ago 8m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (7m) 29s ago 7m 61.4M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (7m) 29s ago 7m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (3m) 46s ago 15m 473M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (3m) 29s ago 12m 447M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (61s) 46s ago 15m 27.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (35s) 29s ago 12m 27.1M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (2m) 46s ago 13m 16.9M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (2m) 29s ago 12m 17.6M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (11m) 46s ago 11m 224M 1898M 17.2.7-805-g6fda41ff a49113eacc58 9adc3a479eeb 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (11m) 46s ago 11m 229M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (10m) 46s ago 10m 199M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (10m) 29s ago 10m 265M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:07:48.277 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (9m) 29s ago 9m 231M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:07:48.278 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (9m) 29s ago 9m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:07:48.278 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (2m) 29s ago 11m 60.4M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:07:48.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.278+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 msgr2=0x7ffa08070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.278+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 0x7ffa08070280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c004560 tx=0x7ffa0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.278+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 msgr2=0x7ffa1810ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.278+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa04002a50 tx=0x7ffa04004600 comp rx=0 tx=0).stop 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 shutdown_connections 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffa0806ddd0 0x7ffa08070280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa18101d20 0x7ffa1810c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 --2- 172.21.15.153:0/3925030355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa181026c0 0x7ffa1810ceb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 >> 172.21.15.153:0/3925030355 conn(0x7ffa180f7870 msgr2=0x7ffa180f8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 shutdown_connections 2024-04-05T03:07:48.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.279+0000 7ffa1ef91700 1 -- 172.21.15.153:0/3925030355 wait complete. 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 -- 172.21.15.153:0/944517376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af00fb570 msgr2=0x7f8af00fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 --2- 172.21.15.153:0/944517376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af00fb570 0x7f8af00fb9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4004940 tx=0x7f8ae4009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 -- 172.21.15.153:0/944517376 shutdown_connections 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 --2- 172.21.15.153:0/944517376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af00fb570 0x7f8af00fb9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 --2- 172.21.15.153:0/944517376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af0101530 0x7f8af0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 -- 172.21.15.153:0/944517376 >> 172.21.15.153:0/944517376 conn(0x7f8af00714c0 msgr2=0x7f8af00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.378+0000 7f8af738f700 1 -- 172.21.15.153:0/944517376 shutdown_connections 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 -- 172.21.15.153:0/944517376 wait complete. 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 Processor -- start 2024-04-05T03:07:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 -- start start 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af00fb570 0x7f8af0112870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0113490 con 0x7f8af0101530 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af738f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af010ca00 con 0x7f8af00fb570 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af492a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.379+0000 7f8af492a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42660/0 (socket says 172.21.15.153:42660) 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af492a700 1 -- 172.21.15.153:0/1407104569 learned_addr learned my addr 172.21.15.153:0/1407104569 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af512b700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af00fb570 0x7f8af0112870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af492a700 1 -- 172.21.15.153:0/1407104569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af00fb570 msgr2=0x7f8af0112870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af492a700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af00fb570 0x7f8af0112870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af492a700 1 -- 172.21.15.153:0/1407104569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4005040 con 0x7f8af0101530 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af492a700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400b3b0 tx=0x7f8ae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae400e040 con 0x7f8af0101530 2024-04-05T03:07:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ae400bd00 con 0x7f8af0101530 2024-04-05T03:07:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af010cc80 con 0x7f8af0101530 2024-04-05T03:07:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae4013660 con 0x7f8af0101530 2024-04-05T03:07:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.380+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af010d1e0 con 0x7f8af0101530 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.382+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af00fcc90 con 0x7f8af0101530 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.382+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f8ae401e420 con 0x7f8af0101530 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.382+0000 7f8aedffb700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 0x7f8ad8070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.382+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8ae40915f0 con 0x7f8af0101530 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.382+0000 7f8af512b700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 0x7f8ad8070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.383+0000 7f8af512b700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 0x7f8ad8070280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009bf0 tx=0x7f8ae00091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.388+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ae405e530 con 0x7f8af0101530 2024-04-05T03:07:48.540 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:48 smithi203 ceph-mon[95988]: pgmap v28: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:48.540 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:48.540 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:48.540 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:48.540 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:48.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:48 smithi153 ceph-mon[106102]: pgmap v28: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:48.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:48.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:48.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:07:48.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.674+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8af004c710 con 0x7f8af0101530 2024-04-05T03:07:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.675+0000 7f8aedffb700 1 -- 172.21.15.153:0/1407104569 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f8ae405e0c0 con 0x7f8af0101530 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:07:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 10, 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:07:48.676 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:07:48.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 msgr2=0x7f8ad8070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 0x7f8ad8070280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0009bf0 tx=0x7f8ae00091a0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 msgr2=0x7f8af0112db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400b3b0 tx=0x7f8ae4009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 shutdown_connections 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.677+0000 7f8af738f700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ad806ddd0 0x7f8ad8070280 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.678+0000 7f8af738f700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8af00fb570 0x7f8af0112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.678+0000 7f8af738f700 1 --2- 172.21.15.153:0/1407104569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8af0101530 0x7f8af0112db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.678+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 >> 172.21.15.153:0/1407104569 conn(0x7f8af00714c0 msgr2=0x7f8af00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.678+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 shutdown_connections 2024-04-05T03:07:48.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.678+0000 7f8af738f700 1 -- 172.21.15.153:0/1407104569 wait complete. 2024-04-05T03:07:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 -- 172.21.15.153:0/2143273939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb8102e70 msgr2=0x7f8eb8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 --2- 172.21.15.153:0/2143273939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb8102e70 0x7f8eb8103240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8004940 tx=0x7f8ea8009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 -- 172.21.15.153:0/2143273939 shutdown_connections 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 --2- 172.21.15.153:0/2143273939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 --2- 172.21.15.153:0/2143273939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb8102e70 0x7f8eb8103240 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 -- 172.21.15.153:0/2143273939 >> 172.21.15.153:0/2143273939 conn(0x7f8eb80714c0 msgr2=0x7f8eb80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 -- 172.21.15.153:0/2143273939 shutdown_connections 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 -- 172.21.15.153:0/2143273939 wait complete. 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.778+0000 7f8ebce36700 1 Processor -- start 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 -- start start 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb8102e70 0x7f8eb80fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80fd150 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb80fd290 con 0x7f8eb8102e70 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33790/0 (socket says 172.21.15.153:33790) 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 -- 172.21.15.153:0/3691744179 learned_addr learned my addr 172.21.15.153:0/3691744179 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb5d9b700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb8102e70 0x7f8eb80fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 -- 172.21.15.153:0/3691744179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb8102e70 msgr2=0x7f8eb80fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb8102e70 0x7f8eb80fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 -- 172.21.15.153:0/3691744179 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8005040 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8eb659c700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8009150 tx=0x7f8ea8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea800e040 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ea8013420 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb80fd4e0 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.779+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ea801c610 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.780+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb811a280 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.781+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb8100110 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.781+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f8ea800baf0 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.781+0000 7f8ea6ffd700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 0x7f8ea0070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.782+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8ea808e940 con 0x7f8eb80fb4a0 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.782+0000 7f8eb5d9b700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 0x7f8ea0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:48.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.782+0000 7f8eb5d9b700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 0x7f8ea0070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80fdfc0 tx=0x7f8eac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:48.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:48.786+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ea8057dd0 con 0x7f8eb80fb4a0 2024-04-05T03:07:49.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.024+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8eb804bed0 con 0x7f8eb80fb4a0 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.025+0000 7f8ea6ffd700 1 -- 172.21.15.153:0/3691744179 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7f8ea805b490 con 0x7f8eb80fb4a0 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:07:49.025 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:07:49.026 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:49.027 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 msgr2=0x7f8ea0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 0x7f8ea0070270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80fdfc0 tx=0x7f8eac00b040 comp rx=0 tx=0).stop 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 msgr2=0x7f8eb80fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8009150 tx=0x7f8ea8009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 shutdown_connections 2024-04-05T03:07:49.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ea006ddc0 0x7f8ea0070270 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb80fb4a0 0x7f8eb80fc680 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 --2- 172.21.15.153:0/3691744179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb8102e70 0x7f8eb80fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 >> 172.21.15.153:0/3691744179 conn(0x7f8eb80714c0 msgr2=0x7f8eb80fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 shutdown_connections 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.027+0000 7f8ebce36700 1 -- 172.21.15.153:0/3691744179 wait complete. 2024-04-05T03:07:49.029 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:07:49.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 -- 172.21.15.153:0/3820512261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec581026c0 msgr2=0x7fec580fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 --2- 172.21.15.153:0/3820512261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec581026c0 0x7fec580fbd30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004940 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 -- 172.21.15.153:0/3820512261 shutdown_connections 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 --2- 172.21.15.153:0/3820512261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec581026c0 0x7fec580fbd30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 --2- 172.21.15.153:0/3820512261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec58101d20 0x7fec581020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 -- 172.21.15.153:0/3820512261 >> 172.21.15.153:0/3820512261 conn(0x7fec580f78b0 msgr2=0x7fec580f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 -- 172.21.15.153:0/3820512261 shutdown_connections 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.130+0000 7fec5dde6700 1 -- 172.21.15.153:0/3820512261 wait complete. 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 Processor -- start 2024-04-05T03:07:49.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 -- start start 2024-04-05T03:07:49.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec581026c0 0x7fec580fdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580ff940 con 0x7fec58101d20 2024-04-05T03:07:49.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec5dde6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580fe420 con 0x7fec581026c0 2024-04-05T03:07:49.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33806/0 (socket says 172.21.15.153:33806) 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec577fe700 1 -- 172.21.15.153:0/2983602002 learned_addr learned my addr 172.21.15.153:0/2983602002 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.131+0000 7fec56ffd700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec581026c0 0x7fec580fdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec577fe700 1 -- 172.21.15.153:0/2983602002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec581026c0 msgr2=0x7fec580fdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec577fe700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec581026c0 0x7fec580fdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec577fe700 1 -- 172.21.15.153:0/2983602002 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c005040 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec577fe700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c830 tx=0x7fec48004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec480106f0 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec580fe6d0 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec48010d20 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec480113a0 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.132+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5811a130 con 0x7fec58101d20 2024-04-05T03:07:49.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.133+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec580fd470 con 0x7fec58101d20 2024-04-05T03:07:49.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.133+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fec48012400 con 0x7fec58101d20 2024-04-05T03:07:49.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.134+0000 7fec3ffff700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 0x7fec40070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.134+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fec4808d590 con 0x7fec58101d20 2024-04-05T03:07:49.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.134+0000 7fec56ffd700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 0x7fec40070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.134+0000 7fec56ffd700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 0x7fec40070280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004890 tx=0x7fec4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:49.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.138+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fec48056b30 con 0x7fec58101d20 2024-04-05T03:07:49.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.359+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec580669a0 con 0x7fec4006ddd0 2024-04-05T03:07:49.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.360+0000 7fec3ffff700 1 -- 172.21.15.153:0/2983602002 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fec580669a0 con 0x7fec4006ddd0 2024-04-05T03:07:49.359 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr" 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "4/21 daemons upgraded", 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading crash daemons", 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:07:49.360 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 msgr2=0x7fec40070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 0x7fec40070280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fec4c004890 tx=0x7fec4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 msgr2=0x7fec580fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fec4800c830 tx=0x7fec48004470 comp rx=0 tx=0).stop 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 shutdown_connections 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec4006ddd0 0x7fec40070280 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec58101d20 0x7fec580fd9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 --2- 172.21.15.153:0/2983602002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec581026c0 0x7fec580fdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 >> 172.21.15.153:0/2983602002 conn(0x7fec580f78b0 msgr2=0x7fec58066290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:49.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 shutdown_connections 2024-04-05T03:07:49.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.362+0000 7fec5dde6700 1 -- 172.21.15.153:0/2983602002 wait complete. 2024-04-05T03:07:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- 172.21.15.153:0/128007363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4102710 msgr2=0x7f1ea40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 --2- 172.21.15.153:0/128007363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4102710 0x7f1ea40fbd80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1e90004940 tx=0x7f1e90009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- 172.21.15.153:0/128007363 shutdown_connections 2024-04-05T03:07:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 --2- 172.21.15.153:0/128007363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4102710 0x7f1ea40fbd80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 --2- 172.21.15.153:0/128007363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4101d70 0x7f1ea4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- 172.21.15.153:0/128007363 >> 172.21.15.153:0/128007363 conn(0x7f1ea4072bc0 msgr2=0x7f1ea4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- 172.21.15.153:0/128007363 shutdown_connections 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- 172.21.15.153:0/128007363 wait complete. 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 Processor -- start 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.470+0000 7f1eaac42700 1 -- start start 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1eaac42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1eaac42700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4102710 0x7f1ea410f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1eaac42700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea410f6b0 con 0x7f1ea4101d70 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1eaac42700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea410f7f0 con 0x7f1ea4102710 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33818/0 (socket says 172.21.15.153:33818) 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 -- 172.21.15.153:0/2115110268 learned_addr learned my addr 172.21.15.153:0/2115110268 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea3fff700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4102710 0x7f1ea410f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 -- 172.21.15.153:0/2115110268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4102710 msgr2=0x7f1ea410f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4102710 0x7f1ea410f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 -- 172.21.15.153:0/2115110268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90005040 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.471+0000 7f1ea89de700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e98002a50 tx=0x7f1e980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.472+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e98007cc0 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.472+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e9800f040 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.472+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea4117b90 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.472+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9800db70 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.472+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea41180f0 con 0x7f1ea4101d70 2024-04-05T03:07:49.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.473+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea4104240 con 0x7f1ea4101d70 2024-04-05T03:07:49.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.473+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f1e9800dd10 con 0x7f1ea4101d70 2024-04-05T03:07:49.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.474+0000 7f1ea17fa700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 0x7f1e94070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:07:49.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.474+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1e9808ce40 con 0x7f1ea4101d70 2024-04-05T03:07:49.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.474+0000 7f1ea3fff700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 0x7f1e94070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:07:49.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.474+0000 7f1ea3fff700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 0x7f1e94070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4066000 tx=0x7f1e90009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:07:49.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.478+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1e98056250 con 0x7f1ea4101d70 2024-04-05T03:07:49.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: Reconfiguring mds.cephfs.smithi203.hgbifk (monmap changed)... 2024-04-05T03:07:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: Reconfiguring daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T03:07:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1407104569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3691744179' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.548 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]: dispatch 2024-04-05T03:07:49.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]': finished 2024-04-05T03:07:49.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi203"}]: dispatch 2024-04-05T03:07:49.549 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:49 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi203"}]': finished 2024-04-05T03:07:49.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.771+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1ea40fd4c0 con 0x7f1ea4101d70 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: Reconfiguring mds.cephfs.smithi203.hgbifk (monmap changed)... 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: Reconfiguring daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1407104569' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3691744179' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi153"}]': finished 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi203"}]: dispatch 2024-04-05T03:07:49.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:49 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi203"}]': finished 2024-04-05T03:07:49.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.771+0000 7f1ea17fa700 1 -- 172.21.15.153:0/2115110268 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1e98059910 con 0x7f1ea4101d70 2024-04-05T03:07:49.773 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 msgr2=0x7f1e94070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 0x7f1e94070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4066000 tx=0x7f1e90009e30 comp rx=0 tx=0).stop 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 msgr2=0x7f1ea410eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e98002a50 tx=0x7f1e980046b0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 shutdown_connections 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e9406ddc0 0x7f1e94070270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ea4101d70 0x7f1ea410eb20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 --2- 172.21.15.153:0/2115110268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4102710 0x7f1ea410f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:07:49.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 >> 172.21.15.153:0/2115110268 conn(0x7f1ea4072bc0 msgr2=0x7f1ea40fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:07:49.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 shutdown_connections 2024-04-05T03:07:49.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:07:49.774+0000 7f1eaac42700 1 -- 172.21.15.153:0/2115110268 wait complete. 2024-04-05T03:07:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:50 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all mon 2024-04-05T03:07:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:50 smithi203 ceph-mon[95988]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:50 smithi203 ceph-mon[95988]: pgmap v29: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:50 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2115110268' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:07:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:50 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all mon 2024-04-05T03:07:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:50 smithi153 ceph-mon[106102]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:07:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:50 smithi153 ceph-mon[106102]: pgmap v29: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:50.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:50 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2115110268' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:07:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: Upgrade: Updating crash.smithi153 (1/2) 2024-04-05T03:07:52.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:52.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:52.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:52.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: Deploying daemon crash.smithi153 on smithi153 2024-04-05T03:07:52.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:51 smithi153 ceph-mon[106102]: pgmap v30: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: Upgrade: Updating crash.smithi153 (1/2) 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: Deploying daemon crash.smithi153 on smithi153 2024-04-05T03:07:52.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:51 smithi203 ceph-mon[95988]: pgmap v30: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:54 smithi203 ceph-mon[95988]: pgmap v31: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:54 smithi153 ceph-mon[106102]: pgmap v31: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:07:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:56.781 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:56 smithi203 ceph-mon[95988]: pgmap v32: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:07:56.781 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:56.781 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:56.781 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:56 smithi153 ceph-mon[106102]: pgmap v32: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:07:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:07:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi203", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T03:07:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:07:57.714 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:57 smithi203 ceph-mon[95988]: Upgrade: Updating crash.smithi203 (2/2) 2024-04-05T03:07:57.714 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:57 smithi203 ceph-mon[95988]: Deploying daemon crash.smithi203 on smithi203 2024-04-05T03:07:57.714 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:57 smithi153 ceph-mon[106102]: Upgrade: Updating crash.smithi203 (2/2) 2024-04-05T03:07:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:57 smithi153 ceph-mon[106102]: Deploying daemon crash.smithi203 on smithi203 2024-04-05T03:07:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:07:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:07:58 smithi203 ceph-mon[95988]: pgmap v33: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:07:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:07:58 smithi153 ceph-mon[106102]: pgmap v33: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:00.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:00 smithi203 ceph-mon[95988]: pgmap v34: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:00.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:00 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:00.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:00 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:00.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:00 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:00 smithi153 ceph-mon[106102]: pgmap v34: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:00 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:00 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:00 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:02.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:02 smithi203 ceph-mon[95988]: pgmap v35: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:02 smithi153 ceph-mon[106102]: pgmap v35: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:04 smithi203 ceph-mon[95988]: pgmap v36: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:04 smithi153 ceph-mon[106102]: pgmap v36: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:05 smithi203 ceph-mon[95988]: pgmap v37: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:06.064 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.064 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.064 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.065 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:06.065 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:05 smithi153 ceph-mon[106102]: pgmap v37: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:08.660 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:08 smithi203 ceph-mon[95988]: pgmap v38: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:08.660 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:08 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:08.660 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:08 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:08 smithi153 ceph-mon[106102]: pgmap v38: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:08 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:08 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all crash 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]': finished 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi203"}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi203"}]': finished 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T03:08:10.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T03:08:10.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: Upgrade: osd.0 is safe to restart 2024-04-05T03:08:10.048 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:09 smithi203 ceph-mon[95988]: pgmap v39: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:10.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:10.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:08:10.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all crash 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi153"}]': finished 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi203"}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi203"}]': finished 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: Upgrade: osd.0 is safe to restart 2024-04-05T03:08:10.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:09 smithi153 ceph-mon[106102]: pgmap v39: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:11.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:11 smithi203 ceph-mon[95988]: Upgrade: Updating osd.0 2024-04-05T03:08:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:11 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:11 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T03:08:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:11 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:11.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:11 smithi203 ceph-mon[95988]: Deploying daemon osd.0 on smithi153 2024-04-05T03:08:11.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:11 smithi153 ceph-mon[106102]: Upgrade: Updating osd.0 2024-04-05T03:08:11.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:11 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:11.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:11 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T03:08:11.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:11 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:11.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:11 smithi153 ceph-mon[106102]: Deploying daemon osd.0 on smithi153 2024-04-05T03:08:12.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:12 smithi153 ceph-mon[106102]: pgmap v40: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:12.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:12 smithi203 ceph-mon[95988]: pgmap v40: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:14.048 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:13 smithi153 systemd[1]: Stopping Ceph osd.0 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:08:14.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:14 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[46305]: 2024-04-05T03:08:14.047+0000 7fd0a3de2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:08:14.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:14 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[46305]: 2024-04-05T03:08:14.047+0000 7fd0a3de2700 -1 osd.0 48 *** Got signal Terminated *** 2024-04-05T03:08:14.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:14 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[46305]: 2024-04-05T03:08:14.047+0000 7fd0a3de2700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:08:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:14 smithi203 ceph-mon[95988]: pgmap v41: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:14 smithi203 ceph-mon[95988]: osd.0 marked itself down and dead 2024-04-05T03:08:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:14 smithi153 ceph-mon[106102]: pgmap v41: 65 pgs: 65 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:14 smithi153 ceph-mon[106102]: osd.0 marked itself down and dead 2024-04-05T03:08:15.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:15 smithi153 podman[116750]: 2024-04-05 03:08:15.011322023 +0000 UTC m=+1.079505327 container died 9adc3a479eeb2ffa870321d282be07da905cf4762c85c9c515b95273c1e6a660 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=quincy-6fda41f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-05T03:08:15.572 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:15 smithi153 podman[116750]: 2024-04-05 03:08:15.456798096 +0000 UTC m=+1.524981399 container cleanup 9adc3a479eeb2ffa870321d282be07da905cf4762c85c9c515b95273c1e6a660 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=quincy-6fda41f, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:08:15.572 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:15 smithi153 bash[116750]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0 2024-04-05T03:08:15.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:15 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:08:15.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:15 smithi153 ceph-mon[106102]: failed to encode map e49 with expected crc 2024-04-05T03:08:15.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:15 smithi153 ceph-mon[106102]: osdmap e49: 6 total, 5 up, 6 in 2024-04-05T03:08:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:15 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:08:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:15 smithi203 ceph-mon[95988]: failed to encode map e49 with expected crc 2024-04-05T03:08:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:15 smithi203 ceph-mon[95988]: osdmap e49: 6 total, 5 up, 6 in 2024-04-05T03:08:16.072 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:15 smithi153 podman[116768]: 2024-04-05 03:08:15.900362473 +0000 UTC m=+0.892134015 container remove 9adc3a479eeb2ffa870321d282be07da905cf4762c85c9c515b95273c1e6a660 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-6fda41f, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:08:16.513 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.150431685 +0000 UTC m=+0.018417346 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.251195196 +0000 UTC m=+0.119180841 container create d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-05T03:08:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.470874354 +0000 UTC m=+0.338860007 container init d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:08:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.474632424 +0000 UTC m=+0.342618074 container start d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-05T03:08:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.587415329 +0000 UTC m=+0.455400982 container attach d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: pgmap v43: 65 pgs: 9 stale+active+clean, 56 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: osdmap e50: 6 total, 5 up, 6 in 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:16 smithi153 ceph-mon[106102]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: pgmap v43: 65 pgs: 9 stale+active+clean, 56 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: osdmap e50: 6 total, 5 up, 6 in 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:16.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:16 smithi203 ceph-mon[95988]: failed to encode map e50 with expected crc 2024-04-05T03:08:17.072 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:16 smithi153 podman[116870]: 2024-04-05 03:08:16.776686698 +0000 UTC m=+0.644672340 container died d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-05T03:08:17.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 podman[116897]: 2024-04-05 03:08:17.324496723 +0000 UTC m=+0.542950229 container remove d6985127055bf418e34dbfe7c398fb94dbc4b9e1ab4dcedd6eea5a6a5e3b65cd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:08:17.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-04-05T03:08:17.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 systemd[1]: Stopped Ceph osd.0 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:08:17.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.0.service: Consumed 1min 1.633s CPU time. 2024-04-05T03:08:17.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 systemd[1]: Starting Ceph osd.0 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:08:17.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 podman[117002]: 2024-04-05 03:08:17.680690148 +0000 UTC m=+0.018151961 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:18.094 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:17 smithi153 podman[117002]: 2024-04-05 03:08:17.834305489 +0000 UTC m=+0.171767287 container create 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-05T03:08:18.094 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 podman[117002]: 2024-04-05 03:08:18.020240096 +0000 UTC m=+0.357701895 container init 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-05T03:08:18.094 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 podman[117002]: 2024-04-05 03:08:18.023615526 +0000 UTC m=+0.361077335 container start 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-0028525, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-05T03:08:18.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 podman[117002]: 2024-04-05 03:08:18.09502792 +0000 UTC m=+0.432489731 container attach 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:08:18.789 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-mon[106102]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T03:08:18.789 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 bash[117002]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate[117017]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-05T03:08:18.790 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:18 smithi153 podman[117002]: 2024-04-05 03:08:18.491768741 +0000 UTC m=+0.829230535 container died 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-0028525) 2024-04-05T03:08:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:18 smithi203 ceph-mon[95988]: pgmap v45: 65 pgs: 9 stale+active+clean, 56 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:08:19.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 podman[117150]: 2024-04-05 03:08:19.040823449 +0000 UTC m=+0.543619498 container remove 994b52bb49066076abc0cc07588246f62a826f3d558a890d71549c6709a6de63 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-05T03:08:19.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 podman[117209]: 2024-04-05 03:08:19.1584903 +0000 UTC m=+0.018923865 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:19.322 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 podman[117209]: 2024-04-05 03:08:19.267576399 +0000 UTC m=+0.128009964 container create 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:08:19.635 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:19 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 62/318 objects degraded (19.497%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:19.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 podman[117209]: 2024-04-05 03:08:19.51123547 +0000 UTC m=+0.371669043 container init 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-05T03:08:19.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 podman[117209]: 2024-04-05 03:08:19.514558383 +0000 UTC m=+0.374991949 container start 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-0028525, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-05T03:08:19.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 ceph-osd[117227]: -- 172.21.15.153:0/4070146360 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d27b42f680 con 0x55d27b3f8c00 2024-04-05T03:08:19.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 bash[117209]: 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 2024-04-05T03:08:19.870 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:19 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 62/318 objects degraded (19.497%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:19.887 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:19 smithi153 systemd[1]: Started Ceph osd.0 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 -- 172.21.15.153:0/3451618096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 msgr2=0x7f1e34102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/3451618096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e34102140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1e28004940 tx=0x7f1e28009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 -- 172.21.15.153:0/3451618096 shutdown_connections 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/3451618096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 0x7f1e340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/3451618096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e34102140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 -- 172.21.15.153:0/3451618096 >> 172.21.15.153:0/3451618096 conn(0x7f1e34072bc0 msgr2=0x7f1e34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 -- 172.21.15.153:0/3451618096 shutdown_connections 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 -- 172.21.15.153:0/3451618096 wait complete. 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.887+0000 7f1e3acdf700 1 Processor -- start 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e3acdf700 1 -- start start 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e3acdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e3acdf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 0x7f1e3410f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e3acdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3410f670 con 0x7f1e34101d70 2024-04-05T03:08:19.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e3acdf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3410f7b0 con 0x7f1e34102710 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e38a7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e38a7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43114/0 (socket says 172.21.15.153:43114) 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e38a7b700 1 -- 172.21.15.153:0/2599805090 learned_addr learned my addr 172.21.15.153:0/2599805090 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.888+0000 7f1e33fff700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 0x7f1e3410f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e38a7b700 1 -- 172.21.15.153:0/2599805090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 msgr2=0x7f1e3410f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e38a7b700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 0x7f1e3410f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e38a7b700 1 -- 172.21.15.153:0/2599805090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28005040 con 0x7f1e34101d70 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e38a7b700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b3b0 tx=0x7f1e28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2800e040 con 0x7f1e34101d70 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e2800bbc0 con 0x7f1e34101d70 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2801c630 con 0x7f1e34101d70 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e34117b50 con 0x7f1e34101d70 2024-04-05T03:08:19.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.889+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e34118060 con 0x7f1e34101d70 2024-04-05T03:08:19.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.890+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e340ff080 con 0x7f1e34101d70 2024-04-05T03:08:19.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.891+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f1e28013420 con 0x7f1e34101d70 2024-04-05T03:08:19.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.891+0000 7f1e317fa700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 0x7f1e24070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:19.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.892+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1e2808e8f0 con 0x7f1e34101d70 2024-04-05T03:08:19.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.892+0000 7f1e33fff700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 0x7f1e24070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:19.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.892+0000 7f1e33fff700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 0x7f1e24070270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e34066000 tx=0x7f1e20009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:19.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:19.896+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1e28057e20 con 0x7f1e34101d70 2024-04-05T03:08:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.121+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e3404bf40 con 0x7f1e2406ddc0 2024-04-05T03:08:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.122+0000 7f1e317fa700 1 -- 172.21.15.153:0/2599805090 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1e3404bf40 con 0x7f1e2406ddc0 2024-04-05T03:08:20.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.124+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 msgr2=0x7f1e24070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.124+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 0x7f1e24070270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1e34066000 tx=0x7f1e20009ee0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.124+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 msgr2=0x7f1e3410eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.124+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b3b0 tx=0x7f1e28009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 shutdown_connections 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e2406ddc0 0x7f1e24070270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e34101d70 0x7f1e3410eae0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 --2- 172.21.15.153:0/2599805090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e34102710 0x7f1e3410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 >> 172.21.15.153:0/2599805090 conn(0x7f1e34072bc0 msgr2=0x7f1e340fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 shutdown_connections 2024-04-05T03:08:20.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.125+0000 7f1e3acdf700 1 -- 172.21.15.153:0/2599805090 wait complete. 2024-04-05T03:08:20.135 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.251+0000 7fce7860c700 1 -- 172.21.15.153:0/59333961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 msgr2=0x7fce70102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.251+0000 7fce7860c700 1 --2- 172.21.15.153:0/59333961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70102170 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce6c004940 tx=0x7fce6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 -- 172.21.15.153:0/59333961 shutdown_connections 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 --2- 172.21.15.153:0/59333961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce700fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 --2- 172.21.15.153:0/59333961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70102170 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 -- 172.21.15.153:0/59333961 >> 172.21.15.153:0/59333961 conn(0x7fce700714c0 msgr2=0x7fce700718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 -- 172.21.15.153:0/59333961 shutdown_connections 2024-04-05T03:08:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 -- 172.21.15.153:0/59333961 wait complete. 2024-04-05T03:08:20.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:20 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:08:20.059+0000 7f6f3e1587c0 -1 Falling back to public interface 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 Processor -- start 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.252+0000 7fce7860c700 1 -- start start 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce7860c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce70113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce7860c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce7860c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce70114520 con 0x7fce70101da0 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce7860c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7010db40 con 0x7fce700fc710 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce75ba7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce75ba7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43130/0 (socket says 172.21.15.153:43130) 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce75ba7700 1 -- 172.21.15.153:0/2067827501 learned_addr learned my addr 172.21.15.153:0/2067827501 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce763a8700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce70113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce763a8700 1 -- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 msgr2=0x7fce70113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce763a8700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce763a8700 1 -- 172.21.15.153:0/2067827501 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c005040 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.253+0000 7fce763a8700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce70113990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009fd0 tx=0x7fce6c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.254+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6c00e040 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.254+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce6c01a570 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.254+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce6c013660 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.254+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7010ddc0 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.254+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7010e300 con 0x7fce700fc710 2024-04-05T03:08:20.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.255+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce70065fb0 con 0x7fce700fc710 2024-04-05T03:08:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.255+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fce6c01a6e0 con 0x7fce700fc710 2024-04-05T03:08:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.256+0000 7fce66ffd700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 0x7fce5c070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.256+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce6c092470 con 0x7fce700fc710 2024-04-05T03:08:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.256+0000 7fce75ba7700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 0x7fce5c070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.256+0000 7fce75ba7700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 0x7fce5c070280 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce7010f230 tx=0x7fce6000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.261+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fce6c05b860 con 0x7fce700fc710 2024-04-05T03:08:20.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.491+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce7010f050 con 0x7fce5c06ddd0 2024-04-05T03:08:20.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.492+0000 7fce66ffd700 1 -- 172.21.15.153:0/2067827501 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fce7010f050 con 0x7fce5c06ddd0 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 msgr2=0x7fce5c070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 0x7fce5c070280 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce7010f230 tx=0x7fce6000b040 comp rx=0 tx=0).stop 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 msgr2=0x7fce70113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce70113990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009fd0 tx=0x7fce6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 shutdown_connections 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fce5c06ddd0 0x7fce5c070280 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce700fc710 0x7fce70113990 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 --2- 172.21.15.153:0/2067827501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101da0 0x7fce70113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 >> 172.21.15.153:0/2067827501 conn(0x7fce700714c0 msgr2=0x7fce700fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 shutdown_connections 2024-04-05T03:08:20.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.494+0000 7fce7860c700 1 -- 172.21.15.153:0/2067827501 wait complete. 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- 172.21.15.153:0/3480117062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50301026c0 msgr2=0x7f50300fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3480117062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50301026c0 0x7f50300fbd30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5024004940 tx=0x7f5024009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- 172.21.15.153:0/3480117062 shutdown_connections 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3480117062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50301026c0 0x7f50300fbd30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3480117062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030101d20 0x7f50301020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- 172.21.15.153:0/3480117062 >> 172.21.15.153:0/3480117062 conn(0x7f50300f78d0 msgr2=0x7f50300f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- 172.21.15.153:0/3480117062 shutdown_connections 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- 172.21.15.153:0/3480117062 wait complete. 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 Processor -- start 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.601+0000 7f5035e7a700 1 -- start start 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f5035e7a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f5035e7a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 0x7f50301030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f5035e7a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50301036e0 con 0x7f5030101d20 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f5035e7a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5030103820 con 0x7f50301026c0 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43142/0 (socket says 172.21.15.153:43142) 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 -- 172.21.15.153:0/3954974605 learned_addr learned my addr 172.21.15.153:0/3954974605 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502effd700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 0x7f50301030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 -- 172.21.15.153:0/3954974605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 msgr2=0x7f50301030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 0x7f50301030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 -- 172.21.15.153:0/3954974605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024005040 con 0x7f5030101d20 2024-04-05T03:08:20.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502effd700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 0x7f50301030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:08:20.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f502f7fe700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f502000b790 tx=0x7f5020009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.602+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f502000f770 con 0x7f5030101d20 2024-04-05T03:08:20.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.603+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f502000fda0 con 0x7f5030101d20 2024-04-05T03:08:20.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.603+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5030119ef0 con 0x7f5030101d20 2024-04-05T03:08:20.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.603+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f502000d310 con 0x7f5030101d20 2024-04-05T03:08:20.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.603+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503011a3d0 con 0x7f5030101d20 2024-04-05T03:08:20.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.604+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50300fd470 con 0x7f5030101d20 2024-04-05T03:08:20.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.604+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f502000d5b0 con 0x7f5030101d20 2024-04-05T03:08:20.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.605+0000 7f5017fff700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 0x7f5018070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.605+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f502008d550 con 0x7f5030101d20 2024-04-05T03:08:20.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.605+0000 7f502effd700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 0x7f5018070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.606+0000 7f502effd700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 0x7f5018070270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f50301044c0 tx=0x7f5024009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.610+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5020056c00 con 0x7f5030101d20 2024-04-05T03:08:20.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.822+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5030104270 con 0x7f501806ddc0 2024-04-05T03:08:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:20 smithi153 ceph-mon[106102]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:20 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:20 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:20 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:20.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.832+0000 7f5017fff700 1 -- 172.21.15.153:0/3954974605 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f5030104270 con 0x7f501806ddc0 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (2m) 16s ago 14m 26.6M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (26s) 16s ago 15m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (21s) 17s ago 12m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (2m) 16s ago 13m 81.4M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (8m) 16s ago 8m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (8m) 16s ago 8m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (8m) 17s ago 8m 61.4M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (8m) 17s ago 8m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (4m) 16s ago 16m 494M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (3m) 17s ago 12m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:08:20.832 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (93s) 16s ago 16m 41.3M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (68s) 17s ago 12m 36.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (3m) 16s ago 14m 16.7M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (3m) 17s ago 12m 17.3M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 starting - - - 1898M 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (11m) 16s ago 11m 229M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (11m) 16s ago 11m 200M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (10m) 17s ago 10m 265M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (10m) 17s ago 10m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (9m) 17s ago 9m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:08:20.833 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (3m) 17s ago 12m 60.5M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 msgr2=0x7f5018070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 0x7f5018070270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f50301044c0 tx=0x7f5024009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 msgr2=0x7f5030102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f502000b790 tx=0x7f5020009250 comp rx=0 tx=0).stop 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 shutdown_connections 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f501806ddc0 0x7f5018070270 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5030101d20 0x7f5030102b80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 --2- 172.21.15.153:0/3954974605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50301026c0 0x7f50301030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 >> 172.21.15.153:0/3954974605 conn(0x7f50300f78d0 msgr2=0x7f5030065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 shutdown_connections 2024-04-05T03:08:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.834+0000 7f5035e7a700 1 -- 172.21.15.153:0/3954974605 wait complete. 2024-04-05T03:08:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 -- 172.21.15.153:0/1073590678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 msgr2=0x7fe86c0fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 --2- 172.21.15.153:0/1073590678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c0fce30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe860004940 tx=0x7fe860009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:20.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 -- 172.21.15.153:0/1073590678 shutdown_connections 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 --2- 172.21.15.153:0/1073590678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c0fce30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 --2- 172.21.15.153:0/1073590678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 0x7fe86c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 -- 172.21.15.153:0/1073590678 >> 172.21.15.153:0/1073590678 conn(0x7fe86c0f89b0 msgr2=0x7fe86c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 -- 172.21.15.153:0/1073590678 shutdown_connections 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.939+0000 7fe87127b700 1 -- 172.21.15.153:0/1073590678 wait complete. 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 Processor -- start 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 -- start start 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 0x7fe86c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c072210 con 0x7fe86c1037c0 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe87127b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c070d40 con 0x7fe86c102e20 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43162/0 (socket says 172.21.15.153:43162) 2024-04-05T03:08:20.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 -- 172.21.15.153:0/2469642242 learned_addr learned my addr 172.21.15.153:0/2469642242 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86affd700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 0x7fe86c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 -- 172.21.15.153:0/2469642242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 msgr2=0x7fe86c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 0x7fe86c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 -- 172.21.15.153:0/2469642242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860005040 con 0x7fe86c1037c0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.940+0000 7fe86a7fc700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe860009100 tx=0x7fe860009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.941+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86000e040 con 0x7fe86c1037c0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.941+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c070fc0 con 0x7fe86c1037c0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.941+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c11a0f0 con 0x7fe86c1037c0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.941+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe860009af0 con 0x7fe86c1037c0 2024-04-05T03:08:20.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.941+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe860013d10 con 0x7fe86c1037c0 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.942+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8580052b0 con 0x7fe86c1037c0 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.942+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fe86000baf0 con 0x7fe86c1037c0 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.942+0000 7fe8537fe700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 0x7fe854070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.943+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe86008e930 con 0x7fe86c1037c0 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.943+0000 7fe86affd700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 0x7fe854070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.943+0000 7fe86affd700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 0x7fe854070280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe85c004eb0 tx=0x7fe85c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:20.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:20.947+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe860057dc0 con 0x7fe86c1037c0 2024-04-05T03:08:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:20 smithi203 ceph-mon[95988]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:20 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:20 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:20 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:21.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.257+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe858005830 con 0x7fe86c1037c0 2024-04-05T03:08:21.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.258+0000 7fe8537fe700 1 -- 172.21.15.153:0/2469642242 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+735 (secure 0 0 0) 0x7fe86005b480 con 0x7fe86c1037c0 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 5 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:08:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 9, 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:08:21.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 msgr2=0x7fe854070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 0x7fe854070280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe85c004eb0 tx=0x7fe85c00b040 comp rx=0 tx=0).stop 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 msgr2=0x7fe86c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe860009100 tx=0x7fe860009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 shutdown_connections 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe85406ddd0 0x7fe854070280 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe86c102e20 0x7fe86c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 --2- 172.21.15.153:0/2469642242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe86c1037c0 0x7fe86c070800 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 >> 172.21.15.153:0/2469642242 conn(0x7fe86c0f89b0 msgr2=0x7fe86c1021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 shutdown_connections 2024-04-05T03:08:21.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.260+0000 7fe87127b700 1 -- 172.21.15.153:0/2469642242 wait complete. 2024-04-05T03:08:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 -- 172.21.15.153:0/843435791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95040fb4a0 msgr2=0x7f95040fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 --2- 172.21.15.153:0/843435791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95040fb4a0 0x7f95040fb910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004940 tx=0x7f94f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:21.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 -- 172.21.15.153:0/843435791 shutdown_connections 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 --2- 172.21.15.153:0/843435791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95040fb4a0 0x7f95040fb910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 --2- 172.21.15.153:0/843435791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9504102e70 0x7f9504103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 -- 172.21.15.153:0/843435791 >> 172.21.15.153:0/843435791 conn(0x7f95040714c0 msgr2=0x7f95040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 -- 172.21.15.153:0/843435791 shutdown_connections 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.363+0000 7f950b721700 1 -- 172.21.15.153:0/843435791 wait complete. 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 Processor -- start 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 -- start start 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95040fb4a0 0x7f9504113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504114720 con 0x7f9504102e70 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f950b721700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950410db10 con 0x7f95040fb4a0 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f9508cbc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.364+0000 7f9508cbc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43182/0 (socket says 172.21.15.153:43182) 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f9508cbc700 1 -- 172.21.15.153:0/2310146467 learned_addr learned my addr 172.21.15.153:0/2310146467 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f95094bd700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95040fb4a0 0x7f9504113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f9508cbc700 1 -- 172.21.15.153:0/2310146467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95040fb4a0 msgr2=0x7f9504113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f9508cbc700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95040fb4a0 0x7f9504113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f9508cbc700 1 -- 172.21.15.153:0/2310146467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4005040 con 0x7f9504102e70 2024-04-05T03:08:21.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f9508cbc700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f94f400b3b0 tx=0x7f94f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f400e040 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950410dd90 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94f4013420 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.365+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94f401c630 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.366+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950410e350 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.367+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9504100110 con 0x7f9504102e70 2024-04-05T03:08:21.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.367+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7f94f4013590 con 0x7f9504102e70 2024-04-05T03:08:21.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.368+0000 7f94f9ffb700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 0x7f94f0070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.368+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f94f408e930 con 0x7f9504102e70 2024-04-05T03:08:21.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.368+0000 7f95094bd700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 0x7f94f0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.368+0000 7f95094bd700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 0x7f94f0070270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95000041d0 tx=0x7f95000043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:21.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.376+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f94f405b8f0 con 0x7f9504102e70 2024-04-05T03:08:21.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.628+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f950410f1e0 con 0x7f9504102e70 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.628+0000 7f94f9ffb700 1 -- 172.21.15.153:0/2310146467 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7f94f405b480 con 0x7f9504102e70 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:08:21.628 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:08:21.629 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:21.630 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 msgr2=0x7f94f0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 0x7f94f0070270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95000041d0 tx=0x7f95000043c0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 msgr2=0x7f95041140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f94f400b3b0 tx=0x7f94f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 shutdown_connections 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94f006ddc0 0x7f94f0070270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95040fb4a0 0x7f9504113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 --2- 172.21.15.153:0/2310146467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9504102e70 0x7f95041140d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 >> 172.21.15.153:0/2310146467 conn(0x7f95040714c0 msgr2=0x7f95040fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 shutdown_connections 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.631+0000 7f950b721700 1 -- 172.21.15.153:0/2310146467 wait complete. 2024-04-05T03:08:21.632 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:08:21.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/380951923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8101d70 msgr2=0x7fa2f8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/380951923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8101d70 0x7fa2f8102140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8004940 tx=0x7fa2e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:21.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/380951923 shutdown_connections 2024-04-05T03:08:21.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/380951923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8102710 0x7fa2f80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/380951923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8101d70 0x7fa2f8102140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/380951923 >> 172.21.15.153:0/380951923 conn(0x7fa2f8072bc0 msgr2=0x7fa2f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:21.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/380951923 shutdown_connections 2024-04-05T03:08:21.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.737+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/380951923 wait complete. 2024-04-05T03:08:21.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 Processor -- start 2024-04-05T03:08:21.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 -- start start 2024-04-05T03:08:21.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8101d70 0x7fa2f810eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f810f6d0 con 0x7fa2f8102710 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2ffbd9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f810f810 con 0x7fa2f8101d70 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2fd174700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2fd174700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43206/0 (socket says 172.21.15.153:43206) 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2fd174700 1 -- 172.21.15.153:0/4264044739 learned_addr learned my addr 172.21.15.153:0/4264044739 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.738+0000 7fa2fd975700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8101d70 0x7fa2f810eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2fd174700 1 -- 172.21.15.153:0/4264044739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8101d70 msgr2=0x7fa2f810eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2fd174700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8101d70 0x7fa2f810eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2fd174700 1 -- 172.21.15.153:0/4264044739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8005040 con 0x7fa2f8102710 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2fd174700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b790 tx=0x7fa2f4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f400f6f0 con 0x7fa2f8102710 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2f400fd20 con 0x7fa2f8102710 2024-04-05T03:08:21.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f8117b80 con 0x7fa2f8102710 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2f40103a0 con 0x7fa2f8102710 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.739+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f8118090 con 0x7fa2f8102710 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.740+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f80ff080 con 0x7fa2f8102710 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.740+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fa2f4011400 con 0x7fa2f8102710 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.741+0000 7fa2ee7fc700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 0x7fa2e4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.741+0000 7fa2fd975700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 0x7fa2e4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:21.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.741+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa2f408d400 con 0x7fa2f8102710 2024-04-05T03:08:21.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.741+0000 7fa2fd975700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 0x7fa2e4070270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8004f90 tx=0x7fa2e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:21.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.745+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa2f40569a0 con 0x7fa2f8102710 2024-04-05T03:08:21.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.967+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2f8108f30 con 0x7fa2e406ddc0 2024-04-05T03:08:21.967 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:21 smithi153 ceph-mon[106102]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:21.967 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:21 smithi153 ceph-mon[106102]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:21.967 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:21 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2469642242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:21.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.968+0000 7fa2ee7fc700 1 -- 172.21.15.153:0/4264044739 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa2f8108f30 con 0x7fa2e406ddc0 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "6/21 daemons upgraded", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:08:21.968 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:08:21.969 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 msgr2=0x7fa2e4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 0x7fa2e4070270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8004f90 tx=0x7fa2e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 msgr2=0x7fa2f810eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b790 tx=0x7fa2f4009220 comp rx=0 tx=0).stop 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 shutdown_connections 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa2e406ddc0 0x7fa2e4070270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f8101d70 0x7fa2f810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 --2- 172.21.15.153:0/4264044739 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2f8102710 0x7fa2f810eff0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 >> 172.21.15.153:0/4264044739 conn(0x7fa2f8072bc0 msgr2=0x7fa2f80fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 shutdown_connections 2024-04-05T03:08:21.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:21.970+0000 7fa2ffbd9700 1 -- 172.21.15.153:0/4264044739 wait complete. 2024-04-05T03:08:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:21 smithi203 ceph-mon[95988]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:21 smithi203 ceph-mon[95988]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:22.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:21 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2469642242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 -- 172.21.15.153:0/199825875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd574102e20 msgr2=0x7fd5741031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/199825875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd574102e20 0x7fd5741031f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd564004940 tx=0x7fd564009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 -- 172.21.15.153:0/199825875 shutdown_connections 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/199825875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5741037c0 0x7fd5740fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/199825875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd574102e20 0x7fd5741031f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 -- 172.21.15.153:0/199825875 >> 172.21.15.153:0/199825875 conn(0x7fd5740f8970 msgr2=0x7fd5740fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 -- 172.21.15.153:0/199825875 shutdown_connections 2024-04-05T03:08:22.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.073+0000 7fd57aa5a700 1 -- 172.21.15.153:0/199825875 wait complete. 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 Processor -- start 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 -- start start 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 0x7fd574113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd574114700 con 0x7fd5741037c0 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.074+0000 7fd57aa5a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57410daf0 con 0x7fd574102e20 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43224/0 (socket says 172.21.15.153:43224) 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 -- 172.21.15.153:0/2423752060 learned_addr learned my addr 172.21.15.153:0/2423752060 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd573fff700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 0x7fd574113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 -- 172.21.15.153:0/2423752060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 msgr2=0x7fd574113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 0x7fd574113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 -- 172.21.15.153:0/2423752060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564005040 con 0x7fd5741037c0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd573fff700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 0x7fd574113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd5737fe700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd568002a50 tx=0x7fd568004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd568007cc0 con 0x7fd5741037c0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57410ddd0 con 0x7fd5741037c0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57410e390 con 0x7fd5741037c0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56800f040 con 0x7fd5741037c0 2024-04-05T03:08:22.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.075+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56800daf0 con 0x7fd5741037c0 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.076+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd574100130 con 0x7fd5741037c0 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.077+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 39) v1 ==== 91963+0+0 (secure 0 0 0) 0x7fd568009810 con 0x7fd5741037c0 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.077+0000 7fd570ff9700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 0x7fd55c070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.077+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd56808d400 con 0x7fd5741037c0 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.077+0000 7fd573fff700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 0x7fd55c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:22.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.078+0000 7fd573fff700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 0x7fd55c070270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd564004f90 tx=0x7fd564009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:22.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.082+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd56805a3c0 con 0x7fd5741037c0 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.375+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd5740fe570 con 0x7fd5741037c0 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.375+0000 7fd570ff9700 1 -- 172.21.15.153:0/2423752060 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7fd568059f50 con 0x7fd5741037c0 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 62/318 objects degraded (19.497%), 19 pgs degraded 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout: osd.0 (root=default,host=smithi153) is down 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 62/318 objects degraded (19.497%), 19 pgs degraded 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-05T03:08:22.375 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-05T03:08:22.376 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 msgr2=0x7fd55c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 0x7fd55c070270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd564004f90 tx=0x7fd564009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 msgr2=0x7fd5741140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd568002a50 tx=0x7fd568004600 comp rx=0 tx=0).stop 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 shutdown_connections 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd55c06ddc0 0x7fd55c070270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd574102e20 0x7fd574113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 --2- 172.21.15.153:0/2423752060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5741037c0 0x7fd5741140b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 >> 172.21.15.153:0/2423752060 conn(0x7fd5740f8970 msgr2=0x7fd5741073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 shutdown_connections 2024-04-05T03:08:22.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:22.378+0000 7fd57aa5a700 1 -- 172.21.15.153:0/2423752060 wait complete. 2024-04-05T03:08:22.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:22 smithi153 ceph-mon[106102]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:22.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:22 smithi153 ceph-mon[106102]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/318 objects degraded (19.497%) 2024-04-05T03:08:22.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2310146467' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:08:22.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2423752060' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:08:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:22 smithi203 ceph-mon[95988]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:22 smithi203 ceph-mon[95988]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 62/318 objects degraded (19.497%) 2024-04-05T03:08:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2310146467' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:08:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2423752060' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:08:24.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:23 smithi203 ceph-mon[95988]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:24.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:23 smithi153 ceph-mon[106102]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:24 smithi203 ceph-mon[95988]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:24 smithi153 ceph-mon[106102]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:25.072 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:24 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:08:24.824+0000 7f6f3e1587c0 -1 osd.0 48 log_to_monitors true 2024-04-05T03:08:25.865 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:08:25 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:08:25.603+0000 7f6f2f5b0700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:08:25.865 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:25 smithi153 ceph-mon[106102]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T03:08:25.865 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:25.865 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:25 smithi203 ceph-mon[95988]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T03:08:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:26 smithi153 ceph-mon[106102]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:26 smithi153 ceph-mon[106102]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T03:08:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:26 smithi153 ceph-mon[106102]: failed to encode map e51 with expected crc 2024-04-05T03:08:26.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:26 smithi153 ceph-mon[106102]: osdmap e51: 6 total, 5 up, 6 in 2024-04-05T03:08:26.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:26 smithi153 ceph-mon[106102]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:08:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:26 smithi203 ceph-mon[95988]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:26 smithi203 ceph-mon[95988]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T03:08:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:26 smithi203 ceph-mon[95988]: failed to encode map e51 with expected crc 2024-04-05T03:08:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:26 smithi203 ceph-mon[95988]: osdmap e51: 6 total, 5 up, 6 in 2024-04-05T03:08:27.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:26 smithi203 ceph-mon[95988]: from='osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:08:27.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: failed to encode map e52 with expected crc 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135] boot 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: osdmap e52: 6 total, 6 up, 6 in 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:27.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: failed to encode map e52 with expected crc 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: osd.0 [v2:172.21.15.153:6802/190885135,v1:172.21.15.153:6803/190885135] boot 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: osdmap e52: 6 total, 6 up, 6 in 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:28.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: osdmap e53: 6 total, 6 up, 6 in 2024-04-05T03:08:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:28.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:28.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:28.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:28.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:28 smithi153 ceph-mon[106102]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 62/318 objects degraded (19.497%) 2024-04-05T03:08:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: osdmap e53: 6 total, 6 up, 6 in 2024-04-05T03:08:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:29.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:28 smithi203 ceph-mon[95988]: failed to encode map e53 with expected crc 2024-04-05T03:08:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:30.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-05T03:08:30.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:30 smithi203 ceph-mon[95988]: pgmap v54: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s; 11/318 objects degraded (3.459%) 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:30.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:30.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:30.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-05T03:08:30.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:30 smithi153 ceph-mon[106102]: pgmap v54: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s; 11/318 objects degraded (3.459%) 2024-04-05T03:08:31.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:31 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 11/318 objects degraded (3.459%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:31.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:31 smithi153 ceph-mon[106102]: mgrmap e40: smithi153.npzyqy(active, since 92s), standbys: smithi203.iakodz 2024-04-05T03:08:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:31 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 11/318 objects degraded (3.459%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:31.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:31 smithi203 ceph-mon[95988]: mgrmap e40: smithi153.npzyqy(active, since 92s), standbys: smithi203.iakodz 2024-04-05T03:08:32.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:32 smithi203 ceph-mon[95988]: pgmap v55: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-05T03:08:32.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:32 smithi153 ceph-mon[106102]: pgmap v55: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-05T03:08:33.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:33 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/318 objects degraded (3.459%), 3 pgs degraded) 2024-04-05T03:08:33.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:33 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:08:33.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:33 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/318 objects degraded (3.459%), 3 pgs degraded) 2024-04-05T03:08:33.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:33 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:08:34.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:34 smithi203 ceph-mon[95988]: pgmap v56: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 521 B/s wr, 5 op/s 2024-04-05T03:08:34.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:34 smithi153 ceph-mon[106102]: pgmap v56: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.8 KiB/s rd, 521 B/s wr, 5 op/s 2024-04-05T03:08:36.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:36 smithi203 ceph-mon[95988]: pgmap v57: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 462 B/s wr, 4 op/s 2024-04-05T03:08:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:36 smithi153 ceph-mon[106102]: pgmap v57: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 462 B/s wr, 4 op/s 2024-04-05T03:08:38.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:38 smithi203 ceph-mon[95988]: pgmap v58: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 409 B/s wr, 4 op/s 2024-04-05T03:08:38.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:38 smithi153 ceph-mon[106102]: pgmap v58: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 409 B/s wr, 4 op/s 2024-04-05T03:08:38.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:40.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:40 smithi203 ceph-mon[95988]: pgmap v59: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 345 B/s wr, 5 op/s 2024-04-05T03:08:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:40 smithi153 ceph-mon[106102]: pgmap v59: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 345 B/s wr, 5 op/s 2024-04-05T03:08:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:42 smithi203 ceph-mon[95988]: pgmap v60: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T03:08:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:42.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:42 smithi153 ceph-mon[106102]: pgmap v60: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T03:08:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:44.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:44 smithi203 ceph-mon[95988]: pgmap v61: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-05T03:08:44.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:44 smithi153 ceph-mon[106102]: pgmap v61: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-05T03:08:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:45 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:45 smithi203 ceph-mon[95988]: Upgrade: osd.1 is safe to restart 2024-04-05T03:08:45.819 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:45 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T03:08:45.819 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:45 smithi153 ceph-mon[106102]: Upgrade: osd.1 is safe to restart 2024-04-05T03:08:46.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:46 smithi153 ceph-mon[106102]: pgmap v62: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:46.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:46.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T03:08:46.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:46 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:46 smithi203 ceph-mon[95988]: pgmap v62: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T03:08:46.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:46 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:08:47.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:47 smithi153 ceph-mon[106102]: Upgrade: Updating osd.1 2024-04-05T03:08:47.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:47 smithi153 ceph-mon[106102]: Deploying daemon osd.1 on smithi153 2024-04-05T03:08:47.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:47 smithi203 ceph-mon[95988]: Upgrade: Updating osd.1 2024-04-05T03:08:47.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:47 smithi203 ceph-mon[95988]: Deploying daemon osd.1 on smithi153 2024-04-05T03:08:48.675 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:48 smithi153 ceph-mon[106102]: pgmap v63: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:48 smithi203 ceph-mon[95988]: pgmap v63: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:08:49.305 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:49 smithi153 systemd[1]: Stopping Ceph osd.1 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:08:49.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:49 smithi153 ceph-mon[106102]: osd.1 marked itself down and dead 2024-04-05T03:08:49.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:49 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[54573]: 2024-04-05T03:08:49.305+0000 7f3819ff4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:08:49.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:49 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[54573]: 2024-04-05T03:08:49.305+0000 7f3819ff4700 -1 osd.1 53 *** Got signal Terminated *** 2024-04-05T03:08:49.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:49 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[54573]: 2024-04-05T03:08:49.305+0000 7f3819ff4700 -1 osd.1 53 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:08:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:49 smithi203 ceph-mon[95988]: osd.1 marked itself down and dead 2024-04-05T03:08:50.289 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:50 smithi153 podman[122190]: 2024-04-05 03:08:50.030554074 +0000 UTC m=+0.845400783 container died 2aaad36455b5e9fa95c72e39ac9261e0261951cddaca6203e2916a752c055836 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, RELEASE=quincy-6fda41f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-05T03:08:50.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:50 smithi153 podman[122190]: 2024-04-05 03:08:50.438872056 +0000 UTC m=+1.253718764 container cleanup 2aaad36455b5e9fa95c72e39ac9261e0261951cddaca6203e2916a752c055836 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-6fda41f) 2024-04-05T03:08:50.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:50 smithi153 bash[122190]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1 2024-04-05T03:08:50.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: pgmap v64: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:50.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:08:50.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:50 smithi153 ceph-mon[106102]: osdmap e54: 6 total, 5 up, 6 in 2024-04-05T03:08:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: pgmap v64: 65 pgs: 65 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:08:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:08:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: failed to encode map e54 with expected crc 2024-04-05T03:08:50.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:50 smithi203 ceph-mon[95988]: osdmap e54: 6 total, 5 up, 6 in 2024-04-05T03:08:50.982 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:50 smithi153 podman[122207]: 2024-04-05 03:08:50.657347119 +0000 UTC m=+0.624771397 container remove 2aaad36455b5e9fa95c72e39ac9261e0261951cddaca6203e2916a752c055836 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-6fda41f) 2024-04-05T03:08:51.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:50 smithi153 podman[122302]: 2024-04-05 03:08:50.88678574 +0000 UTC m=+0.018162724 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:51.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:51 smithi153 podman[122302]: 2024-04-05 03:08:51.019149137 +0000 UTC m=+0.150526121 container create bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-0028525) 2024-04-05T03:08:51.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:51 smithi153 podman[122302]: 2024-04-05 03:08:51.227770509 +0000 UTC m=+0.359147504 container init bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-05T03:08:51.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:51 smithi153 podman[122302]: 2024-04-05 03:08:51.231263049 +0000 UTC m=+0.362640043 container start bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-05T03:08:51.310 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:51 smithi153 podman[122302]: 2024-04-05 03:08:51.311076965 +0000 UTC m=+0.442453965 container attach bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-0028525) 2024-04-05T03:08:51.572 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:51 smithi153 podman[122302]: 2024-04-05 03:08:51.524252798 +0000 UTC m=+0.655629783 container died bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-0028525) 2024-04-05T03:08:51.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: osdmap e55: 6 total, 5 up, 6 in 2024-04-05T03:08:51.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:51 smithi153 ceph-mon[106102]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: osdmap e55: 6 total, 5 up, 6 in 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:51.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:51 smithi203 ceph-mon[95988]: failed to encode map e55 with expected crc 2024-04-05T03:08:52.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122328]: 2024-04-05 03:08:52.123586703 +0000 UTC m=+0.587133713 container remove bb3f63b4b0968971dcd57451d1b3bdb332760c523441d582085f468f27bc2bcf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-0028525, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-05T03:08:52.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-04-05T03:08:52.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 systemd[1]: Stopped Ceph osd.1 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:08:52.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.1.service: Consumed 1min 7.242s CPU time. 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.499+0000 7f1eedd79700 1 -- 172.21.15.153:0/2223688836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8102e70 msgr2=0x7f1ee8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.499+0000 7f1eedd79700 1 --2- 172.21.15.153:0/2223688836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8102e70 0x7f1ee8103240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8004940 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 -- 172.21.15.153:0/2223688836 shutdown_connections 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 --2- 172.21.15.153:0/2223688836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 --2- 172.21.15.153:0/2223688836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee8102e70 0x7f1ee8103240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 -- 172.21.15.153:0/2223688836 >> 172.21.15.153:0/2223688836 conn(0x7f1ee80714c0 msgr2=0x7f1ee80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 -- 172.21.15.153:0/2223688836 shutdown_connections 2024-04-05T03:08:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 -- 172.21.15.153:0/2223688836 wait complete. 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 Processor -- start 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 -- start start 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.500+0000 7f1eedd79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1eedd79700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8102e70 0x7f1ee80fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1eedd79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee80fd150 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1eedd79700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee80fd290 con 0x7f1ee8102e70 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54950/0 (socket says 172.21.15.153:54950) 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 -- 172.21.15.153:0/3053078609 learned_addr learned my addr 172.21.15.153:0/3053078609 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 -- 172.21.15.153:0/3053078609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8102e70 msgr2=0x7f1ee80fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8102e70 0x7f1ee80fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 -- 172.21.15.153:0/3053078609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8005040 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ee77fe700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800b3b0 tx=0x7f1ed8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed800e040 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ed800bbc0 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee80fd4e0 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ed801c3f0 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.501+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee811a280 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.502+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8100110 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.503+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1ed800bd30 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.503+0000 7f1ecffff700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 0x7f1ed0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.504+0000 7f1ee6ffd700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 0x7f1ed0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.504+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1ed808e680 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.504+0000 7f1ee6ffd700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 0x7f1ed0070150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80fdfc0 tx=0x7f1edc009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.508+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ed8057d30 con 0x7f1ee80fb4a0 2024-04-05T03:08:52.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:52 smithi153 ceph-mon[106102]: pgmap v67: 65 pgs: 2 active+undersized+degraded, 9 stale+active+clean, 3 active+undersized, 51 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/318 objects degraded (1.887%) 2024-04-05T03:08:52.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:52 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 6/318 objects degraded (1.887%), 2 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:52.691 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 systemd[1]: Starting Ceph osd.1 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:08:52.692 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122431]: 2024-04-05 03:08:52.484868612 +0000 UTC m=+0.019465360 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:52.692 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122431]: 2024-04-05 03:08:52.591354049 +0000 UTC m=+0.125950788 container create c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-05T03:08:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.731+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee80fde10 con 0x7f1ed006dca0 2024-04-05T03:08:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.732+0000 7f1ecffff700 1 -- 172.21.15.153:0/3053078609 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1ee80fde10 con 0x7f1ed006dca0 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 msgr2=0x7f1ed0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 0x7f1ed0070150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80fdfc0 tx=0x7f1edc009ce0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 msgr2=0x7f1ee80fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800b3b0 tx=0x7f1ed8009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 shutdown_connections 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ed006dca0 0x7f1ed0070150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ee80fb4a0 0x7f1ee80fc680 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 --2- 172.21.15.153:0/3053078609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8102e70 0x7f1ee80fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 >> 172.21.15.153:0/3053078609 conn(0x7f1ee80714c0 msgr2=0x7f1ee80fa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 shutdown_connections 2024-04-05T03:08:52.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.735+0000 7f1eedd79700 1 -- 172.21.15.153:0/3053078609 wait complete. 2024-04-05T03:08:52.745 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:08:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:52 smithi203 ceph-mon[95988]: pgmap v67: 65 pgs: 2 active+undersized+degraded, 9 stale+active+clean, 3 active+undersized, 51 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6/318 objects degraded (1.887%) 2024-04-05T03:08:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:52 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 6/318 objects degraded (1.887%), 2 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 -- 172.21.15.153:0/1012132649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c481037c0 msgr2=0x7f2c480fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 --2- 172.21.15.153:0/1012132649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c481037c0 0x7f2c480fce30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c38004940 tx=0x7f2c38009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 -- 172.21.15.153:0/1012132649 shutdown_connections 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 --2- 172.21.15.153:0/1012132649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c481037c0 0x7f2c480fce30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 --2- 172.21.15.153:0/1012132649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c48102e20 0x7f2c481031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 -- 172.21.15.153:0/1012132649 >> 172.21.15.153:0/1012132649 conn(0x7f2c480f89d0 msgr2=0x7f2c480fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 -- 172.21.15.153:0/1012132649 shutdown_connections 2024-04-05T03:08:52.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.843+0000 7f2c50585700 1 -- 172.21.15.153:0/1012132649 wait complete. 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 Processor -- start 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 -- start start 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c481037c0 0x7f2c48070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48070d90 con 0x7f2c48102e20 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c50585700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48070ed0 con 0x7f2c481037c0 2024-04-05T03:08:52.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c4e321700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c4e321700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54976/0 (socket says 172.21.15.153:54976) 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c4e321700 1 -- 172.21.15.153:0/1183928308 learned_addr learned my addr 172.21.15.153:0/1183928308 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.844+0000 7f2c4db20700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c481037c0 0x7f2c48070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c4e321700 1 -- 172.21.15.153:0/1183928308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c481037c0 msgr2=0x7f2c48070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c4e321700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c481037c0 0x7f2c48070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c4e321700 1 -- 172.21.15.153:0/1183928308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38005040 con 0x7f2c48102e20 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c4e321700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c44002a50 tx=0x7f2c440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c44007cc0 con 0x7f2c48102e20 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c4400f040 con 0x7f2c48102e20 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4400da30 con 0x7f2c48102e20 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c48119ef0 con 0x7f2c48102e20 2024-04-05T03:08:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.845+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4811a310 con 0x7f2c48102e20 2024-04-05T03:08:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.846+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c481052f0 con 0x7f2c48102e20 2024-04-05T03:08:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.847+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2c4400db90 con 0x7f2c48102e20 2024-04-05T03:08:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.848+0000 7f2c3effd700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 0x7f2c34070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.848+0000 7f2c4db20700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 0x7f2c34070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.848+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2c4408d110 con 0x7f2c48102e20 2024-04-05T03:08:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.848+0000 7f2c4db20700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 0x7f2c34070150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c38004890 tx=0x7f2c38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:52.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:52.853+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2c440567c0 con 0x7f2c48102e20 2024-04-05T03:08:53.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.072+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c48071ab0 con 0x7f2c3406dca0 2024-04-05T03:08:53.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122431]: 2024-04-05 03:08:52.769297246 +0000 UTC m=+0.303893993 container init c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-05T03:08:53.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122431]: 2024-04-05 03:08:52.773031699 +0000 UTC m=+0.307628445 container start c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-05T03:08:53.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:52 smithi153 podman[122431]: 2024-04-05 03:08:52.852181743 +0000 UTC m=+0.386778482 container attach c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:08:53.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.073+0000 7f2c3effd700 1 -- 172.21.15.153:0/1183928308 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2c3800b5c0 con 0x7f2c3406dca0 2024-04-05T03:08:53.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 msgr2=0x7f2c34070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 0x7f2c34070150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c38004890 tx=0x7f2c38009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 msgr2=0x7f2c480702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c44002a50 tx=0x7f2c440046b0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 shutdown_connections 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2c3406dca0 0x7f2c34070150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c48102e20 0x7f2c480702c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 --2- 172.21.15.153:0/1183928308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c481037c0 0x7f2c48070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 >> 172.21.15.153:0/1183928308 conn(0x7f2c480f89d0 msgr2=0x7f2c48102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 shutdown_connections 2024-04-05T03:08:53.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.076+0000 7f2c50585700 1 -- 172.21.15.153:0/1183928308 wait complete. 2024-04-05T03:08:53.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 -- 172.21.15.153:0/2822440989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 msgr2=0x7f94d41031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/2822440989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d41031f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94c4004940 tx=0x7f94c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 -- 172.21.15.153:0/2822440989 shutdown_connections 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/2822440989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d40fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/2822440989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d41031f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.183+0000 7f94dbcc2700 1 -- 172.21.15.153:0/2822440989 >> 172.21.15.153:0/2822440989 conn(0x7f94d40f89d0 msgr2=0x7f94d40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 -- 172.21.15.153:0/2822440989 shutdown_connections 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 -- 172.21.15.153:0/2822440989 wait complete. 2024-04-05T03:08:53.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 Processor -- start 2024-04-05T03:08:53.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 -- start start 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d4114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d41146b0 con 0x7f94d4102e20 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94dbcc2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410daf0 con 0x7f94d41037c0 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94d9a5e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94d9a5e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55006/0 (socket says 172.21.15.153:55006) 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.184+0000 7f94d9a5e700 1 -- 172.21.15.153:0/3346111729 learned_addr learned my addr 172.21.15.153:0/3346111729 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d925d700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d4114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d9a5e700 1 -- 172.21.15.153:0/3346111729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 msgr2=0x7f94d4114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d9a5e700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d4114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d9a5e700 1 -- 172.21.15.153:0/3346111729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4005040 con 0x7f94d4102e20 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d925d700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:08:53.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94d9a5e700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f94c400b8b0 tx=0x7f94c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94c400e040 con 0x7f94d4102e20 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94c4004290 con 0x7f94d4102e20 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d410dd70 con 0x7f94d4102e20 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94c401c3f0 con 0x7f94d4102e20 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.185+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d410e330 con 0x7f94d4102e20 2024-04-05T03:08:53.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.186+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d4100130 con 0x7f94d4102e20 2024-04-05T03:08:53.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.187+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f94c40137b0 con 0x7f94d4102e20 2024-04-05T03:08:53.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.187+0000 7f94ca7fc700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 0x7f94c0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.187+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f94c408e720 con 0x7f94d4102e20 2024-04-05T03:08:53.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.188+0000 7f94d925d700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 0x7f94c0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.188+0000 7f94d925d700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 0x7f94c0070150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009b80 tx=0x7f94d000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.192+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f94c405b9a0 con 0x7f94d4102e20 2024-04-05T03:08:53.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.405+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f94d410f1e0 con 0x7f94c006dca0 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 bash[122431]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T03:08:53.406 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-05T03:08:53.407 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T03:08:53.407 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate[122466]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-05T03:08:53.407 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 podman[122431]: 2024-04-05 03:08:53.256171869 +0000 UTC m=+0.790768624 container died c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-05T03:08:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.416+0000 7f94ca7fc700 1 -- 172.21.15.153:0/3346111729 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f94d410f1e0 con 0x7f94c006dca0 2024-04-05T03:08:53.415 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:08:53.415 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (3m) 29s ago 14m 26.6M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:08:53.415 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (59s) 29s ago 15m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:08:53.415 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (54s) 49s ago 13m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (3m) 29s ago 14m 80.3M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (9m) 29s ago 9m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (9m) 29s ago 9m 61.0M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (8m) 49s ago 8m 61.4M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (9m) 49s ago 9m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (4m) 29s ago 16m 497M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (4m) 49s ago 13m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (2m) 29s ago 17m 42.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (100s) 49s ago 13m 36.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (4m) 29s ago 14m 16.4M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (4m) 49s ago 13m 17.3M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (34s) 29s ago 12m 15.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (12m) 29s ago 12m 229M 1898M 17.2.7-805-g6fda41ff a49113eacc58 2aaad36455b5 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (11m) 29s ago 11m 199M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (11m) 49s ago 11m 265M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:08:53.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (10m) 49s ago 10m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:08:53.417 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (10m) 49s ago 10m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:08:53.417 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (3m) 49s ago 13m 60.5M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 msgr2=0x7f94c0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 0x7f94c0070150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009b80 tx=0x7f94d000b040 comp rx=0 tx=0).stop 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 msgr2=0x7f94d4113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f94c400b8b0 tx=0x7f94c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 shutdown_connections 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f94c006dca0 0x7f94c0070150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94d4102e20 0x7f94d4113b20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 --2- 172.21.15.153:0/3346111729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94d41037c0 0x7f94d4114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 >> 172.21.15.153:0/3346111729 conn(0x7f94d40f89d0 msgr2=0x7f94d41073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 shutdown_connections 2024-04-05T03:08:53.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.418+0000 7f94dbcc2700 1 -- 172.21.15.153:0/3346111729 wait complete. 2024-04-05T03:08:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- 172.21.15.153:0/2738906943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fde80 msgr2=0x7fa6880fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 --2- 172.21.15.153:0/2738906943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fde80 0x7fa6880fe250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa684004940 tx=0x7fa684009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- 172.21.15.153:0/2738906943 shutdown_connections 2024-04-05T03:08:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 --2- 172.21.15.153:0/2738906943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fe820 0x7fa688106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 --2- 172.21.15.153:0/2738906943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fde80 0x7fa6880fe250 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- 172.21.15.153:0/2738906943 >> 172.21.15.153:0/2738906943 conn(0x7fa6880f7870 msgr2=0x7fa6880f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- 172.21.15.153:0/2738906943 shutdown_connections 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- 172.21.15.153:0/2738906943 wait complete. 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 Processor -- start 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- start start 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fde80 0x7fa688113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6881144a0 con 0x7fa6880fe820 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.522+0000 7fa690557700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68810daa0 con 0x7fa6880fde80 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55020/0 (socket says 172.21.15.153:55020) 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 -- 172.21.15.153:0/2046160905 learned_addr learned my addr 172.21.15.153:0/2046160905 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68e2f3700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fde80 0x7fa688113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 -- 172.21.15.153:0/2046160905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fde80 msgr2=0x7fa688113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fde80 0x7fa688113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 -- 172.21.15.153:0/2046160905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684005040 con 0x7fa6880fe820 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa68daf2700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa67800c8c0 tx=0x7fa6780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67800a980 con 0x7fa6880fe820 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68810dd80 con 0x7fa6880fe820 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa67800ce60 con 0x7fa6880fe820 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6780115e0 con 0x7fa6880fe820 2024-04-05T03:08:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.523+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68810e2c0 con 0x7fa6880fe820 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.524+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa688066b10 con 0x7fa6880fe820 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.525+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa678012400 con 0x7fa6880fe820 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.525+0000 7fa67effd700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 0x7fa674070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.525+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa67808d5c0 con 0x7fa6880fe820 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.525+0000 7fa68e2f3700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 0x7fa674070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.526+0000 7fa68e2f3700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 0x7fa674070150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa684004f90 tx=0x7fa684009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.530+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa67805a7a0 con 0x7fa6880fe820 2024-04-05T03:08:53.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.812+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa688109e40 con 0x7fa6880fe820 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.812+0000 7fa67effd700 1 -- 172.21.15.153:0/2046160905 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fa6780126b0 con 0x7fa6880fe820 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:08:53.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4, 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 8, 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 5 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:08:53.813 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 msgr2=0x7fa674070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 0x7fa674070150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa684004f90 tx=0x7fa684009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 msgr2=0x7fa688113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa67800c8c0 tx=0x7fa6780044c0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 shutdown_connections 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa67406dca0 0x7fa674070150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6880fde80 0x7fa688113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 --2- 172.21.15.153:0/2046160905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880fe820 0x7fa688113e50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 >> 172.21.15.153:0/2046160905 conn(0x7fa6880f7870 msgr2=0x7fa6880f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 shutdown_connections 2024-04-05T03:08:53.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.815+0000 7fa690557700 1 -- 172.21.15.153:0/2046160905 wait complete. 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 -- 172.21.15.153:0/16664873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4102e10 msgr2=0x7fa4a41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 --2- 172.21.15.153:0/16664873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4102e10 0x7fa4a41031e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa494004940 tx=0x7fa494009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 -- 172.21.15.153:0/16664873 shutdown_connections 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 --2- 172.21.15.153:0/16664873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a41037b0 0x7fa4a40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 --2- 172.21.15.153:0/16664873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4102e10 0x7fa4a41031e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 -- 172.21.15.153:0/16664873 >> 172.21.15.153:0/16664873 conn(0x7fa4a40f8980 msgr2=0x7fa4a40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 -- 172.21.15.153:0/16664873 shutdown_connections 2024-04-05T03:08:53.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.928+0000 7fa4abe18700 1 -- 172.21.15.153:0/16664873 wait complete. 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 Processor -- start 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 -- start start 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a4102e10 0x7fa4a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4070d90 con 0x7fa4a41037b0 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4abe18700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4070ed0 con 0x7fa4a4102e10 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a93b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a93b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55052/0 (socket says 172.21.15.153:55052) 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a93b3700 1 -- 172.21.15.153:0/4245358255 learned_addr learned my addr 172.21.15.153:0/4245358255 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a9bb4700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a4102e10 0x7fa4a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a93b3700 1 -- 172.21.15.153:0/4245358255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a4102e10 msgr2=0x7fa4a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.929+0000 7fa4a93b3700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a4102e10 0x7fa4a40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa4a93b3700 1 -- 172.21.15.153:0/4245358255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494005040 con 0x7fa4a41037b0 2024-04-05T03:08:53.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa4a93b3700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c890 tx=0x7fa4a00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a000a700 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4119ee0 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4a000ad30 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a00113a0 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.930+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a411a360 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.931+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a4100120 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.932+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa4a0012400 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.932+0000 7fa49a7fc700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 0x7fa490070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.932+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa4a008d450 con 0x7fa4a41037b0 2024-04-05T03:08:53.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.932+0000 7fa4a9bb4700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 0x7fa490070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:53.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.933+0000 7fa4a9bb4700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 0x7fa490070150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa494004f90 tx=0x7fa494009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:53.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:53.937+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa4a0056ba0 con 0x7fa4a41037b0 2024-04-05T03:08:54.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:53 smithi153 podman[122637]: 2024-04-05 03:08:53.831183539 +0000 UTC m=+0.566083280 container remove c3d270705d95dca3afbced62a30c779bf11b13005b978d2b56bd4aa333ea0fd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-activate, RELEASE=squid-0028525, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-05T03:08:54.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 podman[122727]: 2024-04-05 03:08:54.040574188 +0000 UTC m=+0.112615897 container create 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True) 2024-04-05T03:08:54.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 podman[122727]: 2024-04-05 03:08:53.949525268 +0000 UTC m=+0.021567002 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:08:54.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.185+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa4a4109f10 con 0x7fa4a41037b0 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.186+0000 7fa49a7fc700 1 -- 172.21.15.153:0/4245358255 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7fa4a005a260 con 0x7fa4a41037b0 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:08:54.186 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:08:54.187 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:08:54.188 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:54.189 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 msgr2=0x7fa490070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 0x7fa490070150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa494004f90 tx=0x7fa494009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 msgr2=0x7fa4a4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c890 tx=0x7fa4a00044c0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 shutdown_connections 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa49006dca0 0x7fa490070150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a4102e10 0x7fa4a40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 --2- 172.21.15.153:0/4245358255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a41037b0 0x7fa4a4070800 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.188+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 >> 172.21.15.153:0/4245358255 conn(0x7fa4a40f8980 msgr2=0x7fa4a4107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.189+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 shutdown_connections 2024-04-05T03:08:54.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.189+0000 7fa4abe18700 1 -- 172.21.15.153:0/4245358255 wait complete. 2024-04-05T03:08:54.191 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 -- 172.21.15.153:0/4043723347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0102710 msgr2=0x7fa7d00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/4043723347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0102710 0x7fa7d00fbd80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4004940 tx=0x7fa7c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 -- 172.21.15.153:0/4043723347 shutdown_connections 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/4043723347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0102710 0x7fa7d00fbd80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/4043723347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0101d70 0x7fa7d0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 -- 172.21.15.153:0/4043723347 >> 172.21.15.153:0/4043723347 conn(0x7fa7d0072bc0 msgr2=0x7fa7d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 -- 172.21.15.153:0/4043723347 shutdown_connections 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 -- 172.21.15.153:0/4043723347 wait complete. 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.293+0000 7fa7d4fad700 1 Processor -- start 2024-04-05T03:08:54.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7d4fad700 1 -- start start 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7d4fad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7d4fad700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0102710 0x7fa7d010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7d4fad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d010f610 con 0x7fa7d0101d70 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7d4fad700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d010f750 con 0x7fa7d0102710 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55056/0 (socket says 172.21.15.153:55056) 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 -- 172.21.15.153:0/3955619412 learned_addr learned my addr 172.21.15.153:0/3955619412 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7cdd9b700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0102710 0x7fa7d010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 -- 172.21.15.153:0/3955619412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0102710 msgr2=0x7fa7d010efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0102710 0x7fa7d010efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 -- 172.21.15.153:0/3955619412 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4005040 con 0x7fa7d0101d70 2024-04-05T03:08:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7ce59c700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000eb40 tx=0x7fa7c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.294+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7c0007cc0 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.295+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d0117af0 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.295+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7c000f040 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.295+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7c0009620 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.295+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d0117f50 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.295+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d0104240 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.296+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa7c0009780 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.296+0000 7fa7beffd700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 0x7fa7b8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.297+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa7c008d150 con 0x7fa7d0101d70 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.297+0000 7fa7cdd9b700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 0x7fa7b8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.297+0000 7fa7cdd9b700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 0x7fa7b8070150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009b30 tx=0x7fa7c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.301+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa7c0056830 con 0x7fa7d0101d70 2024-04-05T03:08:54.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 podman[122727]: 2024-04-05 03:08:54.318140741 +0000 UTC m=+0.390182453 container init 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True) 2024-04-05T03:08:54.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 podman[122727]: 2024-04-05 03:08:54.321330161 +0000 UTC m=+0.393371868 container start 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-05T03:08:54.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-osd[122787]: -- 172.21.15.153:0/2870430952 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5580ef7e7680 con 0x5580ef7b0c00 2024-04-05T03:08:54.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 bash[122727]: 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d 2024-04-05T03:08:54.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 systemd[1]: Started Ceph osd.1 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:08:54.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.546+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7d004bed0 con 0x7fa7b806dca0 2024-04-05T03:08:54.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.548+0000 7fa7beffd700 1 -- 172.21.15.153:0/3955619412 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa7d004bed0 con 0x7fa7b806dca0 2024-04-05T03:08:54.547 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "7/21 daemons upgraded", 2024-04-05T03:08:54.548 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:08:54.549 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:08:54.549 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 msgr2=0x7fa7b8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 0x7fa7b8070150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009b30 tx=0x7fa7c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 msgr2=0x7fa7d010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000eb40 tx=0x7fa7c000b040 comp rx=0 tx=0).stop 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 shutdown_connections 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa7b806dca0 0x7fa7b8070150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7d0101d70 0x7fa7d010ea80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 --2- 172.21.15.153:0/3955619412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7d0102710 0x7fa7d010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 >> 172.21.15.153:0/3955619412 conn(0x7fa7d0072bc0 msgr2=0x7fa7d00faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 shutdown_connections 2024-04-05T03:08:54.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.550+0000 7fa7d4fad700 1 -- 172.21.15.153:0/3955619412 wait complete. 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 -- 172.21.15.153:0/2454625240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 msgr2=0x7fca8c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 --2- 172.21.15.153:0/2454625240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c1020e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fca84009ce0 tx=0x7fca84004d10 comp rx=0 tx=0).stop 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 -- 172.21.15.153:0/2454625240 shutdown_connections 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 --2- 172.21.15.153:0/2454625240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 0x7fca8c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 --2- 172.21.15.153:0/2454625240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c1020e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 -- 172.21.15.153:0/2454625240 >> 172.21.15.153:0/2454625240 conn(0x7fca8c0f7880 msgr2=0x7fca8c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 -- 172.21.15.153:0/2454625240 shutdown_connections 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.657+0000 7fca94583700 1 -- 172.21.15.153:0/2454625240 wait complete. 2024-04-05T03:08:54.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 Processor -- start 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 -- start start 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 0x7fca8c10f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c10fb10 con 0x7fca8c101d10 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca94583700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c10fc50 con 0x7fca8c1026b0 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca9231f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca9231f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55074/0 (socket says 172.21.15.153:55074) 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca9231f700 1 -- 172.21.15.153:0/2156462130 learned_addr learned my addr 172.21.15.153:0/2156462130 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:08:54.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.658+0000 7fca91b1e700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 0x7fca8c10f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca9231f700 1 -- 172.21.15.153:0/2156462130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 msgr2=0x7fca8c10f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca9231f700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 0x7fca8c10f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca9231f700 1 -- 172.21.15.153:0/2156462130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84009940 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca9231f700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fca84003400 tx=0x7fca84004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8400e040 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca84004610 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c117ff0 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca8401c640 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.659+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c1184d0 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.660+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c0fd460 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.661+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fca84012a70 con 0x7fca8c101d10 2024-04-05T03:08:54.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.661+0000 7fca82ffd700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 0x7fca78070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:08:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.661+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fca8408f240 con 0x7fca8c101d10 2024-04-05T03:08:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.661+0000 7fca91b1e700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 0x7fca78070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:08:54.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.662+0000 7fca91b1e700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 0x7fca78070150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca8c066890 tx=0x7fca7c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:08:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.666+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fca840588f0 con 0x7fca8c101d10 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2046160905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4245358255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:08:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:54.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2046160905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4245358255' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:08:54.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.956+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fca8c108e70 con 0x7fca8c101d10 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 48/318 objects degraded (15.094%), 16 pgs degraded 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: osd.1 (root=default,host=smithi153) is down 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 48/318 objects degraded (15.094%), 16 pgs degraded 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-05T03:08:54.957 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-05T03:08:54.958 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.956+0000 7fca82ffd700 1 -- 172.21.15.153:0/2156462130 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1160 (secure 0 0 0) 0x7fca840204a0 con 0x7fca8c101d10 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 msgr2=0x7fca78070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 0x7fca78070150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca8c066890 tx=0x7fca7c00b040 comp rx=0 tx=0).stop 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 msgr2=0x7fca8c10ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fca84003400 tx=0x7fca84004d10 comp rx=0 tx=0).stop 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 shutdown_connections 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fca7806dca0 0x7fca78070150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fca8c101d10 0x7fca8c10ef80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 --2- 172.21.15.153:0/2156462130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca8c1026b0 0x7fca8c10f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 >> 172.21.15.153:0/2156462130 conn(0x7fca8c0f7880 msgr2=0x7fca8c0f87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 shutdown_connections 2024-04-05T03:08:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:08:54.959+0000 7fca94583700 1 -- 172.21.15.153:0/2156462130 wait complete. 2024-04-05T03:08:55.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:54 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:08:54.869+0000 7f255abe57c0 -1 Falling back to public interface 2024-04-05T03:08:55.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:55 smithi153 ceph-mon[106102]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:55.670 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:55 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2156462130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:08:55.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:55 smithi203 ceph-mon[95988]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:08:55.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:55 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2156462130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:08:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:56 smithi203 ceph-mon[95988]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:56 smithi153 ceph-mon[106102]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:58 smithi203 ceph-mon[95988]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:58 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 48/318 objects degraded (15.094%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:08:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:58 smithi153 ceph-mon[106102]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:08:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:08:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:08:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:58 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 48/318 objects degraded (15.094%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:00.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:59 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:00.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:59 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:00.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:59 smithi203 ceph-mon[95988]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:09:00.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:08:59 smithi203 ceph-mon[95988]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T03:09:00.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:08:59 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:08:59.923+0000 7f255abe57c0 -1 osd.1 53 log_to_monitors true 2024-04-05T03:09:00.322 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:09:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:09:00.015+0000 7f254c03d700 -1 osd.1 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:09:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:59 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:59 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:59 smithi153 ceph-mon[106102]: pgmap v71: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 48/318 objects degraded (15.094%) 2024-04-05T03:09:00.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:08:59 smithi153 ceph-mon[106102]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T03:09:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T03:09:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: osdmap e56: 6 total, 5 up, 6 in 2024-04-05T03:09:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:09:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: failed to encode map e56 with expected crc 2024-04-05T03:09:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: failed to encode map e56 with expected crc 2024-04-05T03:09:01.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:01 smithi153 ceph-mon[106102]: failed to encode map e56 with expected crc 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: osdmap e56: 6 total, 5 up, 6 in 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: from='osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:01.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: failed to encode map e56 with expected crc 2024-04-05T03:09:01.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: failed to encode map e56 with expected crc 2024-04-05T03:09:01.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:01 smithi203 ceph-mon[95988]: failed to encode map e56 with expected crc 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: failed to encode map e56 with expected crc 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: failed to encode map e57 with expected crc 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340] boot 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: osdmap e57: 6 total, 6 up, 6 in 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:09:02.154 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:02 smithi153 ceph-mon[106102]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 48/318 objects degraded (15.094%) 2024-04-05T03:09:02.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: failed to encode map e56 with expected crc 2024-04-05T03:09:02.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:09:02.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: failed to encode map e57 with expected crc 2024-04-05T03:09:02.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: osd.1 [v2:172.21.15.153:6810/2271479340,v1:172.21.15.153:6811/2271479340] boot 2024-04-05T03:09:02.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: osdmap e57: 6 total, 6 up, 6 in 2024-04-05T03:09:02.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T03:09:02.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:02 smithi203 ceph-mon[95988]: pgmap v74: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 48/318 objects degraded (15.094%) 2024-04-05T03:09:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: osdmap e58: 6 total, 6 up, 6 in 2024-04-05T03:09:03.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:03 smithi203 ceph-mon[95988]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: osdmap e58: 6 total, 6 up, 6 in 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:03.311 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:03 smithi153 ceph-mon[106102]: failed to encode map e58 with expected crc 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 20/318 objects degraded (6.289%) 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:04.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.266 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.266 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.266 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:04 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 20/318 objects degraded (6.289%) 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:04.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:04.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:04 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:05 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:05 smithi203 ceph-mon[95988]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-05T03:09:05.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:05 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 20/318 objects degraded (6.289%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:05.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:05 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:05.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:05 smithi153 ceph-mon[106102]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-05T03:09:05.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:05 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 20/318 objects degraded (6.289%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:06.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:06 smithi203 ceph-mon[95988]: pgmap v77: 65 pgs: 2 active+undersized, 63 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-05T03:09:06.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:06 smithi153 ceph-mon[106102]: pgmap v77: 65 pgs: 2 active+undersized, 63 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-05T03:09:07.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:07 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/318 objects degraded (6.289%), 7 pgs degraded) 2024-04-05T03:09:07.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:07 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:09:07.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:07 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/318 objects degraded (6.289%), 7 pgs degraded) 2024-04-05T03:09:07.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:07 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:09:08.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:08 smithi203 ceph-mon[95988]: pgmap v78: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:09:08.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:08 smithi153 ceph-mon[106102]: pgmap v78: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T03:09:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:10 smithi203 ceph-mon[95988]: pgmap v79: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T03:09:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:10 smithi153 ceph-mon[106102]: pgmap v79: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T03:09:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:12 smithi203 ceph-mon[95988]: pgmap v80: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:09:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:12 smithi153 ceph-mon[106102]: pgmap v80: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:09:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:14 smithi203 ceph-mon[95988]: pgmap v81: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-05T03:09:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:14 smithi153 ceph-mon[106102]: pgmap v81: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-05T03:09:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:16 smithi203 ceph-mon[95988]: pgmap v82: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:16 smithi153 ceph-mon[106102]: pgmap v82: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:18 smithi203 ceph-mon[95988]: pgmap v83: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:18 smithi153 ceph-mon[106102]: pgmap v83: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:19 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:19.811 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:19 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:20 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:20 smithi203 ceph-mon[95988]: Upgrade: osd.2 is safe to restart 2024-04-05T03:09:20.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:20 smithi203 ceph-mon[95988]: pgmap v84: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:20 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T03:09:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:20 smithi153 ceph-mon[106102]: Upgrade: osd.2 is safe to restart 2024-04-05T03:09:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:20 smithi153 ceph-mon[106102]: pgmap v84: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: Upgrade: Updating osd.2 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: Deploying daemon osd.2 on smithi153 2024-04-05T03:09:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:21 smithi153 ceph-mon[106102]: pgmap v85: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: Upgrade: Updating osd.2 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: Deploying daemon osd.2 on smithi153 2024-04-05T03:09:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:21 smithi203 ceph-mon[95988]: pgmap v85: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:24.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:23 smithi153 systemd[1]: Stopping Ceph osd.2 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:09:24.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:24 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T03:09:24.083+0000 7fc6af762700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:09:24.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:24 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T03:09:24.083+0000 7fc6af762700 -1 osd.2 58 *** Got signal Terminated *** 2024-04-05T03:09:24.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:24 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[61728]: 2024-04-05T03:09:24.083+0000 7fc6af762700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:09:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:24 smithi203 ceph-mon[95988]: pgmap v86: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:24 smithi203 ceph-mon[95988]: osd.2 marked itself down and dead 2024-04-05T03:09:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:24 smithi153 ceph-mon[106102]: pgmap v86: 65 pgs: 65 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:24 smithi153 ceph-mon[106102]: osd.2 marked itself down and dead 2024-04-05T03:09:25.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 -- 172.21.15.153:0/2804002223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 msgr2=0x7fde240fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 --2- 172.21.15.153:0/2804002223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde240fce10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde18004940 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:25.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 -- 172.21.15.153:0/2804002223 shutdown_connections 2024-04-05T03:09:25.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 --2- 172.21.15.153:0/2804002223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde240fce10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 --2- 172.21.15.153:0/2804002223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde241031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 -- 172.21.15.153:0/2804002223 >> 172.21.15.153:0/2804002223 conn(0x7fde240f89b0 msgr2=0x7fde240fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 -- 172.21.15.153:0/2804002223 shutdown_connections 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.070+0000 7fde293b6700 1 -- 172.21.15.153:0/2804002223 wait complete. 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 Processor -- start 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 -- start start 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde24113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde241144a0 con 0x7fde241037a0 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde293b6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2410daf0 con 0x7fde24102e00 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde227fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde227fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48048/0 (socket says 172.21.15.153:48048) 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.071+0000 7fde227fc700 1 -- 172.21.15.153:0/4140585050 learned_addr learned my addr 172.21.15.153:0/4140585050 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde22ffd700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde24113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde227fc700 1 -- 172.21.15.153:0/4140585050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 msgr2=0x7fde24113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde227fc700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde24113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde227fc700 1 -- 172.21.15.153:0/4140585050 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde18005040 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde22ffd700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde24113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde227fc700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde18009100 tx=0x7fde18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde1800e040 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde18013420 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2410dd70 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde1801c610 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.072+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2410e330 con 0x7fde241037a0 2024-04-05T03:09:25.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.073+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde241052d0 con 0x7fde241037a0 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.074+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fde18013590 con 0x7fde241037a0 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.074+0000 7fde0b7fe700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 0x7fde0c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.074+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fde1808e680 con 0x7fde241037a0 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.074+0000 7fde22ffd700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 0x7fde0c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.075+0000 7fde22ffd700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 0x7fde0c070150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde14009490 tx=0x7fde14009ed0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.079+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fde1805b900 con 0x7fde241037a0 2024-04-05T03:09:25.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:24 smithi153 podman[127772]: 2024-04-05 03:09:24.870518736 +0000 UTC m=+0.882197037 container died baa52caae33b91027046fa6deb1992f86b836dc6b0db52cc1fb9164fa609b468 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-6fda41f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-05T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.295+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde2410ef60 con 0x7fde0c06dca0 2024-04-05T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.296+0000 7fde0b7fe700 1 -- 172.21.15.153:0/4140585050 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fde2410ef60 con 0x7fde0c06dca0 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 msgr2=0x7fde0c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 0x7fde0c070150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde14009490 tx=0x7fde14009ed0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 msgr2=0x7fde24113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde18009100 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 shutdown_connections 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde0c06dca0 0x7fde0c070150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde24102e00 0x7fde24113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 --2- 172.21.15.153:0/4140585050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde241037a0 0x7fde24113e50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 >> 172.21.15.153:0/4140585050 conn(0x7fde240f89b0 msgr2=0x7fde241021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 shutdown_connections 2024-04-05T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.298+0000 7fde293b6700 1 -- 172.21.15.153:0/4140585050 wait complete. 2024-04-05T03:09:25.309 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- 172.21.15.153:0/197528059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c90101530 msgr2=0x7f3c90101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 --2- 172.21.15.153:0/197528059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c90101530 0x7f3c90101900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c80004a50 tx=0x7f3c8000af40 comp rx=0 tx=0).stop 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- 172.21.15.153:0/197528059 shutdown_connections 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 --2- 172.21.15.153:0/197528059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c900fb570 0x7f3c900fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 --2- 172.21.15.153:0/197528059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c90101530 0x7f3c90101900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- 172.21.15.153:0/197528059 >> 172.21.15.153:0/197528059 conn(0x7f3c900714c0 msgr2=0x7f3c900718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- 172.21.15.153:0/197528059 shutdown_connections 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- 172.21.15.153:0/197528059 wait complete. 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 Processor -- start 2024-04-05T03:09:25.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.409+0000 7f3c96005700 1 -- start start 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c96005700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c96005700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c90101530 0x7f3c90113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c96005700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90114500 con 0x7f3c900fb570 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c96005700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9010db50 con 0x7f3c90101530 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48058/0 (socket says 172.21.15.153:48058) 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 -- 172.21.15.153:0/1949161843 learned_addr learned my addr 172.21.15.153:0/1949161843 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c87fff700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c90101530 0x7f3c90113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 -- 172.21.15.153:0/1949161843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c90101530 msgr2=0x7f3c90113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c90101530 0x7f3c90113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 -- 172.21.15.153:0/1949161843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80005040 con 0x7f3c900fb570 2024-04-05T03:09:25.411 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:25 smithi153 podman[127772]: 2024-04-05 03:09:25.292927719 +0000 UTC m=+1.304606020 container cleanup baa52caae33b91027046fa6deb1992f86b836dc6b0db52cc1fb9164fa609b468 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=quincy-6fda41f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-05T03:09:25.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:25 smithi153 bash[127772]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8f7fe700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000a4d0 tx=0x7f3c8000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c8000f070 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c8000a990 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.410+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9010ddd0 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.411+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c80014620 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.411+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9010e330 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.412+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c900fcc90 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.412+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3c8001ca80 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.412+0000 7f3c8cff9700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 0x7f3c7c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.413+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3c8008eaa0 con 0x7f3c900fb570 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.413+0000 7f3c87fff700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 0x7f3c7c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.413+0000 7f3c87fff700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 0x7f3c7c070150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c78004600 tx=0x7f3c7800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.417+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3c800581f0 con 0x7f3c900fb570 2024-04-05T03:09:25.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.662+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c9010eff0 con 0x7f3c7c06dca0 2024-04-05T03:09:25.662 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:25 smithi153 podman[127792]: 2024-04-05 03:09:25.536207295 +0000 UTC m=+0.667883601 container remove baa52caae33b91027046fa6deb1992f86b836dc6b0db52cc1fb9164fa609b468 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=quincy-6fda41f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:09:25.662 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: osdmap e59: 6 total, 5 up, 6 in 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.663 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:25 smithi153 ceph-mon[106102]: osdmap e60: 6 total, 5 up, 6 in 2024-04-05T03:09:25.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.664+0000 7f3c8cff9700 1 -- 172.21.15.153:0/1949161843 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3c9010eff0 con 0x7f3c7c06dca0 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 msgr2=0x7f3c7c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 0x7f3c7c070150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c78004600 tx=0x7f3c7800b040 comp rx=0 tx=0).stop 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 msgr2=0x7f3c90113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000a4d0 tx=0x7f3c8000af40 comp rx=0 tx=0).stop 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 shutdown_connections 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c7c06dca0 0x7f3c7c070150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c900fb570 0x7f3c90113a10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 --2- 172.21.15.153:0/1949161843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c90101530 0x7f3c90113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 >> 172.21.15.153:0/1949161843 conn(0x7f3c900714c0 msgr2=0x7f3c90105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 shutdown_connections 2024-04-05T03:09:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.666+0000 7f3c96005700 1 -- 172.21.15.153:0/1949161843 wait complete. 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 -- 172.21.15.153:0/1859069264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100102710 msgr2=0x7f21000fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 --2- 172.21.15.153:0/1859069264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100102710 0x7f21000fbd80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20ec004940 tx=0x7f20ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 -- 172.21.15.153:0/1859069264 shutdown_connections 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 --2- 172.21.15.153:0/1859069264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100102710 0x7f21000fbd80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 --2- 172.21.15.153:0/1859069264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100101d70 0x7f2100102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.774+0000 7f2106c4a700 1 -- 172.21.15.153:0/1859069264 >> 172.21.15.153:0/1859069264 conn(0x7f2100072bc0 msgr2=0x7f2100072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 -- 172.21.15.153:0/1859069264 shutdown_connections 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 -- 172.21.15.153:0/1859069264 wait complete. 2024-04-05T03:09:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 Processor -- start 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 -- start start 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 0x7f2100114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2100114660 con 0x7f2100101d70 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.775+0000 7f2106c4a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210010da50 con 0x7f2100102710 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48086/0 (socket says 172.21.15.153:48086) 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 -- 172.21.15.153:0/4023360990 learned_addr learned my addr 172.21.15.153:0/4023360990 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f20fffff700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 0x7f2100114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 -- 172.21.15.153:0/4023360990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 msgr2=0x7f2100114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 0x7f2100114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 -- 172.21.15.153:0/4023360990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec005040 con 0x7f2100101d70 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f20fffff700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 0x7f2100114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f21049e6700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f20f400eb40 tx=0x7f20f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20f4007cc0 con 0x7f2100101d70 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210010dcd0 con 0x7f2100101d70 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20f400f040 con 0x7f2100101d70 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20f4009620 con 0x7f2100101d70 2024-04-05T03:09:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.776+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210010e290 con 0x7f2100101d70 2024-04-05T03:09:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.777+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21000fd4c0 con 0x7f2100101d70 2024-04-05T03:09:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.778+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f20f4009780 con 0x7f2100101d70 2024-04-05T03:09:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.778+0000 7f20fd7fa700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 0x7f20f0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.778+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f20f408d150 con 0x7f2100101d70 2024-04-05T03:09:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.779+0000 7f20fffff700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 0x7f20f0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.779+0000 7f20fffff700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 0x7f20f0070150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00b0d0 tx=0x7f20ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:25.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.784+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f20f405a360 con 0x7f2100101d70 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: osdmap e59: 6 total, 5 up, 6 in 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: failed to encode map e59 with expected crc 2024-04-05T03:09:25.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:25 smithi203 ceph-mon[95988]: osdmap e60: 6 total, 5 up, 6 in 2024-04-05T03:09:25.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:25.999+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2100108e50 con 0x7f20f006dca0 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.009+0000 7f20fd7fa700 1 -- 172.21.15.153:0/4023360990 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2100108e50 con 0x7f20f006dca0 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (3m) 27s ago 15m 26.6M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (91s) 27s ago 16m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (86s) 82s ago 14m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (3m) 27s ago 14m 82.1M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (9m) 27s ago 9m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (9m) 27s ago 9m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:09:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (9m) 82s ago 9m 61.4M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (9m) 82s ago 9m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (5m) 27s ago 17m 499M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (4m) 82s ago 13m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (2m) 27s ago 17m 44.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (2m) 82s ago 13m 36.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (4m) 27s ago 15m 17.0M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (4m) 82s ago 13m 17.3M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (66s) 27s ago 13m 88.0M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (32s) 27s ago 12m 14.9M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (12m) 27s ago 12m 199M 1898M 17.2.7-805-g6fda41ff a49113eacc58 baa52caae33b 2024-04-05T03:09:26.010 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (11m) 82s ago 11m 265M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:09:26.011 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (11m) 82s ago 11m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:09:26.011 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (11m) 82s ago 11m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:09:26.011 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (4m) 82s ago 13m 60.5M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:09:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.011+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 msgr2=0x7f20f0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.011+0000 7f2106c4a700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 0x7f20f0070150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00b0d0 tx=0x7f20ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.011+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 msgr2=0x7f2100113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.011+0000 7f2106c4a700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f20f400eb40 tx=0x7f20f400b040 comp rx=0 tx=0).stop 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.011+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 shutdown_connections 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f20f006dca0 0x7f20f0070150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2100101d70 0x7f2100113ad0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 --2- 172.21.15.153:0/4023360990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2100102710 0x7f2100114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 >> 172.21.15.153:0/4023360990 conn(0x7f2100072bc0 msgr2=0x7f2100100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 shutdown_connections 2024-04-05T03:09:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.012+0000 7f2106c4a700 1 -- 172.21.15.153:0/4023360990 wait complete. 2024-04-05T03:09:26.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.115+0000 7f86687ec700 1 -- 172.21.15.153:0/849076730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fe830 msgr2=0x7f8660106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.115+0000 7f86687ec700 1 --2- 172.21.15.153:0/849076730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fe830 0x7f8660106f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8650004940 tx=0x7f8650009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.115+0000 7f86687ec700 1 -- 172.21.15.153:0/849076730 shutdown_connections 2024-04-05T03:09:26.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 --2- 172.21.15.153:0/849076730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fe830 0x7f8660106f00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 --2- 172.21.15.153:0/849076730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fde90 0x7f86600fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- 172.21.15.153:0/849076730 >> 172.21.15.153:0/849076730 conn(0x7f8660072bc0 msgr2=0x7f8660072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- 172.21.15.153:0/849076730 shutdown_connections 2024-04-05T03:09:26.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- 172.21.15.153:0/849076730 wait complete. 2024-04-05T03:09:26.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 Processor -- start 2024-04-05T03:09:26.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- start start 2024-04-05T03:09:26.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fe830 0x7f86600fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86600fc820 con 0x7f86600fde90 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.116+0000 7f86687ec700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86600fc960 con 0x7f86600fe830 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48092/0 (socket says 172.21.15.153:48092) 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 -- 172.21.15.153:0/2972868814 learned_addr learned my addr 172.21.15.153:0/2972868814 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8665d87700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fe830 0x7f86600fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 -- 172.21.15.153:0/2972868814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fe830 msgr2=0x7f86600fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fe830 0x7f86600fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 -- 172.21.15.153:0/2972868814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650005040 con 0x7f86600fde90 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8666588700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f865c00eb40 tx=0x7f865c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865c007cc0 con 0x7f86600fde90 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86600fcbb0 con 0x7f86600fde90 2024-04-05T03:09:26.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f865c00f040 con 0x7f86600fde90 2024-04-05T03:09:26.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865c00ca10 con 0x7f86600fde90 2024-04-05T03:09:26.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.117+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866011a1d0 con 0x7f86600fde90 2024-04-05T03:09:26.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.118+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8660065e30 con 0x7f86600fde90 2024-04-05T03:09:26.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.119+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f865c00cc80 con 0x7f86600fde90 2024-04-05T03:09:26.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.119+0000 7f8656ffd700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 0x7f864c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.119+0000 7f8665d87700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 0x7f864c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.119+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f865c08d090 con 0x7f86600fde90 2024-04-05T03:09:26.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.120+0000 7f8665d87700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 0x7f864c070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86600fd690 tx=0x7f8650009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.124+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f865c056770 con 0x7f86600fde90 2024-04-05T03:09:26.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:25 smithi153 podman[127956]: 2024-04-05 03:09:25.861941301 +0000 UTC m=+0.018513336 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:09:26.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:25 smithi153 podman[127956]: 2024-04-05 03:09:25.996396221 +0000 UTC m=+0.152968289 container create f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-05T03:09:26.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:26 smithi153 podman[127956]: 2024-04-05 03:09:26.239984735 +0000 UTC m=+0.396556755 container init f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, RELEASE=squid-0028525, CEPH_POINT_RELEASE=) 2024-04-05T03:09:26.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.429+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f866005f5f0 con 0x7f86600fde90 2024-04-05T03:09:26.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.430+0000 7f8656ffd700 1 -- 172.21.15.153:0/2972868814 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f865c059e30 con 0x7f86600fde90 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 3, 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:26.430 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 7, 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:09:26.431 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:09:26.433 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 msgr2=0x7f864c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 0x7f864c070150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f86600fd690 tx=0x7f8650009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 msgr2=0x7f86600fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f865c00eb40 tx=0x7f865c00b040 comp rx=0 tx=0).stop 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 shutdown_connections 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f864c06dca0 0x7f864c070150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86600fde90 0x7f86600fbd50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 --2- 172.21.15.153:0/2972868814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86600fe830 0x7f86600fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 >> 172.21.15.153:0/2972868814 conn(0x7f8660072bc0 msgr2=0x7f8660103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 shutdown_connections 2024-04-05T03:09:26.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.432+0000 7f86687ec700 1 -- 172.21.15.153:0/2972868814 wait complete. 2024-04-05T03:09:26.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:26 smithi153 podman[127956]: 2024-04-05 03:09:26.244419576 +0000 UTC m=+0.400991601 container start f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-05T03:09:26.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:26 smithi153 podman[127956]: 2024-04-05 03:09:26.356020022 +0000 UTC m=+0.512592052 container attach f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2044414665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a741037b0 msgr2=0x7f3a740fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2044414665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a741037b0 0x7f3a740fce20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3a64004940 tx=0x7f3a64009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2044414665 shutdown_connections 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2044414665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a741037b0 0x7f3a740fce20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2044414665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a74102e10 0x7f3a741031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2044414665 >> 172.21.15.153:0/2044414665 conn(0x7f3a740f89a0 msgr2=0x7f3a740fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2044414665 shutdown_connections 2024-04-05T03:09:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.541+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2044414665 wait complete. 2024-04-05T03:09:26.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 Processor -- start 2024-04-05T03:09:26.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 -- start start 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a741037b0 0x7f3a741140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a74114730 con 0x7f3a74102e10 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a7a18c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7410db20 con 0x7f3a741037b0 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48108/0 (socket says 172.21.15.153:48108) 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a737fe700 1 -- 172.21.15.153:0/2917010077 learned_addr learned my addr 172.21.15.153:0/2917010077 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.542+0000 7f3a72ffd700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a741037b0 0x7f3a741140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a737fe700 1 -- 172.21.15.153:0/2917010077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a741037b0 msgr2=0x7f3a741140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a737fe700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a741037b0 0x7f3a741140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a737fe700 1 -- 172.21.15.153:0/2917010077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a64005040 con 0x7f3a74102e10 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a737fe700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3a68002a50 tx=0x7f3a680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a68007cc0 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a6800f040 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6800da30 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7410de00 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.543+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7410e390 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.544+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a741052e0 con 0x7f3a74102e10 2024-04-05T03:09:26.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.545+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3a6800db90 con 0x7f3a74102e10 2024-04-05T03:09:26.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.545+0000 7f3a5bfff700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 0x7f3a5c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.545+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3a6808d180 con 0x7f3a74102e10 2024-04-05T03:09:26.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.545+0000 7f3a72ffd700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 0x7f3a5c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.546+0000 7f3a72ffd700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 0x7f3a5c070160 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3a7410ee00 tx=0x7f3a64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.550+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3a6805a360 con 0x7f3a74102e10 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: pgmap v88: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:26 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2972868814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:26.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.798+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3a7410f1f0 con 0x7f3a74102e10 2024-04-05T03:09:26.798 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:26 smithi153 podman[127956]: 2024-04-05 03:09:26.556289099 +0000 UTC m=+0.712861119 container died f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, GIT_BRANCH=HEAD) 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: pgmap v88: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.798 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: failed to encode map e60 with expected crc 2024-04-05T03:09:26.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:26 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2972868814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.798+0000 7f3a5bfff700 1 -- 172.21.15.153:0/2917010077 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7f3a68059ef0 con 0x7f3a74102e10 2024-04-05T03:09:26.799 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:09:26.799 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:09:26.799 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:09:26.800 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:09:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:26.802 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 msgr2=0x7f3a5c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 0x7f3a5c070160 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3a7410ee00 tx=0x7f3a64009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 msgr2=0x7f3a74113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3a68002a50 tx=0x7f3a680046b0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 shutdown_connections 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5c06dcb0 0x7f3a5c070160 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a74102e10 0x7f3a74113ba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 --2- 172.21.15.153:0/2917010077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a741037b0 0x7f3a741140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.801+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 >> 172.21.15.153:0/2917010077 conn(0x7f3a740f89a0 msgr2=0x7f3a74102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.802+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 shutdown_connections 2024-04-05T03:09:26.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.802+0000 7f3a7a18c700 1 -- 172.21.15.153:0/2917010077 wait complete. 2024-04-05T03:09:26.804 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 -- 172.21.15.153:0/3062663945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 msgr2=0x7f48840fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 --2- 172.21.15.153:0/3062663945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f48840fce10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f486c004940 tx=0x7f486c009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 -- 172.21.15.153:0/3062663945 shutdown_connections 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 --2- 172.21.15.153:0/3062663945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f48840fce10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 --2- 172.21.15.153:0/3062663945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 0x7f48841031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 -- 172.21.15.153:0/3062663945 >> 172.21.15.153:0/3062663945 conn(0x7f48840f89f0 msgr2=0x7f48840fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 -- 172.21.15.153:0/3062663945 shutdown_connections 2024-04-05T03:09:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.905+0000 7f4889cd0700 1 -- 172.21.15.153:0/3062663945 wait complete. 2024-04-05T03:09:26.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 Processor -- start 2024-04-05T03:09:26.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 -- start start 2024-04-05T03:09:26.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 0x7f48840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884072210 con 0x7f48841037a0 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f4889cd0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884070d40 con 0x7f4884102e00 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48128/0 (socket says 172.21.15.153:48128) 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 -- 172.21.15.153:0/1342559489 learned_addr learned my addr 172.21.15.153:0/1342559489 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f48837fe700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 0x7f48840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 -- 172.21.15.153:0/1342559489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 msgr2=0x7f48840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 0x7f48840702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.906+0000 7f487bfff700 1 -- 172.21.15.153:0/1342559489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c005040 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f487bfff700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f486c009100 tx=0x7f486c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486c00e040 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f486c013420 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4884070fc0 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486c01c610 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.907+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488411a0b0 con 0x7f48841037a0 2024-04-05T03:09:26.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.908+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48841052d0 con 0x7f48841037a0 2024-04-05T03:09:26.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.908+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f486c013590 con 0x7f48841037a0 2024-04-05T03:09:26.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.909+0000 7f4880ff9700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 0x7f4870070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:26.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.909+0000 7f48837fe700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 0x7f4870070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:26.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.909+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f486c08e770 con 0x7f48841037a0 2024-04-05T03:09:26.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.909+0000 7f48837fe700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 0x7f4870070150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4874009fd0 tx=0x7f4874009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:26.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:26.914+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f486c057e20 con 0x7f48841037a0 2024-04-05T03:09:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.133+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4884109ee0 con 0x7f487006dca0 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.134+0000 7f4880ff9700 1 -- 172.21.15.153:0/1342559489 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f4884109ee0 con 0x7f487006dca0 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:09:27.134 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "8/21 daemons upgraded", 2024-04-05T03:09:27.135 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:09:27.135 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:09:27.135 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:09:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 msgr2=0x7f4870070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 0x7f4870070150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4874009fd0 tx=0x7f4874009e50 comp rx=0 tx=0).stop 2024-04-05T03:09:27.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 msgr2=0x7f4884070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f486c009100 tx=0x7f486c009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 shutdown_connections 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f487006dca0 0x7f4870070150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4884102e00 0x7f48840702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 --2- 172.21.15.153:0/1342559489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48841037a0 0x7f4884070800 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 >> 172.21.15.153:0/1342559489 conn(0x7f48840f89f0 msgr2=0x7f4884102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 shutdown_connections 2024-04-05T03:09:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.137+0000 7f4889cd0700 1 -- 172.21.15.153:0/1342559489 wait complete. 2024-04-05T03:09:27.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 -- 172.21.15.153:0/4249515118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 msgr2=0x7f78f41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/4249515118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f41031e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004940 tx=0x7f78e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 -- 172.21.15.153:0/4249515118 shutdown_connections 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/4249515118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 0x7f78f40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/4249515118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f41031e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 -- 172.21.15.153:0/4249515118 >> 172.21.15.153:0/4249515118 conn(0x7f78f40f89a0 msgr2=0x7f78f40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 -- 172.21.15.153:0/4249515118 shutdown_connections 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 -- 172.21.15.153:0/4249515118 wait complete. 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.249+0000 7f78fbfd1700 1 Processor -- start 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78fbfd1700 1 -- start start 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78fbfd1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78fbfd1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 0x7f78f41140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78fbfd1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41146f0 con 0x7f78f4102e10 2024-04-05T03:09:27.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78fbfd1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f410dae0 con 0x7f78f41037b0 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48140/0 (socket says 172.21.15.153:48140) 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 -- 172.21.15.153:0/3955215590 learned_addr learned my addr 172.21.15.153:0/3955215590 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f956c700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 0x7f78f41140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 -- 172.21.15.153:0/3955215590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 msgr2=0x7f78f41140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 0x7f78f41140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 -- 172.21.15.153:0/3955215590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4005040 con 0x7f78f4102e10 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78f9d6d700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009150 tx=0x7f78e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78e4021070 con 0x7f78f4102e10 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.250+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78e4013c00 con 0x7f78f4102e10 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.251+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f410dd60 con 0x7f78f4102e10 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.251+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78e400e070 con 0x7f78f4102e10 2024-04-05T03:09:27.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.251+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f410e240 con 0x7f78f4102e10 2024-04-05T03:09:27.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.252+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f4100120 con 0x7f78f4102e10 2024-04-05T03:09:27.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.252+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f78e401c400 con 0x7f78f4102e10 2024-04-05T03:09:27.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.253+0000 7f78ea7fc700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 0x7f78e0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.253+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f78e408e8b0 con 0x7f78f4102e10 2024-04-05T03:09:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.253+0000 7f78f956c700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 0x7f78e0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:27.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.253+0000 7f78f956c700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 0x7f78e0070160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78f0004600 tx=0x7f78f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.257+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f78e405bb30 con 0x7f78f4102e10 2024-04-05T03:09:27.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128028]: 2024-04-05 03:09:27.159940115 +0000 UTC m=+0.598214739 container remove f660c9ce574c5b8ff643c52279882626a17b440433b84865528b5cb98942b819 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-05T03:09:27.390 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-04-05T03:09:27.390 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 systemd[1]: Stopped Ceph osd.2 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:09:27.390 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.2.service: Consumed 55.799s CPU time. 2024-04-05T03:09:27.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.566+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f78f410f1b0 con 0x7f78f4102e10 2024-04-05T03:09:27.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.566+0000 7f78ea7fc700 1 -- 172.21.15.153:0/3955215590 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+830 (secure 0 0 0) 0x7f78e405b6c0 con 0x7f78f4102e10 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 3 pgs peering; Degraded data redundancy: 13/318 objects degraded (4.088%), 4 pgs degraded 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout: osd.2 (root=default,host=smithi153) is down 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1 is stuck peering for 5m, current state peering, last acting [1,0] 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout: pg 3.7 is stuck peering for 5m, current state peering, last acting [1,4] 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout: pg 3.8 is stuck peering for 5m, current state peering, last acting [1,5] 2024-04-05T03:09:27.567 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 13/318 objects degraded (4.088%), 4 pgs degraded 2024-04-05T03:09:27.568 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.e is active+undersized+degraded, acting [0,3] 2024-04-05T03:09:27.568 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-04-05T03:09:27.568 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.19 is active+undersized+degraded, acting [0,4] 2024-04-05T03:09:27.568 INFO:teuthology.orchestra.run.smithi153.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 msgr2=0x7f78e0070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 0x7f78e0070160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78f0004600 tx=0x7f78f000b040 comp rx=0 tx=0).stop 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 msgr2=0x7f78f4113b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009150 tx=0x7f78e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 shutdown_connections 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78e006dcb0 0x7f78e0070160 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78f4102e10 0x7f78f4113b60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 --2- 172.21.15.153:0/3955215590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78f41037b0 0x7f78f41140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 >> 172.21.15.153:0/3955215590 conn(0x7f78f40f89a0 msgr2=0x7f78f4107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 shutdown_connections 2024-04-05T03:09:27.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:27.569+0000 7f78fbfd1700 1 -- 172.21.15.153:0/3955215590 wait complete. 2024-04-05T03:09:27.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:27 smithi153 ceph-mon[106102]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:27.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:27 smithi153 ceph-mon[106102]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:27.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:27 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2917010077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:09:27.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:27.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:27.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 systemd[1]: Starting Ceph osd.2 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:09:27.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128178]: 2024-04-05 03:09:27.522988302 +0000 UTC m=+0.023581564 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:09:27.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128178]: 2024-04-05 03:09:27.636360567 +0000 UTC m=+0.136953801 container create 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.build-date=20240403, RELEASE=squid-0028525, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:09:27.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:27 smithi203 ceph-mon[95988]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:27.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:27 smithi203 ceph-mon[95988]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:27.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:27 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2917010077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:09:27.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:27.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:28.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128178]: 2024-04-05 03:09:27.838240161 +0000 UTC m=+0.338833411 container init 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-05T03:09:28.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128178]: 2024-04-05 03:09:27.841704398 +0000 UTC m=+0.342297642 container start 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-05T03:09:28.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:27 smithi153 podman[128178]: 2024-04-05 03:09:27.912749272 +0000 UTC m=+0.413342516 container attach 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-0028525, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-05T03:09:28.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-mon[106102]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:28.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-mon[106102]: pgmap v90: 65 pgs: 4 active+undersized+degraded, 7 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 13/318 objects degraded (4.088%) 2024-04-05T03:09:28.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 13/318 objects degraded (4.088%), 4 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:28.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3955215590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:09:28.572 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T03:09:28.572 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 bash[128178]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate[128196]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-05T03:09:28.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 podman[128178]: 2024-04-05 03:09:28.309869594 +0000 UTC m=+0.810462832 container died 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-0028525, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-05T03:09:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:28 smithi203 ceph-mon[95988]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:28 smithi203 ceph-mon[95988]: pgmap v90: 65 pgs: 4 active+undersized+degraded, 7 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 13/318 objects degraded (4.088%) 2024-04-05T03:09:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:28 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 13/318 objects degraded (4.088%), 4 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:28 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3955215590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:09:29.291 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:28 smithi153 podman[128333]: 2024-04-05 03:09:28.98376471 +0000 UTC m=+0.663756592 container remove 6da0f7b9afc8610c4d4c71f15a9287b7116c99227af296c7f5bc91ffffbde8cf (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-0028525, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:09:29.291 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 podman[128396]: 2024-04-05 03:09:29.184511302 +0000 UTC m=+0.104872098 container create fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403) 2024-04-05T03:09:29.292 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 podman[128396]: 2024-04-05 03:09:29.097671313 +0000 UTC m=+0.018032134 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:09:29.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 podman[128396]: 2024-04-05 03:09:29.412285238 +0000 UTC m=+0.332646045 container init fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-05T03:09:29.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 podman[128396]: 2024-04-05 03:09:29.415524072 +0000 UTC m=+0.335884866 container start fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, ceph=True, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-05T03:09:29.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 ceph-osd[128415]: -- 172.21.15.153:0/2624643151 <== mon.0 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d01639b680 con 0x55d016364c00 2024-04-05T03:09:29.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 bash[128396]: fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 2024-04-05T03:09:29.823 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:29 smithi153 systemd[1]: Started Ceph osd.2 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:09:30.322 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:30 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:09:29.961+0000 7f5e1e4e57c0 -1 Falling back to public interface 2024-04-05T03:09:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:30 smithi153 ceph-mon[106102]: pgmap v91: 65 pgs: 9 active+undersized+degraded, 7 peering, 11 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27/318 objects degraded (8.491%) 2024-04-05T03:09:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:30 smithi203 ceph-mon[95988]: pgmap v91: 65 pgs: 9 active+undersized+degraded, 7 peering, 11 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 27/318 objects degraded (8.491%) 2024-04-05T03:09:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:31.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:31 smithi153 ceph-mon[106102]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-05T03:09:32.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:31 smithi203 ceph-mon[95988]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-05T03:09:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:32 smithi153 ceph-mon[106102]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:32 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 39/318 objects degraded (12.264%), 12 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:32 smithi203 ceph-mon[95988]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:32 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 39/318 objects degraded (12.264%), 12 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:34 smithi203 ceph-mon[95988]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:34 smithi203 ceph-mon[95988]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T03:09:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-mon[106102]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:35.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-mon[106102]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T03:09:35.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:09:34.575+0000 7f5e1e4e57c0 -1 osd.2 58 log_to_monitors true 2024-04-05T03:09:35.072 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:09:34 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:09:34.822+0000 7f5e0f93d700 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:09:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T03:09:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: osdmap e61: 6 total, 5 up, 6 in 2024-04-05T03:09:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:09:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: failed to encode map e61 with expected crc 2024-04-05T03:09:36.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:35 smithi153 ceph-mon[106102]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: osdmap e61: 6 total, 5 up, 6 in 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: from='osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: failed to encode map e61 with expected crc 2024-04-05T03:09:36.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:35 smithi203 ceph-mon[95988]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/318 objects degraded (12.264%) 2024-04-05T03:09:36.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: failed to encode map e61 with expected crc 2024-04-05T03:09:36.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: failed to encode map e61 with expected crc 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: failed to encode map e62 with expected crc 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589] boot 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: osdmap e62: 6 total, 6 up, 6 in 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: failed to encode map e62 with expected crc 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: failed to encode map e62 with expected crc 2024-04-05T03:09:36.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:36 smithi153 ceph-mon[106102]: osdmap e63: 6 total, 6 up, 6 in 2024-04-05T03:09:37.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: failed to encode map e61 with expected crc 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: failed to encode map e61 with expected crc 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: failed to encode map e62 with expected crc 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: osd.2 [v2:172.21.15.153:6818/2475488589,v1:172.21.15.153:6819/2475488589] boot 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: osdmap e62: 6 total, 6 up, 6 in 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: failed to encode map e62 with expected crc 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: failed to encode map e62 with expected crc 2024-04-05T03:09:37.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:36 smithi203 ceph-mon[95988]: osdmap e63: 6 total, 6 up, 6 in 2024-04-05T03:09:38.197 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.198 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.198 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.198 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.198 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.198 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:37 smithi203 ceph-mon[95988]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 39/318 objects degraded (12.264%) 2024-04-05T03:09:38.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: failed to encode map e63 with expected crc 2024-04-05T03:09:38.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:37 smithi153 ceph-mon[106102]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 39/318 objects degraded (12.264%) 2024-04-05T03:09:40.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:40.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T03:09:40.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:39 smithi203 ceph-mon[95988]: pgmap v99: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 25/318 objects degraded (7.862%) 2024-04-05T03:09:40.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:40.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:09:40.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T03:09:40.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:39 smithi153 ceph-mon[106102]: pgmap v99: 65 pgs: 7 active+undersized+degraded, 10 active+undersized, 48 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 25/318 objects degraded (7.862%) 2024-04-05T03:09:41.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:40 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 25/318 objects degraded (7.862%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:41.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:40 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 25/318 objects degraded (7.862%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T03:09:42.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:41 smithi203 ceph-mon[95988]: pgmap v100: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-05T03:09:42.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:41 smithi153 ceph-mon[106102]: pgmap v100: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-05T03:09:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:42 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/318 objects degraded (7.862%), 7 pgs degraded) 2024-04-05T03:09:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:42 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:09:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:43.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:42 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/318 objects degraded (7.862%), 7 pgs degraded) 2024-04-05T03:09:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:42 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:09:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:43.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:44.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:43 smithi153 ceph-mon[106102]: pgmap v101: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 2 op/s 2024-04-05T03:09:44.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:43 smithi203 ceph-mon[95988]: pgmap v101: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 2 op/s 2024-04-05T03:09:46.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:46 smithi203 ceph-mon[95988]: pgmap v102: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-05T03:09:46.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:46 smithi153 ceph-mon[106102]: pgmap v102: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-05T03:09:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:48 smithi203 ceph-mon[95988]: pgmap v103: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-05T03:09:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:48 smithi153 ceph-mon[106102]: pgmap v103: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-05T03:09:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:50 smithi203 ceph-mon[95988]: pgmap v104: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 2 op/s 2024-04-05T03:09:50.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:50 smithi153 ceph-mon[106102]: pgmap v104: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 2 op/s 2024-04-05T03:09:52.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:52 smithi203 ceph-mon[95988]: pgmap v105: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-05T03:09:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:52 smithi153 ceph-mon[106102]: pgmap v105: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-05T03:09:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:54 smithi203 ceph-mon[95988]: pgmap v106: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:54.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:54 smithi153 ceph-mon[106102]: pgmap v106: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:09:54.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:55.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:55 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:55.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:55 smithi203 ceph-mon[95988]: Upgrade: osd.3 is safe to restart 2024-04-05T03:09:55.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:55 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T03:09:55.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:55 smithi153 ceph-mon[106102]: Upgrade: osd.3 is safe to restart 2024-04-05T03:09:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:56 smithi203 ceph-mon[95988]: pgmap v107: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:09:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:56.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T03:09:56.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:56 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:56 smithi153 ceph-mon[106102]: pgmap v107: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:09:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:09:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T03:09:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:56 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 -- 172.21.15.153:0/1211208047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fde20 msgr2=0x7fb9600fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 --2- 172.21.15.153:0/1211208047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fde20 0x7fb9600fe1f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004940 tx=0x7fb95c009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 -- 172.21.15.153:0/1211208047 shutdown_connections 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 --2- 172.21.15.153:0/1211208047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fe7c0 0x7fb960106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 --2- 172.21.15.153:0/1211208047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fde20 0x7fb9600fe1f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 -- 172.21.15.153:0/1211208047 >> 172.21.15.153:0/1211208047 conn(0x7fb9600f7870 msgr2=0x7fb9600f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 -- 172.21.15.153:0/1211208047 shutdown_connections 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.679+0000 7fb967ba1700 1 -- 172.21.15.153:0/1211208047 wait complete. 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 Processor -- start 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 -- start start 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 0x7fb9600fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600fdba0 con 0x7fb9600fe7c0 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb967ba1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600fc6d0 con 0x7fb9600fde20 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96513c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96513c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41890/0 (socket says 172.21.15.153:41890) 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96513c700 1 -- 172.21.15.153:0/2335262432 learned_addr learned my addr 172.21.15.153:0/2335262432 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:57.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96593d700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 0x7fb9600fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96593d700 1 -- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 msgr2=0x7fb9600fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96593d700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96593d700 1 -- 172.21.15.153:0/2335262432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c005040 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.680+0000 7fb96513c700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb96593d700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 0x7fb9600fbc50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004940 tx=0x7fb95c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95c021070 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9600fc950 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb95c013c00 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95c00e070 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.681+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb960119fe0 con 0x7fb9600fde20 2024-04-05T03:09:57.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.682+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb960066b70 con 0x7fb9600fde20 2024-04-05T03:09:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.682+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb95c01c960 con 0x7fb9600fde20 2024-04-05T03:09:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.683+0000 7fb9567fc700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 0x7fb94c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.683+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb95c08e970 con 0x7fb9600fde20 2024-04-05T03:09:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.683+0000 7fb96513c700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 0x7fb94c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:57.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.683+0000 7fb96513c700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 0x7fb94c070150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9600fd380 tx=0x7fb95000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.688+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb95c057f80 con 0x7fb9600fde20 2024-04-05T03:09:57.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:57 smithi203 ceph-mon[95988]: Upgrade: Updating osd.3 2024-04-05T03:09:57.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:57 smithi203 ceph-mon[95988]: Deploying daemon osd.3 on smithi203 2024-04-05T03:09:57.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:57 smithi153 ceph-mon[106102]: Upgrade: Updating osd.3 2024-04-05T03:09:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:57 smithi153 ceph-mon[106102]: Deploying daemon osd.3 on smithi203 2024-04-05T03:09:57.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:09:57.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.903+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9600f9430 con 0x7fb94c06dca0 2024-04-05T03:09:57.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.904+0000 7fb9567fc700 1 -- 172.21.15.153:0/2335262432 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb9600f9430 con 0x7fb94c06dca0 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 msgr2=0x7fb94c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 0x7fb94c070150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb9600fd380 tx=0x7fb95000b040 comp rx=0 tx=0).stop 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 msgr2=0x7fb9600fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 0x7fb9600fbc50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004940 tx=0x7fb95c009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 shutdown_connections 2024-04-05T03:09:57.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb94c06dca0 0x7fb94c070150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9600fde20 0x7fb9600fbc50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 --2- 172.21.15.153:0/2335262432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9600fe7c0 0x7fb9600fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:57.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.906+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 >> 172.21.15.153:0/2335262432 conn(0x7fb9600f7870 msgr2=0x7fb9600f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:57.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.907+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 shutdown_connections 2024-04-05T03:09:57.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:57.907+0000 7fb967ba1700 1 -- 172.21.15.153:0/2335262432 wait complete. 2024-04-05T03:09:57.917 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:09:58.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.009+0000 7f51ab203700 1 -- 172.21.15.153:0/3496734928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a4102e00 msgr2=0x7f51a41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.009+0000 7f51ab203700 1 --2- 172.21.15.153:0/3496734928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a4102e00 0x7f51a41031d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5198004940 tx=0x7f5198009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 -- 172.21.15.153:0/3496734928 shutdown_connections 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 --2- 172.21.15.153:0/3496734928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a41037a0 0x7f51a40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 --2- 172.21.15.153:0/3496734928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a4102e00 0x7f51a41031d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 -- 172.21.15.153:0/3496734928 >> 172.21.15.153:0/3496734928 conn(0x7f51a40f89b0 msgr2=0x7f51a40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 -- 172.21.15.153:0/3496734928 shutdown_connections 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 -- 172.21.15.153:0/3496734928 wait complete. 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 Processor -- start 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.010+0000 7f51ab203700 1 -- start start 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a4102e00 0x7f51a41138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4114450 con 0x7f51a41037a0 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a410da50 con 0x7f51a4102e00 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41898/0 (socket says 172.21.15.153:41898) 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 -- 172.21.15.153:0/2842391763 learned_addr learned my addr 172.21.15.153:0/2842391763 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a8f9f700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a4102e00 0x7f51a41138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 -- 172.21.15.153:0/2842391763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a4102e00 msgr2=0x7f51a41138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a4102e00 0x7f51a41138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 -- 172.21.15.153:0/2842391763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5198005040 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a3fff700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f519000eb40 tx=0x7f519000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5190007cc0 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f519000f040 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a410dcd0 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519000cad0 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.011+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a410e210 con 0x7f51a41037a0 2024-04-05T03:09:58.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.012+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4100110 con 0x7f51a41037a0 2024-04-05T03:09:58.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.013+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f519000cc30 con 0x7f51a41037a0 2024-04-05T03:09:58.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.013+0000 7f51a17fa700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 0x7f5194070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.013+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f519008d000 con 0x7f51a41037a0 2024-04-05T03:09:58.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.013+0000 7f51a8f9f700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 0x7f5194070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.014+0000 7f51a8f9f700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 0x7f5194070150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5198009b30 tx=0x7f5198009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.018+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f519005a160 con 0x7f51a41037a0 2024-04-05T03:09:58.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.233+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51a410efb0 con 0x7f519406dca0 2024-04-05T03:09:58.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.234+0000 7f51a17fa700 1 -- 172.21.15.153:0/2842391763 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f51a410efb0 con 0x7f519406dca0 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 msgr2=0x7f5194070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 0x7f5194070150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5198009b30 tx=0x7f5198009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 msgr2=0x7f51a4113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f519000eb40 tx=0x7f519000b040 comp rx=0 tx=0).stop 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 shutdown_connections 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519406dca0 0x7f5194070150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51a4102e00 0x7f51a41138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 --2- 172.21.15.153:0/2842391763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51a41037a0 0x7f51a4113e00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 >> 172.21.15.153:0/2842391763 conn(0x7f51a40f89b0 msgr2=0x7f51a40fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 shutdown_connections 2024-04-05T03:09:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.237+0000 7f51ab203700 1 -- 172.21.15.153:0/2842391763 wait complete. 2024-04-05T03:09:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 -- 172.21.15.153:0/1591660181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 msgr2=0x7efbfc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 --2- 172.21.15.153:0/1591660181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc0fbd80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efbec004940 tx=0x7efbec009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 -- 172.21.15.153:0/1591660181 shutdown_connections 2024-04-05T03:09:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 --2- 172.21.15.153:0/1591660181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc0fbd80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 --2- 172.21.15.153:0/1591660181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 -- 172.21.15.153:0/1591660181 >> 172.21.15.153:0/1591660181 conn(0x7efbfc072bc0 msgr2=0x7efbfc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 -- 172.21.15.153:0/1591660181 shutdown_connections 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.338+0000 7efc03549700 1 -- 172.21.15.153:0/1591660181 wait complete. 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 Processor -- start 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 -- start start 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc10eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc10f6d0 con 0x7efbfc102710 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc03549700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc10f810 con 0x7efbfc101d70 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41918/0 (socket says 172.21.15.153:41918) 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 -- 172.21.15.153:0/2386892209 learned_addr learned my addr 172.21.15.153:0/2386892209 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc012e5700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc10eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 -- 172.21.15.153:0/2386892209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 msgr2=0x7efbfc10eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc10eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 -- 172.21.15.153:0/2386892209 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec005040 con 0x7efbfc102710 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc012e5700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc10eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.339+0000 7efc00ae4700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efbec009100 tx=0x7efbec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.340+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbec00e040 con 0x7efbfc102710 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.340+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbec013420 con 0x7efbfc102710 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.340+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbec01c610 con 0x7efbfc102710 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.340+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc117bb0 con 0x7efbfc102710 2024-04-05T03:09:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.340+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc1180c0 con 0x7efbfc102710 2024-04-05T03:09:58.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.341+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc104240 con 0x7efbfc102710 2024-04-05T03:09:58.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.341+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efbec013590 con 0x7efbfc102710 2024-04-05T03:09:58.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.342+0000 7efbf1ffb700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 0x7efbe8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.342+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efbec08e730 con 0x7efbfc102710 2024-04-05T03:09:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.342+0000 7efc012e5700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 0x7efbe8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.342+0000 7efc012e5700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 0x7efbe8070150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efbf40041d0 tx=0x7efbf4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.346+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efbec057de0 con 0x7efbfc102710 2024-04-05T03:09:58.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.555+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efbfc065e20 con 0x7efbe806dca0 2024-04-05T03:09:58.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.566+0000 7efbf1ffb700 1 -- 172.21.15.153:0/2386892209 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7efbfc065e20 con 0x7efbe806dca0 2024-04-05T03:09:58.565 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:09:58.565 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (4m) 24s ago 15m 26.0M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (2m) 24s ago 16m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (119s) 114s ago 14m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (4m) 24s ago 15m 80.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (10m) 24s ago 10m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (10m) 24s ago 10m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (10m) 114s ago 10m 61.4M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (10m) 114s ago 10m 113M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (6m) 24s ago 18m 504M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (5m) 114s ago 14m 449M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:09:58.566 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (3m) 24s ago 18m 45.7M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (2m) 114s ago 14m 36.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 24s ago 15m 16.7M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (5m) 114s ago 14m 17.3M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (99s) 24s ago 13m 88.5M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (64s) 24s ago 13m 92.7M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (29s) 24s ago 12m 12.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (12m) 114s ago 12m 265M 2240M 17.2.7-805-g6fda41ff a49113eacc58 48d718f52df0 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (12m) 114s ago 12m 232M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (11m) 114s ago 11m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:09:58.567 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (4m) 114s ago 14m 60.5M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:09:58.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 msgr2=0x7efbe8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 0x7efbe8070150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efbf40041d0 tx=0x7efbf4009040 comp rx=0 tx=0).stop 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 msgr2=0x7efbfc10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efbec009100 tx=0x7efbec009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 shutdown_connections 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efbe806dca0 0x7efbe8070150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efbfc101d70 0x7efbfc10eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 --2- 172.21.15.153:0/2386892209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbfc102710 0x7efbfc10f080 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 >> 172.21.15.153:0/2386892209 conn(0x7efbfc072bc0 msgr2=0x7efbfc0fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 shutdown_connections 2024-04-05T03:09:58.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.568+0000 7efc03549700 1 -- 172.21.15.153:0/2386892209 wait complete. 2024-04-05T03:09:58.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 -- 172.21.15.153:0/551368218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 msgr2=0x7fc0d00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/551368218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d00fbd80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0004940 tx=0x7fc0c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 -- 172.21.15.153:0/551368218 shutdown_connections 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/551368218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d00fbd80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/551368218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 -- 172.21.15.153:0/551368218 >> 172.21.15.153:0/551368218 conn(0x7fc0d0072bc0 msgr2=0x7fc0d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 -- 172.21.15.153:0/551368218 shutdown_connections 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.669+0000 7fc0d84d1700 1 -- 172.21.15.153:0/551368218 wait complete. 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 Processor -- start 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 -- start start 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d01129e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0113570 con 0x7fc0d0102710 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d84d1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d010c960 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d5a6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d626d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d01129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d5a6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48654/0 (socket says 172.21.15.153:48654) 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d5a6c700 1 -- 172.21.15.153:0/2597474716 learned_addr learned my addr 172.21.15.153:0/2597474716 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d626d700 1 -- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 msgr2=0x7fc0d0112f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d626d700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d626d700 1 -- 172.21.15.153:0/2597474716 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0005040 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d5a6c700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.670+0000 7fc0d626d700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d01129e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009fd0 tx=0x7fc0cc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0cc004e90 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d010cbe0 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0cc00f040 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0cc00ca50 con 0x7fc0d0101d70 2024-04-05T03:09:58.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d010d120 con 0x7fc0d0101d70 2024-04-05T03:09:58.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.671+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d0104240 con 0x7fc0d0101d70 2024-04-05T03:09:58.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.672+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc0cc007cc0 con 0x7fc0d0101d70 2024-04-05T03:09:58.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.672+0000 7fc0c6ffd700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 0x7fc0bc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:58.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.673+0000 7fc0d5a6c700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 0x7fc0bc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:58.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.673+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc0cc08cf60 con 0x7fc0d0101d70 2024-04-05T03:09:58.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.673+0000 7fc0d5a6c700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 0x7fc0bc070150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0007f50 tx=0x7fc0c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:58.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.678+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc0cc05a110 con 0x7fc0d0101d70 2024-04-05T03:09:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:58 smithi203 ceph-mon[95988]: pgmap v108: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:58 smithi153 ceph-mon[106102]: pgmap v108: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:09:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.957+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc0d010e090 con 0x7fc0d0101d70 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.958+0000 7fc0c6ffd700 1 -- 172.21.15.153:0/2597474716 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fc0cc059ca0 con 0x7fc0d0101d70 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 3, 2024-04-05T03:09:58.958 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 3 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 7, 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 7 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:09:58.959 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 msgr2=0x7fc0bc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 0x7fc0bc070150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0007f50 tx=0x7fc0c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 msgr2=0x7fc0d01129e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d01129e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009fd0 tx=0x7fc0cc00b040 comp rx=0 tx=0).stop 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 shutdown_connections 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc0bc06dca0 0x7fc0bc070150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d0101d70 0x7fc0d01129e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.960+0000 7fc0d84d1700 1 --2- 172.21.15.153:0/2597474716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0d0102710 0x7fc0d0112f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:58.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.961+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 >> 172.21.15.153:0/2597474716 conn(0x7fc0d0072bc0 msgr2=0x7fc0d00faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:58.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.961+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 shutdown_connections 2024-04-05T03:09:58.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:58.961+0000 7fc0d84d1700 1 -- 172.21.15.153:0/2597474716 wait complete. 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 -- 172.21.15.153:0/4045164564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fde90 msgr2=0x7f8eec0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 --2- 172.21.15.153:0/4045164564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fde90 0x7f8eec0fe260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8edc004940 tx=0x7f8edc009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 -- 172.21.15.153:0/4045164564 shutdown_connections 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 --2- 172.21.15.153:0/4045164564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fe830 0x7f8eec106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 --2- 172.21.15.153:0/4045164564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fde90 0x7f8eec0fe260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 -- 172.21.15.153:0/4045164564 >> 172.21.15.153:0/4045164564 conn(0x7f8eec072bc0 msgr2=0x7f8eec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 -- 172.21.15.153:0/4045164564 shutdown_connections 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 -- 172.21.15.153:0/4045164564 wait complete. 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.062+0000 7f8ef184c700 1 Processor -- start 2024-04-05T03:09:59.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ef184c700 1 -- start start 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ef184c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fde90 0x7f8eec113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ef184c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ef184c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec1146c0 con 0x7f8eec0fe830 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ef184c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec10dab0 con 0x7f8eec0fde90 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48668/0 (socket says 172.21.15.153:48668) 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 -- 172.21.15.153:0/670465054 learned_addr learned my addr 172.21.15.153:0/670465054 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:59.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8eeaffd700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fde90 0x7f8eec113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 -- 172.21.15.153:0/670465054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fde90 msgr2=0x7f8eec113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fde90 0x7f8eec113b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 -- 172.21.15.153:0/670465054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc005040 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.063+0000 7f8ee27fc700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400eb40 tx=0x7f8ed400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.064+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed4007cc0 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.064+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ed400f040 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.064+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec10dd30 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.064+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed400ca10 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.064+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec10e270 con 0x7f8eec0fe830 2024-04-05T03:09:59.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.065+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec065e30 con 0x7f8eec0fe830 2024-04-05T03:09:59.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.065+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8ed400cc80 con 0x7f8eec0fe830 2024-04-05T03:09:59.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.066+0000 7f8ee3fff700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 0x7f8ed8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.066+0000 7f8eeaffd700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 0x7f8ed8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.066+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8ed408d090 con 0x7f8eec0fe830 2024-04-05T03:09:59.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.066+0000 7f8eeaffd700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 0x7f8ed8070150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00b0d0 tx=0x7f8edc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.070+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ed405a2a0 con 0x7f8eec0fe830 2024-04-05T03:09:59.078 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:09:58 smithi203 systemd[1]: Stopping Ceph osd.3 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:09:59.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.307+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8eec109f00 con 0x7f8eec0fe830 2024-04-05T03:09:59.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.308+0000 7f8ee3fff700 1 -- 172.21.15.153:0/670465054 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7f8ed4059e30 con 0x7f8eec0fe830 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:09:59.308 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:09:59.309 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:09:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 msgr2=0x7f8ed8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 0x7f8ed8070150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00b0d0 tx=0x7f8edc009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 msgr2=0x7f8eec114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400eb40 tx=0x7f8ed400b040 comp rx=0 tx=0).stop 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 shutdown_connections 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ed806dca0 0x7f8ed8070150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eec0fde90 0x7f8eec113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 --2- 172.21.15.153:0/670465054 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eec0fe830 0x7f8eec114070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 >> 172.21.15.153:0/670465054 conn(0x7f8eec072bc0 msgr2=0x7f8eec1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 shutdown_connections 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.310+0000 7f8ef184c700 1 -- 172.21.15.153:0/670465054 wait complete. 2024-04-05T03:09:59.312 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:09:59.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 -- 172.21.15.153:0/3356337414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fde90 msgr2=0x7fa75c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 --2- 172.21.15.153:0/3356337414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fde90 0x7fa75c0fe260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa754009ce0 tx=0x7fa754004d10 comp rx=0 tx=0).stop 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 -- 172.21.15.153:0/3356337414 shutdown_connections 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 --2- 172.21.15.153:0/3356337414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fe830 0x7fa75c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 --2- 172.21.15.153:0/3356337414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fde90 0x7fa75c0fe260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 -- 172.21.15.153:0/3356337414 >> 172.21.15.153:0/3356337414 conn(0x7fa75c072bc0 msgr2=0x7fa75c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 -- 172.21.15.153:0/3356337414 shutdown_connections 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.412+0000 7fa763d13700 1 -- 172.21.15.153:0/3356337414 wait complete. 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 Processor -- start 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 -- start start 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 0x7fa75c1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c114720 con 0x7fa75c0fe830 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa763d13700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c10db30 con 0x7fa75c0fde90 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa761aaf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa761aaf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:43384/0 (socket says 172.21.15.153:43384) 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.413+0000 7fa761aaf700 1 -- 172.21.15.153:0/2315617317 learned_addr learned my addr 172.21.15.153:0/2315617317 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa761aaf700 1 -- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 msgr2=0x7fa75c1140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa7612ae700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 0x7fa75c1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa761aaf700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 0x7fa75c1140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa761aaf700 1 -- 172.21.15.153:0/2315617317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754009940 con 0x7fa75c0fde90 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa7612ae700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 0x7fa75c1140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa761aaf700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa75400b740 tx=0x7fa754004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75400e040 con 0x7fa75c0fde90 2024-04-05T03:09:59.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7540049b0 con 0x7fa75c0fde90 2024-04-05T03:09:59.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c10ddb0 con 0x7fa75c0fde90 2024-04-05T03:09:59.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75401c6d0 con 0x7fa75c0fde90 2024-04-05T03:09:59.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.414+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c10e2f0 con 0x7fa75c0fde90 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.415+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c101980 con 0x7fa75c0fde90 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.415+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa754012b00 con 0x7fa75c0fde90 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.416+0000 7fa7527fc700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 0x7fa748070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.416+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa75408f130 con 0x7fa75c0fde90 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.416+0000 7fa7612ae700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 0x7fa748070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.416+0000 7fa7612ae700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 0x7fa748070160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009fd0 tx=0x7fa74c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.421+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa754058760 con 0x7fa75c0fde90 2024-04-05T03:09:59.544 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[46382]: 2024-04-05T03:09:59.078+0000 7fe911a43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:09:59.545 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[46382]: 2024-04-05T03:09:59.078+0000 7fe911a43700 -1 osd.3 63 *** Got signal Terminated *** 2024-04-05T03:09:59.545 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[46382]: 2024-04-05T03:09:59.078+0000 7fe911a43700 -1 osd.3 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2597474716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: osd.3 marked itself down and dead 2024-04-05T03:09:59.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:09:59 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/670465054' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:09:59.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.643+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa75c109fa0 con 0x7fa74806dcb0 2024-04-05T03:09:59.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.644+0000 7fa7527fc700 1 -- 172.21.15.153:0/2315617317 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa74c00f040 con 0x7fa74806dcb0 2024-04-05T03:09:59.643 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:09:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:09:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:09:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "9/21 daemons upgraded", 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:09:59.644 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 msgr2=0x7fa748070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 0x7fa748070160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009fd0 tx=0x7fa74c00b040 comp rx=0 tx=0).stop 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 msgr2=0x7fa75c113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa75400b740 tx=0x7fa754004d10 comp rx=0 tx=0).stop 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 shutdown_connections 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74806dcb0 0x7fa748070160 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c0fde90 0x7fa75c113b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 --2- 172.21.15.153:0/2315617317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa75c0fe830 0x7fa75c1140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 >> 172.21.15.153:0/2315617317 conn(0x7fa75c072bc0 msgr2=0x7fa75c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:59.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 shutdown_connections 2024-04-05T03:09:59.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.646+0000 7fa763d13700 1 -- 172.21.15.153:0/2315617317 wait complete. 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.747+0000 7f7c086cd700 1 -- 172.21.15.153:0/3198651556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 msgr2=0x7f7c000fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.747+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3198651556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c000fce10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc004940 tx=0x7f7bfc009e30 comp rx=0 tx=0).stop 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- 172.21.15.153:0/3198651556 shutdown_connections 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3198651556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c000fce10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3198651556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c001031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- 172.21.15.153:0/3198651556 >> 172.21.15.153:0/3198651556 conn(0x7f7c000f89f0 msgr2=0x7f7c000fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- 172.21.15.153:0/3198651556 shutdown_connections 2024-04-05T03:09:59.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- 172.21.15.153:0/3198651556 wait complete. 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 Processor -- start 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- start start 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.748+0000 7f7c086cd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c001144b0 con 0x7f7c001037a0 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c086cd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0010dab0 con 0x7f7c00102e00 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c06469700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48702/0 (socket says 172.21.15.153:48702) 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c06469700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:43410/0 (socket says 172.21.15.153:43410) 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 -- 172.21.15.153:0/3207826259 learned_addr learned my addr 172.21.15.153:0/3207826259 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 -- 172.21.15.153:0/3207826259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 msgr2=0x7f7c00113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:09:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 -- 172.21.15.153:0/3207826259 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc005040 con 0x7f7c001037a0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c06469700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c05c68700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000c00 tx=0x7f7bfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bfc021070 con 0x7f7c001037a0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bfc013c00 con 0x7f7c001037a0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0010dd30 con 0x7f7c001037a0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bfc00e070 con 0x7f7c001037a0 2024-04-05T03:09:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.749+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0010e210 con 0x7f7c001037a0 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.750+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c001052d0 con 0x7f7c001037a0 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.751+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7bfc01ca60 con 0x7f7c001037a0 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.751+0000 7f7bf6ffd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 0x7f7bec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.751+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7bfc08e8c0 con 0x7f7c001037a0 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.751+0000 7f7c06469700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 0x7f7bec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:09:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.752+0000 7f7c06469700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 0x7f7bec070150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009b80 tx=0x7f7bf000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:09:59.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:09:59.756+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7bfc05baa0 con 0x7f7c001037a0 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2597474716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: osd.3 marked itself down and dead 2024-04-05T03:09:59.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:09:59 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/670465054' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:10:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.033+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7c000fe550 con 0x7f7c001037a0 2024-04-05T03:10:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.033+0000 7f7bf6ffd700 1 -- 172.21.15.153:0/3207826259 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7f7bfc004490 con 0x7f7c001037a0 2024-04-05T03:10:00.033 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_WARN 1 osds down 2024-04-05T03:10:00.033 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T03:10:00.033 INFO:teuthology.orchestra.run.smithi153.stdout: osd.3 (root=default,host=smithi203) is down 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 msgr2=0x7f7bec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 0x7f7bec070150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009b80 tx=0x7f7bf000b040 comp rx=0 tx=0).stop 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 msgr2=0x7f7c00113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000c00 tx=0x7f7bfc009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 shutdown_connections 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7bec06dca0 0x7f7bec070150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c00102e00 0x7f7c00113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 --2- 172.21.15.153:0/3207826259 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c001037a0 0x7f7c00113e60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 >> 172.21.15.153:0/3207826259 conn(0x7f7c000f89f0 msgr2=0x7f7c000fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 shutdown_connections 2024-04-05T03:10:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:00.035+0000 7f7c086cd700 1 -- 172.21.15.153:0/3207826259 wait complete. 2024-04-05T03:10:00.296 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:00 smithi203 podman[104961]: 2024-04-05 03:10:00.003974138 +0000 UTC m=+1.035208214 container died 48d718f52df0bca32be0b720d90a28e3b450757cca4cf5f90b529e66cccd8911 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-6fda41f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-05T03:10:00.730 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:00 smithi203 podman[104961]: 2024-04-05 03:10:00.463419325 +0000 UTC m=+1.494653442 container cleanup 48d718f52df0bca32be0b720d90a28e3b450757cca4cf5f90b529e66cccd8911 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-6fda41f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0) 2024-04-05T03:10:00.730 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:00 smithi203 bash[104961]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: pgmap v109: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: failed to encode map e64 with expected crc 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: osdmap e64: 6 total, 5 up, 6 in 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: Health detail: HEALTH_WARN 1 osds down 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: [WRN] OSD_DOWN: 1 osds down 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: osd.3 (root=default,host=smithi203) is down 2024-04-05T03:10:00.731 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:00 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3207826259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: pgmap v109: 65 pgs: 65 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: failed to encode map e64 with expected crc 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: osdmap e64: 6 total, 5 up, 6 in 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: Health detail: HEALTH_WARN 1 osds down 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: [WRN] OSD_DOWN: 1 osds down 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: osd.3 (root=default,host=smithi203) is down 2024-04-05T03:10:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:00 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3207826259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:10:01.046 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:00 smithi203 podman[104988]: 2024-04-05 03:10:00.731857017 +0000 UTC m=+0.729472773 container remove 48d718f52df0bca32be0b720d90a28e3b450757cca4cf5f90b529e66cccd8911 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=quincy-6fda41f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-05T03:10:01.385 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:00.987863194 +0000 UTC m=+0.018486096 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:01.385 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:01.099984317 +0000 UTC m=+0.130607238 container create 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-04-05T03:10:01.385 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:01.310800947 +0000 UTC m=+0.341423841 container init 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:10:01.385 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:01.314015746 +0000 UTC m=+0.344638632 container start 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-05T03:10:01.385 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:01.385724186 +0000 UTC m=+0.416347065 container attach 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-05T03:10:01.796 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:01 smithi203 podman[105084]: 2024-04-05 03:10:01.605672389 +0000 UTC m=+0.636295272 container died 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-0028525, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-05T03:10:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:01.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: osdmap e65: 6 total, 5 up, 6 in 2024-04-05T03:10:01.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:01 smithi203 ceph-mon[95988]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: osdmap e65: 6 total, 5 up, 6 in 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: failed to encode map e65 with expected crc 2024-04-05T03:10:01.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:01 smithi153 ceph-mon[106102]: failed to encode map e65 with expected crc 2024-04-05T03:10:02.785 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:02 smithi203 ceph-mon[95988]: pgmap v112: 65 pgs: 3 active+undersized+degraded, 13 peering, 10 stale+active+clean, 4 active+undersized, 35 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7/318 objects degraded (2.201%) 2024-04-05T03:10:02.785 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:02 smithi203 ceph-mon[95988]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-05T03:10:02.785 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:02 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 7/318 objects degraded (2.201%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:02.785 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 podman[105110]: 2024-04-05 03:10:02.332314219 +0000 UTC m=+0.721403650 container remove 7272ff4d313aae1d5fce61be1850ce17908e2af92eb3f5609b5b1dc12b91b42f (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-05T03:10:02.786 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-04-05T03:10:02.786 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 systemd[1]: Stopped Ceph osd.3 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:10:02.786 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.3.service: Consumed 1min 18.280s CPU time. 2024-04-05T03:10:02.786 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 systemd[1]: Starting Ceph osd.3 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:10:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:02 smithi153 ceph-mon[106102]: pgmap v112: 65 pgs: 3 active+undersized+degraded, 13 peering, 10 stale+active+clean, 4 active+undersized, 35 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7/318 objects degraded (2.201%) 2024-04-05T03:10:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:02 smithi153 ceph-mon[106102]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-05T03:10:02.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:02 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 7/318 objects degraded (2.201%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:03.035 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 podman[105218]: 2024-04-05 03:10:02.688924082 +0000 UTC m=+0.017857376 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:03.035 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:02 smithi203 podman[105218]: 2024-04-05 03:10:02.822499712 +0000 UTC m=+0.151432983 container create c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-05T03:10:03.035 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 podman[105218]: 2024-04-05 03:10:03.036699171 +0000 UTC m=+0.365632443 container init c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=squid-0028525, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:10:03.296 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 podman[105218]: 2024-04-05 03:10:03.040010935 +0000 UTC m=+0.368944216 container start c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-05T03:10:03.296 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 podman[105218]: 2024-04-05 03:10:03.11162468 +0000 UTC m=+0.440557966 container attach c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.build-date=20240403, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-05T03:10:03.796 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 bash[105218]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate[105232]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-05T03:10:03.797 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:03 smithi203 podman[105218]: 2024-04-05 03:10:03.516276272 +0000 UTC m=+0.845209546 container died c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=squid-0028525, org.label-schema.license=GPLv2) 2024-04-05T03:10:04.457 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 podman[105357]: 2024-04-05 03:10:04.132558331 +0000 UTC m=+0.610941988 container remove c21272229b2d4980eceaf196787c028eb0a76c65576728dd0f5d743a00f407a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:10:04.458 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 podman[105424]: 2024-04-05 03:10:04.249225508 +0000 UTC m=+0.018416866 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:04.458 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 podman[105424]: 2024-04-05 03:10:04.350316591 +0000 UTC m=+0.119507943 container create 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-05T03:10:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:04 smithi153 ceph-mon[106102]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 45/318 objects degraded (14.151%) 2024-04-05T03:10:05.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:04 smithi203 ceph-mon[95988]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 13 peering, 13 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 45/318 objects degraded (14.151%) 2024-04-05T03:10:05.046 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 podman[105424]: 2024-04-05 03:10:04.553440966 +0000 UTC m=+0.322632317 container init 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-05T03:10:05.046 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 podman[105424]: 2024-04-05 03:10:04.55650554 +0000 UTC m=+0.325696895 container start 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-05T03:10:05.046 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 ceph-osd[105442]: -- 172.21.15.203:0/4145337002 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c65f815680 con 0x55c65f7dec00 2024-04-05T03:10:05.047 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 bash[105424]: 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 2024-04-05T03:10:05.047 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:04 smithi203 systemd[1]: Started Ceph osd.3 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:10:05.546 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:05 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:10:05.098+0000 7f9865b817c0 -1 Falling back to public interface 2024-04-05T03:10:05.956 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:05.956 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:05.956 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:05.956 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:05 smithi203 ceph-mon[95988]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:06.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:06.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:06.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:06.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:05 smithi153 ceph-mon[106102]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:07.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:06 smithi203 ceph-mon[95988]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-05T03:10:07.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:06 smithi153 ceph-mon[106102]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-04-05T03:10:08.036 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:07 smithi203 ceph-mon[95988]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:08.036 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:07 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 66/318 objects degraded (20.755%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:08.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:07 smithi153 ceph-mon[106102]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:08.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:07 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 66/318 objects degraded (20.755%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:10.546 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:10:10.334+0000 7f9865b817c0 -1 osd.3 63 log_to_monitors true 2024-04-05T03:10:10.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-mon[95988]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:10.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:10.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:10.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-mon[95988]: from='osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T03:10:10.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:10 smithi203 ceph-mon[95988]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T03:10:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:10 smithi153 ceph-mon[106102]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:10 smithi153 ceph-mon[106102]: from='osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T03:10:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:10 smithi153 ceph-mon[106102]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T03:10:11.603 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:10:11.454+0000 7f9852fd1700 -1 osd.3 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: from='osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: osdmap e66: 6 total, 5 up, 6 in 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: failed to encode map e66 with expected crc 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: from='osd.3 ' entity='osd.3' 2024-04-05T03:10:11.896 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:11 smithi203 ceph-mon[95988]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: from='osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: osdmap e66: 6 total, 5 up, 6 in 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: failed to encode map e66 with expected crc 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: from='osd.3 ' entity='osd.3' 2024-04-05T03:10:12.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:11 smithi153 ceph-mon[106102]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 66/318 objects degraded (20.755%) 2024-04-05T03:10:12.973 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: failed to encode map e66 with expected crc 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: failed to encode map e67 with expected crc 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898] boot 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: osdmap e67: 6 total, 6 up, 6 in 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:12.974 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: failed to encode map e66 with expected crc 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: failed to encode map e67 with expected crc 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: osd.3 [v2:172.21.15.203:6800/2106212898,v1:172.21.15.203:6801/2106212898] boot 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: osdmap e67: 6 total, 6 up, 6 in 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: osdmap e68: 6 total, 6 up, 6 in 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: pgmap v121: 65 pgs: 8 peering, 16 active+undersized+degraded, 14 active+undersized, 27 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 57/318 objects degraded (17.925%) 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:14.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:14 smithi153 ceph-mon[106102]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:14.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: osdmap e68: 6 total, 6 up, 6 in 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: failed to encode map e68 with expected crc 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: pgmap v121: 65 pgs: 8 peering, 16 active+undersized+degraded, 14 active+undersized, 27 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 57/318 objects degraded (17.925%) 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:14.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:14 smithi203 ceph-mon[95988]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-05T03:10:15.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:15 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 57/318 objects degraded (17.925%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:15.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:15 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 57/318 objects degraded (17.925%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:16 smithi203 ceph-mon[95988]: pgmap v122: 65 pgs: 8 peering, 11 active+undersized+degraded, 9 active+undersized, 37 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 3 op/s; 43/318 objects degraded (13.522%) 2024-04-05T03:10:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:16 smithi153 ceph-mon[106102]: pgmap v122: 65 pgs: 8 peering, 11 active+undersized+degraded, 9 active+undersized, 37 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 3 op/s; 43/318 objects degraded (13.522%) 2024-04-05T03:10:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:17 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/318 objects degraded (13.522%), 11 pgs degraded) 2024-04-05T03:10:17.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:17 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:10:18.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:17 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/318 objects degraded (13.522%), 11 pgs degraded) 2024-04-05T03:10:18.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:17 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:10:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:18 smithi203 ceph-mon[95988]: pgmap v123: 65 pgs: 8 peering, 57 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-05T03:10:19.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:18 smithi153 ceph-mon[106102]: pgmap v123: 65 pgs: 8 peering, 57 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-05T03:10:20.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:20 smithi153 ceph-mon[106102]: pgmap v124: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-05T03:10:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:20 smithi203 ceph-mon[95988]: pgmap v124: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-04-05T03:10:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:22 smithi203 ceph-mon[95988]: pgmap v125: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T03:10:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:22 smithi153 ceph-mon[106102]: pgmap v125: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T03:10:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:24 smithi203 ceph-mon[95988]: pgmap v126: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:25.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:24 smithi153 ceph-mon[106102]: pgmap v126: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:26 smithi203 ceph-mon[95988]: pgmap v127: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:10:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:26 smithi153 ceph-mon[106102]: pgmap v127: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:10:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:28 smithi203 ceph-mon[95988]: pgmap v128: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:28 smithi153 ceph-mon[106102]: pgmap v128: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:29 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:29 smithi203 ceph-mon[95988]: Upgrade: osd.4 is safe to restart 2024-04-05T03:10:29.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:29.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:29 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T03:10:29.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:29 smithi153 ceph-mon[106102]: Upgrade: osd.4 is safe to restart 2024-04-05T03:10:30.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 -- 172.21.15.153:0/90389776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 msgr2=0x7f61480fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 --2- 172.21.15.153:0/90389776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f61480fe250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6138004940 tx=0x7f6138009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 -- 172.21.15.153:0/90389776 shutdown_connections 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 --2- 172.21.15.153:0/90389776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 --2- 172.21.15.153:0/90389776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f61480fe250 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 -- 172.21.15.153:0/90389776 >> 172.21.15.153:0/90389776 conn(0x7f61480f7870 msgr2=0x7f61480f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 -- 172.21.15.153:0/90389776 shutdown_connections 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 -- 172.21.15.153:0/90389776 wait complete. 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.152+0000 7f614dbcd700 1 Processor -- start 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f614dbcd700 1 -- start start 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f614dbcd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f614dbcd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f614dbcd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61481144a0 con 0x7f61480fde80 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f614dbcd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614810daa0 con 0x7f61480fe820 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48644/0 (socket says 172.21.15.153:48644) 2024-04-05T03:10:30.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 -- 172.21.15.153:0/36870109 learned_addr learned my addr 172.21.15.153:0/36870109 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f6146ffd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 -- 172.21.15.153:0/36870109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 msgr2=0x7f6148113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148113e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 -- 172.21.15.153:0/36870109 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138005040 con 0x7f61480fde80 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f6146ffd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:10:30.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.153+0000 7f61477fe700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6138004890 tx=0x7f6138009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.154+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613800e040 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.154+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6138009af0 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.154+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614810dd80 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.154+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613801c630 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.154+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614810e2c0 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.155+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148066b10 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.155+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f613800bbc0 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.156+0000 7f612ffff700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 0x7f6130070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.156+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f613808e6c0 con 0x7f61480fde80 2024-04-05T03:10:30.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.156+0000 7f6146ffd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 0x7f6130070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.156+0000 7f6146ffd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 0x7f6130070150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f613c009fd0 tx=0x7f613c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.160+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f613805b940 con 0x7f61480fde80 2024-04-05T03:10:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.374+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6148109e90 con 0x7f613006dca0 2024-04-05T03:10:30.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.375+0000 7f612ffff700 1 -- 172.21.15.153:0/36870109 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f6148109e90 con 0x7f613006dca0 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 msgr2=0x7f6130070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 0x7f6130070150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f613c009fd0 tx=0x7f613c009ee0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 msgr2=0x7f6148113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6138004890 tx=0x7f6138009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 shutdown_connections 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f613006dca0 0x7f6130070150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61480fde80 0x7f6148113910 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 --2- 172.21.15.153:0/36870109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61480fe820 0x7f6148113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 >> 172.21.15.153:0/36870109 conn(0x7f61480f7870 msgr2=0x7f61480f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:30.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 shutdown_connections 2024-04-05T03:10:30.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.377+0000 7f614dbcd700 1 -- 172.21.15.153:0/36870109 wait complete. 2024-04-05T03:10:30.387 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 -- 172.21.15.153:0/1559450759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 msgr2=0x7f8ec41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 --2- 172.21.15.153:0/1559450759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec41031e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4004940 tx=0x7f8eb4009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 -- 172.21.15.153:0/1559450759 shutdown_connections 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 --2- 172.21.15.153:0/1559450759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 0x7f8ec40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 --2- 172.21.15.153:0/1559450759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec41031e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 -- 172.21.15.153:0/1559450759 >> 172.21.15.153:0/1559450759 conn(0x7f8ec40f8980 msgr2=0x7f8ec40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 -- 172.21.15.153:0/1559450759 shutdown_connections 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 -- 172.21.15.153:0/1559450759 wait complete. 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.480+0000 7f8ec929b700 1 Processor -- start 2024-04-05T03:10:30.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec929b700 1 -- start start 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec929b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec929b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 0x7f8ec41140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec929b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4114730 con 0x7f8ec4102e10 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec929b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec410db40 con 0x7f8ec41037b0 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48664/0 (socket says 172.21.15.153:48664) 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 -- 172.21.15.153:0/3629834646 learned_addr learned my addr 172.21.15.153:0/3629834646 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8eba7fc700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 0x7f8ec41140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 -- 172.21.15.153:0/3629834646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 msgr2=0x7f8ec41140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 0x7f8ec41140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 -- 172.21.15.153:0/3629834646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4005040 con 0x7f8ec4102e10 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.481+0000 7f8ec2ffd700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009150 tx=0x7f8eb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.482+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb400e040 con 0x7f8ec4102e10 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.482+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec410ddc0 con 0x7f8ec4102e10 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.482+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec410e380 con 0x7f8ec4102e10 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.482+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8eb4013420 con 0x7f8ec4102e10 2024-04-05T03:10:30.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.482+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb401c610 con 0x7f8ec4102e10 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.483+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea40052b0 con 0x7f8ec4102e10 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.483+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8eb4013a90 con 0x7f8ec4102e10 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.484+0000 7f8ebbfff700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 0x7f8eb0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.484+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8eb408e580 con 0x7f8ec4102e10 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.484+0000 7f8eba7fc700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 0x7f8eb0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.484+0000 7f8eba7fc700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 0x7f8eb0070160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ec410f440 tx=0x7f8eac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.489+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8eb4057cd0 con 0x7f8ec4102e10 2024-04-05T03:10:30.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.703+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ea4000bc0 con 0x7f8eb006dcb0 2024-04-05T03:10:30.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.704+0000 7f8ebbfff700 1 -- 172.21.15.153:0/3629834646 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8ea4000bc0 con 0x7f8eb006dcb0 2024-04-05T03:10:30.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.706+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 msgr2=0x7f8eb0070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.706+0000 7f8ec929b700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 0x7f8eb0070160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ec410f440 tx=0x7f8eac00b040 comp rx=0 tx=0).stop 2024-04-05T03:10:30.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.706+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 msgr2=0x7f8ec4113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.706+0000 7f8ec929b700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009150 tx=0x7f8eb4009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 shutdown_connections 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8eb006dcb0 0x7f8eb0070160 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ec4102e10 0x7f8ec4113ba0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 --2- 172.21.15.153:0/3629834646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ec41037b0 0x7f8ec41140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 >> 172.21.15.153:0/3629834646 conn(0x7f8ec40f8980 msgr2=0x7f8ec4107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 shutdown_connections 2024-04-05T03:10:30.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.707+0000 7f8ec929b700 1 -- 172.21.15.153:0/3629834646 wait complete. 2024-04-05T03:10:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:30 smithi203 ceph-mon[95988]: pgmap v129: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T03:10:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:30.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.809+0000 7f827279d700 1 -- 172.21.15.153:0/567771526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 msgr2=0x7f826c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.809+0000 7f827279d700 1 --2- 172.21.15.153:0/567771526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c106c90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8268004940 tx=0x7f8268009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:30.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- 172.21.15.153:0/567771526 shutdown_connections 2024-04-05T03:10:30.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 --2- 172.21.15.153:0/567771526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c106c90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 --2- 172.21.15.153:0/567771526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 0x7f826c0fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- 172.21.15.153:0/567771526 >> 172.21.15.153:0/567771526 conn(0x7f826c0f76a0 msgr2=0x7f826c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:30.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- 172.21.15.153:0/567771526 shutdown_connections 2024-04-05T03:10:30.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- 172.21.15.153:0/567771526 wait complete. 2024-04-05T03:10:30.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:30 smithi153 ceph-mon[106102]: pgmap v129: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:30.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:30.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T03:10:30.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 Processor -- start 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- start start 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 0x7f826c113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1144d0 con 0x7f826c0fe5c0 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.810+0000 7f827279d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c10d8c0 con 0x7f826c0fdc20 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f827179b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 0x7f826c113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48688/0 (socket says 172.21.15.153:48688) 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 -- 172.21.15.153:0/698329559 learned_addr learned my addr 172.21.15.153:0/698329559 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 -- 172.21.15.153:0/698329559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 msgr2=0x7f826c113940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 0x7f826c113940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 -- 172.21.15.153:0/698329559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8268005040 con 0x7f826c0fe5c0 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8270f9a700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8268000c00 tx=0x7f8268009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8268021070 con 0x7f826c0fe5c0 2024-04-05T03:10:30.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8268013c00 con 0x7f826c0fe5c0 2024-04-05T03:10:30.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f826800e070 con 0x7f826c0fe5c0 2024-04-05T03:10:30.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.811+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c10db40 con 0x7f826c0fe5c0 2024-04-05T03:10:30.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.812+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c10e080 con 0x7f826c0fe5c0 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.813+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c0668d0 con 0x7f826c0fe5c0 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.813+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f826801c400 con 0x7f826c0fe5c0 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.814+0000 7f8261ffb700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 0x7f8258070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.814+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f826808e8d0 con 0x7f826c0fe5c0 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.814+0000 7f827179b700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 0x7f8258070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:30.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.814+0000 7f827179b700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 0x7f8258070150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f825c009b80 tx=0x7f825c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:30.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:30.818+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f826805bab0 con 0x7f826c0fe5c0 2024-04-05T03:10:31.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.025+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f826c109cc0 con 0x7f825806dca0 2024-04-05T03:10:31.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.035+0000 7f8261ffb700 1 -- 172.21.15.153:0/698329559 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f826c109cc0 con 0x7f825806dca0 2024-04-05T03:10:31.034 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 57s ago 16m 26.0M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (2m) 57s ago 17m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (2m) 22s ago 15m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (4m) 57s ago 16m 80.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (10m) 57s ago 10m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (10m) 57s ago 10m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (10m) 22s ago 10m 61.3M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (10m) 22s ago 10m 112M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (6m) 57s ago 18m 504M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (5m) 22s ago 15m 451M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (3m) 57s ago 18m 45.7M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (3m) 22s ago 14m 39.6M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 57s ago 16m 16.7M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:10:31.035 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (5m) 22s ago 15m 17.6M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (2m) 57s ago 14m 88.5M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (97s) 57s ago 13m 92.7M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (61s) 57s ago 13m 12.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (26s) 22s ago 13m 15.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (12m) 22s ago 12m 234M 2240M 17.2.7-805-g6fda41ff a49113eacc58 8f1829157ed2 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (12m) 22s ago 12m 235M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:10:31.036 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (5m) 22s ago 14m 63.7M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 msgr2=0x7f8258070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 0x7f8258070150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f825c009b80 tx=0x7f825c00b040 comp rx=0 tx=0).stop 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 msgr2=0x7f826c113e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8268000c00 tx=0x7f8268009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 shutdown_connections 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f825806dca0 0x7f8258070150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826c0fdc20 0x7f826c113940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 --2- 172.21.15.153:0/698329559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f826c0fe5c0 0x7f826c113e80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 >> 172.21.15.153:0/698329559 conn(0x7f826c0f76a0 msgr2=0x7f826c105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 shutdown_connections 2024-04-05T03:10:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.037+0000 7f827279d700 1 -- 172.21.15.153:0/698329559 wait complete. 2024-04-05T03:10:31.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- 172.21.15.153:0/609572099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a881037a0 msgr2=0x7f2a880fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 --2- 172.21.15.153:0/609572099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a881037a0 0x7f2a880fce10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c004940 tx=0x7f2a7c009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:31.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- 172.21.15.153:0/609572099 shutdown_connections 2024-04-05T03:10:31.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 --2- 172.21.15.153:0/609572099 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a881037a0 0x7f2a880fce10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 --2- 172.21.15.153:0/609572099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a88102e00 0x7f2a881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- 172.21.15.153:0/609572099 >> 172.21.15.153:0/609572099 conn(0x7f2a880f89b0 msgr2=0x7f2a880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- 172.21.15.153:0/609572099 shutdown_connections 2024-04-05T03:10:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- 172.21.15.153:0/609572099 wait complete. 2024-04-05T03:10:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 Processor -- start 2024-04-05T03:10:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.138+0000 7f2a8e325700 1 -- start start 2024-04-05T03:10:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a8e325700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a8e325700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a881037a0 0x7f2a88113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a8e325700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88114510 con 0x7f2a88102e00 2024-04-05T03:10:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a8e325700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8810db10 con 0x7f2a881037a0 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48712/0 (socket says 172.21.15.153:48712) 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 -- 172.21.15.153:0/2991148767 learned_addr learned my addr 172.21.15.153:0/2991148767 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a877fe700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a881037a0 0x7f2a88113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 -- 172.21.15.153:0/2991148767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a881037a0 msgr2=0x7f2a88113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a881037a0 0x7f2a88113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 -- 172.21.15.153:0/2991148767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c005040 con 0x7f2a88102e00 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a87fff700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c8c0 tx=0x7f2a780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a7800a760 con 0x7f2a88102e00 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8810ddf0 con 0x7f2a88102e00 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a7800ad90 con 0x7f2a88102e00 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.139+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a780113a0 con 0x7f2a88102e00 2024-04-05T03:10:31.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.140+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8810e380 con 0x7f2a88102e00 2024-04-05T03:10:31.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.140+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a881052d0 con 0x7f2a88102e00 2024-04-05T03:10:31.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.141+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2a78012400 con 0x7f2a88102e00 2024-04-05T03:10:31.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.141+0000 7f2a84ff9700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 0x7f2a70070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.141+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2a7808d4d0 con 0x7f2a88102e00 2024-04-05T03:10:31.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.141+0000 7f2a877fe700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 0x7f2a70070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.142+0000 7f2a877fe700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 0x7f2a70070150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2a8810edf0 tx=0x7f2a7c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.146+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2a7805a6b0 con 0x7f2a88102e00 2024-04-05T03:10:31.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.422+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2a8810efd0 con 0x7f2a88102e00 2024-04-05T03:10:31.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.423+0000 7f2a84ff9700 1 -- 172.21.15.153:0/2991148767 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f2a7805a240 con 0x7f2a88102e00 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 2, 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:10:31.423 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:10:31.424 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 6, 2024-04-05T03:10:31.424 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 8 2024-04-05T03:10:31.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:10:31.424 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:10:31.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 msgr2=0x7f2a70070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 0x7f2a70070150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2a8810edf0 tx=0x7f2a7c009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:31.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 msgr2=0x7f2a88113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c8c0 tx=0x7f2a780044c0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 shutdown_connections 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a7006dca0 0x7f2a70070150 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a88102e00 0x7f2a88113980 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 --2- 172.21.15.153:0/2991148767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a881037a0 0x7f2a88113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 >> 172.21.15.153:0/2991148767 conn(0x7f2a880f89b0 msgr2=0x7f2a881021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 shutdown_connections 2024-04-05T03:10:31.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.426+0000 7f2a8e325700 1 -- 172.21.15.153:0/2991148767 wait complete. 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/2936141377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 msgr2=0x7fdb84102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/2936141377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb84102140 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009ce0 tx=0x7fdb80004d10 comp rx=0 tx=0).stop 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/2936141377 shutdown_connections 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/2936141377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 0x7fdb840fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/2936141377 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb84102140 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/2936141377 >> 172.21.15.153:0/2936141377 conn(0x7fdb84072bc0 msgr2=0x7fdb84072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/2936141377 shutdown_connections 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/2936141377 wait complete. 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 Processor -- start 2024-04-05T03:10:31.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.530+0000 7fdb8ae9f700 1 -- start start 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb8ae9f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb8ae9f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 0x7fdb8410f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb8ae9f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8410f670 con 0x7fdb84101d70 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb8ae9f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8410f7b0 con 0x7fdb84102710 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48724/0 (socket says 172.21.15.153:48724) 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 -- 172.21.15.153:0/3435499272 learned_addr learned my addr 172.21.15.153:0/3435499272 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb7bfff700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 0x7fdb8410f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 -- 172.21.15.153:0/3435499272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 msgr2=0x7fdb8410f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 0x7fdb8410f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 -- 172.21.15.153:0/3435499272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb80009940 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.531+0000 7fdb88c3b700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000ba50 tx=0x7fdb80004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.532+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8000e040 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.532+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb8001cb20 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.532+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb84117b50 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.532+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb80012990 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.532+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb84118060 con 0x7fdb84101d70 2024-04-05T03:10:31.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.533+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb840ff080 con 0x7fdb84101d70 2024-04-05T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.534+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fdb8001cc90 con 0x7fdb84101d70 2024-04-05T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.534+0000 7fdb797fa700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 0x7fdb6c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.534+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdb8008f7e0 con 0x7fdb84101d70 2024-04-05T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.534+0000 7fdb7bfff700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 0x7fdb6c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.535+0000 7fdb7bfff700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 0x7fdb6c070150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdb84066000 tx=0x7fdb7400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.543+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdb80058e10 con 0x7fdb84101d70 2024-04-05T03:10:31.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.783+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdb8404bed0 con 0x7fdb84101d70 2024-04-05T03:10:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:31 smithi153 ceph-mon[106102]: Upgrade: Updating osd.4 2024-04-05T03:10:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:31 smithi153 ceph-mon[106102]: Deploying daemon osd.4 on smithi203 2024-04-05T03:10:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:31 smithi153 ceph-mon[106102]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:31.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2991148767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:31.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.783+0000 7fdb797fa700 1 -- 172.21.15.153:0/3435499272 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7fdb8005c4d0 con 0x7fdb84101d70 2024-04-05T03:10:31.783 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:10:31.783 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:10:31.784 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:10:31.785 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:10:31.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 msgr2=0x7fdb6c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 0x7fdb6c070150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdb84066000 tx=0x7fdb7400b040 comp rx=0 tx=0).stop 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 msgr2=0x7fdb8410eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000ba50 tx=0x7fdb80004d10 comp rx=0 tx=0).stop 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 shutdown_connections 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb6c06dca0 0x7fdb6c070150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.785+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb84101d70 0x7fdb8410eae0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.786+0000 7fdb8ae9f700 1 --2- 172.21.15.153:0/3435499272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb84102710 0x7fdb8410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.786+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 >> 172.21.15.153:0/3435499272 conn(0x7fdb84072bc0 msgr2=0x7fdb840fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.786+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 shutdown_connections 2024-04-05T03:10:31.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.786+0000 7fdb8ae9f700 1 -- 172.21.15.153:0/3435499272 wait complete. 2024-04-05T03:10:31.788 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:10:31.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:31 smithi203 ceph-mon[95988]: Upgrade: Updating osd.4 2024-04-05T03:10:31.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:31 smithi203 ceph-mon[95988]: Deploying daemon osd.4 on smithi203 2024-04-05T03:10:31.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:31 smithi203 ceph-mon[95988]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:31.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2991148767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 -- 172.21.15.153:0/1211208322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc0fb3b0 msgr2=0x7fa1dc0fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 --2- 172.21.15.153:0/1211208322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc0fb820 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0004940 tx=0x7fa1d0009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 -- 172.21.15.153:0/1211208322 shutdown_connections 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 --2- 172.21.15.153:0/1211208322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc0fb820 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 --2- 172.21.15.153:0/1211208322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc102f70 0x7fa1dc103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 -- 172.21.15.153:0/1211208322 >> 172.21.15.153:0/1211208322 conn(0x7fa1dc072310 msgr2=0x7fa1dc072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.887+0000 7fa1e0880700 1 -- 172.21.15.153:0/1211208322 shutdown_connections 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 -- 172.21.15.153:0/1211208322 wait complete. 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 Processor -- start 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 -- start start 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc1144f0 con 0x7fa1dc102f70 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1e0880700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc10daf0 con 0x7fa1dc0fb3b0 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1d9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1d9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48736/0 (socket says 172.21.15.153:48736) 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1d9d9b700 1 -- 172.21.15.153:0/3371962732 learned_addr learned my addr 172.21.15.153:0/3371962732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.888+0000 7fa1da59c700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1d9d9b700 1 -- 172.21.15.153:0/3371962732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc0fb3b0 msgr2=0x7fa1dc113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1d9d9b700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1d9d9b700 1 -- 172.21.15.153:0/3371962732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0005040 con 0x7fa1dc102f70 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1d9d9b700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000b3b0 tx=0x7fa1d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d000e040 con 0x7fa1dc102f70 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1d0013420 con 0x7fa1dc102f70 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc10dd70 con 0x7fa1dc102f70 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d001c630 con 0x7fa1dc102f70 2024-04-05T03:10:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.889+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc10e330 con 0x7fa1dc102f70 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.890+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc105350 con 0x7fa1dc102f70 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.890+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa1d0013590 con 0x7fa1dc102f70 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.891+0000 7fa1caffd700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 0x7fa1c4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.891+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa1d008e6b0 con 0x7fa1dc102f70 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.891+0000 7fa1da59c700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 0x7fa1c4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:31.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.891+0000 7fa1da59c700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 0x7fa1c4070150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc004f40 tx=0x7fa1cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:31.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:31.895+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa1d005b890 con 0x7fa1dc102f70 2024-04-05T03:10:32.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.108+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1dc10f020 con 0x7fa1c406dca0 2024-04-05T03:10:32.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.109+0000 7fa1caffd700 1 -- 172.21.15.153:0/3371962732 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fa1dc10f020 con 0x7fa1c406dca0 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "10/21 daemons upgraded", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:10:32.109 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 msgr2=0x7fa1c4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 0x7fa1c4070150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc004f40 tx=0x7fa1cc00b040 comp rx=0 tx=0).stop 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 msgr2=0x7fa1dc113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000b3b0 tx=0x7fa1d0009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 shutdown_connections 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa1c406dca0 0x7fa1c4070150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1dc0fb3b0 0x7fa1dc113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 --2- 172.21.15.153:0/3371962732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1dc102f70 0x7fa1dc113ea0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.111+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 >> 172.21.15.153:0/3371962732 conn(0x7fa1dc072310 msgr2=0x7fa1dc0f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.112+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 shutdown_connections 2024-04-05T03:10:32.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.112+0000 7fa1e0880700 1 -- 172.21.15.153:0/3371962732 wait complete. 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 -- 172.21.15.153:0/2457500903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fe830 msgr2=0x7f8744106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2457500903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fe830 0x7f8744106f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8740004940 tx=0x7f8740009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 -- 172.21.15.153:0/2457500903 shutdown_connections 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2457500903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fe830 0x7f8744106f00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2457500903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fde90 0x7f87440fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 -- 172.21.15.153:0/2457500903 >> 172.21.15.153:0/2457500903 conn(0x7f8744072bc0 msgr2=0x7f8744072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:32.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 -- 172.21.15.153:0/2457500903 shutdown_connections 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.213+0000 7f874c5fd700 1 -- 172.21.15.153:0/2457500903 wait complete. 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 Processor -- start 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 -- start start 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fe830 0x7f8744114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87441146b0 con 0x7f87440fde90 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874c5fd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874410daa0 con 0x7f87440fe830 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48758/0 (socket says 172.21.15.153:48758) 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 -- 172.21.15.153:0/2254529012 learned_addr learned my addr 172.21.15.153:0/2254529012 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f8749b98700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fe830 0x7f8744114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 -- 172.21.15.153:0/2254529012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fe830 msgr2=0x7f8744114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fe830 0x7f8744114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 -- 172.21.15.153:0/2254529012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740005040 con 0x7f87440fde90 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.214+0000 7f874a399700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f873400eb40 tx=0x7f873400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.215+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8734007cc0 con 0x7f87440fde90 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.215+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f873400f040 con 0x7f87440fde90 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.215+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f873400ca10 con 0x7f87440fde90 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.215+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874410dd20 con 0x7f87440fde90 2024-04-05T03:10:32.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.215+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874410e260 con 0x7f87440fde90 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.216+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8744065e30 con 0x7f87440fde90 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.216+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f873400cb70 con 0x7f87440fde90 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.216+0000 7f873affd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 0x7f8730070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.217+0000 7f8749b98700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 0x7f8730070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.217+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f873408cfa0 con 0x7f87440fde90 2024-04-05T03:10:32.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.217+0000 7f8749b98700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 0x7f8730070160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f874410ec60 tx=0x7f8740009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:10:32.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.221+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f873405a230 con 0x7f87440fde90 2024-04-05T03:10:32.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.494+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8744109f10 con 0x7f87440fde90 2024-04-05T03:10:32.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.494+0000 7f873affd700 1 -- 172.21.15.153:0/2254529012 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f8734059dc0 con 0x7f87440fde90 2024-04-05T03:10:32.494 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 msgr2=0x7f8730070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 0x7f8730070160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f874410ec60 tx=0x7f8740009e30 comp rx=0 tx=0).stop 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 msgr2=0x7f8744113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f873400eb40 tx=0x7f873400b040 comp rx=0 tx=0).stop 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 shutdown_connections 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f873006dcb0 0x7f8730070160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87440fde90 0x7f8744113b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 --2- 172.21.15.153:0/2254529012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87440fe830 0x7f8744114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 >> 172.21.15.153:0/2254529012 conn(0x7f8744072bc0 msgr2=0x7f8744103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 shutdown_connections 2024-04-05T03:10:32.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:10:32.496+0000 7f874c5fd700 1 -- 172.21.15.153:0/2254529012 wait complete. 2024-04-05T03:10:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:32 smithi203 ceph-mon[95988]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:32 smithi203 ceph-mon[95988]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:32 smithi203 ceph-mon[95988]: pgmap v130: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:10:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:32 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3435499272' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:10:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:32 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2254529012' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:10:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:32 smithi153 ceph-mon[106102]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:32 smithi153 ceph-mon[106102]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:32 smithi153 ceph-mon[106102]: pgmap v130: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:10:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:32 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3435499272' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:10:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:32 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2254529012' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:10:33.792 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:33 smithi203 ceph-mon[95988]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:33.793 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:33 smithi203 systemd[1]: Stopping Ceph osd.4 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:10:33.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:33 smithi153 ceph-mon[106102]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:10:34.046 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:33 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[53454]: 2024-04-05T03:10:33.792+0000 7f45fb9fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:10:34.046 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:33 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[53454]: 2024-04-05T03:10:33.792+0000 7f45fb9fa700 -1 osd.4 68 *** Got signal Terminated *** 2024-04-05T03:10:34.046 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:33 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[53454]: 2024-04-05T03:10:33.792+0000 7f45fb9fa700 -1 osd.4 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:10:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:34 smithi153 ceph-mon[106102]: pgmap v131: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:34 smithi153 ceph-mon[106102]: osd.4 marked itself down and dead 2024-04-05T03:10:34.972 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:34 smithi203 ceph-mon[95988]: pgmap v131: 65 pgs: 65 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:34.972 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:34 smithi203 ceph-mon[95988]: osd.4 marked itself down and dead 2024-04-05T03:10:35.225 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:34 smithi203 podman[110361]: 2024-04-05 03:10:34.973294798 +0000 UTC m=+1.266578312 container died 8f1829157ed2ba371c03077ff1bb247e6258d0c1f2bb886240277ca323192970 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=quincy-6fda41f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-05T03:10:35.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:35 smithi203 podman[110361]: 2024-04-05 03:10:35.325827048 +0000 UTC m=+1.619110568 container cleanup 8f1829157ed2ba371c03077ff1bb247e6258d0c1f2bb886240277ca323192970 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, RELEASE=quincy-6fda41f, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-05T03:10:35.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:35 smithi203 bash[110361]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4 2024-04-05T03:10:35.986 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:35 smithi203 podman[110391]: 2024-04-05 03:10:35.636228234 +0000 UTC m=+0.661808210 container remove 8f1829157ed2ba371c03077ff1bb247e6258d0c1f2bb886240277ca323192970 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, RELEASE=quincy-6fda41f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: failed to encode map e69 with expected crc 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: failed to encode map e69 with expected crc 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: osdmap e69: 6 total, 5 up, 6 in 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: failed to encode map e69 with expected crc 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: failed to encode map e69 with expected crc 2024-04-05T03:10:35.987 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:35 smithi203 ceph-mon[95988]: osdmap e70: 6 total, 5 up, 6 in 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: failed to encode map e69 with expected crc 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: failed to encode map e69 with expected crc 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: osdmap e69: 6 total, 5 up, 6 in 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: failed to encode map e69 with expected crc 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: failed to encode map e69 with expected crc 2024-04-05T03:10:36.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:35 smithi153 ceph-mon[106102]: osdmap e70: 6 total, 5 up, 6 in 2024-04-05T03:10:36.255 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:35 smithi203 podman[110479]: 2024-04-05 03:10:35.890923032 +0000 UTC m=+0.017928704 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:36.255 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:36 smithi203 podman[110479]: 2024-04-05 03:10:36.007773411 +0000 UTC m=+0.134779082 container create edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=squid-0028525, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-05T03:10:36.255 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:36 smithi203 podman[110479]: 2024-04-05 03:10:36.256217019 +0000 UTC m=+0.383222691 container init edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-05T03:10:36.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:36 smithi203 podman[110479]: 2024-04-05 03:10:36.259725856 +0000 UTC m=+0.386731538 container start edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-05T03:10:36.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:36 smithi203 podman[110479]: 2024-04-05 03:10:36.339395027 +0000 UTC m=+0.466400700 container attach edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-05T03:10:36.883 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:36 smithi203 podman[110479]: 2024-04-05 03:10:36.548974553 +0000 UTC m=+0.675980234 container died edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: pgmap v133: 65 pgs: 10 peering, 4 stale+active+clean, 51 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: failed to encode map e70 with expected crc 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: failed to encode map e70 with expected crc 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: failed to encode map e70 with expected crc 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: failed to encode map e70 with expected crc 2024-04-05T03:10:36.883 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:36 smithi203 ceph-mon[95988]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: pgmap v133: 65 pgs: 10 peering, 4 stale+active+clean, 51 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: failed to encode map e70 with expected crc 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: failed to encode map e70 with expected crc 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: failed to encode map e70 with expected crc 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: failed to encode map e70 with expected crc 2024-04-05T03:10:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:36 smithi153 ceph-mon[106102]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T03:10:37.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110505]: 2024-04-05 03:10:37.14301346 +0000 UTC m=+0.588686171 container remove edbc5e75dae3ef0aeb392b5aee2c59aeb6ac0656ee97cd1655a5aff21fc27dd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-05T03:10:37.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.4.service: Deactivated successfully. 2024-04-05T03:10:37.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 systemd[1]: Stopped Ceph osd.4 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:10:37.547 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.4.service: Consumed 1min 4.302s CPU time. 2024-04-05T03:10:37.547 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 systemd[1]: Starting Ceph osd.4 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:10:37.896 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110611]: 2024-04-05 03:10:37.483895049 +0000 UTC m=+0.018664925 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:37.896 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110611]: 2024-04-05 03:10:37.611127672 +0000 UTC m=+0.145897534 container create 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True) 2024-04-05T03:10:37.896 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110611]: 2024-04-05 03:10:37.897124781 +0000 UTC m=+0.431894643 container init 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:10:38.296 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110611]: 2024-04-05 03:10:37.900702244 +0000 UTC m=+0.435472107 container start 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-05T03:10:38.296 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:37 smithi203 podman[110611]: 2024-04-05 03:10:37.988640479 +0000 UTC m=+0.523410356 container attach 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:10:38.549 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T03:10:38.549 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T03:10:38.549 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T03:10:38.549 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 bash[110611]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate[110626]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-05T03:10:38.550 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 podman[110611]: 2024-04-05 03:10:38.366471206 +0000 UTC m=+0.901241063 container died 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:10:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:38 smithi153 ceph-mon[106102]: pgmap v135: 65 pgs: 10 active+undersized+degraded, 10 peering, 1 stale+active+clean, 9 active+undersized, 35 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 37/318 objects degraded (11.635%) 2024-04-05T03:10:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:38 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 37/318 objects degraded (11.635%), 10 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:38.941 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-mon[95988]: pgmap v135: 65 pgs: 10 active+undersized+degraded, 10 peering, 1 stale+active+clean, 9 active+undersized, 35 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 37/318 objects degraded (11.635%) 2024-04-05T03:10:38.941 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:38 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 37/318 objects degraded (11.635%), 10 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:39.268 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:38 smithi203 podman[110752]: 2024-04-05 03:10:38.9424939 +0000 UTC m=+0.566190309 container remove 1ad3f8fd35728d8f3587315f0e2827c241156f1f64efd295d8fb860920d56783 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, org.label-schema.build-date=20240403) 2024-04-05T03:10:39.269 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 podman[110818]: 2024-04-05 03:10:39.05921798 +0000 UTC m=+0.018092498 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:10:39.269 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 podman[110818]: 2024-04-05 03:10:39.160314331 +0000 UTC m=+0.119188836 container create 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-05T03:10:39.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 podman[110818]: 2024-04-05 03:10:39.396575432 +0000 UTC m=+0.355449937 container init 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-05T03:10:39.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 podman[110818]: 2024-04-05 03:10:39.399739407 +0000 UTC m=+0.358613913 container start 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-05T03:10:39.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 ceph-osd[110838]: -- 172.21.15.203:0/511690309 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55940dd9d680 con 0x55940dd67000 2024-04-05T03:10:39.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 bash[110818]: 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce 2024-04-05T03:10:39.546 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 systemd[1]: Started Ceph osd.4 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:10:40.296 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:39 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:10:39.943+0000 7f99f79787c0 -1 Falling back to public interface 2024-04-05T03:10:40.656 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:40 smithi203 ceph-mon[95988]: pgmap v136: 65 pgs: 12 active+undersized+degraded, 10 peering, 12 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 42/318 objects degraded (13.208%) 2024-04-05T03:10:40.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:40 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:40.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:40 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:40.657 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:40 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:40 smithi153 ceph-mon[106102]: pgmap v136: 65 pgs: 12 active+undersized+degraded, 10 peering, 12 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 42/318 objects degraded (13.208%) 2024-04-05T03:10:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:40 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:40 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:40.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:40 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:41.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:41 smithi153 ceph-mon[106102]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T03:10:41.895 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:41 smithi203 ceph-mon[95988]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T03:10:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:42 smithi203 ceph-mon[95988]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:42 smithi153 ceph-mon[106102]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:44.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:44 smithi203 ceph-mon[95988]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:44.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:44.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:44.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:44 smithi153 ceph-mon[106102]: pgmap v138: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:45.296 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:10:45.040+0000 7f99f79787c0 -1 osd.4 68 log_to_monitors true 2024-04-05T03:10:45.796 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:10:45.402+0000 7f99e8dd0700 -1 osd.4 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:10:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-mon[95988]: from='osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T03:10:45.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:45 smithi203 ceph-mon[95988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T03:10:45.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:45.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:45 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:45.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:45 smithi153 ceph-mon[106102]: from='osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T03:10:45.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:45 smithi153 ceph-mon[106102]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: failed to encode map e71 with expected crc 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: osdmap e71: 6 total, 5 up, 6 in 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: from='osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:46.629 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:46 smithi203 ceph-mon[95988]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: failed to encode map e71 with expected crc 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: osdmap e71: 6 total, 5 up, 6 in 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: from='osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:10:46.813 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:46 smithi153 ceph-mon[106102]: pgmap v140: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:47 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:10:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:47 smithi203 ceph-mon[95988]: failed to encode map e72 with expected crc 2024-04-05T03:10:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:47 smithi203 ceph-mon[95988]: osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120] boot 2024-04-05T03:10:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:47 smithi203 ceph-mon[95988]: osdmap e72: 6 total, 6 up, 6 in 2024-04-05T03:10:47.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:47 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:10:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:47 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:10:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:47 smithi153 ceph-mon[106102]: failed to encode map e72 with expected crc 2024-04-05T03:10:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:47 smithi153 ceph-mon[106102]: osd.4 [v2:172.21.15.203:6808/2710960120,v1:172.21.15.203:6809/2710960120] boot 2024-04-05T03:10:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:47 smithi153 ceph-mon[106102]: osdmap e72: 6 total, 6 up, 6 in 2024-04-05T03:10:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:47 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: osdmap e73: 6 total, 6 up, 6 in 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: pgmap v143: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:48.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 52/318 objects degraded (16.352%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:48 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: osdmap e73: 6 total, 6 up, 6 in 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: failed to encode map e73 with expected crc 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: pgmap v143: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 52/318 objects degraded (16.352%) 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 52/318 objects degraded (16.352%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T03:10:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:10:48.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:48 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:10:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:49 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:10:49.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:49 smithi203 ceph-mon[95988]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T03:10:49.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:49 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:10:49.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:49 smithi153 ceph-mon[106102]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T03:10:50.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:50 smithi203 ceph-mon[95988]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 15 active+undersized, 35 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 49/318 objects degraded (15.409%) 2024-04-05T03:10:50.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:50 smithi153 ceph-mon[106102]: pgmap v144: 65 pgs: 15 active+undersized+degraded, 15 active+undersized, 35 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 49/318 objects degraded (15.409%) 2024-04-05T03:10:52.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:51 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/318 objects degraded (15.409%), 15 pgs degraded) 2024-04-05T03:10:52.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:51 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:10:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:51 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/318 objects degraded (15.409%), 15 pgs degraded) 2024-04-05T03:10:52.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:51 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:10:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:52 smithi203 ceph-mon[95988]: pgmap v145: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:52 smithi153 ceph-mon[106102]: pgmap v145: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:10:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:54 smithi203 ceph-mon[95988]: pgmap v146: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-05T03:10:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:54 smithi153 ceph-mon[106102]: pgmap v146: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-05T03:10:56.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:56 smithi153 ceph-mon[106102]: pgmap v147: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s 2024-04-05T03:10:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:56 smithi203 ceph-mon[95988]: pgmap v147: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 3 op/s 2024-04-05T03:10:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:58 smithi203 ceph-mon[95988]: pgmap v148: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T03:10:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:58.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:10:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:58 smithi153 ceph-mon[106102]: pgmap v148: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T03:10:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:10:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:10:58.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:10:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:00.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:00 smithi203 ceph-mon[95988]: pgmap v149: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:11:00.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:00 smithi153 ceph-mon[106102]: pgmap v149: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:11:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.612+0000 7fb910eb4700 1 -- 172.21.15.153:0/301162686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 msgr2=0x7fb90c106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.612+0000 7fb910eb4700 1 --2- 172.21.15.153:0/301162686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c106e80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc004940 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- 172.21.15.153:0/301162686 shutdown_connections 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 --2- 172.21.15.153:0/301162686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c106e80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 --2- 172.21.15.153:0/301162686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 0x7fb90c0fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- 172.21.15.153:0/301162686 >> 172.21.15.153:0/301162686 conn(0x7fb90c0f7850 msgr2=0x7fb90c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- 172.21.15.153:0/301162686 shutdown_connections 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- 172.21.15.153:0/301162686 wait complete. 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 Processor -- start 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- start start 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 0x7fb90c113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c114500 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.613+0000 7fb910eb4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c10db00 con 0x7fb90c0fde10 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51120/0 (socket says 172.21.15.153:51120) 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb90a59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 0x7fb90c113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 -- 172.21.15.153:0/1738768367 learned_addr learned my addr 172.21.15.153:0/1738768367 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 -- 172.21.15.153:0/1738768367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 msgr2=0x7fb90c113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 0x7fb90c113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 -- 172.21.15.153:0/1738768367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc005040 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb909d9b700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009380 tx=0x7fb8fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8fc00e040 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8fc009af0 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8fc01c3f0 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c10dd80 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.614+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c10e2c0 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.616+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c066ad0 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.616+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb8fc013420 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.616+0000 7fb902ffd700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 0x7fb8f8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.617+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb8fc08e750 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.617+0000 7fb90a59c700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 0x7fb8f8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.617+0000 7fb90a59c700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 0x7fb8f8070150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4004f40 tx=0x7fb8f400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:02.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.621+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb8fc05b930 con 0x7fb90c0fe7b0 2024-04-05T03:11:02.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:02 smithi203 ceph-mon[95988]: pgmap v150: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:11:02.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:02 smithi153 ceph-mon[106102]: pgmap v150: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-05T03:11:02.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.835+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb90c109ea0 con 0x7fb8f806dca0 2024-04-05T03:11:02.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.836+0000 7fb902ffd700 1 -- 172.21.15.153:0/1738768367 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb90c109ea0 con 0x7fb8f806dca0 2024-04-05T03:11:02.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 msgr2=0x7fb8f8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 0x7fb8f8070150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4004f40 tx=0x7fb8f400b040 comp rx=0 tx=0).stop 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 msgr2=0x7fb90c113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009380 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 shutdown_connections 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb8f806dca0 0x7fb8f8070150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c0fde10 0x7fb90c113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 --2- 172.21.15.153:0/1738768367 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb90c0fe7b0 0x7fb90c113eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 >> 172.21.15.153:0/1738768367 conn(0x7fb90c0f7850 msgr2=0x7fb90c0f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 shutdown_connections 2024-04-05T03:11:02.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.839+0000 7fb910eb4700 1 -- 172.21.15.153:0/1738768367 wait complete. 2024-04-05T03:11:02.849 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:11:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- 172.21.15.153:0/438862774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fc5b0 msgr2=0x7f62ac100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 --2- 172.21.15.153:0/438862774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac100bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f629c004940 tx=0x7f629c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- 172.21.15.153:0/438862774 shutdown_connections 2024-04-05T03:11:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 --2- 172.21.15.153:0/438862774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac100bd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 --2- 172.21.15.153:0/438862774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fbc10 0x7f62ac0fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- 172.21.15.153:0/438862774 >> 172.21.15.153:0/438862774 conn(0x7f62ac0f7870 msgr2=0x7f62ac0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- 172.21.15.153:0/438862774 shutdown_connections 2024-04-05T03:11:02.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- 172.21.15.153:0/438862774 wait complete. 2024-04-05T03:11:02.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 Processor -- start 2024-04-05T03:11:02.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.941+0000 7f62b36c3700 1 -- start start 2024-04-05T03:11:02.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac11a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac11ac10 con 0x7f62ac0fbc10 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ac114230 con 0x7f62ac0fc5b0 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51130/0 (socket says 172.21.15.153:51130) 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 -- 172.21.15.153:0/363357546 learned_addr learned my addr 172.21.15.153:0/363357546 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b0c5e700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac11a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 -- 172.21.15.153:0/363357546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fc5b0 msgr2=0x7f62ac11a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac11a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 -- 172.21.15.153:0/363357546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c005040 con 0x7f62ac0fbc10 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b145f700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f62a8002a50 tx=0x7f62a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a8007cc0 con 0x7f62ac0fbc10 2024-04-05T03:11:02.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ac114510 con 0x7f62ac0fbc10 2024-04-05T03:11:02.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62a800f040 con 0x7f62ac0fbc10 2024-04-05T03:11:02.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a800da70 con 0x7f62ac0fbc10 2024-04-05T03:11:02.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.942+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ac114a50 con 0x7f62ac0fbc10 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.944+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac066b70 con 0x7f62ac0fbc10 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.944+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f62a800dc10 con 0x7f62ac0fbc10 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.944+0000 7f62a1ffb700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 0x7f6298070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.944+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f62a808cb10 con 0x7f62ac0fbc10 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.944+0000 7f62b0c5e700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 0x7f6298070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:02.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.945+0000 7f62b0c5e700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 0x7f6298070150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62ac115960 tx=0x7f629c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:02.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:02.949+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f62a8056140 con 0x7f62ac0fbc10 2024-04-05T03:11:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.164+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62ac115710 con 0x7f629806dca0 2024-04-05T03:11:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.165+0000 7f62a1ffb700 1 -- 172.21.15.153:0/363357546 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f62ac115710 con 0x7f629806dca0 2024-04-05T03:11:03.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.167+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 msgr2=0x7f6298070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.167+0000 7f62b36c3700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 0x7f6298070150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f62ac115960 tx=0x7f629c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.167+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 msgr2=0x7f62ac11a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.167+0000 7f62b36c3700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f62a8002a50 tx=0x7f62a80046b0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.167+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 shutdown_connections 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f629806dca0 0x7f6298070150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f62ac0fbc10 0x7f62ac11a080 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 --2- 172.21.15.153:0/363357546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62ac0fc5b0 0x7f62ac11a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 >> 172.21.15.153:0/363357546 conn(0x7f62ac0f7870 msgr2=0x7f62ac0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 shutdown_connections 2024-04-05T03:11:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.168+0000 7f62b36c3700 1 -- 172.21.15.153:0/363357546 wait complete. 2024-04-05T03:11:03.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 -- 172.21.15.153:0/3391537987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5200fb3b0 msgr2=0x7fd5200fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 --2- 172.21.15.153:0/3391537987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5200fb3b0 0x7fd5200fb820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd50c004940 tx=0x7fd50c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 -- 172.21.15.153:0/3391537987 shutdown_connections 2024-04-05T03:11:03.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 --2- 172.21.15.153:0/3391537987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5200fb3b0 0x7fd5200fb820 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 --2- 172.21.15.153:0/3391537987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd520102f70 0x7fd520103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 -- 172.21.15.153:0/3391537987 >> 172.21.15.153:0/3391537987 conn(0x7fd520072310 msgr2=0x7fd520072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 -- 172.21.15.153:0/3391537987 shutdown_connections 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.269+0000 7fd526e40700 1 -- 172.21.15.153:0/3391537987 wait complete. 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 Processor -- start 2024-04-05T03:11:03.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 -- start start 2024-04-05T03:11:03.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5200fb3b0 0x7fd520113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201144f0 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.270+0000 7fd526e40700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52010daf0 con 0x7fd5200fb3b0 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51148/0 (socket says 172.21.15.153:51148) 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 -- 172.21.15.153:0/4064367863 learned_addr learned my addr 172.21.15.153:0/4064367863 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd524bdc700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5200fb3b0 0x7fd520113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 -- 172.21.15.153:0/4064367863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5200fb3b0 msgr2=0x7fd520113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5200fb3b0 0x7fd520113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 -- 172.21.15.153:0/4064367863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c005040 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51ffff700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00b3b0 tx=0x7fd50c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd50c00e040 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd50c013420 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52010dd70 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd50c01c630 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.271+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52010e330 con 0x7fd520102f70 2024-04-05T03:11:03.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.272+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520105350 con 0x7fd520102f70 2024-04-05T03:11:03.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.273+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd50c013590 con 0x7fd520102f70 2024-04-05T03:11:03.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.273+0000 7fd51d7fa700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 0x7fd510070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.273+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd50c08e6b0 con 0x7fd520102f70 2024-04-05T03:11:03.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.273+0000 7fd524bdc700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 0x7fd510070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.273+0000 7fd524bdc700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 0x7fd510070150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd514004f40 tx=0x7fd51400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.278+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd50c05b890 con 0x7fd520102f70 2024-04-05T03:11:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.487+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd520113630 con 0x7fd51006dca0 2024-04-05T03:11:03.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.497+0000 7fd51d7fa700 1 -- 172.21.15.153:0/4064367863 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd520113630 con 0x7fd51006dca0 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 89s ago 16m 26.0M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (3m) 89s ago 17m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (3m) 20s ago 15m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 89s ago 16m 80.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (11m) 89s ago 11m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (11m) 89s ago 11m 60.8M - 17.2.7-805-g6fda41ff a49113eacc58 1e0be1ed0a99 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (11m) 20s ago 11m 61.3M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (11m) 20s ago 11m 112M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (7m) 89s ago 19m 504M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (6m) 20s ago 15m 451M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (4m) 89s ago 19m 45.7M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:11:03.497 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (3m) 20s ago 15m 39.6M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 89s ago 16m 16.7M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (6m) 20s ago 15m 17.2M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (2m) 89s ago 14m 88.5M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 89s ago 14m 92.7M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (94s) 89s ago 13m 12.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (59s) 20s ago 13m 108M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (24s) 20s ago 13m 14.7M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (12m) 20s ago 12m 236M 2240M 17.2.7-805-g6fda41ff a49113eacc58 2a93448a42cb 2024-04-05T03:11:03.498 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (5m) 20s ago 15m 63.7M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 msgr2=0x7fd510070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 0x7fd510070150 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd514004f40 tx=0x7fd51400b040 comp rx=0 tx=0).stop 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 msgr2=0x7fd520113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00b3b0 tx=0x7fd50c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 shutdown_connections 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd51006dca0 0x7fd510070150 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5200fb3b0 0x7fd520113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 --2- 172.21.15.153:0/4064367863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd520102f70 0x7fd520113ea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 >> 172.21.15.153:0/4064367863 conn(0x7fd520072310 msgr2=0x7fd5200f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 shutdown_connections 2024-04-05T03:11:03.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.499+0000 7fd526e40700 1 -- 172.21.15.153:0/4064367863 wait complete. 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- 172.21.15.153:0/2423417863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 msgr2=0x7f03ac1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 --2- 172.21.15.153:0/2423417863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac1031e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f03a0004940 tx=0x7f03a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- 172.21.15.153:0/2423417863 shutdown_connections 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 --2- 172.21.15.153:0/2423417863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 0x7f03ac0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 --2- 172.21.15.153:0/2423417863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac1031e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- 172.21.15.153:0/2423417863 >> 172.21.15.153:0/2423417863 conn(0x7f03ac0f8980 msgr2=0x7f03ac0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- 172.21.15.153:0/2423417863 shutdown_connections 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- 172.21.15.153:0/2423417863 wait complete. 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 Processor -- start 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.600+0000 7f03b08af700 1 -- start start 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03b08af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03b08af700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 0x7f03ac114160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03b08af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac1147b0 con 0x7f03ac102e10 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03b08af700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac10dba0 con 0x7f03ac1037b0 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51176/0 (socket says 172.21.15.153:51176) 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 -- 172.21.15.153:0/349809216 learned_addr learned my addr 172.21.15.153:0/349809216 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03a9d9b700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 0x7f03ac114160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 -- 172.21.15.153:0/349809216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 msgr2=0x7f03ac114160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 0x7f03ac114160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 -- 172.21.15.153:0/349809216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0005040 con 0x7f03ac102e10 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03aa59c700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03a000b3b0 tx=0x7f03a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a000e040 con 0x7f03ac102e10 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a000bbc0 con 0x7f03ac102e10 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ac10de20 con 0x7f03ac102e10 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.601+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a001c3f0 con 0x7f03ac102e10 2024-04-05T03:11:03.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.602+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ac10e3e0 con 0x7f03ac102e10 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.603+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ac100120 con 0x7f03ac102e10 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.603+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f03a000bd30 con 0x7f03ac102e10 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.603+0000 7f039affd700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 0x7f0394070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.603+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f03a008e780 con 0x7f03ac102e10 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.603+0000 7f03a9d9b700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 0x7f0394070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.604+0000 7f03a9d9b700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 0x7f0394070150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f039c007fd0 tx=0x7f039c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.608+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f03a005b8e0 con 0x7f03ac102e10 2024-04-05T03:11:03.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:03 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:11:03.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:03 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:11:03.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.886+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03ac04bed0 con 0x7f03ac102e10 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.887+0000 7f039affd700 1 -- 172.21.15.153:0/349809216 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f03a005b470 con 0x7f03ac102e10 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:03.886 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 5 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 4 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 5, 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 9 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:11:03.887 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 msgr2=0x7f0394070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 0x7f0394070150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f039c007fd0 tx=0x7f039c009040 comp rx=0 tx=0).stop 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 msgr2=0x7f03ac113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f03a000b3b0 tx=0x7f03a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 shutdown_connections 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f039406dca0 0x7f0394070150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ac102e10 0x7f03ac113c20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 --2- 172.21.15.153:0/349809216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03ac1037b0 0x7f03ac114160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 >> 172.21.15.153:0/349809216 conn(0x7f03ac0f8980 msgr2=0x7f03ac107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 shutdown_connections 2024-04-05T03:11:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.889+0000 7f03b08af700 1 -- 172.21.15.153:0/349809216 wait complete. 2024-04-05T03:11:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/2934992515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 msgr2=0x7fa6c8103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/2934992515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8103340 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc004940 tx=0x7fa6bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:03.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/2934992515 shutdown_connections 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/2934992515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 0x7fa6c80fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/2934992515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8103340 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/2934992515 >> 172.21.15.153:0/2934992515 conn(0x7fa6c8072310 msgr2=0x7fa6c8072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/2934992515 shutdown_connections 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.991+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/2934992515 wait complete. 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 Processor -- start 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 -- start start 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 0x7fa6c8113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c81144d0 con 0x7fa6c8102f70 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6ccbb6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c810dad0 con 0x7fa6c80fb3b0 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6c5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6c5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51204/0 (socket says 172.21.15.153:51204) 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6c5d9b700 1 -- 172.21.15.153:0/3537654262 learned_addr learned my addr 172.21.15.153:0/3537654262 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:03.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.992+0000 7fa6c659c700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 0x7fa6c8113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6c5d9b700 1 -- 172.21.15.153:0/3537654262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 msgr2=0x7fa6c8113940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6c5d9b700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 0x7fa6c8113940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6c5d9b700 1 -- 172.21.15.153:0/3537654262 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc005040 con 0x7fa6c8102f70 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6c5d9b700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800c830 tx=0x7fa6b8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b80106f0 con 0x7fa6c8102f70 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c810ddb0 con 0x7fa6c8102f70 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6b8010d20 con 0x7fa6c8102f70 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b80113a0 con 0x7fa6c8102f70 2024-04-05T03:11:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.993+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c810e370 con 0x7fa6c8102f70 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.994+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80fe6d0 con 0x7fa6c8102f70 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.994+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa6b8012400 con 0x7fa6c8102f70 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.995+0000 7fa6b6ffd700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 0x7fa6b0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.995+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa6b808d420 con 0x7fa6c8102f70 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.995+0000 7fa6c659c700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 0x7fa6b0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:03.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:03.995+0000 7fa6c659c700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 0x7fa6b0070150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc004f90 tx=0x7fa6bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:03.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.000+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa6b805a600 con 0x7fa6c8102f70 2024-04-05T03:11:04.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.237+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa6c804c710 con 0x7fa6c8102f70 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.237+0000 7fa6b6ffd700 1 -- 172.21.15.153:0/3537654262 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1906 (secure 0 0 0) 0x7fa6b805a190 con 0x7fa6c8102f70 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout:e25 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:04.237 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 25 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:06:24.861523+0000 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:11:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 45 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14506,1=24261} 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:11:04.239 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{0:14506} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/1509596739,v1:172.21.15.153:6829/1509596739] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:04.240 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:14518} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1440749410,v1:172.21.15.153:6827/1440749410] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:04.240 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.203:6824/1771218289,v1:172.21.15.203:6825/1771218289] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:04.240 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:24271} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.203:6826/215598905,v1:172.21.15.203:6827/215598905] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:04.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:04.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 msgr2=0x7fa6b0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 0x7fa6b0070150 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc004f90 tx=0x7fa6bc009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 msgr2=0x7fa6c8113e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800c830 tx=0x7fa6b8004470 comp rx=0 tx=0).stop 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 shutdown_connections 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa6b006dca0 0x7fa6b0070150 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6c80fb3b0 0x7fa6c8113940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 --2- 172.21.15.153:0/3537654262 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6c8102f70 0x7fa6c8113e80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 >> 172.21.15.153:0/3537654262 conn(0x7fa6c8072310 msgr2=0x7fa6c80f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:04.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 shutdown_connections 2024-04-05T03:11:04.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.240+0000 7fa6ccbb6700 1 -- 172.21.15.153:0/3537654262 wait complete. 2024-04-05T03:11:04.242 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 -- 172.21.15.153:0/2278012636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c101d50 msgr2=0x7fb39c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 --2- 172.21.15.153:0/2278012636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c101d50 0x7fb39c102120 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb38c004940 tx=0x7fb38c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 -- 172.21.15.153:0/2278012636 shutdown_connections 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 --2- 172.21.15.153:0/2278012636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c102660 0x7fb39c0fc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 --2- 172.21.15.153:0/2278012636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c101d50 0x7fb39c102120 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 -- 172.21.15.153:0/2278012636 >> 172.21.15.153:0/2278012636 conn(0x7fb39c0714c0 msgr2=0x7fb39c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 -- 172.21.15.153:0/2278012636 shutdown_connections 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 -- 172.21.15.153:0/2278012636 wait complete. 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.341+0000 7fb3a2333700 1 Processor -- start 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb3a2333700 1 -- start start 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb3a2333700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c101d50 0x7fb39c102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb3a2333700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb3a2333700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c103630 con 0x7fb39c102660 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb3a2333700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c103770 con 0x7fb39c101d50 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51212/0 (socket says 172.21.15.153:51212) 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 -- 172.21.15.153:0/1121675775 learned_addr learned my addr 172.21.15.153:0/1121675775 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39bfff700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c101d50 0x7fb39c102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 -- 172.21.15.153:0/1121675775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c101d50 msgr2=0x7fb39c102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c101d50 0x7fb39c102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 -- 172.21.15.153:0/1121675775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c005040 con 0x7fb39c102660 2024-04-05T03:11:04.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.342+0000 7fb39b7fe700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb390002a50 tx=0x7fb3900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:04.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.343+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb390007cc0 con 0x7fb39c102660 2024-04-05T03:11:04.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.343+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb39000f040 con 0x7fb39c102660 2024-04-05T03:11:04.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.343+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c119f70 con 0x7fb39c102660 2024-04-05T03:11:04.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.343+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb39000db70 con 0x7fb39c102660 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.343+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c11a370 con 0x7fb39c102660 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.344+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c0fd5c0 con 0x7fb39c102660 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.344+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb39000dd10 con 0x7fb39c102660 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.345+0000 7fb398ff9700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 0x7fb384070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.345+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb39008cb00 con 0x7fb39c102660 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.345+0000 7fb39bfff700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 0x7fb384070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.345+0000 7fb39bfff700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 0x7fb384070150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b3b0 tx=0x7fb38c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:04.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.349+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb3900561b0 con 0x7fb39c102660 2024-04-05T03:11:04.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.565+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb39c04bed0 con 0x7fb38406dca0 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.566+0000 7fb398ff9700 1 -- 172.21.15.153:0/1121675775 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fb39c04bed0 con 0x7fb38406dca0 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:11:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "11/21 daemons upgraded", 2024-04-05T03:11:04.567 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T03:11:04.567 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:11:04.567 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 msgr2=0x7fb384070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 0x7fb384070150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00b3b0 tx=0x7fb38c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 msgr2=0x7fb39c1030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb390002a50 tx=0x7fb3900046b0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 shutdown_connections 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb38406dca0 0x7fb384070150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb39c101d50 0x7fb39c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 --2- 172.21.15.153:0/1121675775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c102660 0x7fb39c1030a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.568+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 >> 172.21.15.153:0/1121675775 conn(0x7fb39c0714c0 msgr2=0x7fb39c0fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:04.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.569+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 shutdown_connections 2024-04-05T03:11:04.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.569+0000 7fb3a2333700 1 -- 172.21.15.153:0/1121675775 wait complete. 2024-04-05T03:11:04.653 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.653 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: Upgrade: osd.5 is safe to restart 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: pgmap v151: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/349809216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:04.654 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:04 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3537654262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:11:04.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 -- 172.21.15.153:0/1859088808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c1026b0 msgr2=0x7f1f6c0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 --2- 172.21.15.153:0/1859088808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c0fbd20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1f60004940 tx=0x7f1f60009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:04.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 -- 172.21.15.153:0/1859088808 shutdown_connections 2024-04-05T03:11:04.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 --2- 172.21.15.153:0/1859088808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c0fbd20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 --2- 172.21.15.153:0/1859088808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c101d10 0x7f1f6c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 -- 172.21.15.153:0/1859088808 >> 172.21.15.153:0/1859088808 conn(0x7f1f6c0f7880 msgr2=0x7f1f6c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:04.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 -- 172.21.15.153:0/1859088808 shutdown_connections 2024-04-05T03:11:04.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.670+0000 7f1f7277f700 1 -- 172.21.15.153:0/1859088808 wait complete. 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 Processor -- start 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 -- start start 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c10f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c10fa20 con 0x7f1f6c101d10 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f7277f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c10fb60 con 0x7f1f6c1026b0 2024-04-05T03:11:04.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51234/0 (socket says 172.21.15.153:51234) 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 -- 172.21.15.153:0/2629353936 learned_addr learned my addr 172.21.15.153:0/2629353936 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6b7fe700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c10f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 -- 172.21.15.153:0/2629353936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c1026b0 msgr2=0x7f1f6c10f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c10f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.671+0000 7f1f6bfff700 1 -- 172.21.15.153:0/2629353936 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60005040 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f6bfff700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c002a50 tx=0x7f1f5c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5c007cc0 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f5c00f040 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c117e50 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f5c00da30 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.672+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c118380 con 0x7f1f6c101d10 2024-04-05T03:11:04.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.673+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c0fd460 con 0x7f1f6c101d10 2024-04-05T03:11:04.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.673+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1f5c00dbb0 con 0x7f1f6c101d10 2024-04-05T03:11:04.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.674+0000 7f1f68ff9700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 0x7f1f54070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:04.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.674+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1f5c08d1f0 con 0x7f1f6c101d10 2024-04-05T03:11:04.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.674+0000 7f1f6b7fe700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 0x7f1f54070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:04.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.674+0000 7f1f6b7fe700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 0x7f1f54070150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f60004890 tx=0x7f1f60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:04.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.678+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1f5c056820 con 0x7f1f6c101d10 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: Upgrade: osd.5 is safe to restart 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: pgmap v151: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/349809216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:04 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3537654262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:11:04.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.958+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1f6c108e20 con 0x7f1f6c101d10 2024-04-05T03:11:04.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.958+0000 7f1f68ff9700 1 -- 172.21.15.153:0/2629353936 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1f5c059ee0 con 0x7f1f6c101d10 2024-04-05T03:11:04.958 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 msgr2=0x7f1f54070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 0x7f1f54070150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f60004890 tx=0x7f1f60009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 msgr2=0x7f1f6c10efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c002a50 tx=0x7f1f5c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 shutdown_connections 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f5406dca0 0x7f1f54070150 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f6c101d10 0x7f1f6c10efa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 --2- 172.21.15.153:0/2629353936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6c1026b0 0x7f1f6c10f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.960+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 >> 172.21.15.153:0/2629353936 conn(0x7f1f6c0f7880 msgr2=0x7f1f6c0f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.961+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 shutdown_connections 2024-04-05T03:11:04.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:04.961+0000 7f1f7277f700 1 -- 172.21.15.153:0/2629353936 wait complete. 2024-04-05T03:11:05.531 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:05 smithi203 ceph-mon[95988]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:05.531 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:05.531 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T03:11:05.531 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:05 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:05.531 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:05 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2629353936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:11:05.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:05 smithi153 ceph-mon[106102]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:05.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:05.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T03:11:05.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:05 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:05.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:05 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2629353936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:11:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:06 smithi203 ceph-mon[95988]: Upgrade: Updating osd.5 2024-04-05T03:11:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:06 smithi203 ceph-mon[95988]: Deploying daemon osd.5 on smithi203 2024-04-05T03:11:06.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:06 smithi203 ceph-mon[95988]: pgmap v152: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:11:06.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:06 smithi153 ceph-mon[106102]: Upgrade: Updating osd.5 2024-04-05T03:11:06.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:06 smithi153 ceph-mon[106102]: Deploying daemon osd.5 on smithi203 2024-04-05T03:11:06.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:06 smithi153 ceph-mon[106102]: pgmap v152: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T03:11:08.248 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:07 smithi203 systemd[1]: Stopping Ceph osd.5 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:11:08.513 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[60518]: 2024-04-05T03:11:08.248+0000 7f9a5b0c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:11:08.513 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[60518]: 2024-04-05T03:11:08.248+0000 7f9a5b0c4700 -1 osd.5 73 *** Got signal Terminated *** 2024-04-05T03:11:08.514 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[60518]: 2024-04-05T03:11:08.248+0000 7f9a5b0c4700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:11:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:08 smithi203 ceph-mon[95988]: pgmap v153: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:11:08.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:08 smithi203 ceph-mon[95988]: osd.5 marked itself down and dead 2024-04-05T03:11:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:08 smithi153 ceph-mon[106102]: pgmap v153: 65 pgs: 65 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:11:08.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:08 smithi153 ceph-mon[106102]: osd.5 marked itself down and dead 2024-04-05T03:11:09.226 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:08 smithi203 podman[115346]: 2024-04-05 03:11:08.974837639 +0000 UTC m=+0.848701387 container died 2a93448a42cbf69daa2a5d745ac9df27dcb9d3868cf3739a9718328121d17522 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-6fda41f, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-05T03:11:09.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:09 smithi203 podman[115346]: 2024-04-05 03:11:09.37375077 +0000 UTC m=+1.247614518 container cleanup 2a93448a42cbf69daa2a5d745ac9df27dcb9d3868cf3739a9718328121d17522 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=quincy-6fda41f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-05T03:11:09.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:09 smithi203 bash[115346]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5 2024-04-05T03:11:09.796 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:09 smithi203 podman[115366]: 2024-04-05 03:11:09.659117173 +0000 UTC m=+0.676515795 container remove 2a93448a42cbf69daa2a5d745ac9df27dcb9d3868cf3739a9718328121d17522 (image=quay.ceph.io/ceph-ci/ceph@sha256:bc7712ec89fede52a2d1211e734d3862c877e03d01eec5cccb6e6be746353030, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-6fda41f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-05T03:11:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:09 smithi203 ceph-mon[95988]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:11:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:09 smithi203 ceph-mon[95988]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-05T03:11:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:09 smithi203 ceph-mon[95988]: osdmap e74: 6 total, 5 up, 6 in 2024-04-05T03:11:09.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:09 smithi203 ceph-mon[95988]: osdmap e75: 6 total, 5 up, 6 in 2024-04-05T03:11:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:09 smithi153 ceph-mon[106102]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T03:11:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:09 smithi153 ceph-mon[106102]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-05T03:11:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:09 smithi153 ceph-mon[106102]: osdmap e74: 6 total, 5 up, 6 in 2024-04-05T03:11:09.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:09 smithi153 ceph-mon[106102]: osdmap e75: 6 total, 5 up, 6 in 2024-04-05T03:11:10.244 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:09 smithi203 podman[115460]: 2024-04-05 03:11:09.856345607 +0000 UTC m=+0.017952120 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:11:10.245 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:09 smithi203 podman[115460]: 2024-04-05 03:11:09.976684133 +0000 UTC m=+0.138290635 container create 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.license=GPLv2, ceph=True) 2024-04-05T03:11:10.245 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:10 smithi203 podman[115460]: 2024-04-05 03:11:10.162440368 +0000 UTC m=+0.324046871 container init 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-05T03:11:10.245 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:10 smithi203 podman[115460]: 2024-04-05 03:11:10.165601566 +0000 UTC m=+0.327208075 container start 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-0028525, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-05T03:11:10.245 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:10 smithi203 podman[115460]: 2024-04-05 03:11:10.245792647 +0000 UTC m=+0.407399166 container attach 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:11:10.524 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:10 smithi203 podman[115460]: 2024-04-05 03:11:10.458383486 +0000 UTC m=+0.619989991 container died 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, ceph=True) 2024-04-05T03:11:10.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:10 smithi203 ceph-mon[95988]: pgmap v155: 65 pgs: 13 stale+active+clean, 52 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:11:10.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:10 smithi153 ceph-mon[106102]: pgmap v155: 65 pgs: 13 stale+active+clean, 52 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:11:11.282 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115487]: 2024-04-05 03:11:11.032853435 +0000 UTC m=+0.569258221 container remove 36d0c187b3b9d3c827a0742b16e45055e50efd2e7f7ce212af2fb8214834a01e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525) 2024-04-05T03:11:11.282 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.5.service: Deactivated successfully. 2024-04-05T03:11:11.282 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 systemd[1]: Stopped Ceph osd.5 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:11:11.282 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 systemd[1]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.5.service: Consumed 1min 10.068s CPU time. 2024-04-05T03:11:11.282 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 systemd[1]: Starting Ceph osd.5 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:11:11.761 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:11 smithi203 ceph-mon[95988]: Health check failed: Degraded data redundancy: 51/318 objects degraded (16.038%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T03:11:11.762 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115593]: 2024-04-05 03:11:11.400893028 +0000 UTC m=+0.017636750 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:11:11.762 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115593]: 2024-04-05 03:11:11.525861693 +0000 UTC m=+0.142605406 container create f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=squid-0028525, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-05T03:11:11.762 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115593]: 2024-04-05 03:11:11.762485947 +0000 UTC m=+0.379229667 container init f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:11:12.046 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115593]: 2024-04-05 03:11:11.766276482 +0000 UTC m=+0.383020202 container start f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, org.label-schema.license=GPLv2, RELEASE=squid-0028525, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:11:12.046 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:11 smithi203 podman[115593]: 2024-04-05 03:11:11.861945919 +0000 UTC m=+0.478689650 container attach f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-05T03:11:12.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:11 smithi153 ceph-mon[106102]: Health check failed: Degraded data redundancy: 51/318 objects degraded (16.038%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T03:11:12.522 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T03:11:12.522 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T03:11:12.522 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 bash[115593]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate[115608]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-05T03:11:12.523 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 podman[115593]: 2024-04-05 03:11:12.233439206 +0000 UTC m=+0.850182919 container died f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-05T03:11:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-mon[95988]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:12.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:12 smithi153 ceph-mon[106102]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:13.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:13.107 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:12 smithi203 podman[115738]: 2024-04-05 03:11:12.849081411 +0000 UTC m=+0.610675843 container remove f3f5503cfbc85aa06e43de9a1430d480a4bd2e449051a74cb719c3816ae49255 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-05T03:11:13.107 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 podman[115804]: 2024-04-05 03:11:12.96451158 +0000 UTC m=+0.018078288 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:11:13.107 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 podman[115804]: 2024-04-05 03:11:13.108590872 +0000 UTC m=+0.162157573 container create 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, RELEASE=squid-0028525, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-05T03:11:13.358 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 podman[115804]: 2024-04-05 03:11:13.336287491 +0000 UTC m=+0.389854189 container init 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-05T03:11:13.358 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 podman[115804]: 2024-04-05 03:11:13.339304937 +0000 UTC m=+0.392871640 container start 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-05T03:11:13.796 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 ceph-osd[115824]: -- 172.21.15.203:0/1701383914 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55726ea07680 con 0x55726e9d1000 2024-04-05T03:11:13.796 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 bash[115804]: 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 2024-04-05T03:11:13.796 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 systemd[1]: Started Ceph osd.5 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4. 2024-04-05T03:11:14.296 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:13 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:11:13.882+0000 7f4b153587c0 -1 Falling back to public interface 2024-04-05T03:11:14.763 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:14 smithi203 ceph-mon[95988]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:14.763 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:14.763 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:14.763 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:14 smithi153 ceph-mon[106102]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:16.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:16 smithi203 ceph-mon[95988]: pgmap v159: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:16 smithi153 ceph-mon[106102]: pgmap v159: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:18 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:18 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:18.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:18 smithi203 ceph-mon[95988]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:18 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:18 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:18 smithi153 ceph-mon[106102]: pgmap v160: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:19.158 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:18 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:11:18.796+0000 7f4b153587c0 -1 osd.5 73 log_to_monitors true 2024-04-05T03:11:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:19 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:19 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:19 smithi203 ceph-mon[95988]: from='osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T03:11:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:19 smithi203 ceph-mon[95988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T03:11:19.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:19 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:19 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:19 smithi153 ceph-mon[106102]: from='osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T03:11:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:19 smithi153 ceph-mon[106102]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T03:11:20.430 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:11:20.404+0000 7f4b027a8700 -1 osd.5 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: from='osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: osdmap e76: 6 total, 5 up, 6 in 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:11:20.702 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:20 smithi203 ceph-mon[95988]: from='osd.5 ' entity='osd.5' 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/318 objects degraded (16.038%) 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: from='osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: osdmap e76: 6 total, 5 up, 6 in 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-05T03:11:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:20 smithi153 ceph-mon[106102]: from='osd.5 ' entity='osd.5' 2024-04-05T03:11:21.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:21 smithi203 ceph-mon[95988]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:11:21.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:21 smithi203 ceph-mon[95988]: osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398] boot 2024-04-05T03:11:21.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:21 smithi203 ceph-mon[95988]: osdmap e77: 6 total, 6 up, 6 in 2024-04-05T03:11:21.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:21 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:11:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:21 smithi153 ceph-mon[106102]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T03:11:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:21 smithi153 ceph-mon[106102]: osd.5 [v2:172.21.15.203:6816/2060217398,v1:172.21.15.203:6817/2060217398] boot 2024-04-05T03:11:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:21 smithi153 ceph-mon[106102]: osdmap e77: 6 total, 6 up, 6 in 2024-04-05T03:11:21.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:21 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 962 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 51/318 objects degraded (16.038%) 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-05T03:11:22.798 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-05T03:11:22.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-05T03:11:22.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: osdmap e78: 6 total, 6 up, 6 in 2024-04-05T03:11:22.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.799 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-05T03:11:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: pgmap v164: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 962 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 51/318 objects degraded (16.038%) 2024-04-05T03:11:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-05T03:11:22.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-05T03:11:22.824 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: osdmap e78: 6 total, 6 up, 6 in 2024-04-05T03:11:22.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:22.825 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all osd 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: Upgrade: Setting require_osd_release to 19 squid 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: osdmap e79: 6 total, 6 up, 6 in 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-05T03:11:23.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T03:11:23.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:23.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:23.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:23.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:23.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all osd 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: Upgrade: Setting require_osd_release to 19 squid 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: osdmap e79: 6 total, 6 up, 6 in 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: fsmap cephfs:2 {0=cephfs.smithi153.fmpmvd=up:active,1=cephfs.smithi203.xpteyc=up:active} 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:23.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:23.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:23.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:11:23.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:23 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:11:23.582+0000 7f9e30ab0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T03:11:24.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: pgmap v167: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 962 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 51/318 objects degraded (16.038%) 2024-04-05T03:11:24.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:24.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: osdmap e80: 6 total, 6 up, 6 in 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.203:6826/2394111038,v1:172.21.15.203:6827/2394111038] up:boot 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/646956490,v1:172.21.15.153:6827/646956490] up:boot 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:11:24.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: pgmap v167: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 252 MiB data, 962 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 51/318 objects degraded (16.038%) 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:24.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: osdmap e80: 6 total, 6 up, 6 in 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.203:6826/2394111038,v1:172.21.15.203:6827/2394111038] up:boot 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/646956490,v1:172.21.15.153:6827/646956490] up:boot 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:11:24.797 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.203:6824/4013577824,v1:172.21.15.203:6825/4013577824] up:boot 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/1079014242,v1:172.21.15.153:6829/1079014242] up:boot 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:11:25.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:25 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.203:6824/4013577824,v1:172.21.15.203:6825/4013577824] up:boot 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/1079014242,v1:172.21.15.153:6829/1079014242] up:boot 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.vicaky", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:11:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:25 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:26.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:26 smithi153 ceph-mon[106102]: Upgrade: Updating mds.cephfs.smithi153.vicaky 2024-04-05T03:11:26.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:26 smithi153 ceph-mon[106102]: Deploying daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T03:11:26.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:26 smithi153 ceph-mon[106102]: pgmap v169: 65 pgs: 5 active+undersized+degraded, 3 active+undersized, 57 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail; 26/318 objects degraded (8.176%) 2024-04-05T03:11:26.843 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:26 smithi153 ceph-mon[106102]: Health check update: Degraded data redundancy: 26/318 objects degraded (8.176%), 5 pgs degraded (PG_DEGRADED) 2024-04-05T03:11:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:26 smithi203 ceph-mon[95988]: Upgrade: Updating mds.cephfs.smithi153.vicaky 2024-04-05T03:11:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:26 smithi203 ceph-mon[95988]: Deploying daemon mds.cephfs.smithi153.vicaky on smithi153 2024-04-05T03:11:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:26 smithi203 ceph-mon[95988]: pgmap v169: 65 pgs: 5 active+undersized+degraded, 3 active+undersized, 57 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail; 26/318 objects degraded (8.176%) 2024-04-05T03:11:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:26 smithi203 ceph-mon[95988]: Health check update: Degraded data redundancy: 26/318 objects degraded (8.176%), 5 pgs degraded (PG_DEGRADED) 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: pgmap v170: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: osdmap e81: 6 total, 6 up, 6 in 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/318 objects degraded (8.176%), 5 pgs degraded) 2024-04-05T03:11:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:28 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: pgmap v170: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: osdmap e81: 6 total, 6 up, 6 in 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/318 objects degraded (8.176%), 5 pgs degraded) 2024-04-05T03:11:28.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:28 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:11:29.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:29.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:30.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:30 smithi203 ceph-mon[95988]: pgmap v172: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:30.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:30 smithi153 ceph-mon[106102]: pgmap v172: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:32.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:32 smithi153 ceph-mon[106102]: pgmap v173: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:32.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:32 smithi203 ceph-mon[95988]: pgmap v173: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:33.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:33.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:33.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:33.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:34.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:34 smithi153 ceph-mon[106102]: pgmap v174: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:34 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:boot 2024-04-05T03:11:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:34 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:34 smithi203 ceph-mon[95988]: pgmap v174: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:34 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:boot 2024-04-05T03:11:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:34 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 -- 172.21.15.153:0/1680824597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 msgr2=0x7fba900fcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1680824597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba900fcdb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba84004940 tx=0x7fba84009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 -- 172.21.15.153:0/1680824597 shutdown_connections 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1680824597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba900fcdb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1680824597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba90103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 -- 172.21.15.153:0/1680824597 >> 172.21.15.153:0/1680824597 conn(0x7fba900f8930 msgr2=0x7fba900fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 -- 172.21.15.153:0/1680824597 shutdown_connections 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.075+0000 7fba96e5c700 1 -- 172.21.15.153:0/1680824597 wait complete. 2024-04-05T03:11:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 Processor -- start 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 -- start start 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90070d90 con 0x7fba90103740 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba96e5c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90070ed0 con 0x7fba90102da0 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51210/0 (socket says 172.21.15.153:51210) 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 -- 172.21.15.153:0/1413712627 learned_addr learned my addr 172.21.15.153:0/1413712627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba94bf8700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 -- 172.21.15.153:0/1413712627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 msgr2=0x7fba900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba8ffff700 1 -- 172.21.15.153:0/1413712627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84005040 con 0x7fba90103740 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.076+0000 7fba94bf8700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:11:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.077+0000 7fba8ffff700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fba84009100 tx=0x7fba84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.077+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8400e040 con 0x7fba90103740 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.077+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba84004290 con 0x7fba90103740 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.077+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba90071120 con 0x7fba90103740 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.077+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8401c3f0 con 0x7fba90103740 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.078+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9011a260 con 0x7fba90103740 2024-04-05T03:11:35.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.079+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fba84013790 con 0x7fba90103740 2024-04-05T03:11:35.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.079+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba900fe4f0 con 0x7fba90103740 2024-04-05T03:11:35.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.079+0000 7fba8d7fa700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 0x7fba78070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.079+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fba8408e860 con 0x7fba90103740 2024-04-05T03:11:35.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.079+0000 7fba94bf8700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 0x7fba78070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.080+0000 7fba94bf8700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 0x7fba78070160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba80009fd0 tx=0x7fba80009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.084+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba84057e60 con 0x7fba90103740 2024-04-05T03:11:35.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.318+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba90109f20 con 0x7fba7806dcb0 2024-04-05T03:11:35.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.320+0000 7fba8d7fa700 1 -- 172.21.15.153:0/1413712627 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fba80009970 con 0x7fba7806dcb0 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.322+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 msgr2=0x7fba78070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.322+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 0x7fba78070160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fba80009fd0 tx=0x7fba80009ee0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.322+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 msgr2=0x7fba90070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.322+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fba84009100 tx=0x7fba84009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 shutdown_connections 2024-04-05T03:11:35.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba7806dcb0 0x7fba78070160 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba90102da0 0x7fba900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 --2- 172.21.15.153:0/1413712627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba90103740 0x7fba90070800 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 >> 172.21.15.153:0/1413712627 conn(0x7fba900f8930 msgr2=0x7fba900f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 shutdown_connections 2024-04-05T03:11:35.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.323+0000 7fba96e5c700 1 -- 172.21.15.153:0/1413712627 wait complete. 2024-04-05T03:11:35.334 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:11:35.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.431+0000 7f6624718700 1 -- 172.21.15.153:0/970670862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 msgr2=0x7f661c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.431+0000 7f6624718700 1 --2- 172.21.15.153:0/970670862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c0fbd80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f660c004940 tx=0x7f660c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- 172.21.15.153:0/970670862 shutdown_connections 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 --2- 172.21.15.153:0/970670862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c0fbd80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 --2- 172.21.15.153:0/970670862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 0x7f661c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- 172.21.15.153:0/970670862 >> 172.21.15.153:0/970670862 conn(0x7f661c072bc0 msgr2=0x7f661c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- 172.21.15.153:0/970670862 shutdown_connections 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- 172.21.15.153:0/970670862 wait complete. 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 Processor -- start 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- start start 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 0x7f661c113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c114730 con 0x7f661c102710 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.432+0000 7f6624718700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c10db20 con 0x7f661c101d70 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51232/0 (socket says 172.21.15.153:51232) 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 -- 172.21.15.153:0/3492651817 learned_addr learned my addr 172.21.15.153:0/3492651817 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f66224b4700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 0x7f661c113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 -- 172.21.15.153:0/3492651817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 msgr2=0x7f661c113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 0x7f661c113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 -- 172.21.15.153:0/3492651817 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c005040 con 0x7f661c102710 2024-04-05T03:11:35.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6621cb3700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f660c009180 tx=0x7f660c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.433+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660c00e040 con 0x7f661c102710 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.434+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f660c00bdd0 con 0x7f661c102710 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.434+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c10dda0 con 0x7f661c102710 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.434+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660c0138a0 con 0x7f661c102710 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.434+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c10e360 con 0x7f661c102710 2024-04-05T03:11:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.435+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0fd4c0 con 0x7f661c102710 2024-04-05T03:11:35.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.437+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f660c022020 con 0x7f661c102710 2024-04-05T03:11:35.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.437+0000 7f6612ffd700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 0x7f6608070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.437+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f660c092a60 con 0x7f661c102710 2024-04-05T03:11:35.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.437+0000 7f66224b4700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 0x7f6608070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.438+0000 7f66224b4700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 0x7f6608070150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6618007fd0 tx=0x7f6618009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.442+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f660c05fc30 con 0x7f661c102710 2024-04-05T03:11:35.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.684+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f661c0666a0 con 0x7f660806dca0 2024-04-05T03:11:35.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.686+0000 7f6612ffd700 1 -- 172.21.15.153:0/3492651817 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f661c0666a0 con 0x7f660806dca0 2024-04-05T03:11:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 msgr2=0x7f6608070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 0x7f6608070150 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6618007fd0 tx=0x7f6618009040 comp rx=0 tx=0).stop 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 msgr2=0x7f661c1140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f660c009180 tx=0x7f660c009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 shutdown_connections 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f660806dca0 0x7f6608070150 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f661c101d70 0x7f661c113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 --2- 172.21.15.153:0/3492651817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f661c102710 0x7f661c1140e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 >> 172.21.15.153:0/3492651817 conn(0x7f661c072bc0 msgr2=0x7f661c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 shutdown_connections 2024-04-05T03:11:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.688+0000 7f6624718700 1 -- 172.21.15.153:0/3492651817 wait complete. 2024-04-05T03:11:35.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 -- 172.21.15.153:0/1695115304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 msgr2=0x7f6ff8106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 --2- 172.21.15.153:0/1695115304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8106f20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6fec004970 tx=0x7f6fec009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:35.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 -- 172.21.15.153:0/1695115304 shutdown_connections 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 --2- 172.21.15.153:0/1695115304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8106f20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 --2- 172.21.15.153:0/1695115304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 0x7f6ff80fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 -- 172.21.15.153:0/1695115304 >> 172.21.15.153:0/1695115304 conn(0x7f6ff8072bc0 msgr2=0x7f6ff8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 -- 172.21.15.153:0/1695115304 shutdown_connections 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.797+0000 7f6ffea10700 1 -- 172.21.15.153:0/1695115304 wait complete. 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 Processor -- start 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 -- start start 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 0x7f6ff8113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8114770 con 0x7f6ff80fe850 2024-04-05T03:11:35.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ffea10700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810dad0 con 0x7f6ff80fdeb0 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ff77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ff77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51254/0 (socket says 172.21.15.153:51254) 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.798+0000 7f6ff77fe700 1 -- 172.21.15.153:0/2166526880 learned_addr learned my addr 172.21.15.153:0/2166526880 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff7fff700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 0x7f6ff8113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff77fe700 1 -- 172.21.15.153:0/2166526880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 msgr2=0x7f6ff8113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff77fe700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 0x7f6ff8113b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:35.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff77fe700 1 -- 172.21.15.153:0/2166526880 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec005040 con 0x7f6ff80fe850 2024-04-05T03:11:35.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff77fe700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fec00e040 con 0x7f6ff80fe850 2024-04-05T03:11:35.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fec00bc10 con 0x7f6ff80fe850 2024-04-05T03:11:35.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff810dd50 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fec01c630 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.799+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff810e290 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.800+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8065e10 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.801+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6fec00bd80 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.801+0000 7f6ff4ff9700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 0x7f6fe0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.802+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6fec08e820 con 0x7f6ff80fe850 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.802+0000 7f6ff7fff700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 0x7f6fe0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.802+0000 7f6ff7fff700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 0x7f6fe0070150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8006fd0 tx=0x7f6fe8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:35.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:35.806+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6fec05b950 con 0x7f6ff80fe850 2024-04-05T03:11:36.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.024+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ff8109fe0 con 0x7f6fe006dca0 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.034+0000 7f6ff4ff9700 1 -- 172.21.15.153:0/2166526880 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6ff8109fe0 con 0x7f6fe006dca0 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 2m ago 17m 26.0M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (3m) 2m ago 18m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (3m) 18s ago 16m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 2m ago 17m 80.9M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:11:36.034 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (11m) 2m ago 11m 116M - 17.2.7-805-g6fda41ff a49113eacc58 4ca7ef52cc4c 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 starting - - - - 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (11m) 18s ago 11m 61.3M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (11m) 18s ago 11m 112M - 17.2.7-805-g6fda41ff a49113eacc58 0862a50fde5f 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (7m) 2m ago 19m 504M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (7m) 18s ago 16m 451M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (4m) 2m ago 19m 45.7M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (4m) 18s ago 16m 44.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 2m ago 17m 16.7M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (6m) 18s ago 16m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 2m ago 15m 88.5M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 2m ago 14m 92.7M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 2m ago 14m 12.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (91s) 18s ago 14m 109M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (56s) 18s ago 13m 94.0M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (23s) 18s ago 13m 15.0M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:11:36.035 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (6m) 18s ago 15m 63.7M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:11:36.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 msgr2=0x7f6fe0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 0x7f6fe0070150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8006fd0 tx=0x7f6fe8008040 comp rx=0 tx=0).stop 2024-04-05T03:11:36.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 msgr2=0x7f6ff8114090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6fec000c00 tx=0x7f6fec009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 shutdown_connections 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6fe006dca0 0x7f6fe0070150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ff80fdeb0 0x7f6ff8113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 --2- 172.21.15.153:0/2166526880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ff80fe850 0x7f6ff8114090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 >> 172.21.15.153:0/2166526880 conn(0x7f6ff8072bc0 msgr2=0x7f6ff8103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 shutdown_connections 2024-04-05T03:11:36.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.037+0000 7f6ffea10700 1 -- 172.21.15.153:0/2166526880 wait complete. 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.139+0000 7fe3395bc700 1 -- 172.21.15.153:0/3509736952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334102710 msgr2=0x7fe3340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.139+0000 7fe3395bc700 1 --2- 172.21.15.153:0/3509736952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334102710 0x7fe3340fbd80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe328004940 tx=0x7fe328009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 -- 172.21.15.153:0/3509736952 shutdown_connections 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 --2- 172.21.15.153:0/3509736952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334102710 0x7fe3340fbd80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 --2- 172.21.15.153:0/3509736952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334101d70 0x7fe334102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 -- 172.21.15.153:0/3509736952 >> 172.21.15.153:0/3509736952 conn(0x7fe334072bc0 msgr2=0x7fe334072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 -- 172.21.15.153:0/3509736952 shutdown_connections 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 -- 172.21.15.153:0/3509736952 wait complete. 2024-04-05T03:11:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 Processor -- start 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.140+0000 7fe3395bc700 1 -- start start 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3395bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3395bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334102710 0x7fe3340fdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3395bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340fe4c0 con 0x7fe334101d70 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3395bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340fe600 con 0x7fe334102710 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51260/0 (socket says 172.21.15.153:51260) 2024-04-05T03:11:36.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 -- 172.21.15.153:0/314684990 learned_addr learned my addr 172.21.15.153:0/314684990 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3327fc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334102710 0x7fe3340fdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 -- 172.21.15.153:0/314684990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334102710 msgr2=0x7fe3340fdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334102710 0x7fe3340fdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 -- 172.21.15.153:0/314684990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe328005040 con 0x7fe334101d70 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe332ffd700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c830 tx=0x7fe324004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3240106f0 con 0x7fe334101d70 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe324010d20 con 0x7fe334101d70 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.141+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe334119ec0 con 0x7fe334101d70 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.142+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3240113a0 con 0x7fe334101d70 2024-04-05T03:11:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.142+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33411a2e0 con 0x7fe334101d70 2024-04-05T03:11:36.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.143+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340fd4c0 con 0x7fe334101d70 2024-04-05T03:11:36.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.143+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe324011500 con 0x7fe334101d70 2024-04-05T03:11:36.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.143+0000 7fe31b7fe700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 0x7fe31c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.143+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe32408d4a0 con 0x7fe334101d70 2024-04-05T03:11:36.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.143+0000 7fe3327fc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 0x7fe31c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.144+0000 7fe3327fc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 0x7fe31c070150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe328004890 tx=0x7fe328009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.148+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe324056b80 con 0x7fe334101d70 2024-04-05T03:11:36.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.457+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe334108eb0 con 0x7fe334101d70 2024-04-05T03:11:36.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.458+0000 7fe31b7fe700 1 -- 172.21.15.153:0/314684990 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7fe32405a240 con 0x7fe334101d70 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:36.458 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 3, 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 1 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 3, 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 11 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:11:36.459 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 msgr2=0x7fe31c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 0x7fe31c070150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe328004890 tx=0x7fe328009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 msgr2=0x7fe3340fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe32400c830 tx=0x7fe324004470 comp rx=0 tx=0).stop 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 shutdown_connections 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe31c06dca0 0x7fe31c070150 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe334101d70 0x7fe3340fd9f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 --2- 172.21.15.153:0/314684990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe334102710 0x7fe3340fdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 >> 172.21.15.153:0/314684990 conn(0x7fe334072bc0 msgr2=0x7fe3340659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 shutdown_connections 2024-04-05T03:11:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.460+0000 7fe3395bc700 1 -- 172.21.15.153:0/314684990 wait complete. 2024-04-05T03:11:36.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 -- 172.21.15.153:0/1499146676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 msgr2=0x7f5070106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 --2- 172.21.15.153:0/1499146676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070106f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5064004940 tx=0x7f5064009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 -- 172.21.15.153:0/1499146676 shutdown_connections 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 --2- 172.21.15.153:0/1499146676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070106f00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 --2- 172.21.15.153:0/1499146676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 0x7f50700fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 -- 172.21.15.153:0/1499146676 >> 172.21.15.153:0/1499146676 conn(0x7f5070072bc0 msgr2=0x7f5070072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 -- 172.21.15.153:0/1499146676 shutdown_connections 2024-04-05T03:11:36.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.562+0000 7f507686e700 1 -- 172.21.15.153:0/1499146676 wait complete. 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 Processor -- start 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 -- start start 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 0x7f5070113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50701146b0 con 0x7f50700fe830 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f507686e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507010daa0 con 0x7f50700fde90 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51280/0 (socket says 172.21.15.153:51280) 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 -- 172.21.15.153:0/721866304 learned_addr learned my addr 172.21.15.153:0/721866304 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506ffff700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 0x7f5070113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 -- 172.21.15.153:0/721866304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 msgr2=0x7f5070113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 0x7f5070113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.563+0000 7f506f7fe700 1 -- 172.21.15.153:0/721866304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5064005040 con 0x7f50700fe830 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f506f7fe700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5064009360 tx=0x7f5064009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f506400e040 con 0x7f50700fe830 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5064013420 con 0x7f50700fe830 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507010dd20 con 0x7f50700fe830 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f506401c630 con 0x7f50700fe830 2024-04-05T03:11:36.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.564+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507010e260 con 0x7f50700fe830 2024-04-05T03:11:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.565+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5070065e30 con 0x7f50700fe830 2024-04-05T03:11:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.565+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5064013a90 con 0x7f50700fe830 2024-04-05T03:11:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.566+0000 7f506cff9700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 0x7f5058070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.566+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f506408e7a0 con 0x7f50700fe830 2024-04-05T03:11:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.566+0000 7f506ffff700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 0x7f5058070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.566+0000 7f506ffff700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 0x7f5058070150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5060009fd0 tx=0x7f5060009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.570+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f506405b8d0 con 0x7f50700fe830 2024-04-05T03:11:36.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:36 smithi153 ceph-mon[106102]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:36.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:36 smithi153 ceph-mon[106102]: pgmap v175: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:36.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:36 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/314684990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.821+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f507010f1a0 con 0x7f50700fe830 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.822+0000 7f506cff9700 1 -- 172.21.15.153:0/721866304 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f506405b460 con 0x7f50700fe830 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:e30 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:11:36.822 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:11:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:11:36.824 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34384} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/1079014242,v1:172.21.15.153:6829/1079014242] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/2394111038,v1:172.21.15.203:6827/2394111038] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:36.825 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/4013577824,v1:172.21.15.203:6825/4013577824] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 msgr2=0x7f5058070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 0x7f5058070150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5060009fd0 tx=0x7f5060009f60 comp rx=0 tx=0).stop 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 msgr2=0x7f5070114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5064009360 tx=0x7f5064009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 shutdown_connections 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f505806dca0 0x7f5058070150 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50700fde90 0x7f5070113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 --2- 172.21.15.153:0/721866304 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50700fe830 0x7f5070114060 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 >> 172.21.15.153:0/721866304 conn(0x7f5070072bc0 msgr2=0x7f5070103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 shutdown_connections 2024-04-05T03:11:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.825+0000 7f507686e700 1 -- 172.21.15.153:0/721866304 wait complete. 2024-04-05T03:11:36.827 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 30 2024-04-05T03:11:36.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 -- 172.21.15.153:0/3313016111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 msgr2=0x7f63f40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 --2- 172.21.15.153:0/3313016111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f40fce10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f63e4004940 tx=0x7f63e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:36.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 -- 172.21.15.153:0/3313016111 shutdown_connections 2024-04-05T03:11:36.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 --2- 172.21.15.153:0/3313016111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f40fce10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 --2- 172.21.15.153:0/3313016111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 0x7f63f41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 -- 172.21.15.153:0/3313016111 >> 172.21.15.153:0/3313016111 conn(0x7f63f40f89b0 msgr2=0x7f63f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 -- 172.21.15.153:0/3313016111 shutdown_connections 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.928+0000 7f63fbba4700 1 -- 172.21.15.153:0/3313016111 wait complete. 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 Processor -- start 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 -- start start 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 0x7f63f40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f4072210 con 0x7f63f41037a0 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63fbba4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f4070d40 con 0x7f63f4102e00 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51306/0 (socket says 172.21.15.153:51306) 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 -- 172.21.15.153:0/1455634047 learned_addr learned my addr 172.21.15.153:0/1455634047 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f9940700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 0x7f63f40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 -- 172.21.15.153:0/1455634047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 msgr2=0x7f63f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 0x7f63f40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.929+0000 7f63f913f700 1 -- 172.21.15.153:0/1455634047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e4005040 con 0x7f63f41037a0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63f913f700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009100 tx=0x7f63e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e400e040 con 0x7f63f41037a0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63e4009af0 con 0x7f63f41037a0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f4070fc0 con 0x7f63f41037a0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63e401c3f0 con 0x7f63f41037a0 2024-04-05T03:11:36.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.930+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f411a0d0 con 0x7f63f41037a0 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.931+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f41052d0 con 0x7f63f41037a0 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.931+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f63e400baf0 con 0x7f63f41037a0 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.932+0000 7f63ea7fc700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 0x7f63e0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.932+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f63e408e700 con 0x7f63f41037a0 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.932+0000 7f63f9940700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 0x7f63e0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:36.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.932+0000 7f63f9940700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 0x7f63e0070150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f63f0004eb0 tx=0x7f63f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:36.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:36.936+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f63e4057da0 con 0x7f63f41037a0 2024-04-05T03:11:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:36 smithi203 ceph-mon[95988]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:36 smithi203 ceph-mon[95988]: pgmap v175: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:36 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/314684990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:37.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.154+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63f4109ee0 con 0x7f63e006dca0 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.160+0000 7f63ea7fc700 1 -- 172.21.15.153:0/1455634047 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f63f4109ee0 con 0x7f63e006dca0 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:11:37.160 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:11:37.161 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "12/21 daemons upgraded", 2024-04-05T03:11:37.161 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:11:37.161 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:11:37.161 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:11:37.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 msgr2=0x7f63e0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 0x7f63e0070150 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f63f0004eb0 tx=0x7f63f000b040 comp rx=0 tx=0).stop 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 msgr2=0x7f63f4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009100 tx=0x7f63e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 shutdown_connections 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f63e006dca0 0x7f63e0070150 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f4102e00 0x7f63f40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 --2- 172.21.15.153:0/1455634047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63f41037a0 0x7f63f4070800 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 >> 172.21.15.153:0/1455634047 conn(0x7f63f40f89b0 msgr2=0x7f63f41021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 shutdown_connections 2024-04-05T03:11:37.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.163+0000 7f63fbba4700 1 -- 172.21.15.153:0/1455634047 wait complete. 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 -- 172.21.15.153:0/1441502791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 msgr2=0x7f3a6c0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 --2- 172.21.15.153:0/1441502791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c0fce10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3a60004940 tx=0x7f3a60009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 -- 172.21.15.153:0/1441502791 shutdown_connections 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 --2- 172.21.15.153:0/1441502791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c0fce10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 --2- 172.21.15.153:0/1441502791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 0x7f3a6c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 -- 172.21.15.153:0/1441502791 >> 172.21.15.153:0/1441502791 conn(0x7f3a6c0f89b0 msgr2=0x7f3a6c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 -- 172.21.15.153:0/1441502791 shutdown_connections 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.269+0000 7f3a71a84700 1 -- 172.21.15.153:0/1441502791 wait complete. 2024-04-05T03:11:37.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 Processor -- start 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 -- start start 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 0x7f3a6c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c070e20 con 0x7f3a6c1037a0 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a71a84700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c070f60 con 0x7f3a6c102e00 2024-04-05T03:11:37.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a6affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51326/0 (socket says 172.21.15.153:51326) 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.270+0000 7f3a6affd700 1 -- 172.21.15.153:0/606753087 learned_addr learned my addr 172.21.15.153:0/606753087 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a6b7fe700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 0x7f3a6c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a6affd700 1 -- 172.21.15.153:0/606753087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 msgr2=0x7f3a6c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a6affd700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 0x7f3a6c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a6affd700 1 -- 172.21.15.153:0/606753087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60005040 con 0x7f3a6c1037a0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a6affd700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009100 tx=0x7f3a60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6000e040 con 0x7f3a6c1037a0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c119ee0 con 0x7f3a6c1037a0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a60004290 con 0x7f3a6c1037a0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c11a3c0 con 0x7f3a6c1037a0 2024-04-05T03:11:37.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.271+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6001c3f0 con 0x7f3a6c1037a0 2024-04-05T03:11:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.272+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c100110 con 0x7f3a6c1037a0 2024-04-05T03:11:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.273+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3a60013790 con 0x7f3a6c1037a0 2024-04-05T03:11:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.273+0000 7f3a53fff700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 0x7f3a54070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:11:37.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.273+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3a6008e860 con 0x7f3a6c1037a0 2024-04-05T03:11:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.273+0000 7f3a6b7fe700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 0x7f3a54070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:11:37.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.274+0000 7f3a6b7fe700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 0x7f3a54070160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:11:37.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.280+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3a60057e60 con 0x7f3a6c1037a0 2024-04-05T03:11:37.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.567+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3a6c0710f0 con 0x7f3a6c1037a0 2024-04-05T03:11:37.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.567+0000 7f3a53fff700 1 -- 172.21.15.153:0/606753087 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3a6c0710f0 con 0x7f3a6c1037a0 2024-04-05T03:11:37.567 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:11:37.567 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:11:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:11:37.568 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:11:37.568 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:11:37.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 msgr2=0x7f3a54070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 0x7f3a54070160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c008040 comp rx=0 tx=0).stop 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 msgr2=0x7f3a6c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009100 tx=0x7f3a60009e30 comp rx=0 tx=0).stop 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 shutdown_connections 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3a5406dcb0 0x7f3a54070160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a6c102e00 0x7f3a6c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 --2- 172.21.15.153:0/606753087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a6c1037a0 0x7f3a6c070800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.569+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 >> 172.21.15.153:0/606753087 conn(0x7f3a6c0f89b0 msgr2=0x7f3a6c0fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.570+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 shutdown_connections 2024-04-05T03:11:37.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:11:37.570+0000 7f3a71a84700 1 -- 172.21.15.153:0/606753087 wait complete. 2024-04-05T03:11:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:37 smithi153 ceph-mon[106102]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:37 smithi153 ceph-mon[106102]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:37 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/721866304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:11:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:37 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:37.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:37 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:37 smithi203 ceph-mon[95988]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:37 smithi203 ceph-mon[95988]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:37 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/721866304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:11:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:37 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:37 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:38 smithi153 ceph-mon[106102]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:38 smithi153 ceph-mon[106102]: pgmap v176: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:38 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/606753087' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:11:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:38 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:38 smithi203 ceph-mon[95988]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:11:38.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:38 smithi203 ceph-mon[95988]: pgmap v176: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:38.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:38 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/606753087' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:11:38.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:38.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:38 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:40.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:40 smithi153 ceph-mon[106102]: pgmap v177: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:40 smithi203 ceph-mon[95988]: pgmap v177: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: pgmap v178: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:42.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: pgmap v178: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:43.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:43.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:43.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:43.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:43.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:44 smithi153 ceph-mon[106102]: pgmap v179: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:11:44.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:44 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:44 smithi203 ceph-mon[95988]: pgmap v179: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.fmpmvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:11:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:44 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:45.849 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:45 smithi153 ceph-mon[106102]: Upgrade: Updating mds.cephfs.smithi153.fmpmvd 2024-04-05T03:11:45.849 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:45 smithi153 ceph-mon[106102]: Deploying daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T03:11:46.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:45 smithi203 ceph-mon[95988]: Upgrade: Updating mds.cephfs.smithi153.fmpmvd 2024-04-05T03:11:46.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:45 smithi203 ceph-mon[95988]: Deploying daemon mds.cephfs.smithi153.fmpmvd on smithi153 2024-04-05T03:11:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:46 smithi203 ceph-mon[95988]: pgmap v180: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:46 smithi153 ceph-mon[106102]: pgmap v180: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:47.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:47 smithi153 ceph-mon[106102]: osdmap e82: 6 total, 6 up, 6 in 2024-04-05T03:11:47.899 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:47 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:11:48.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:47 smithi203 ceph-mon[95988]: osdmap e82: 6 total, 6 up, 6 in 2024-04-05T03:11:48.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:47 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:11:48.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:48 smithi153 ceph-mon[106102]: pgmap v182: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:48 smithi203 ceph-mon[95988]: pgmap v182: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:50.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:50.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:50.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:50.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: pgmap v183: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:50.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:boot 2024-04-05T03:11:50.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:50.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:50 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: pgmap v183: 65 pgs: 65 active+clean; 252 MiB data, 966 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:boot 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:11:50.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:50 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:11:52.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:52 smithi153 ceph-mon[106102]: pgmap v184: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:52 smithi203 ceph-mon[95988]: pgmap v184: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:53 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:53 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:53 smithi203 ceph-mon[95988]: pgmap v185: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:54.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:53 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:53 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:54.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:53 smithi153 ceph-mon[106102]: pgmap v185: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:55.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:54 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:55.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:54 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:56.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:55 smithi153 ceph-mon[106102]: pgmap v186: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:56.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:55 smithi203 ceph-mon[95988]: pgmap v186: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:58.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:58.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:58.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:58.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:58 smithi203 ceph-mon[95988]: pgmap v187: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:11:58.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:11:58.573 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:58 smithi153 ceph-mon[106102]: pgmap v187: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:00.184 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: Upgrade: Updating mds.cephfs.smithi203.xpteyc 2024-04-05T03:12:00.185 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:00.185 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:12:00.185 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:00.185 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: Deploying daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T03:12:00.185 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:11:59 smithi203 ceph-mon[95988]: pgmap v188: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: Upgrade: Updating mds.cephfs.smithi203.xpteyc 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.xpteyc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: Deploying daemon mds.cephfs.smithi203.xpteyc on smithi203 2024-04-05T03:12:00.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:11:59 smithi153 ceph-mon[106102]: pgmap v188: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:02.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:02 smithi203 ceph-mon[95988]: osdmap e83: 6 total, 6 up, 6 in 2024-04-05T03:12:02.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:02 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:12:02.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:02 smithi203 ceph-mon[95988]: pgmap v190: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:02 smithi153 ceph-mon[106102]: osdmap e83: 6 total, 6 up, 6 in 2024-04-05T03:12:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:02 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:12:02.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:02 smithi153 ceph-mon[106102]: pgmap v190: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:04.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:04 smithi153 ceph-mon[106102]: pgmap v191: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:04.823 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:04 smithi203 ceph-mon[95988]: pgmap v191: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:06.373 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: pgmap v192: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] up:boot 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:12:06.374 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:06 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: pgmap v192: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] up:boot 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:12:06.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:06 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:12:07.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 -- 172.21.15.153:0/2505080509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c071f90 msgr2=0x7fb60c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 --2- 172.21.15.153:0/2505080509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c071f90 0x7fb60c070360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb608009d10 tx=0x7fb608004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 -- 172.21.15.153:0/2505080509 shutdown_connections 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 --2- 172.21.15.153:0/2505080509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c0708a0 0x7fb60c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 --2- 172.21.15.153:0/2505080509 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c071f90 0x7fb60c070360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 -- 172.21.15.153:0/2505080509 >> 172.21.15.153:0/2505080509 conn(0x7fb60c0f89b0 msgr2=0x7fb60c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 -- 172.21.15.153:0/2505080509 shutdown_connections 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.686+0000 7fb613e9d700 1 -- 172.21.15.153:0/2505080509 wait complete. 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 Processor -- start 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 -- start start 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c071f90 0x7fb60c11a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c11aad0 con 0x7fb60c0708a0 2024-04-05T03:12:07.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb613e9d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c114160 con 0x7fb60c071f90 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb611c39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb611c39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47258/0 (socket says 172.21.15.153:47258) 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb611c39700 1 -- 172.21.15.153:0/1275199206 learned_addr learned my addr 172.21.15.153:0/1275199206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.687+0000 7fb611438700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c071f90 0x7fb60c11a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb611c39700 1 -- 172.21.15.153:0/1275199206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c071f90 msgr2=0x7fb60c11a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb611c39700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c071f90 0x7fb60c11a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb611c39700 1 -- 172.21.15.153:0/1275199206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608009940 con 0x7fb60c0708a0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb611c39700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb608008df0 tx=0x7fb608004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60800e040 con 0x7fb60c0708a0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb60800bdd0 con 0x7fb60c0708a0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c1143e0 con 0x7fb60c0708a0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb60801c640 con 0x7fb60c0708a0 2024-04-05T03:12:07.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.688+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c114920 con 0x7fb60c0708a0 2024-04-05T03:12:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.689+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c071aa0 con 0x7fb60c0708a0 2024-04-05T03:12:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.690+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb608012970 con 0x7fb60c0708a0 2024-04-05T03:12:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.690+0000 7fb6027fc700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 0x7fb5f8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.690+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb60808f140 con 0x7fb60c0708a0 2024-04-05T03:12:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.690+0000 7fb611438700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 0x7fb5f8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.691+0000 7fb611438700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 0x7fb5f8070150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb60c10cf40 tx=0x7fb5fc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:07.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.695+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb608058740 con 0x7fb60c0708a0 2024-04-05T03:12:07.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.907+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb60c110520 con 0x7fb5f806dca0 2024-04-05T03:12:07.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.908+0000 7fb6027fc700 1 -- 172.21.15.153:0/1275199206 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb60c110520 con 0x7fb5f806dca0 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.910+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 msgr2=0x7fb5f8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.910+0000 7fb613e9d700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 0x7fb5f8070150 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb60c10cf40 tx=0x7fb5fc00b040 comp rx=0 tx=0).stop 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.910+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 msgr2=0x7fb60c119f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.910+0000 7fb613e9d700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb608008df0 tx=0x7fb608004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 shutdown_connections 2024-04-05T03:12:07.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb5f806dca0 0x7fb5f8070150 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb60c0708a0 0x7fb60c119f40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 --2- 172.21.15.153:0/1275199206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb60c071f90 0x7fb60c11a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:07.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 >> 172.21.15.153:0/1275199206 conn(0x7fb60c0f89b0 msgr2=0x7fb60c0f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:07.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 shutdown_connections 2024-04-05T03:12:07.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:07.911+0000 7fb613e9d700 1 -- 172.21.15.153:0/1275199206 wait complete. 2024-04-05T03:12:07.921 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.013+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/2849609934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 msgr2=0x7f2b181020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.013+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/2849609934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b181020f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b08004940 tx=0x7f2b08009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/2849609934 shutdown_connections 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/2849609934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b180fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/2849609934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b181020f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/2849609934 >> 172.21.15.153:0/2849609934 conn(0x7f2b180f7870 msgr2=0x7f2b180f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/2849609934 shutdown_connections 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/2849609934 wait complete. 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 Processor -- start 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- start start 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b18116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b18116670 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.014+0000 7f2b1cc0b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1810fc50 con 0x7f2b181026c0 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47278/0 (socket says 172.21.15.153:47278) 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 -- 172.21.15.153:0/879483848 learned_addr learned my addr 172.21.15.153:0/879483848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b15d9b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b18116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 -- 172.21.15.153:0/879483848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 msgr2=0x7f2b18116020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b18116020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 -- 172.21.15.153:0/879483848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08005040 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b15d9b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b18116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1659c700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800b8b0 tx=0x7f2b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b0800e040 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1810fed0 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b18110490 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b08004290 con 0x7f2b18101d20 2024-04-05T03:12:08.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.015+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b0801c3f0 con 0x7f2b18101d20 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.017+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b18106220 con 0x7f2b18101d20 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.017+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2b080137b0 con 0x7f2b18101d20 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.017+0000 7f2b06ffd700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 0x7f2b00070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.018+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2b0808e810 con 0x7f2b18101d20 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.018+0000 7f2b15d9b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 0x7f2b00070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.018+0000 7f2b15d9b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 0x7f2b00070150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c007fd0 tx=0x7f2b0c009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.022+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2b0805b9e0 con 0x7f2b18101d20 2024-04-05T03:12:08.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.234+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b18108e20 con 0x7f2b0006dca0 2024-04-05T03:12:08.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.235+0000 7f2b06ffd700 1 -- 172.21.15.153:0/879483848 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2b18108e20 con 0x7f2b0006dca0 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.237+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 msgr2=0x7f2b00070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.237+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 0x7f2b00070150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c007fd0 tx=0x7f2b0c009f60 comp rx=0 tx=0).stop 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.237+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 msgr2=0x7f2b18115ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.237+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800b8b0 tx=0x7f2b08009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 shutdown_connections 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2b0006dca0 0x7f2b00070150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18101d20 0x7f2b18115ae0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 --2- 172.21.15.153:0/879483848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b181026c0 0x7f2b18116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 >> 172.21.15.153:0/879483848 conn(0x7f2b180f7870 msgr2=0x7f2b180f97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 shutdown_connections 2024-04-05T03:12:08.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.238+0000 7f2b1cc0b700 1 -- 172.21.15.153:0/879483848 wait complete. 2024-04-05T03:12:08.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 -- 172.21.15.153:0/142094826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 msgr2=0x7fe6a0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 --2- 172.21.15.153:0/142094826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0103320 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe69c004940 tx=0x7fe69c009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 -- 172.21.15.153:0/142094826 shutdown_connections 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 --2- 172.21.15.153:0/142094826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 0x7fe6a00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 --2- 172.21.15.153:0/142094826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0103320 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.339+0000 7fe6a7290700 1 -- 172.21.15.153:0/142094826 >> 172.21.15.153:0/142094826 conn(0x7fe6a00714c0 msgr2=0x7fe6a00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 -- 172.21.15.153:0/142094826 shutdown_connections 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 -- 172.21.15.153:0/142094826 wait complete. 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 Processor -- start 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 -- start start 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 0x7fe6a0113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a01145b0 con 0x7fe6a0102f50 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a7290700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a010db90 con 0x7fe6a00fb4d0 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a482b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a482b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47290/0 (socket says 172.21.15.153:47290) 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a482b700 1 -- 172.21.15.153:0/2204116792 learned_addr learned my addr 172.21.15.153:0/2204116792 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.340+0000 7fe6a502c700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 0x7fe6a0113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a482b700 1 -- 172.21.15.153:0/2204116792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 msgr2=0x7fe6a0113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a482b700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 0x7fe6a0113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a482b700 1 -- 172.21.15.153:0/2204116792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c005040 con 0x7fe6a0102f50 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a482b700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe690002a50 tx=0x7fe6900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe690007cc0 con 0x7fe6a0102f50 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a010de70 con 0x7fe6a0102f50 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe69000f040 con 0x7fe6a0102f50 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe69000da30 con 0x7fe6a0102f50 2024-04-05T03:12:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.341+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a010e3b0 con 0x7fe6a0102f50 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.342+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a0100200 con 0x7fe6a0102f50 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.343+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe69000db90 con 0x7fe6a0102f50 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.343+0000 7fe695ffb700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 0x7fe68c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.343+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe69008d310 con 0x7fe6a0102f50 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.343+0000 7fe6a502c700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 0x7fe68c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.343+0000 7fe6a502c700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 0x7fe68c070150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe69c004890 tx=0x7fe69c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.348+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe69005a3c0 con 0x7fe6a0102f50 2024-04-05T03:12:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.551+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6a010a0a0 con 0x7fe68c06dca0 2024-04-05T03:12:08.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.561+0000 7fe695ffb700 1 -- 172.21.15.153:0/2204116792 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe6a010a0a0 con 0x7fe68c06dca0 2024-04-05T03:12:08.560 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:12:08.560 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (6m) 15s ago 17m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (4m) 15s ago 18m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (4m) 51s ago 16m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 15s ago 17m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (19s) 15s ago 12m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (36s) 15s ago 12m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (12m) 51s ago 12m 61.3M - 17.2.7-805-g6fda41ff a49113eacc58 6078fa87a326 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 starting - - - - 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (8m) 15s ago 20m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (7m) 51s ago 16m 451M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (5m) 15s ago 20m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (4m) 51s ago 16m 44.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 15s ago 18m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (7m) 51s ago 16m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 15s ago 16m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:12:08.561 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (3m) 15s ago 15m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:12:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 15s ago 15m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:12:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (2m) 51s ago 14m 109M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:12:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (89s) 51s ago 14m 94.0M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:12:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (55s) 51s ago 13m 15.0M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:12:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (6m) 51s ago 16m 63.7M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 msgr2=0x7fe68c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 0x7fe68c070150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe69c004890 tx=0x7fe69c009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 msgr2=0x7fe6a0113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe690002a50 tx=0x7fe6900046b0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 shutdown_connections 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe68c06dca0 0x7fe68c070150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a00fb4d0 0x7fe6a0113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 --2- 172.21.15.153:0/2204116792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6a0102f50 0x7fe6a0113f60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 >> 172.21.15.153:0/2204116792 conn(0x7fe6a00714c0 msgr2=0x7fe6a00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 shutdown_connections 2024-04-05T03:12:08.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.564+0000 7fe6a7290700 1 -- 172.21.15.153:0/2204116792 wait complete. 2024-04-05T03:12:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 -- 172.21.15.153:0/291266463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc101d70 msgr2=0x7f8fcc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/291266463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc101d70 0x7f8fcc102140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4004940 tx=0x7f8fc4009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.665 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:08 smithi153 ceph-mon[106102]: pgmap v193: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 -- 172.21.15.153:0/291266463 shutdown_connections 2024-04-05T03:12:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/291266463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc102710 0x7f8fcc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/291266463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc101d70 0x7f8fcc102140 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 -- 172.21.15.153:0/291266463 >> 172.21.15.153:0/291266463 conn(0x7f8fcc072bc0 msgr2=0x7f8fcc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 -- 172.21.15.153:0/291266463 shutdown_connections 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.665+0000 7f8fd45f2700 1 -- 172.21.15.153:0/291266463 wait complete. 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 Processor -- start 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 -- start start 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc101d70 0x7f8fcc112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc1135f0 con 0x7f8fcc102710 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd45f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc10c9e0 con 0x7f8fcc101d70 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53786/0 (socket says 172.21.15.153:53786) 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 -- 172.21.15.153:0/2814398801 learned_addr learned my addr 172.21.15.153:0/2814398801 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd238e700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc101d70 0x7f8fcc112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 -- 172.21.15.153:0/2814398801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc101d70 msgr2=0x7f8fcc112a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc101d70 0x7f8fcc112a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.666+0000 7f8fd1b8d700 1 -- 172.21.15.153:0/2814398801 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4005040 con 0x7f8fcc102710 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fd1b8d700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00fd10 tx=0x7f8fbc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fbc00dcf0 con 0x7f8fcc102710 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fbc00de50 con 0x7f8fcc102710 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc10cc60 con 0x7f8fcc102710 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fbc0126a0 con 0x7f8fcc102710 2024-04-05T03:12:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.667+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc10d220 con 0x7f8fcc102710 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.668+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc0fd4c0 con 0x7f8fcc102710 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.669+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8fbc007ab0 con 0x7f8fcc102710 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.669+0000 7f8fc2ffd700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 0x7f8fb8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.669+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8fbc08f9c0 con 0x7f8fcc102710 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.669+0000 7f8fd238e700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 0x7f8fb8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:08.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.669+0000 7f8fd238e700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 0x7f8fb8070150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4004f90 tx=0x7f8fc4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:08.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.676+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8fbc05ca70 con 0x7f8fcc102710 2024-04-05T03:12:08.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.952+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8fcc05f5f0 con 0x7f8fcc102710 2024-04-05T03:12:08.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.953+0000 7f8fc2ffd700 1 -- 172.21.15.153:0/2814398801 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f8fbc05c600 con 0x7f8fcc102710 2024-04-05T03:12:08.952 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:12:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:12:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 3 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:12:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-805-g6fda41ff (6fda41ff74b7f038ecae6f40cc5fca107a3d4207) quincy (stable)": 1, 2024-04-05T03:12:08.954 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 13 2024-04-05T03:12:08.954 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:12:08.954 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:12:08.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 msgr2=0x7f8fb8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 0x7f8fb8070150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4004f90 tx=0x7f8fc4009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 msgr2=0x7f8fcc112fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc00fd10 tx=0x7f8fbc004630 comp rx=0 tx=0).stop 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 shutdown_connections 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8fb806dca0 0x7f8fb8070150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fcc101d70 0x7f8fcc112a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 --2- 172.21.15.153:0/2814398801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fcc102710 0x7f8fcc112fa0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 >> 172.21.15.153:0/2814398801 conn(0x7f8fcc072bc0 msgr2=0x7f8fcc0fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 shutdown_connections 2024-04-05T03:12:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:08.955+0000 7f8fd45f2700 1 -- 172.21.15.153:0/2814398801 wait complete. 2024-04-05T03:12:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:08 smithi203 ceph-mon[95988]: pgmap v193: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 -- 172.21.15.153:0/3752058096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fde70 msgr2=0x7f2f000fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 --2- 172.21.15.153:0/3752058096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fde70 0x7f2f000fe240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2efc004940 tx=0x7f2efc009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 -- 172.21.15.153:0/3752058096 shutdown_connections 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 --2- 172.21.15.153:0/3752058096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fe780 0x7f2f00106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 --2- 172.21.15.153:0/3752058096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fde70 0x7f2f000fe240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.055+0000 7f2f073c8700 1 -- 172.21.15.153:0/3752058096 >> 172.21.15.153:0/3752058096 conn(0x7f2f000f7860 msgr2=0x7f2f000f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:09.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 -- 172.21.15.153:0/3752058096 shutdown_connections 2024-04-05T03:12:09.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 -- 172.21.15.153:0/3752058096 wait complete. 2024-04-05T03:12:09.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 Processor -- start 2024-04-05T03:12:09.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 -- start start 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 0x7f2f00113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00114700 con 0x7f2f000fe780 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.056+0000 7f2f073c8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0010daf0 con 0x7f2f000fde70 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f04963700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f04963700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53818/0 (socket says 172.21.15.153:53818) 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f05164700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 0x7f2f00113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f04963700 1 -- 172.21.15.153:0/452913681 learned_addr learned my addr 172.21.15.153:0/452913681 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f05164700 1 -- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 msgr2=0x7f2f001140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f05164700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f05164700 1 -- 172.21.15.153:0/452913681 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc005040 con 0x7f2f000fde70 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f04963700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:12:09.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f05164700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 0x7f2f00113b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2efc000c00 tx=0x7f2efc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2efc021070 con 0x7f2f000fde70 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0010dd70 con 0x7f2f000fde70 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2efc013c00 con 0x7f2f000fde70 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.057+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2efc00e070 con 0x7f2f000fde70 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.058+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0010e250 con 0x7f2f000fde70 2024-04-05T03:12:09.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.058+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f00066710 con 0x7f2f000fde70 2024-04-05T03:12:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.059+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2efc01c9d0 con 0x7f2f000fde70 2024-04-05T03:12:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.059+0000 7f2ef5ffb700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 0x7f2eec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.059+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2efc08e860 con 0x7f2f000fde70 2024-04-05T03:12:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.059+0000 7f2f04963700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 0x7f2eec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.060+0000 7f2f04963700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 0x7f2eec070150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2f0010ecb0 tx=0x7f2ef000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.065+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2efc05b990 con 0x7f2f000fde70 2024-04-05T03:12:09.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.305+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2f00109f00 con 0x7f2f000fde70 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.305+0000 7f2ef5ffb700 1 -- 172.21.15.153:0/452913681 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1878 (secure 0 0 0) 0x7f2efc05b520 con 0x7f2f000fde70 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:e34 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:12:09.305 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:12:09.306 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:09.307 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:12:09.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:09.308 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:09.308 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:09.308 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/2394111038,v1:172.21.15.203:6827/2394111038] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T03:12:09.308 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 msgr2=0x7f2eec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 0x7f2eec070150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2f0010ecb0 tx=0x7f2ef000b040 comp rx=0 tx=0).stop 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 msgr2=0x7f2f00113b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 0x7f2f00113b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2efc000c00 tx=0x7f2efc009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 shutdown_connections 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2eec06dca0 0x7f2eec070150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fde70 0x7f2f00113b70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 --2- 172.21.15.153:0/452913681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f000fe780 0x7f2f001140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 >> 172.21.15.153:0/452913681 conn(0x7f2f000f7860 msgr2=0x7f2f001036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 shutdown_connections 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.308+0000 7f2f073c8700 1 -- 172.21.15.153:0/452913681 wait complete. 2024-04-05T03:12:09.309 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 34 2024-04-05T03:12:09.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 -- 172.21.15.153:0/1912119167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 msgr2=0x7f27001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 --2- 172.21.15.153:0/1912119167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f27001031d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009ce0 tx=0x7f26fc004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:09.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 -- 172.21.15.153:0/1912119167 shutdown_connections 2024-04-05T03:12:09.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 --2- 172.21.15.153:0/1912119167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 0x7f27000fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 --2- 172.21.15.153:0/1912119167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f27001031d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 -- 172.21.15.153:0/1912119167 >> 172.21.15.153:0/1912119167 conn(0x7f27000f89b0 msgr2=0x7f27000fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 -- 172.21.15.153:0/1912119167 shutdown_connections 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.409+0000 7f27082dd700 1 -- 172.21.15.153:0/1912119167 wait complete. 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 Processor -- start 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 -- start start 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 0x7f27001140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2700114730 con 0x7f2700102e00 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f27082dd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270010db70 con 0x7f27001037a0 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53834/0 (socket says 172.21.15.153:53834) 2024-04-05T03:12:09.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 -- 172.21.15.153:0/1980524776 learned_addr learned my addr 172.21.15.153:0/1980524776 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2705878700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 0x7f27001140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 -- 172.21.15.153:0/1980524776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 msgr2=0x7f27001140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 0x7f27001140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 -- 172.21.15.153:0/1980524776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc009940 con 0x7f2700102e00 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f2706079700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00ba50 tx=0x7f26fc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.410+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26fc00e040 con 0x7f2700102e00 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.411+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26fc01cb20 con 0x7f2700102e00 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.411+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270010ddf0 con 0x7f2700102e00 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.411+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26fc012990 con 0x7f2700102e00 2024-04-05T03:12:09.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.411+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270010e3b0 con 0x7f2700102e00 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.412+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2700100110 con 0x7f2700102e00 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.412+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f26fc01cc90 con 0x7f2700102e00 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.413+0000 7f26f6ffd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 0x7f26ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.413+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26fc08f8f0 con 0x7f2700102e00 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.413+0000 7f2705878700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 0x7f26ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.413+0000 7f2705878700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 0x7f26ec070150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f270010ece0 tx=0x7f26f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.417+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f26fc05c9a0 con 0x7f2700102e00 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2814398801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:09.572 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:09 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/452913681' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:12:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.632+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f270010f1f0 con 0x7f26ec06dca0 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.634+0000 7f26f6ffd700 1 -- 172.21.15.153:0/1980524776 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f270010f1f0 con 0x7f26ec06dca0 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mon", 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:12:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "crash" 2024-04-05T03:12:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:12:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "15/21 daemons upgraded", 2024-04-05T03:12:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:12:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:12:09.634 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:12:09.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 msgr2=0x7f26ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 0x7f26ec070150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f270010ece0 tx=0x7f26f000b040 comp rx=0 tx=0).stop 2024-04-05T03:12:09.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 msgr2=0x7f2700113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00ba50 tx=0x7f26fc004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 shutdown_connections 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dca0 0x7f26ec070150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2700102e00 0x7f2700113ba0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 --2- 172.21.15.153:0/1980524776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27001037a0 0x7f27001140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 >> 172.21.15.153:0/1980524776 conn(0x7f27000f89b0 msgr2=0x7f27001073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 shutdown_connections 2024-04-05T03:12:09.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.636+0000 7f27082dd700 1 -- 172.21.15.153:0/1980524776 wait complete. 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- 172.21.15.153:0/3872808174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6644102e10 msgr2=0x7f66441031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 --2- 172.21.15.153:0/3872808174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6644102e10 0x7f66441031e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6638004940 tx=0x7f6638009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- 172.21.15.153:0/3872808174 shutdown_connections 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 --2- 172.21.15.153:0/3872808174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66441037b0 0x7f66440fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 --2- 172.21.15.153:0/3872808174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6644102e10 0x7f66441031e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- 172.21.15.153:0/3872808174 >> 172.21.15.153:0/3872808174 conn(0x7f66440f8980 msgr2=0x7f66440fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- 172.21.15.153:0/3872808174 shutdown_connections 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- 172.21.15.153:0/3872808174 wait complete. 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 Processor -- start 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.737+0000 7f664acac700 1 -- start start 2024-04-05T03:12:09.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f664acac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f664acac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f664acac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66441146b0 con 0x7f66441037b0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f664acac700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664410daa0 con 0x7f6644102e10 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6648a48700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53856/0 (socket says 172.21.15.153:53856) 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6648a48700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:47794/0 (socket says 172.21.15.153:47794) 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 -- 172.21.15.153:0/2201478156 learned_addr learned my addr 172.21.15.153:0/2201478156 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 -- 172.21.15.153:0/2201478156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 msgr2=0x7f6644113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 -- 172.21.15.153:0/2201478156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638005040 con 0x7f66441037b0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6648a48700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f6643fff700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f663000eb40 tx=0x7f663000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6630007cc0 con 0x7f66441037b0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f663000f040 con 0x7f66441037b0 2024-04-05T03:12:09.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.738+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664410dd20 con 0x7f66441037b0 2024-04-05T03:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.739+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663000c9d0 con 0x7f66441037b0 2024-04-05T03:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.739+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664410e260 con 0x7f66441037b0 2024-04-05T03:12:09.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.740+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644100120 con 0x7f66441037b0 2024-04-05T03:12:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.740+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f663000cc40 con 0x7f66441037b0 2024-04-05T03:12:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.740+0000 7f66417fa700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 0x7f6634070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.741+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f663008cfe0 con 0x7f66441037b0 2024-04-05T03:12:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.741+0000 7f6648a48700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 0x7f6634070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.741+0000 7f6648a48700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 0x7f6634070150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6638009b30 tx=0x7f6638009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:09.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:09.745+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f663005a140 con 0x7f66441037b0 2024-04-05T03:12:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.023+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f664410f170 con 0x7f66441037b0 2024-04-05T03:12:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.023+0000 7f66417fa700 1 -- 172.21.15.153:0/2201478156 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6630059cd0 con 0x7f66441037b0 2024-04-05T03:12:10.023 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:12:10.023 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:12:10.023 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:12:10.023 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:12:10.023 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 msgr2=0x7f6634070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 0x7f6634070150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6638009b30 tx=0x7f6638009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 msgr2=0x7f6644114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f663000eb40 tx=0x7f663000b040 comp rx=0 tx=0).stop 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 shutdown_connections 2024-04-05T03:12:10.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f663406dca0 0x7f6634070150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6644102e10 0x7f6644113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 --2- 172.21.15.153:0/2201478156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66441037b0 0x7f6644114060 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 >> 172.21.15.153:0/2201478156 conn(0x7f66440f8980 msgr2=0x7f66441073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 shutdown_connections 2024-04-05T03:12:10.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:10.026+0000 7f664acac700 1 -- 172.21.15.153:0/2201478156 wait complete. 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2814398801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:10.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:09 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/452913681' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:12:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:10 smithi153 ceph-mon[106102]: pgmap v194: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:10 smithi153 ceph-mon[106102]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:10 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2201478156' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:12:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:10.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:10 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:10.894 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:10 smithi203 ceph-mon[95988]: pgmap v194: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:10.894 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:10 smithi203 ceph-mon[95988]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:10.894 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:10 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2201478156' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:12:10.894 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:10 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:10.895 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:10 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:12 smithi153 ceph-mon[106102]: pgmap v195: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:12.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:12 smithi203 ceph-mon[95988]: pgmap v195: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:14.793 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: pgmap v196: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:14.793 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.793 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:14 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: pgmap v196: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:14.823 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:14 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:16.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: Upgrade: Updating mds.cephfs.smithi203.hgbifk 2024-04-05T03:12:16.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:16.728 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:12:16.729 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:16.729 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: Deploying daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T03:12:16.729 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:16 smithi203 ceph-mon[95988]: pgmap v197: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: Upgrade: Updating mds.cephfs.smithi203.hgbifk 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi203.hgbifk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: Deploying daemon mds.cephfs.smithi203.hgbifk on smithi203 2024-04-05T03:12:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:16 smithi153 ceph-mon[106102]: pgmap v197: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:18.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:18 smithi153 ceph-mon[106102]: pgmap v198: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:18 smithi203 ceph-mon[95988]: pgmap v198: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:19 smithi203 ceph-mon[95988]: osdmap e84: 6 total, 6 up, 6 in 2024-04-05T03:12:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:19 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:12:19.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:19 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:12:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:19 smithi153 ceph-mon[106102]: osdmap e84: 6 total, 6 up, 6 in 2024-04-05T03:12:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:19 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T03:12:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:19 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:12:20.761 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:20 smithi203 ceph-mon[95988]: pgmap v200: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:20.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:20 smithi153 ceph-mon[106102]: pgmap v200: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: pgmap v201: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] up:boot 2024-04-05T03:12:22.546 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:12:22.547 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: pgmap v201: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] up:boot 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T03:12:22.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:12:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:24 smithi153 ceph-mon[106102]: pgmap v202: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:24.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:24 smithi203 ceph-mon[95988]: pgmap v202: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:26 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:26 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:26.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:26 smithi203 ceph-mon[95988]: pgmap v203: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:26 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:26 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:12:26.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:26 smithi153 ceph-mon[106102]: pgmap v203: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:27.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:27.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:28.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:28 smithi203 ceph-mon[95988]: pgmap v204: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:28 smithi153 ceph-mon[106102]: pgmap v204: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:30 smithi153 ceph-mon[106102]: pgmap v205: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:30 smithi203 ceph-mon[95988]: pgmap v205: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:32.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:32 smithi153 ceph-mon[106102]: pgmap v206: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:32 smithi203 ceph-mon[95988]: pgmap v206: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:34.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:34 smithi153 ceph-mon[106102]: pgmap v207: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:34 smithi203 ceph-mon[95988]: pgmap v207: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:36.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:36 smithi153 ceph-mon[106102]: pgmap v208: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:36 smithi203 ceph-mon[95988]: pgmap v208: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:38.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:38 smithi153 ceph-mon[106102]: pgmap v209: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:38 smithi203 ceph-mon[95988]: pgmap v209: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 -- 172.21.15.153:0/1394010603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 msgr2=0x7f2bec0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/1394010603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec0fce10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2be0004940 tx=0x7f2be0009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 -- 172.21.15.153:0/1394010603 shutdown_connections 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/1394010603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec0fce10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/1394010603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 0x7f2bec1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 -- 172.21.15.153:0/1394010603 >> 172.21.15.153:0/1394010603 conn(0x7f2bec0f89b0 msgr2=0x7f2bec0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 -- 172.21.15.153:0/1394010603 shutdown_connections 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.142+0000 7f2bf1f87700 1 -- 172.21.15.153:0/1394010603 wait complete. 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 Processor -- start 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 -- start start 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 0x7f2bec113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec1144a0 con 0x7f2bec1037a0 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2bf1f87700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec10daf0 con 0x7f2bec102e00 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2beaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2beaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54980/0 (socket says 172.21.15.153:54980) 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2beaffd700 1 -- 172.21.15.153:0/2487719694 learned_addr learned my addr 172.21.15.153:0/2487719694 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:40.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.143+0000 7f2beb7fe700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 0x7f2bec113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2beaffd700 1 -- 172.21.15.153:0/2487719694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 msgr2=0x7f2bec113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2beaffd700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 0x7f2bec113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2beaffd700 1 -- 172.21.15.153:0/2487719694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be0005040 con 0x7f2bec1037a0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2beaffd700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009180 tx=0x7f2be0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be000e040 con 0x7f2bec1037a0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec10dd70 con 0x7f2bec1037a0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2be0013420 con 0x7f2bec1037a0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be001c610 con 0x7f2bec1037a0 2024-04-05T03:12:40.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.144+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec10e330 con 0x7f2bec1037a0 2024-04-05T03:12:40.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.145+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec1052d0 con 0x7f2bec1037a0 2024-04-05T03:12:40.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.146+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2be0013a00 con 0x7f2bec1037a0 2024-04-05T03:12:40.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.146+0000 7f2bd3fff700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 0x7f2bd4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.146+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2be008e690 con 0x7f2bec1037a0 2024-04-05T03:12:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.146+0000 7f2beb7fe700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 0x7f2bd4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.147+0000 7f2beb7fe700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 0x7f2bd4070150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009fd0 tx=0x7f2bdc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.151+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2be005b870 con 0x7f2bec1037a0 2024-04-05T03:12:40.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.364+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bec10ef60 con 0x7f2bd406dca0 2024-04-05T03:12:40.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.365+0000 7f2bd3fff700 1 -- 172.21.15.153:0/2487719694 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2bec10ef60 con 0x7f2bd406dca0 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.367+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 msgr2=0x7f2bd4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.367+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 0x7f2bd4070150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009fd0 tx=0x7f2bdc00b040 comp rx=0 tx=0).stop 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.367+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 msgr2=0x7f2bec113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.367+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009180 tx=0x7f2be0009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 shutdown_connections 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2bd406dca0 0x7f2bd4070150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bec102e00 0x7f2bec113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 --2- 172.21.15.153:0/2487719694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bec1037a0 0x7f2bec113e50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 >> 172.21.15.153:0/2487719694 conn(0x7f2bec0f89b0 msgr2=0x7f2bec1021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:40.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 shutdown_connections 2024-04-05T03:12:40.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.368+0000 7f2bf1f87700 1 -- 172.21.15.153:0/2487719694 wait complete. 2024-04-05T03:12:40.378 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 -- 172.21.15.153:0/1655430430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 msgr2=0x7f39b40fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 --2- 172.21.15.153:0/1655430430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b40fcdf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39a8004940 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 -- 172.21.15.153:0/1655430430 shutdown_connections 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 --2- 172.21.15.153:0/1655430430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b40fcdf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 --2- 172.21.15.153:0/1655430430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 0x7f39b41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 -- 172.21.15.153:0/1655430430 >> 172.21.15.153:0/1655430430 conn(0x7f39b40f8970 msgr2=0x7f39b40fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 -- 172.21.15.153:0/1655430430 shutdown_connections 2024-04-05T03:12:40.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.470+0000 7f39b96ad700 1 -- 172.21.15.153:0/1655430430 wait complete. 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.471+0000 7f39b96ad700 1 Processor -- start 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.471+0000 7f39b96ad700 1 -- start start 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.471+0000 7f39b96ad700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 0x7f39b4113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b96ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b96ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b41146f0 con 0x7f39b4103780 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b96ad700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b410dae0 con 0x7f39b4102de0 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55002/0 (socket says 172.21.15.153:55002) 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 -- 172.21.15.153:0/4075882017 learned_addr learned my addr 172.21.15.153:0/4075882017 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b2ffd700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 0x7f39b4113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 -- 172.21.15.153:0/4075882017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 msgr2=0x7f39b4113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 0x7f39b4113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 -- 172.21.15.153:0/4075882017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8005040 con 0x7f39b4103780 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b27fc700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009100 tx=0x7f39a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a800e040 con 0x7f39b4103780 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39a8009af0 con 0x7f39b4103780 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b410dd60 con 0x7f39b4103780 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.472+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a801c3f0 con 0x7f39b4103780 2024-04-05T03:12:40.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.473+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b410e320 con 0x7f39b4103780 2024-04-05T03:12:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.473+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b40fe530 con 0x7f39b4103780 2024-04-05T03:12:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.474+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f39a800baf0 con 0x7f39b4103780 2024-04-05T03:12:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.474+0000 7f399b7fe700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 0x7f399c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.475+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f39a808e7f0 con 0x7f39b4103780 2024-04-05T03:12:40.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.475+0000 7f39b2ffd700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 0x7f399c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.475+0000 7f39b2ffd700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 0x7f399c070150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39ac007970 tx=0x7f39ac009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.479+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f39a805b920 con 0x7f39b4103780 2024-04-05T03:12:40.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.697+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39b4109ec0 con 0x7f399c06dca0 2024-04-05T03:12:40.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.698+0000 7f399b7fe700 1 -- 172.21.15.153:0/4075882017 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f39b4109ec0 con 0x7f399c06dca0 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 msgr2=0x7f399c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 0x7f399c070150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39ac007970 tx=0x7f39ac009da0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 msgr2=0x7f39b41140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009100 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 shutdown_connections 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f399c06dca0 0x7f399c070150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b4102de0 0x7f39b4113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 --2- 172.21.15.153:0/4075882017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39b4103780 0x7f39b41140a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 >> 172.21.15.153:0/4075882017 conn(0x7f39b40f8970 msgr2=0x7f39b40f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 shutdown_connections 2024-04-05T03:12:40.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.700+0000 7f39b96ad700 1 -- 172.21.15.153:0/4075882017 wait complete. 2024-04-05T03:12:40.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 -- 172.21.15.153:0/2224656780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 msgr2=0x7f1064102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 --2- 172.21.15.153:0/2224656780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f1064102140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f105c004940 tx=0x7f105c009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:40.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 -- 172.21.15.153:0/2224656780 shutdown_connections 2024-04-05T03:12:40.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 --2- 172.21.15.153:0/2224656780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 0x7f10640fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 --2- 172.21.15.153:0/2224656780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f1064102140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 -- 172.21.15.153:0/2224656780 >> 172.21.15.153:0/2224656780 conn(0x7f1064072bc0 msgr2=0x7f1064072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:40.802 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:40 smithi153 ceph-mon[106102]: pgmap v210: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 -- 172.21.15.153:0/2224656780 shutdown_connections 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.802+0000 7f106b37d700 1 -- 172.21.15.153:0/2224656780 wait complete. 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 Processor -- start 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 -- start start 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 0x7f106410f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410f670 con 0x7f1064101d70 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f106b37d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106410f7b0 con 0x7f1064102710 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55018/0 (socket says 172.21.15.153:55018) 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 -- 172.21.15.153:0/1628757232 learned_addr learned my addr 172.21.15.153:0/1628757232 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1068918700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 0x7f106410f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 -- 172.21.15.153:0/1628757232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 msgr2=0x7f106410f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 0x7f106410f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.803+0000 7f1069119700 1 -- 172.21.15.153:0/1628757232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c005040 con 0x7f1064101d70 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f1069119700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f105c00b3b0 tx=0x7f105c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f105c00e040 con 0x7f1064101d70 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064117b50 con 0x7f1064101d70 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f105c00bbc0 con 0x7f1064101d70 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f105c01c3f0 con 0x7f1064101d70 2024-04-05T03:12:40.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.804+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1064118060 con 0x7f1064101d70 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.805+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10640ff080 con 0x7f1064101d70 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.805+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f105c00bd30 con 0x7f1064101d70 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.806+0000 7f1061ffb700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 0x7f104c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.806+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f105c08e890 con 0x7f1064101d70 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.806+0000 7f1068918700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 0x7f104c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:40.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.806+0000 7f1068918700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 0x7f104c070150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1064066000 tx=0x7f105800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:40.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:40.811+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f105c057db0 con 0x7f1064101d70 2024-04-05T03:12:40.847 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:40 smithi203 ceph-mon[95988]: pgmap v210: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:41.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.015+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f106404bed0 con 0x7f104c06dca0 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.025+0000 7f1061ffb700 1 -- 172.21.15.153:0/1628757232 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f106404bed0 con 0x7f104c06dca0 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (7m) 48s ago 18m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (4m) 48s ago 19m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (4m) 16s ago 17m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (6m) 48s ago 18m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (52s) 48s ago 12m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (68s) 48s ago 12m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (20s) 16s ago 12m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (36s) 16s ago 12m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:12:41.025 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (8m) 48s ago 20m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (8m) 16s ago 17m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (5m) 48s ago 20m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (5m) 16s ago 17m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 48s ago 18m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (7m) 16s ago 17m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 48s ago 16m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (3m) 48s ago 16m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (3m) 48s ago 15m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (2m) 16s ago 15m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (2m) 16s ago 14m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (88s) 16s ago 14m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:12:41.026 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (7m) 16s ago 16m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 msgr2=0x7f104c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 0x7f104c070150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1064066000 tx=0x7f105800b040 comp rx=0 tx=0).stop 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 msgr2=0x7f106410eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f105c00b3b0 tx=0x7f105c009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 shutdown_connections 2024-04-05T03:12:41.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f104c06dca0 0x7f104c070150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064101d70 0x7f106410eae0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 --2- 172.21.15.153:0/1628757232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1064102710 0x7f106410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.027+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 >> 172.21.15.153:0/1628757232 conn(0x7f1064072bc0 msgr2=0x7f10640fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.028+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 shutdown_connections 2024-04-05T03:12:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.028+0000 7f106b37d700 1 -- 172.21.15.153:0/1628757232 wait complete. 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/775883239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01037a0 msgr2=0x7fc7b00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/775883239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01037a0 0x7fc7b00fce10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0004940 tx=0x7fc7a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/775883239 shutdown_connections 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/775883239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b01037a0 0x7fc7b00fce10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/775883239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b0102e00 0x7fc7b01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/775883239 >> 172.21.15.153:0/775883239 conn(0x7fc7b00f89b0 msgr2=0x7fc7b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/775883239 shutdown_connections 2024-04-05T03:12:41.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/775883239 wait complete. 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 Processor -- start 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 -- start start 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.129+0000 7fc7b4ffa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7b4ffa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b01037a0 0x7fc7b01140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7b4ffa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b01147c0 con 0x7fc7b0102e00 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7b4ffa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b010db40 con 0x7fc7b01037a0 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55038/0 (socket says 172.21.15.153:55038) 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 -- 172.21.15.153:0/4273219886 learned_addr learned my addr 172.21.15.153:0/4273219886 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7add9b700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b01037a0 0x7fc7b01140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 -- 172.21.15.153:0/4273219886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b01037a0 msgr2=0x7fc7b01140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b01037a0 0x7fc7b01140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 -- 172.21.15.153:0/4273219886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0005040 con 0x7fc7b0102e00 2024-04-05T03:12:41.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7ae59c700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b790 tx=0x7fc7a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a400f6f0 con 0x7fc7b0102e00 2024-04-05T03:12:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7a400fd20 con 0x7fc7b0102e00 2024-04-05T03:12:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b010de20 con 0x7fc7b0102e00 2024-04-05T03:12:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a40103a0 con 0x7fc7b0102e00 2024-04-05T03:12:41.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.130+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b010e3b0 con 0x7fc7b0102e00 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.131+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0100110 con 0x7fc7b0102e00 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.132+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc7a4011400 con 0x7fc7b0102e00 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.132+0000 7fc79effd700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 0x7fc798070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.132+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc7a408d220 con 0x7fc7b0102e00 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.132+0000 7fc7add9b700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 0x7fc798070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.133+0000 7fc7add9b700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 0x7fc798070150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0004890 tx=0x7fc7a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.137+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc7a40568c0 con 0x7fc7b0102e00 2024-04-05T03:12:41.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.411+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc7b0109fb0 con 0x7fc7b0102e00 2024-04-05T03:12:41.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.412+0000 7fc79effd700 1 -- 172.21.15.153:0/4273219886 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc7a4059f80 con 0x7fc7b0102e00 2024-04-05T03:12:41.411 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:12:41.411 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:12:41.412 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:12:41.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 msgr2=0x7fc798070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 0x7fc798070150 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0004890 tx=0x7fc7a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 msgr2=0x7fc7b0113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b790 tx=0x7fc7a4009250 comp rx=0 tx=0).stop 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 shutdown_connections 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc79806dca0 0x7fc798070150 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7b0102e00 0x7fc7b0113ba0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 --2- 172.21.15.153:0/4273219886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b01037a0 0x7fc7b01140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 >> 172.21.15.153:0/4273219886 conn(0x7fc7b00f89b0 msgr2=0x7fc7b00fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 shutdown_connections 2024-04-05T03:12:41.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.415+0000 7fc7b4ffa700 1 -- 172.21.15.153:0/4273219886 wait complete. 2024-04-05T03:12:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- 172.21.15.153:0/3411261336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 msgr2=0x7f55501031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 --2- 172.21.15.153:0/3411261336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f55501031b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5548009ce0 tx=0x7f5548004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- 172.21.15.153:0/3411261336 shutdown_connections 2024-04-05T03:12:41.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 --2- 172.21.15.153:0/3411261336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f55500fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 --2- 172.21.15.153:0/3411261336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f55501031b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- 172.21.15.153:0/3411261336 >> 172.21.15.153:0/3411261336 conn(0x7f55500f8950 msgr2=0x7f55500fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- 172.21.15.153:0/3411261336 shutdown_connections 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- 172.21.15.153:0/3411261336 wait complete. 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 Processor -- start 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- start start 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f5550114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55501146f0 con 0x7f5550102de0 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.516+0000 7f5558482700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555010dc40 con 0x7f5550103780 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55052/0 (socket says 172.21.15.153:55052) 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 -- 172.21.15.153:0/646961650 learned_addr learned my addr 172.21.15.153:0/646961650 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5555a1d700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f5550114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 -- 172.21.15.153:0/646961650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 msgr2=0x7f5550114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f5550114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 -- 172.21.15.153:0/646961650 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548009940 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5555a1d700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f5550114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f555621e700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f554800be90 tx=0x7f5548004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554800e040 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555010de60 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5548008ea0 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554801c810 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.517+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555010e420 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.518+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55501000f0 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.519+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5548012c00 con 0x7f5550102de0 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.519+0000 7f5546ffd700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 0x7f553c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.519+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f554808f190 con 0x7f5550102de0 2024-04-05T03:12:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.519+0000 7f5555a1d700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 0x7f553c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.520+0000 7f5555a1d700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 0x7f553c070150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f555010f370 tx=0x7f5540008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.524+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f554805c240 con 0x7f5550102de0 2024-04-05T03:12:41.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.765+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5550109f50 con 0x7f5550102de0 2024-04-05T03:12:41.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:41 smithi153 ceph-mon[106102]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:41.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:41 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4273219886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:41.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.766+0000 7f5546ffd700 1 -- 172.21.15.153:0/646961650 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f5548004e10 con 0x7f5550102de0 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:12:41.766 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:12:41.767 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:12:41.768 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 msgr2=0x7f553c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 0x7f553c070150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f555010f370 tx=0x7f5540008040 comp rx=0 tx=0).stop 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 msgr2=0x7f5550113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f554800be90 tx=0x7f5548004d10 comp rx=0 tx=0).stop 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 shutdown_connections 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f553c06dca0 0x7f553c070150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5550102de0 0x7f5550113ad0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 --2- 172.21.15.153:0/646961650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5550103780 0x7f5550114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 >> 172.21.15.153:0/646961650 conn(0x7f55500f8950 msgr2=0x7f5550107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 shutdown_connections 2024-04-05T03:12:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.769+0000 7f5558482700 1 -- 172.21.15.153:0/646961650 wait complete. 2024-04-05T03:12:41.771 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:12:41.862 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:41 smithi203 ceph-mon[95988]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:41.863 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:41 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4273219886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.870+0000 7fe505375700 1 -- 172.21.15.153:0/226908188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500101cf0 msgr2=0x7fe5001020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.870+0000 7fe505375700 1 --2- 172.21.15.153:0/226908188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500101cf0 0x7fe5001020c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4004940 tx=0x7fe4f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- 172.21.15.153:0/226908188 shutdown_connections 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 --2- 172.21.15.153:0/226908188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500102690 0x7fe5000fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 --2- 172.21.15.153:0/226908188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500101cf0 0x7fe5001020c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- 172.21.15.153:0/226908188 >> 172.21.15.153:0/226908188 conn(0x7fe5000f7860 msgr2=0x7fe5000f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- 172.21.15.153:0/226908188 shutdown_connections 2024-04-05T03:12:41.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- 172.21.15.153:0/226908188 wait complete. 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 Processor -- start 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- start start 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500101cf0 0x7fe50010eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50010f6b0 con 0x7fe500102690 2024-04-05T03:12:41.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.871+0000 7fe505375700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50010f7f0 con 0x7fe500101cf0 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55068/0 (socket says 172.21.15.153:55068) 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 -- 172.21.15.153:0/4131665484 learned_addr learned my addr 172.21.15.153:0/4131665484 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4feffd700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500101cf0 0x7fe50010eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 -- 172.21.15.153:0/4131665484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500101cf0 msgr2=0x7fe50010eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500101cf0 0x7fe50010eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 -- 172.21.15.153:0/4131665484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4005040 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4fe7fc700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0002a50 tx=0x7fe4f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f0007cc0 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4f000f040 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe500117b90 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4f000d9c0 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.872+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5001180f0 con 0x7fe500102690 2024-04-05T03:12:41.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.873+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5000fd440 con 0x7fe500102690 2024-04-05T03:12:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.874+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe4f0009730 con 0x7fe500102690 2024-04-05T03:12:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.874+0000 7fe4e77fe700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 0x7fe4e8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.874+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4f008d2e0 con 0x7fe500102690 2024-04-05T03:12:41.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.874+0000 7fe4feffd700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 0x7fe4e8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:41.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.875+0000 7fe4feffd700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 0x7fe4e8070150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4004f90 tx=0x7fe4f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:41.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:41.879+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe4f0056860 con 0x7fe500102690 2024-04-05T03:12:42.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.095+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe500108e60 con 0x7fe4e806dca0 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.096+0000 7fe4e77fe700 1 -- 172.21.15.153:0/4131665484 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe500108e60 con 0x7fe4e806dca0 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:12:42.096 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:12:42.097 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:12:42.097 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:12:42.097 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:12:42.097 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 msgr2=0x7fe4e8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 0x7fe4e8070150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4004f90 tx=0x7fe4f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 msgr2=0x7fe50010f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0002a50 tx=0x7fe4f0004600 comp rx=0 tx=0).stop 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 shutdown_connections 2024-04-05T03:12:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4e806dca0 0x7fe4e8070150 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe500101cf0 0x7fe50010eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 --2- 172.21.15.153:0/4131665484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe500102690 0x7fe50010f060 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 >> 172.21.15.153:0/4131665484 conn(0x7fe5000f7860 msgr2=0x7fe5000f88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 shutdown_connections 2024-04-05T03:12:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.099+0000 7fe505375700 1 -- 172.21.15.153:0/4131665484 wait complete. 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 -- 172.21.15.153:0/1782485850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 msgr2=0x7fa25c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 --2- 172.21.15.153:0/1782485850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c1031e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa258004940 tx=0x7fa258009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 -- 172.21.15.153:0/1782485850 shutdown_connections 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 --2- 172.21.15.153:0/1782485850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 0x7fa25c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 --2- 172.21.15.153:0/1782485850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c1031e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 -- 172.21.15.153:0/1782485850 >> 172.21.15.153:0/1782485850 conn(0x7fa25c0f89a0 msgr2=0x7fa25c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 -- 172.21.15.153:0/1782485850 shutdown_connections 2024-04-05T03:12:42.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.200+0000 7fa26378e700 1 -- 172.21.15.153:0/1782485850 wait complete. 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 Processor -- start 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 -- start start 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 0x7fa25c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c1146b0 con 0x7fa25c102e10 2024-04-05T03:12:42.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26378e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c10daa0 con 0x7fa25c1037b0 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26152a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26152a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55098/0 (socket says 172.21.15.153:55098) 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.201+0000 7fa26152a700 1 -- 172.21.15.153:0/1424335361 learned_addr learned my addr 172.21.15.153:0/1424335361 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa260d29700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 0x7fa25c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26152a700 1 -- 172.21.15.153:0/1424335361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 msgr2=0x7fa25c114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26152a700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 0x7fa25c114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26152a700 1 -- 172.21.15.153:0/1424335361 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258005040 con 0x7fa25c102e10 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26152a700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa258009b30 tx=0x7fa258009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa258021070 con 0x7fa25c102e10 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa258013c00 con 0x7fa25c102e10 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c10dd20 con 0x7fa25c102e10 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25800e070 con 0x7fa25c102e10 2024-04-05T03:12:42.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.202+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c10e200 con 0x7fa25c102e10 2024-04-05T03:12:42.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.203+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c100120 con 0x7fa25c102e10 2024-04-05T03:12:42.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.204+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa25801cad0 con 0x7fa25c102e10 2024-04-05T03:12:42.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.204+0000 7fa251ffb700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 0x7fa248070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:12:42.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.204+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa25808e9d0 con 0x7fa25c102e10 2024-04-05T03:12:42.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.204+0000 7fa260d29700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 0x7fa248070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:12:42.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.205+0000 7fa260d29700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 0x7fa248070150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa25c10ec60 tx=0x7fa24c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:12:42.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.209+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa25805bb00 con 0x7fa25c102e10 2024-04-05T03:12:42.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.490+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa25c10f170 con 0x7fa25c102e10 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.490+0000 7fa251ffb700 1 -- 172.21.15.153:0/1424335361 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa258004490 con 0x7fa25c102e10 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:12:42.490 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 msgr2=0x7fa248070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 0x7fa248070150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa25c10ec60 tx=0x7fa24c00b040 comp rx=0 tx=0).stop 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 msgr2=0x7fa25c113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa258009b30 tx=0x7fa258009e30 comp rx=0 tx=0).stop 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 shutdown_connections 2024-04-05T03:12:42.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa24806dca0 0x7fa248070150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa25c102e10 0x7fa25c113b20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 --2- 172.21.15.153:0/1424335361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa25c1037b0 0x7fa25c114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:12:42.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 >> 172.21.15.153:0/1424335361 conn(0x7fa25c0f89a0 msgr2=0x7fa25c107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:12:42.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 shutdown_connections 2024-04-05T03:12:42.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:12:42.493+0000 7fa26378e700 1 -- 172.21.15.153:0/1424335361 wait complete. 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: pgmap v211: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/646961650' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:42.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:42 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1424335361' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:12:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: pgmap v211: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/646961650' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:12:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:42 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1424335361' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:12:44.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:43 smithi203 ceph-mon[95988]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:44.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:43 smithi153 ceph-mon[106102]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:12:44.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:44 smithi153 ceph-mon[106102]: pgmap v212: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:45.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:44 smithi203 ceph-mon[95988]: pgmap v212: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:46 smithi203 ceph-mon[95988]: pgmap v213: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:46 smithi153 ceph-mon[106102]: pgmap v213: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:48 smithi203 ceph-mon[95988]: pgmap v214: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:48 smithi153 ceph-mon[106102]: pgmap v214: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:50.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:50 smithi153 ceph-mon[106102]: pgmap v215: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:51.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:50 smithi203 ceph-mon[95988]: pgmap v215: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:52 smithi203 ceph-mon[95988]: pgmap v216: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:52 smithi153 ceph-mon[106102]: pgmap v216: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:54 smithi203 ceph-mon[95988]: pgmap v217: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:54 smithi153 ceph-mon[106102]: pgmap v217: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:56 smithi203 ceph-mon[95988]: pgmap v218: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:57.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:56 smithi153 ceph-mon[106102]: pgmap v218: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:12:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:12:58 smithi203 ceph-mon[95988]: pgmap v219: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:12:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:12:58 smithi153 ceph-mon[106102]: pgmap v219: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:00.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:00 smithi153 ceph-mon[106102]: pgmap v220: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:00 smithi203 ceph-mon[95988]: pgmap v220: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:03.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:02 smithi203 ceph-mon[95988]: pgmap v221: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:03.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:02 smithi153 ceph-mon[106102]: pgmap v221: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:05.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:04 smithi203 ceph-mon[95988]: pgmap v222: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:04 smithi153 ceph-mon[106102]: pgmap v222: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:06 smithi203 ceph-mon[95988]: pgmap v223: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:06 smithi153 ceph-mon[106102]: pgmap v223: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:08 smithi203 ceph-mon[95988]: pgmap v224: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:08 smithi153 ceph-mon[106102]: pgmap v224: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:10.885 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:10 smithi153 ceph-mon[106102]: pgmap v225: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:10 smithi203 ceph-mon[95988]: pgmap v225: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 -- 172.21.15.153:0/2696384319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 msgr2=0x7f4fcc1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/2696384319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc1020f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8004940 tx=0x7f4fc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 -- 172.21.15.153:0/2696384319 shutdown_connections 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/2696384319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 0x7f4fcc0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/2696384319 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc1020f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 -- 172.21.15.153:0/2696384319 >> 172.21.15.153:0/2696384319 conn(0x7f4fcc0f78d0 msgr2=0x7f4fcc0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 -- 172.21.15.153:0/2696384319 shutdown_connections 2024-04-05T03:13:12.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.613+0000 7f4fd41d6700 1 -- 172.21.15.153:0/2696384319 wait complete. 2024-04-05T03:13:12.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 Processor -- start 2024-04-05T03:13:12.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 -- start start 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 0x7f4fcc1030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc103670 con 0x7f4fcc101d20 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd41d6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc1037e0 con 0x7f4fcc1026c0 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd1f72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd1f72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59942/0 (socket says 172.21.15.153:59942) 2024-04-05T03:13:12.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.614+0000 7f4fd1f72700 1 -- 172.21.15.153:0/1686956692 learned_addr learned my addr 172.21.15.153:0/1686956692 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd1771700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 0x7f4fcc1030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd1f72700 1 -- 172.21.15.153:0/1686956692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 msgr2=0x7f4fcc1030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd1f72700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 0x7f4fcc1030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd1f72700 1 -- 172.21.15.153:0/1686956692 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc8005040 con 0x7f4fcc101d20 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd1f72700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009150 tx=0x7f4fc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc800e040 con 0x7f4fcc101d20 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc119f40 con 0x7f4fcc101d20 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fc8013420 con 0x7f4fcc101d20 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fc801c630 con 0x7f4fcc101d20 2024-04-05T03:13:12.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.615+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc11a350 con 0x7f4fcc101d20 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.616+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc0fd470 con 0x7f4fcc101d20 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.617+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4fc800baf0 con 0x7f4fcc101d20 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.617+0000 7f4fbe7fc700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 0x7f4fb8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.617+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4fc808e810 con 0x7f4fcc101d20 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.617+0000 7f4fd1771700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 0x7f4fb8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.618+0000 7f4fd1771700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 0x7f4fb8070160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0007fd0 tx=0x7f4fc0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:12.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.622+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4fc8057e10 con 0x7f4fcc101d20 2024-04-05T03:13:12.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.844+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fcc04bf70 con 0x7f4fb806dcb0 2024-04-05T03:13:12.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.846+0000 7f4fbe7fc700 1 -- 172.21.15.153:0/1686956692 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4fc0004db0 con 0x7f4fb806dcb0 2024-04-05T03:13:12.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 msgr2=0x7f4fb8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 0x7f4fb8070160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0007fd0 tx=0x7f4fc0009040 comp rx=0 tx=0).stop 2024-04-05T03:13:12.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 msgr2=0x7f4fcc102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009150 tx=0x7f4fc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 shutdown_connections 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4fb806dcb0 0x7f4fb8070160 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4fcc101d20 0x7f4fcc102b80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 --2- 172.21.15.153:0/1686956692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fcc1026c0 0x7f4fcc1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 >> 172.21.15.153:0/1686956692 conn(0x7f4fcc0f78d0 msgr2=0x7f4fcc0f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 shutdown_connections 2024-04-05T03:13:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.848+0000 7f4fd41d6700 1 -- 172.21.15.153:0/1686956692 wait complete. 2024-04-05T03:13:12.858 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:13:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- 172.21.15.153:0/2196568336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47541037a0 msgr2=0x7f47540fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 --2- 172.21.15.153:0/2196568336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47541037a0 0x7f47540fce10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4748004940 tx=0x7f4748009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- 172.21.15.153:0/2196568336 shutdown_connections 2024-04-05T03:13:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 --2- 172.21.15.153:0/2196568336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47541037a0 0x7f47540fce10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 --2- 172.21.15.153:0/2196568336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4754102e00 0x7f47541031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- 172.21.15.153:0/2196568336 >> 172.21.15.153:0/2196568336 conn(0x7f47540f89b0 msgr2=0x7f47540fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:12.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- 172.21.15.153:0/2196568336 shutdown_connections 2024-04-05T03:13:12.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- 172.21.15.153:0/2196568336 wait complete. 2024-04-05T03:13:12.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:12 smithi153 ceph-mon[106102]: pgmap v226: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:12.952 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 Processor -- start 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.952+0000 7f475a958700 1 -- start start 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f475a958700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f475a958700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47541037a0 0x7f4754070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f475a958700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540721f0 con 0x7f4754102e00 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f475a958700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4754070d40 con 0x7f47541037a0 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59970/0 (socket says 172.21.15.153:59970) 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 -- 172.21.15.153:0/3726909260 learned_addr learned my addr 172.21.15.153:0/3726909260 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:12.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f47537fe700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47541037a0 0x7f4754070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 -- 172.21.15.153:0/3726909260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47541037a0 msgr2=0x7f4754070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47541037a0 0x7f4754070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 -- 172.21.15.153:0/3726909260 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748005040 con 0x7f4754102e00 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.953+0000 7f4753fff700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4744002a50 tx=0x7f47440046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.954+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4744007cc0 con 0x7f4754102e00 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.954+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4754071020 con 0x7f4754102e00 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.954+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f474400f040 con 0x7f4754102e00 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.954+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474400da30 con 0x7f4754102e00 2024-04-05T03:13:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.954+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475411a100 con 0x7f4754102e00 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.955+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f474400db90 con 0x7f4754102e00 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.955+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47541052d0 con 0x7f4754102e00 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.956+0000 7f4750ff9700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 0x7f473c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.956+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f474408d230 con 0x7f4754102e00 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.956+0000 7f47537fe700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 0x7f473c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:12.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.956+0000 7f47537fe700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 0x7f473c070150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f47540fc3e0 tx=0x7f4748009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:12.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:12.961+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4744056830 con 0x7f4754102e00 2024-04-05T03:13:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:12 smithi203 ceph-mon[95988]: pgmap v226: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:13.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.170+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4754109ee0 con 0x7f473c06dca0 2024-04-05T03:13:13.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.172+0000 7f4750ff9700 1 -- 172.21.15.153:0/3726909260 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4754109ee0 con 0x7f473c06dca0 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 msgr2=0x7f473c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 0x7f473c070150 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f47540fc3e0 tx=0x7f4748009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 msgr2=0x7f47540702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4744002a50 tx=0x7f47440046b0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 shutdown_connections 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f473c06dca0 0x7f473c070150 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4754102e00 0x7f47540702c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 --2- 172.21.15.153:0/3726909260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47541037a0 0x7f4754070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 >> 172.21.15.153:0/3726909260 conn(0x7f47540f89b0 msgr2=0x7f47541021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 shutdown_connections 2024-04-05T03:13:13.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.174+0000 7f475a958700 1 -- 172.21.15.153:0/3726909260 wait complete. 2024-04-05T03:13:13.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 -- 172.21.15.153:0/474330992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 msgr2=0x7f4ca81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 --2- 172.21.15.153:0/474330992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca81020f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4004940 tx=0x7f4ca4009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 -- 172.21.15.153:0/474330992 shutdown_connections 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 --2- 172.21.15.153:0/474330992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 --2- 172.21.15.153:0/474330992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca81020f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.275+0000 7f4caf919700 1 -- 172.21.15.153:0/474330992 >> 172.21.15.153:0/474330992 conn(0x7f4ca80f7870 msgr2=0x7f4ca80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 -- 172.21.15.153:0/474330992 shutdown_connections 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 -- 172.21.15.153:0/474330992 wait complete. 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 Processor -- start 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 -- start start 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca8113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8114520 con 0x7f4ca8101d20 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4caf919700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca810db20 con 0x7f4ca81026c0 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4cad6b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4cad6b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59982/0 (socket says 172.21.15.153:59982) 2024-04-05T03:13:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.276+0000 7f4cad6b5700 1 -- 172.21.15.153:0/1033361637 learned_addr learned my addr 172.21.15.153:0/1033361637 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4caceb4700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca8113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4cad6b5700 1 -- 172.21.15.153:0/1033361637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 msgr2=0x7f4ca8113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4cad6b5700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca8113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4cad6b5700 1 -- 172.21.15.153:0/1033361637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4005040 con 0x7f4ca8101d20 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4caceb4700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca8113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4cad6b5700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400b3b0 tx=0x7f4ca4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca400e040 con 0x7f4ca8101d20 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ca400bbc0 con 0x7f4ca8101d20 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca810dda0 con 0x7f4ca8101d20 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca401c3f0 con 0x7f4ca8101d20 2024-04-05T03:13:13.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.277+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca810e360 con 0x7f4ca8101d20 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.278+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca80fd470 con 0x7f4ca8101d20 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.279+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4ca400bd30 con 0x7f4ca8101d20 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.279+0000 7f4c9dffb700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 0x7f4c94070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.279+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4ca408e740 con 0x7f4ca8101d20 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.279+0000 7f4caceb4700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 0x7f4c94070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.280+0000 7f4caceb4700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 0x7f4c94070150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007fd0 tx=0x7f4c98009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.284+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4ca405b870 con 0x7f4ca8101d20 2024-04-05T03:13:13.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.487+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ca8108e20 con 0x7f4c9406dca0 2024-04-05T03:13:13.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.497+0000 7f4c9dffb700 1 -- 172.21.15.153:0/1033361637 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f4ca8108e20 con 0x7f4c9406dca0 2024-04-05T03:13:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:13:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (7m) 80s ago 18m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:13:13.496 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (5m) 80s ago 20m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (5m) 49s ago 17m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (7m) 80s ago 18m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (84s) 80s ago 13m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (101s) 80s ago 13m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (52s) 49s ago 13m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (68s) 49s ago 13m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (9m) 80s ago 21m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (8m) 49s ago 17m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (6m) 80s ago 21m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (6m) 49s ago 17m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 80s ago 19m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (8m) 49s ago 17m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (4m) 80s ago 17m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 80s ago 16m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:13:13.497 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (3m) 80s ago 16m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:13:13.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (3m) 49s ago 15m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:13:13.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (2m) 49s ago 15m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:13:13.498 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (2m) 49s ago 14m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:13:13.498 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (8m) 49s ago 17m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 msgr2=0x7f4c94070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 0x7f4c94070150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007fd0 tx=0x7f4c98009040 comp rx=0 tx=0).stop 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 msgr2=0x7f4ca8113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400b3b0 tx=0x7f4ca4009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 shutdown_connections 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c9406dca0 0x7f4c94070150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca8101d20 0x7f4ca8113990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 --2- 172.21.15.153:0/1033361637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ca81026c0 0x7f4ca8113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 >> 172.21.15.153:0/1033361637 conn(0x7f4ca80f7870 msgr2=0x7f4ca8066200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 shutdown_connections 2024-04-05T03:13:13.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.499+0000 7f4caf919700 1 -- 172.21.15.153:0/1033361637 wait complete. 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 -- 172.21.15.153:0/3413100412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f74102f70 msgr2=0x7f8f74103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 --2- 172.21.15.153:0/3413100412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f74102f70 0x7f8f74103340 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8f70004940 tx=0x7f8f70009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 -- 172.21.15.153:0/3413100412 shutdown_connections 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 --2- 172.21.15.153:0/3413100412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 --2- 172.21.15.153:0/3413100412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f74102f70 0x7f8f74103340 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.603+0000 7f8f7b681700 1 -- 172.21.15.153:0/3413100412 >> 172.21.15.153:0/3413100412 conn(0x7f8f74072310 msgr2=0x7f8f74072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 -- 172.21.15.153:0/3413100412 shutdown_connections 2024-04-05T03:13:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 -- 172.21.15.153:0/3413100412 wait complete. 2024-04-05T03:13:13.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 Processor -- start 2024-04-05T03:13:13.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 -- start start 2024-04-05T03:13:13.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f74102f70 0x7f8f740fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740fe530 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7b681700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740fd010 con 0x7f8f74102f70 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7941d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.604+0000 7f8f7941d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60002/0 (socket says 172.21.15.153:60002) 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7941d700 1 -- 172.21.15.153:0/2537471215 learned_addr learned my addr 172.21.15.153:0/2537471215 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f78c1c700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f74102f70 0x7f8f740fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7941d700 1 -- 172.21.15.153:0/2537471215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f74102f70 msgr2=0x7f8f740fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7941d700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f74102f70 0x7f8f740fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7941d700 1 -- 172.21.15.153:0/2537471215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f70005040 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7941d700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009100 tx=0x7f8f70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f7000e040 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f70013420 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f7001c630 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f740fd260 con 0x7f8f740fb3b0 2024-04-05T03:13:13.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.605+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7411a060 con 0x7f8f740fb3b0 2024-04-05T03:13:13.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.606+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f741053b0 con 0x7f8f740fb3b0 2024-04-05T03:13:13.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.607+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8f70013590 con 0x7f8f740fb3b0 2024-04-05T03:13:13.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.607+0000 7f8f69ffb700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 0x7f8f60070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.607+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8f7008df60 con 0x7f8f740fb3b0 2024-04-05T03:13:13.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.607+0000 7f8f78c1c700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 0x7f8f60070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.608+0000 7f8f78c1c700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 0x7f8f60070150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f740fdcc0 tx=0x7f8f64009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.612+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8f700c2280 con 0x7f8f740fb3b0 2024-04-05T03:13:13.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.886+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8f7410a090 con 0x7f8f740fb3b0 2024-04-05T03:13:13.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.887+0000 7f8f69ffb700 1 -- 172.21.15.153:0/2537471215 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8f70057600 con 0x7f8f740fb3b0 2024-04-05T03:13:13.886 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:13:13.886 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:13:13.887 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.889+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 msgr2=0x7f8f60070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.889+0000 7f8f7b681700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 0x7f8f60070150 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8f740fdcc0 tx=0x7f8f64009040 comp rx=0 tx=0).stop 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.889+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 msgr2=0x7f8f740fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.889+0000 7f8f7b681700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009100 tx=0x7f8f70009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 shutdown_connections 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8f6006dca0 0x7f8f60070150 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f740fb3b0 0x7f8f740fc590 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 --2- 172.21.15.153:0/2537471215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f74102f70 0x7f8f740fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 >> 172.21.15.153:0/2537471215 conn(0x7f8f74072310 msgr2=0x7f8f740f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 shutdown_connections 2024-04-05T03:13:13.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.890+0000 7f8f7b681700 1 -- 172.21.15.153:0/2537471215 wait complete. 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/3169909255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 msgr2=0x7f8c04102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/3169909255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c04102100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4004940 tx=0x7f8bf4009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/3169909255 shutdown_connections 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/3169909255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 0x7f8c040fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/3169909255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c04102100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/3169909255 >> 172.21.15.153:0/3169909255 conn(0x7f8c040f78c0 msgr2=0x7f8c040f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/3169909255 shutdown_connections 2024-04-05T03:13:13.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.991+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/3169909255 wait complete. 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 Processor -- start 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 -- start start 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 0x7f8c0410f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0410f5b0 con 0x7f8c04101d30 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c0c0bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0410f6f0 con 0x7f8c041026d0 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60024/0 (socket says 172.21.15.153:60024) 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 -- 172.21.15.153:0/1902732524 learned_addr learned my addr 172.21.15.153:0/1902732524 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09657700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 0x7f8c0410f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 -- 172.21.15.153:0/1902732524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 msgr2=0x7f8c0410f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 0x7f8c0410f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 -- 172.21.15.153:0/1902732524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4005040 con 0x7f8c04101d30 2024-04-05T03:13:13.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.992+0000 7f8c09e58700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400b3b0 tx=0x7f8bf4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.993+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf400e040 con 0x7f8c04101d30 2024-04-05T03:13:13.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.993+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8bf400bbc0 con 0x7f8c04101d30 2024-04-05T03:13:13.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.993+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bf401c3f0 con 0x7f8c04101d30 2024-04-05T03:13:13.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.993+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0410f970 con 0x7f8c04101d30 2024-04-05T03:13:13.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.993+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c04117ef0 con 0x7f8c04101d30 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.994+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c040fd480 con 0x7f8c04101d30 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.994+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8bf400bd30 con 0x7f8c04101d30 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.994+0000 7f8bfa7fc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 0x7f8bf0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.995+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8bf408e710 con 0x7f8c04101d30 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.995+0000 7f8c09657700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 0x7f8bf0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:13.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.995+0000 7f8c09657700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 0x7f8bf0070150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8c00007fd0 tx=0x7f8c00009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:13.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:13.999+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8bf4057d10 con 0x7f8c04101d30 2024-04-05T03:13:14.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.235+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8c04066870 con 0x7f8c04101d30 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.235+0000 7f8bfa7fc700 1 -- 172.21.15.153:0/1902732524 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f8bf405b3d0 con 0x7f8c04101d30 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:13:14.235 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:13:14.236 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:14.237 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:14.238 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:14.238 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:14.238 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.237+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 msgr2=0x7f8bf0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.237+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 0x7f8bf0070150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8c00007fd0 tx=0x7f8c00009040 comp rx=0 tx=0).stop 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.237+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 msgr2=0x7f8c0410eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.237+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400b3b0 tx=0x7f8bf4009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 shutdown_connections 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8bf006dca0 0x7f8bf0070150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c04101d30 0x7f8c0410eb30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 --2- 172.21.15.153:0/1902732524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c041026d0 0x7f8c0410f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 >> 172.21.15.153:0/1902732524 conn(0x7f8c040f78c0 msgr2=0x7f8c040f8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 shutdown_connections 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.238+0000 7f8c0c0bc700 1 -- 172.21.15.153:0/1902732524 wait complete. 2024-04-05T03:13:14.239 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- 172.21.15.153:0/881431959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 msgr2=0x7fe3240fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 --2- 172.21.15.153:0/881431959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe3240fbd30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe318004940 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- 172.21.15.153:0/881431959 shutdown_connections 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 --2- 172.21.15.153:0/881431959 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe3240fbd30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 --2- 172.21.15.153:0/881431959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 0x7fe3241020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- 172.21.15.153:0/881431959 >> 172.21.15.153:0/881431959 conn(0x7fe3240f7870 msgr2=0x7fe3240f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- 172.21.15.153:0/881431959 shutdown_connections 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- 172.21.15.153:0/881431959 wait complete. 2024-04-05T03:13:14.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 Processor -- start 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.340+0000 7fe328fa2700 1 -- start start 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 0x7fe32410eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32410f5e0 con 0x7fe3241026c0 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32410f720 con 0x7fe324101d20 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60046/0 (socket says 172.21.15.153:60046) 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 -- 172.21.15.153:0/83997718 learned_addr learned my addr 172.21.15.153:0/83997718 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe32259c700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 0x7fe32410eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 -- 172.21.15.153:0/83997718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 msgr2=0x7fe32410eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 0x7fe32410eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 -- 172.21.15.153:0/83997718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318005040 con 0x7fe3241026c0 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe321d9b700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe318009100 tx=0x7fe318009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:14.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31800e040 con 0x7fe3241026c0 2024-04-05T03:13:14.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe318013420 con 0x7fe3241026c0 2024-04-05T03:13:14.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32410f9a0 con 0x7fe3241026c0 2024-04-05T03:13:14.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe31801c630 con 0x7fe3241026c0 2024-04-05T03:13:14.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.341+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324117ed0 con 0x7fe3241026c0 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.342+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3240fd470 con 0x7fe3241026c0 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.343+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe318013a00 con 0x7fe3241026c0 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.343+0000 7fe312ffd700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 0x7fe30c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.343+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe31808e660 con 0x7fe3241026c0 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.344+0000 7fe32259c700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 0x7fe30c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.344+0000 7fe32259c700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 0x7fe30c070150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe3140041d0 tx=0x7fe3140043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:14.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.348+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe318057c60 con 0x7fe3241026c0 2024-04-05T03:13:14.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.559+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3240f8f70 con 0x7fe30c06dca0 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.561+0000 7fe312ffd700 1 -- 172.21.15.153:0/83997718 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3240f8f70 con 0x7fe30c06dca0 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:13:14.560 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:13:14.561 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:13:14.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 msgr2=0x7fe30c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 0x7fe30c070150 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe3140041d0 tx=0x7fe3140043c0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 msgr2=0x7fe32410f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe318009100 tx=0x7fe318009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 shutdown_connections 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe30c06dca0 0x7fe30c070150 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe324101d20 0x7fe32410eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 --2- 172.21.15.153:0/83997718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3241026c0 0x7fe32410f0a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 >> 172.21.15.153:0/83997718 conn(0x7fe3240f7870 msgr2=0x7fe3240f8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 shutdown_connections 2024-04-05T03:13:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.563+0000 7fe328fa2700 1 -- 172.21.15.153:0/83997718 wait complete. 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 -- 172.21.15.153:0/1753314188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 msgr2=0x7f57b0102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 --2- 172.21.15.153:0/1753314188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b0102150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57a8009ce0 tx=0x7f57a8004d10 comp rx=0 tx=0).stop 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 -- 172.21.15.153:0/1753314188 shutdown_connections 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 --2- 172.21.15.153:0/1753314188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 0x7f57b00fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 --2- 172.21.15.153:0/1753314188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b0102150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 -- 172.21.15.153:0/1753314188 >> 172.21.15.153:0/1753314188 conn(0x7f57b00f78f0 msgr2=0x7f57b00f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 -- 172.21.15.153:0/1753314188 shutdown_connections 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.665+0000 7f57b5574700 1 -- 172.21.15.153:0/1753314188 wait complete. 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 Processor -- start 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 -- start start 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 0x7f57b010f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b010f650 con 0x7f57b0101d80 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57b5574700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b010f790 con 0x7f57b0102720 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60074/0 (socket says 172.21.15.153:60074) 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 -- 172.21.15.153:0/1285877196 learned_addr learned my addr 172.21.15.153:0/1285877196 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57ae7fc700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 0x7f57b010f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 -- 172.21.15.153:0/1285877196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 msgr2=0x7f57b010f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 0x7f57b010f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.666+0000 7f57aeffd700 1 -- 172.21.15.153:0/1285877196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a8009940 con 0x7f57b0101d80 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57aeffd700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57a8003400 tx=0x7f57a8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a800e040 con 0x7f57b0101d80 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b010fa10 con 0x7f57b0101d80 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57a8004610 con 0x7f57b0101d80 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a801c640 con 0x7f57b0101d80 2024-04-05T03:13:14.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.667+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b0117fe0 con 0x7f57b0101d80 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.668+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b00ff090 con 0x7f57b0101d80 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.668+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f57a8012a70 con 0x7f57b0101d80 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.669+0000 7f57a77fe700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 0x7f579c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.669+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f57a808ef80 con 0x7f57b0101d80 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.669+0000 7f57ae7fc700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 0x7f579c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:14.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.669+0000 7f57ae7fc700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 0x7f579c070150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f57b0066a60 tx=0x7f579800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:14.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.673+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f57a8058500 con 0x7f57b0101d80 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: pgmap v227: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2537471215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:13:14.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:14 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1902732524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.948+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f57b0108f10 con 0x7f57b0101d80 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.948+0000 7f57a77fe700 1 -- 172.21.15.153:0/1285877196 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f57a8004e10 con 0x7f57b0101d80 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:13:14.948 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 msgr2=0x7f579c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 0x7f579c070150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f57b0066a60 tx=0x7f579800b040 comp rx=0 tx=0).stop 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 msgr2=0x7f57b010ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57a8003400 tx=0x7f57a8004d10 comp rx=0 tx=0).stop 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 shutdown_connections 2024-04-05T03:13:14.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f579c06dca0 0x7f579c070150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57b0101d80 0x7f57b010ebd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 --2- 172.21.15.153:0/1285877196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b0102720 0x7f57b010f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:14.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 >> 172.21.15.153:0/1285877196 conn(0x7f57b00f78f0 msgr2=0x7f57b0106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:14.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 shutdown_connections 2024-04-05T03:13:14.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:14.951+0000 7f57b5574700 1 -- 172.21.15.153:0/1285877196 wait complete. 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: pgmap v227: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2537471215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:13:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:14 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1902732524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:13:16.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:15 smithi203 ceph-mon[95988]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:16.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:15 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1285877196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:13:16.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:15 smithi153 ceph-mon[106102]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:16.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:15 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1285877196' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:13:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:16 smithi203 ceph-mon[95988]: pgmap v228: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:16 smithi153 ceph-mon[106102]: pgmap v228: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:18 smithi203 ceph-mon[95988]: pgmap v229: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:18 smithi153 ceph-mon[106102]: pgmap v229: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:20 smithi203 ceph-mon[95988]: pgmap v230: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:20 smithi153 ceph-mon[106102]: pgmap v230: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:22 smithi203 ceph-mon[95988]: pgmap v231: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:23.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:22 smithi153 ceph-mon[106102]: pgmap v231: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:24 smithi203 ceph-mon[95988]: pgmap v232: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:24 smithi153 ceph-mon[106102]: pgmap v232: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:25 smithi203 ceph-mon[95988]: pgmap v233: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:26.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:25 smithi153 ceph-mon[106102]: pgmap v233: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:27.796 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:27.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:28 smithi153 ceph-mon[106102]: pgmap v234: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:28 smithi203 ceph-mon[95988]: pgmap v234: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:30.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:30 smithi153 ceph-mon[106102]: pgmap v235: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:30 smithi203 ceph-mon[95988]: pgmap v235: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:32 smithi203 ceph-mon[95988]: pgmap v236: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:32 smithi153 ceph-mon[106102]: pgmap v236: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:34 smithi203 ceph-mon[95988]: pgmap v237: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:34 smithi153 ceph-mon[106102]: pgmap v237: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:36 smithi203 ceph-mon[95988]: pgmap v238: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:36 smithi153 ceph-mon[106102]: pgmap v238: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:38 smithi203 ceph-mon[95988]: pgmap v239: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:38 smithi153 ceph-mon[106102]: pgmap v239: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:40.848 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:40 smithi203 ceph-mon[95988]: pgmap v240: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:40.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:40 smithi153 ceph-mon[106102]: pgmap v240: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:42 smithi203 ceph-mon[95988]: pgmap v241: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:42 smithi153 ceph-mon[106102]: pgmap v241: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:44.874 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:44 smithi153 ceph-mon[106102]: pgmap v242: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:44 smithi203 ceph-mon[95988]: pgmap v242: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 -- 172.21.15.153:0/2784287165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 msgr2=0x7fb1941031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 --2- 172.21.15.153:0/2784287165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb1941031e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb184009ce0 tx=0x7fb184004d10 comp rx=0 tx=0).stop 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 -- 172.21.15.153:0/2784287165 shutdown_connections 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 --2- 172.21.15.153:0/2784287165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 0x7fb1940fb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 --2- 172.21.15.153:0/2784287165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb1941031e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 -- 172.21.15.153:0/2784287165 >> 172.21.15.153:0/2784287165 conn(0x7fb194072310 msgr2=0x7fb194072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 -- 172.21.15.153:0/2784287165 shutdown_connections 2024-04-05T03:13:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.066+0000 7fb199be5700 1 -- 172.21.15.153:0/2784287165 wait complete. 2024-04-05T03:13:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 Processor -- start 2024-04-05T03:13:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 -- start start 2024-04-05T03:13:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 0x7fb194113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb194114500 con 0x7fb194102e10 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.067+0000 7fb199be5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19410db50 con 0x7fb1941037b0 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49578/0 (socket says 172.21.15.153:49578) 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 -- 172.21.15.153:0/1749868755 learned_addr learned my addr 172.21.15.153:0/1749868755 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb192ffd700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 0x7fb194113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 -- 172.21.15.153:0/1749868755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 msgr2=0x7fb194113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 0x7fb194113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 -- 172.21.15.153:0/1749868755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb184009940 con 0x7fb194102e10 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb1937fe700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb184000f80 tx=0x7fb184004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb18400e040 con 0x7fb194102e10 2024-04-05T03:13:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb18401cb20 con 0x7fb194102e10 2024-04-05T03:13:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19410ddd0 con 0x7fb194102e10 2024-04-05T03:13:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb184012990 con 0x7fb194102e10 2024-04-05T03:13:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.068+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19410e310 con 0x7fb194102e10 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.069+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb194100150 con 0x7fb194102e10 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.070+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb18401cc90 con 0x7fb194102e10 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.070+0000 7fb17bfff700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 0x7fb17c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.070+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb18408f860 con 0x7fb194102e10 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.070+0000 7fb192ffd700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 0x7fb17c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.071+0000 7fb192ffd700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 0x7fb17c070150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb1941078a0 tx=0x7fb18800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.075+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb184058de0 con 0x7fb194102e10 2024-04-05T03:13:45.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.290+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb194109f40 con 0x7fb17c06dca0 2024-04-05T03:13:45.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.292+0000 7fb17bfff700 1 -- 172.21.15.153:0/1749868755 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb194109f40 con 0x7fb17c06dca0 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 msgr2=0x7fb17c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 0x7fb17c070150 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb1941078a0 tx=0x7fb18800b040 comp rx=0 tx=0).stop 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 msgr2=0x7fb194113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb184000f80 tx=0x7fb184004d10 comp rx=0 tx=0).stop 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 shutdown_connections 2024-04-05T03:13:45.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb17c06dca0 0x7fb17c070150 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb194102e10 0x7fb194113a10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 --2- 172.21.15.153:0/1749868755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1941037b0 0x7fb194113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 >> 172.21.15.153:0/1749868755 conn(0x7fb194072310 msgr2=0x7fb1940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 shutdown_connections 2024-04-05T03:13:45.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.294+0000 7fb199be5700 1 -- 172.21.15.153:0/1749868755 wait complete. 2024-04-05T03:13:45.304 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:13:45.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- 172.21.15.153:0/1204486547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 msgr2=0x7fae080fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1204486547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae080fcdf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fadfc004940 tx=0x7fadfc009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:45.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- 172.21.15.153:0/1204486547 shutdown_connections 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1204486547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae080fcdf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1204486547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae081031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- 172.21.15.153:0/1204486547 >> 172.21.15.153:0/1204486547 conn(0x7fae080f8950 msgr2=0x7fae080fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- 172.21.15.153:0/1204486547 shutdown_connections 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- 172.21.15.153:0/1204486547 wait complete. 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 Processor -- start 2024-04-05T03:13:45.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.396+0000 7fae0dc69700 1 -- start start 2024-04-05T03:13:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae0dc69700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae08113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae0dc69700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae0dc69700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae08114500 con 0x7fae08103780 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae0dc69700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0810db00 con 0x7fae08102de0 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae077fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae08113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49590/0 (socket says 172.21.15.153:49590) 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 -- 172.21.15.153:0/1390477245 learned_addr learned my addr 172.21.15.153:0/1390477245 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 -- 172.21.15.153:0/1390477245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 msgr2=0x7fae08113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae08113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 -- 172.21.15.153:0/1390477245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc005040 con 0x7fae08103780 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae077fe700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae08113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae06ffd700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009100 tx=0x7fadfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadfc00e040 con 0x7fae08103780 2024-04-05T03:13:45.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0810dd80 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadfc009af0 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.397+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadfc01c3f0 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.398+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0810e340 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.398+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae080fe530 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.399+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fadfc00baf0 con 0x7fae08103780 2024-04-05T03:13:45.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.399+0000 7fadeffff700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 0x7fadf0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.400+0000 7fae077fe700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 0x7fadf0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.400+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fadfc08e7c0 con 0x7fae08103780 2024-04-05T03:13:45.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.400+0000 7fae077fe700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 0x7fadf0070150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fadf8004fd0 tx=0x7fadf800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.404+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fadfc05b8f0 con 0x7fae08103780 2024-04-05T03:13:45.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.620+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae08109ec0 con 0x7fadf006dca0 2024-04-05T03:13:45.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.621+0000 7fadeffff700 1 -- 172.21.15.153:0/1390477245 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fae08109ec0 con 0x7fadf006dca0 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 msgr2=0x7fadf0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 0x7fadf0070150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fadf8004fd0 tx=0x7fadf800b040 comp rx=0 tx=0).stop 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 msgr2=0x7fae08113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009100 tx=0x7fadfc009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 shutdown_connections 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fadf006dca0 0x7fadf0070150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae08102de0 0x7fae08113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 --2- 172.21.15.153:0/1390477245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae08103780 0x7fae08113eb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.623+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 >> 172.21.15.153:0/1390477245 conn(0x7fae080f8950 msgr2=0x7fae080f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.624+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 shutdown_connections 2024-04-05T03:13:45.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.624+0000 7fae0dc69700 1 -- 172.21.15.153:0/1390477245 wait complete. 2024-04-05T03:13:45.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- 172.21.15.153:0/889361391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 msgr2=0x7f408c0fc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 --2- 172.21.15.153:0/889361391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c0fc930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4074004940 tx=0x7f4074009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- 172.21.15.153:0/889361391 shutdown_connections 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 --2- 172.21.15.153:0/889361391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c0fc930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 --2- 172.21.15.153:0/889361391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- 172.21.15.153:0/889361391 >> 172.21.15.153:0/889361391 conn(0x7f408c072310 msgr2=0x7f408c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- 172.21.15.153:0/889361391 shutdown_connections 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- 172.21.15.153:0/889361391 wait complete. 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 Processor -- start 2024-04-05T03:13:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.726+0000 7f4090a8a700 1 -- start start 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c114580 con 0x7f408c0fc4c0 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10db80 con 0x7f408c101db0 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49598/0 (socket says 172.21.15.153:49598) 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 -- 172.21.15.153:0/2276972827 learned_addr learned my addr 172.21.15.153:0/2276972827 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4083fff700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 -- 172.21.15.153:0/2276972827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 msgr2=0x7f408c113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 -- 172.21.15.153:0/2276972827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4074005040 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4083fff700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f408a59c700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f407c002a50 tx=0x7f407c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407c007cc0 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c10de60 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f407c00f040 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407c009640 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.727+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c10e3a0 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.728+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c1000a0 con 0x7f408c0fc4c0 2024-04-05T03:13:45.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.729+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f407c0097c0 con 0x7f408c0fc4c0 2024-04-05T03:13:45.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.729+0000 7f4082ffd700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 0x7f4078070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:45.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.729+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f407c08d260 con 0x7f408c0fc4c0 2024-04-05T03:13:45.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.729+0000 7f4083fff700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 0x7f4078070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:45.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.730+0000 7f4083fff700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 0x7f4078070150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f407400b3b0 tx=0x7f4074009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.734+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f407c05a390 con 0x7f408c0fc4c0 2024-04-05T03:13:45.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.940+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f408c108e90 con 0x7f407806dca0 2024-04-05T03:13:45.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.951+0000 7f4082ffd700 1 -- 172.21.15.153:0/2276972827 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f408c108e90 con 0x7f407806dca0 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (8m) 113s ago 19m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (5m) 113s ago 20m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (5m) 81s ago 18m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (8m) 113s ago 19m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (117s) 113s ago 13m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (2m) 113s ago 13m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (85s) 81s ago 13m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (101s) 81s ago 13m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (9m) 113s ago 21m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (9m) 81s ago 18m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (6m) 113s ago 21m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:13:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (6m) 81s ago 18m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (8m) 113s ago 19m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (8m) 81s ago 18m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 113s ago 17m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 113s ago 17m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 113s ago 16m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (3m) 81s ago 16m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (3m) 81s ago 15m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (2m) 81s ago 15m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:13:45.952 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (8m) 81s ago 17m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 msgr2=0x7f4078070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 0x7f4078070150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f407400b3b0 tx=0x7f4074009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 msgr2=0x7f408c1139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f407c002a50 tx=0x7f407c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 shutdown_connections 2024-04-05T03:13:45.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f407806dca0 0x7f4078070150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0fc4c0 0x7f408c1139f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.953+0000 7f4090a8a700 1 --2- 172.21.15.153:0/2276972827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f408c101db0 0x7f408c113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:45.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.954+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 >> 172.21.15.153:0/2276972827 conn(0x7f408c072310 msgr2=0x7f408c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:45.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.954+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 shutdown_connections 2024-04-05T03:13:45.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:45.954+0000 7f4090a8a700 1 -- 172.21.15.153:0/2276972827 wait complete. 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.053+0000 7f060a324700 1 -- 172.21.15.153:0/3646001851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 msgr2=0x7f06040fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.053+0000 7f060a324700 1 --2- 172.21.15.153:0/3646001851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06040fbd30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f05f8004940 tx=0x7f05f8009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 -- 172.21.15.153:0/3646001851 shutdown_connections 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 --2- 172.21.15.153:0/3646001851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06040fbd30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 --2- 172.21.15.153:0/3646001851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f06041020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 -- 172.21.15.153:0/3646001851 >> 172.21.15.153:0/3646001851 conn(0x7f06040f7870 msgr2=0x7f06040f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 -- 172.21.15.153:0/3646001851 shutdown_connections 2024-04-05T03:13:46.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 -- 172.21.15.153:0/3646001851 wait complete. 2024-04-05T03:13:46.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 Processor -- start 2024-04-05T03:13:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 -- start start 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f0604102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.054+0000 7f060a324700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f060a324700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0604104b60 con 0x7f06041026c0 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f060a324700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0604103600 con 0x7f0604101d20 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49606/0 (socket says 172.21.15.153:49606) 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 -- 172.21.15.153:0/31058418 learned_addr learned my addr 172.21.15.153:0/31058418 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f0603fff700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f0604102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 -- 172.21.15.153:0/31058418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 msgr2=0x7f0604102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f0604102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 -- 172.21.15.153:0/31058418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f8005040 con 0x7f06041026c0 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f0603fff700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f0604102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:13:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f06037fe700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009100 tx=0x7f05f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05f800e040 con 0x7f06041026c0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0604103880 con 0x7f06041026c0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05f8013420 con 0x7f06041026c0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05f801c610 con 0x7f06041026c0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.055+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060411a1b0 con 0x7f06041026c0 2024-04-05T03:13:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.056+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06040fd470 con 0x7f06041026c0 2024-04-05T03:13:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.057+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f05f800baf0 con 0x7f06041026c0 2024-04-05T03:13:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.057+0000 7f0600ff9700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 0x7f05ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.057+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05f808e7f0 con 0x7f06041026c0 2024-04-05T03:13:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.057+0000 7f0603fff700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 0x7f05ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.058+0000 7f0603fff700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 0x7f05ec070150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05fc001250 tx=0x7f05fc009610 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.062+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f05f8057df0 con 0x7f06041026c0 2024-04-05T03:13:46.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.355+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0604108ea0 con 0x7f06041026c0 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.356+0000 7f0600ff9700 1 -- 172.21.15.153:0/31058418 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f05f805b4b0 con 0x7f06041026c0 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:13:46.356 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:13:46.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 msgr2=0x7f05ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 0x7f05ec070150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05fc001250 tx=0x7f05fc009610 comp rx=0 tx=0).stop 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 msgr2=0x7f06041030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009100 tx=0x7f05f8009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 shutdown_connections 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05ec06dca0 0x7f05ec070150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604101d20 0x7f0604102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 --2- 172.21.15.153:0/31058418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06041026c0 0x7f06041030c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 >> 172.21.15.153:0/31058418 conn(0x7f06040f7870 msgr2=0x7f06040f88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 shutdown_connections 2024-04-05T03:13:46.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.358+0000 7f060a324700 1 -- 172.21.15.153:0/31058418 wait complete. 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/1089287797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 msgr2=0x7fd1b0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/1089287797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0106f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0004940 tx=0x7fd1a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/1089287797 shutdown_connections 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/1089287797 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0106f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/1089287797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 0x7fd1b00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/1089287797 >> 172.21.15.153:0/1089287797 conn(0x7fd1b0072bc0 msgr2=0x7fd1b0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/1089287797 shutdown_connections 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.460+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/1089287797 wait complete. 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 Processor -- start 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 -- start start 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 0x7fd1b0113b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01146d0 con 0x7fd1b00fe830 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b7ec3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b010dac0 con 0x7fd1b00fde90 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49624/0 (socket says 172.21.15.153:49624) 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 -- 172.21.15.153:0/2656068859 learned_addr learned my addr 172.21.15.153:0/2656068859 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b5c5f700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 0x7fd1b0113b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 -- 172.21.15.153:0/2656068859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 msgr2=0x7fd1b0113b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 0x7fd1b0113b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 -- 172.21.15.153:0/2656068859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a0005040 con 0x7fd1b00fe830 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.461+0000 7fd1b545e700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0000c00 tx=0x7fd1a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.462+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a0021070 con 0x7fd1b00fe830 2024-04-05T03:13:46.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.462+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1a0013c00 con 0x7fd1b00fe830 2024-04-05T03:13:46.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.462+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b010dd40 con 0x7fd1b00fe830 2024-04-05T03:13:46.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.462+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1a000e070 con 0x7fd1b00fe830 2024-04-05T03:13:46.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.462+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b010e280 con 0x7fd1b00fe830 2024-04-05T03:13:46.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.463+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b0101840 con 0x7fd1b00fe830 2024-04-05T03:13:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.463+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd1a001c9d0 con 0x7fd1b00fe830 2024-04-05T03:13:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.464+0000 7fd1a67fc700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 0x7fd19c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.464+0000 7fd1b5c5f700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 0x7fd19c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.464+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd1a008e880 con 0x7fd1b00fe830 2024-04-05T03:13:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.464+0000 7fd1b5c5f700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 0x7fd19c070160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009b80 tx=0x7fd1ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.468+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd1a005b9b0 con 0x7fd1b00fe830 2024-04-05T03:13:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.710+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd1b010f190 con 0x7fd1b00fe830 2024-04-05T03:13:46.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:46 smithi153 ceph-mon[106102]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:46.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:46 smithi153 ceph-mon[106102]: pgmap v243: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:46.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:46 smithi153 ceph-mon[106102]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:46.710 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:46 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/31058418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:13:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.710+0000 7fd1a67fc700 1 -- 172.21.15.153:0/2656068859 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fd1a005b540 con 0x7fd1b00fe830 2024-04-05T03:13:46.710 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:13:46.710 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:13:46.710 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:13:46.711 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:13:46.712 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:46.713 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:46.714 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:46.714 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:13:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 msgr2=0x7fd19c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 0x7fd19c070160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009b80 tx=0x7fd1ac00b040 comp rx=0 tx=0).stop 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 msgr2=0x7fd1b0114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0000c00 tx=0x7fd1a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 shutdown_connections 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd19c06dcb0 0x7fd19c070160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b00fde90 0x7fd1b0113b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 --2- 172.21.15.153:0/2656068859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1b00fe830 0x7fd1b0114080 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 >> 172.21.15.153:0/2656068859 conn(0x7fd1b0072bc0 msgr2=0x7fd1b00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 shutdown_connections 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.712+0000 7fd1b7ec3700 1 -- 172.21.15.153:0/2656068859 wait complete. 2024-04-05T03:13:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 -- 172.21.15.153:0/3630478821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 msgr2=0x7ffb1c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 --2- 172.21.15.153:0/3630478821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c0fe260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c004940 tx=0x7ffb0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 -- 172.21.15.153:0/3630478821 shutdown_connections 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 --2- 172.21.15.153:0/3630478821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 0x7ffb1c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 --2- 172.21.15.153:0/3630478821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c0fe260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 -- 172.21.15.153:0/3630478821 >> 172.21.15.153:0/3630478821 conn(0x7ffb1c072bc0 msgr2=0x7ffb1c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 -- 172.21.15.153:0/3630478821 shutdown_connections 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 -- 172.21.15.153:0/3630478821 wait complete. 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.815+0000 7ffb2138c700 1 Processor -- start 2024-04-05T03:13:46.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb2138c700 1 -- start start 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb2138c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb2138c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 0x7ffb1c1162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb2138c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c116910 con 0x7ffb1c0fde90 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb2138c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c10fd20 con 0x7ffb1c0fe830 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49636/0 (socket says 172.21.15.153:49636) 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 -- 172.21.15.153:0/2141327668 learned_addr learned my addr 172.21.15.153:0/2141327668 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb127fc700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 0x7ffb1c1162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 -- 172.21.15.153:0/2141327668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 msgr2=0x7ffb1c1162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 0x7ffb1c1162c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 -- 172.21.15.153:0/2141327668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c005040 con 0x7ffb1c0fde90 2024-04-05T03:13:46.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb1affd700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c000c00 tx=0x7ffb0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.816+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0c00e040 con 0x7ffb1c0fde90 2024-04-05T03:13:46.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.817+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c10ffa0 con 0x7ffb1c0fde90 2024-04-05T03:13:46.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.817+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb0c013420 con 0x7ffb1c0fde90 2024-04-05T03:13:46.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.817+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb0c01c610 con 0x7ffb1c0fde90 2024-04-05T03:13:46.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.817+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c1104e0 con 0x7ffb1c0fde90 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.817+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c106a20 con 0x7ffb1c0fde90 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.818+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ffb0c013590 con 0x7ffb1c0fde90 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.818+0000 7ffb13fff700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 0x7ffb08070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.818+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ffb0c08e830 con 0x7ffb1c0fde90 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.819+0000 7ffb127fc700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 0x7ffb08070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:46.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.819+0000 7ffb127fc700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 0x7ffb08070160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c111660 tx=0x7ffb04009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:46.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:46.823+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffb0c057db0 con 0x7ffb1c0fde90 2024-04-05T03:13:47.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.035+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb1c109f00 con 0x7ffb0806dcb0 2024-04-05T03:13:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.036+0000 7ffb13fff700 1 -- 172.21.15.153:0/2141327668 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffb1c109f00 con 0x7ffb0806dcb0 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:13:47.036 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:13:47.037 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:13:47.037 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:13:47.037 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:13:47.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 msgr2=0x7ffb08070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 0x7ffb08070160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c111660 tx=0x7ffb04009040 comp rx=0 tx=0).stop 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 msgr2=0x7ffb1c115d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c000c00 tx=0x7ffb0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 shutdown_connections 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ffb0806dcb0 0x7ffb08070160 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb1c0fde90 0x7ffb1c115d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 --2- 172.21.15.153:0/2141327668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb1c0fe830 0x7ffb1c1162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 >> 172.21.15.153:0/2141327668 conn(0x7ffb1c072bc0 msgr2=0x7ffb1c0fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 shutdown_connections 2024-04-05T03:13:47.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.039+0000 7ffb2138c700 1 -- 172.21.15.153:0/2141327668 wait complete. 2024-04-05T03:13:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:46 smithi203 ceph-mon[95988]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:46 smithi203 ceph-mon[95988]: pgmap v243: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:46 smithi203 ceph-mon[95988]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:46 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/31058418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- 172.21.15.153:0/1472119750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c102710 msgr2=0x7f647c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 --2- 172.21.15.153:0/1472119750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c102710 0x7f647c0fbd80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f646c004940 tx=0x7f646c009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- 172.21.15.153:0/1472119750 shutdown_connections 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 --2- 172.21.15.153:0/1472119750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c102710 0x7f647c0fbd80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 --2- 172.21.15.153:0/1472119750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c101d70 0x7f647c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- 172.21.15.153:0/1472119750 >> 172.21.15.153:0/1472119750 conn(0x7f647c072bc0 msgr2=0x7f647c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:47.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- 172.21.15.153:0/1472119750 shutdown_connections 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- 172.21.15.153:0/1472119750 wait complete. 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 Processor -- start 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.140+0000 7f64846ac700 1 -- start start 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c102710 0x7f647c1140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c114710 con 0x7f647c101d70 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c10db20 con 0x7f647c102710 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49656/0 (socket says 172.21.15.153:49656) 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 -- 172.21.15.153:0/542919183 learned_addr learned my addr 172.21.15.153:0/542919183 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6481c47700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c102710 0x7f647c1140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:47.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 -- 172.21.15.153:0/542919183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c102710 msgr2=0x7f647c1140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c102710 0x7f647c1140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 -- 172.21.15.153:0/542919183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c005040 con 0x7f647c101d70 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6482448700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6478002a50 tx=0x7f64780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6478007cc0 con 0x7f647c101d70 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f647800f040 con 0x7f647c101d70 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c10de00 con 0x7f647c101d70 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f647800da30 con 0x7f647c101d70 2024-04-05T03:13:47.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.141+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c10e390 con 0x7f647c101d70 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.142+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c0fd4c0 con 0x7f647c101d70 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.143+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f647800dbb0 con 0x7f647c101d70 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.143+0000 7f6472ffd700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 0x7f6468070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.143+0000 7f6481c47700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 0x7f6468070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.143+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f647808d230 con 0x7f647c101d70 2024-04-05T03:13:47.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.144+0000 7f6481c47700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 0x7f6468070150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f647c10f490 tx=0x7f646c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:13:47.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.148+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6478056830 con 0x7f647c101d70 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.420+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f647c04bed0 con 0x7f647c101d70 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.421+0000 7f6472ffd700 1 -- 172.21.15.153:0/542919183 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6478059ef0 con 0x7f647c101d70 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:13:47.420 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:13:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 msgr2=0x7f6468070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 0x7f6468070150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f647c10f490 tx=0x7f646c009e30 comp rx=0 tx=0).stop 2024-04-05T03:13:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 msgr2=0x7f647c113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6478002a50 tx=0x7f64780046b0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 shutdown_connections 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f646806dca0 0x7f6468070150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f647c101d70 0x7f647c113b80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 --2- 172.21.15.153:0/542919183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f647c102710 0x7f647c1140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 >> 172.21.15.153:0/542919183 conn(0x7f647c072bc0 msgr2=0x7f647c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 shutdown_connections 2024-04-05T03:13:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:13:47.423+0000 7f64846ac700 1 -- 172.21.15.153:0/542919183 wait complete. 2024-04-05T03:13:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:47 smithi153 ceph-mon[106102]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:47 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2656068859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:13:47.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:47 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/542919183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:13:48.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:47 smithi203 ceph-mon[95988]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:48.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:47 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2656068859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:13:48.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:47 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/542919183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:13:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:48 smithi203 ceph-mon[95988]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:48 smithi203 ceph-mon[95988]: pgmap v244: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:48 smithi153 ceph-mon[106102]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:13:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:48 smithi153 ceph-mon[106102]: pgmap v244: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:50.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:50 smithi153 ceph-mon[106102]: pgmap v245: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:50 smithi203 ceph-mon[95988]: pgmap v245: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:52 smithi203 ceph-mon[95988]: pgmap v246: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:52 smithi153 ceph-mon[106102]: pgmap v246: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:54 smithi203 ceph-mon[95988]: pgmap v247: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:54 smithi153 ceph-mon[106102]: pgmap v247: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:56 smithi203 ceph-mon[95988]: pgmap v248: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:56 smithi153 ceph-mon[106102]: pgmap v248: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:58.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:13:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:13:58 smithi203 ceph-mon[95988]: pgmap v249: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:13:59.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:13:58 smithi153 ceph-mon[106102]: pgmap v249: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:00.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:00 smithi153 ceph-mon[106102]: pgmap v250: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:01.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:00 smithi203 ceph-mon[95988]: pgmap v250: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:02 smithi203 ceph-mon[95988]: pgmap v251: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:02 smithi153 ceph-mon[106102]: pgmap v251: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:04 smithi203 ceph-mon[95988]: pgmap v252: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:04 smithi153 ceph-mon[106102]: pgmap v252: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:06 smithi203 ceph-mon[95988]: pgmap v253: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:06 smithi153 ceph-mon[106102]: pgmap v253: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:09.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:08 smithi203 ceph-mon[95988]: pgmap v254: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:08 smithi153 ceph-mon[106102]: pgmap v254: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:10 smithi203 ceph-mon[95988]: pgmap v255: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:11.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:10 smithi153 ceph-mon[106102]: pgmap v255: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:12 smithi203 ceph-mon[95988]: pgmap v256: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:12 smithi153 ceph-mon[106102]: pgmap v256: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:14 smithi203 ceph-mon[95988]: pgmap v257: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:14 smithi153 ceph-mon[106102]: pgmap v257: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:16 smithi203 ceph-mon[95988]: pgmap v258: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:16 smithi153 ceph-mon[106102]: pgmap v258: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 -- 172.21.15.153:0/1806486880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 msgr2=0x7f5660103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 --2- 172.21.15.153:0/1806486880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660103320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5650004940 tx=0x7f5650009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 -- 172.21.15.153:0/1806486880 shutdown_connections 2024-04-05T03:14:17.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 --2- 172.21.15.153:0/1806486880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 0x7f56600fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 --2- 172.21.15.153:0/1806486880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660103320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 -- 172.21.15.153:0/1806486880 >> 172.21.15.153:0/1806486880 conn(0x7f56600714c0 msgr2=0x7f56600718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 -- 172.21.15.153:0/1806486880 shutdown_connections 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.538+0000 7f56657fc700 1 -- 172.21.15.153:0/1806486880 wait complete. 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 Processor -- start 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 -- start start 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 0x7f56601139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5660114530 con 0x7f5660102f50 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f56657fc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566010db30 con 0x7f56600fb4d0 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60532/0 (socket says 172.21.15.153:60532) 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 -- 172.21.15.153:0/3108368996 learned_addr learned my addr 172.21.15.153:0/3108368996 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f565effd700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 0x7f56601139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 -- 172.21.15.153:0/3108368996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 msgr2=0x7f56601139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 0x7f56601139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 -- 172.21.15.153:0/3108368996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5650005040 con 0x7f5660102f50 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.539+0000 7f5657fff700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f564800c8c0 tx=0x7f56480044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.540+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f564800a980 con 0x7f5660102f50 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.540+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f564800ce60 con 0x7f5660102f50 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.540+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56480115e0 con 0x7f5660102f50 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.540+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566010de10 con 0x7f5660102f50 2024-04-05T03:14:17.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.540+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566010e350 con 0x7f5660102f50 2024-04-05T03:14:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.541+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5660100200 con 0x7f5660102f50 2024-04-05T03:14:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.541+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5648012400 con 0x7f5660102f50 2024-04-05T03:14:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.542+0000 7f56577fe700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 0x7f564c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.542+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f564808d710 con 0x7f5660102f50 2024-04-05T03:14:17.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.542+0000 7f565effd700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 0x7f564c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.542+0000 7f565effd700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 0x7f564c070150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f565000b020 tx=0x7f5650009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:17.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.546+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f564805a7a0 con 0x7f5660102f50 2024-04-05T03:14:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.760+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f566010eff0 con 0x7f564c06dca0 2024-04-05T03:14:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.761+0000 7f56577fe700 1 -- 172.21.15.153:0/3108368996 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f566010eff0 con 0x7f564c06dca0 2024-04-05T03:14:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.763+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 msgr2=0x7f564c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.763+0000 7f56657fc700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 0x7f564c070150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f565000b020 tx=0x7f5650009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.763+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 msgr2=0x7f5660113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.763+0000 7f56657fc700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f564800c8c0 tx=0x7f56480044c0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 shutdown_connections 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f564c06dca0 0x7f564c070150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56600fb4d0 0x7f56601139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 --2- 172.21.15.153:0/3108368996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5660102f50 0x7f5660113ee0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 >> 172.21.15.153:0/3108368996 conn(0x7f56600714c0 msgr2=0x7f56600fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:17.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 shutdown_connections 2024-04-05T03:14:17.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.764+0000 7f56657fc700 1 -- 172.21.15.153:0/3108368996 wait complete. 2024-04-05T03:14:17.773 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 -- 172.21.15.153:0/946833471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 msgr2=0x7f9d34106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/946833471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34106e80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9d28004940 tx=0x7f9d28009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 -- 172.21.15.153:0/946833471 shutdown_connections 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/946833471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34106e80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/946833471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d340fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 -- 172.21.15.153:0/946833471 >> 172.21.15.153:0/946833471 conn(0x7f9d340f78b0 msgr2=0x7f9d340f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 -- 172.21.15.153:0/946833471 shutdown_connections 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.867+0000 7f9d3b80d700 1 -- 172.21.15.153:0/946833471 wait complete. 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 Processor -- start 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 -- start start 2024-04-05T03:14:17.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d34113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d341144b0 con 0x7f9d340fe7b0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d3b80d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3410daf0 con 0x7f9d340fde10 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60556/0 (socket says 172.21.15.153:60556) 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 -- 172.21.15.153:0/2977195561 learned_addr learned my addr 172.21.15.153:0/2977195561 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d395a9700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d34113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 -- 172.21.15.153:0/2977195561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 msgr2=0x7f9d34113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d34113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 -- 172.21.15.153:0/2977195561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28005040 con 0x7f9d340fe7b0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d395a9700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d34113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.868+0000 7f9d38da8700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9d28000c00 tx=0x7f9d28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.869+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d28021070 con 0x7f9d340fe7b0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.869+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3410dd70 con 0x7f9d340fe7b0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.869+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d28013cd0 con 0x7f9d340fe7b0 2024-04-05T03:14:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.869+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d2800e070 con 0x7f9d340fe7b0 2024-04-05T03:14:17.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.869+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3410e2b0 con 0x7f9d340fe7b0 2024-04-05T03:14:17.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.870+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d34066af0 con 0x7f9d340fe7b0 2024-04-05T03:14:17.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.870+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9d2801c9d0 con 0x7f9d340fe7b0 2024-04-05T03:14:17.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.871+0000 7f9d31ffb700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 0x7f9d1c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:17.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.871+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9d2808ea10 con 0x7f9d340fe7b0 2024-04-05T03:14:17.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.871+0000 7f9d395a9700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 0x7f9d1c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:17.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.871+0000 7f9d395a9700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 0x7f9d1c070150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009f10 tx=0x7f9d2c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:17.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:17.875+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9d2805bb40 con 0x7f9d340fe7b0 2024-04-05T03:14:18.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.086+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d34109ea0 con 0x7f9d1c06dca0 2024-04-05T03:14:18.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.087+0000 7f9d31ffb700 1 -- 172.21.15.153:0/2977195561 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9d34109ea0 con 0x7f9d1c06dca0 2024-04-05T03:14:18.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 msgr2=0x7f9d1c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 0x7f9d1c070150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009f10 tx=0x7f9d2c00b040 comp rx=0 tx=0).stop 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 msgr2=0x7f9d34113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9d28000c00 tx=0x7f9d28009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 shutdown_connections 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9d1c06dca0 0x7f9d1c070150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d340fde10 0x7f9d34113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 --2- 172.21.15.153:0/2977195561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d340fe7b0 0x7f9d34113e60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 >> 172.21.15.153:0/2977195561 conn(0x7f9d340f78b0 msgr2=0x7f9d340f8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 shutdown_connections 2024-04-05T03:14:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.089+0000 7f9d3b80d700 1 -- 172.21.15.153:0/2977195561 wait complete. 2024-04-05T03:14:18.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.190+0000 7f900e8da700 1 -- 172.21.15.153:0/4207337420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fe7b0 msgr2=0x7f9008106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.190+0000 7f900e8da700 1 --2- 172.21.15.153:0/4207337420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fe7b0 0x7f9008106e80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc004940 tx=0x7f8ffc009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 -- 172.21.15.153:0/4207337420 shutdown_connections 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 --2- 172.21.15.153:0/4207337420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fe7b0 0x7f9008106e80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 --2- 172.21.15.153:0/4207337420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fde10 0x7f90080fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 -- 172.21.15.153:0/4207337420 >> 172.21.15.153:0/4207337420 conn(0x7f90080f7870 msgr2=0x7f90080f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 -- 172.21.15.153:0/4207337420 shutdown_connections 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 -- 172.21.15.153:0/4207337420 wait complete. 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 Processor -- start 2024-04-05T03:14:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.191+0000 7f900e8da700 1 -- start start 2024-04-05T03:14:18.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f900e8da700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f900e8da700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fe7b0 0x7f90080fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f900e8da700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080fdc20 con 0x7f90080fde10 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f900e8da700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080fc750 con 0x7f90080fe7b0 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60574/0 (socket says 172.21.15.153:60574) 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 -- 172.21.15.153:0/1048487427 learned_addr learned my addr 172.21.15.153:0/1048487427 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f90077fe700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fe7b0 0x7f90080fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 -- 172.21.15.153:0/1048487427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fe7b0 msgr2=0x7f90080fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fe7b0 0x7f90080fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 -- 172.21.15.153:0/1048487427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc005040 con 0x7f90080fde10 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9007fff700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8002a50 tx=0x7f8ff80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ff8007cc0 con 0x7f90080fde10 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.192+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90080fca30 con 0x7f90080fde10 2024-04-05T03:14:18.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.193+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ff800f040 con 0x7f90080fde10 2024-04-05T03:14:18.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.193+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900811a100 con 0x7f90080fde10 2024-04-05T03:14:18.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.193+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ff8009640 con 0x7f90080fde10 2024-04-05T03:14:18.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.193+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff40052b0 con 0x7f90080fde10 2024-04-05T03:14:18.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.194+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8ff800d690 con 0x7f90080fde10 2024-04-05T03:14:18.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.195+0000 7f9004ff9700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 0x7f8ff0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.195+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8ff808d200 con 0x7f90080fde10 2024-04-05T03:14:18.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.195+0000 7f90077fe700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 0x7f8ff0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.195+0000 7f90077fe700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 0x7f8ff0070150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f90080fd400 tx=0x7f8ffc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.199+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ff8056800 con 0x7f90080fde10 2024-04-05T03:14:18.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.403+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff4000bc0 con 0x7f8ff006dca0 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.413+0000 7f9004ff9700 1 -- 172.21.15.153:0/1048487427 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f8ff4000bc0 con 0x7f8ff006dca0 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (8m) 2m ago 20m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (6m) 2m ago 21m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (6m) 114s ago 18m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (8m) 2m ago 19m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (2m) 2m ago 14m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (2m) 2m ago 14m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:14:18.412 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (117s) 114s ago 14m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (2m) 114s ago 14m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (10m) 2m ago 22m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (9m) 114s ago 18m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (7m) 2m ago 22m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (7m) 114s ago 18m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (9m) 2m ago 20m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (9m) 114s ago 18m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 2m ago 18m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 2m ago 17m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 2m ago 17m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (4m) 114s ago 16m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (3m) 114s ago 16m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (3m) 114s ago 15m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:14:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (9m) 114s ago 18m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 msgr2=0x7f8ff0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 0x7f8ff0070150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f90080fd400 tx=0x7f8ffc009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 msgr2=0x7f90080fbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8002a50 tx=0x7f8ff80046b0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 shutdown_connections 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8ff006dca0 0x7f8ff0070150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90080fde10 0x7f90080fbcd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 --2- 172.21.15.153:0/1048487427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90080fe7b0 0x7f90080fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 >> 172.21.15.153:0/1048487427 conn(0x7f90080f7870 msgr2=0x7f90080f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 shutdown_connections 2024-04-05T03:14:18.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.415+0000 7f900e8da700 1 -- 172.21.15.153:0/1048487427 wait complete. 2024-04-05T03:14:18.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 -- 172.21.15.153:0/1562518311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc101d70 msgr2=0x7f9ddc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 --2- 172.21.15.153:0/1562518311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc101d70 0x7f9ddc102140 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004940 tx=0x7f9dd0009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 -- 172.21.15.153:0/1562518311 shutdown_connections 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 --2- 172.21.15.153:0/1562518311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102710 0x7f9ddc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 --2- 172.21.15.153:0/1562518311 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc101d70 0x7f9ddc102140 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 -- 172.21.15.153:0/1562518311 >> 172.21.15.153:0/1562518311 conn(0x7f9ddc072bc0 msgr2=0x7f9ddc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 -- 172.21.15.153:0/1562518311 shutdown_connections 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.518+0000 7f9de2983700 1 -- 172.21.15.153:0/1562518311 wait complete. 2024-04-05T03:14:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 Processor -- start 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 -- start start 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc101d70 0x7f9ddc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc114670 con 0x7f9ddc102710 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9de2983700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc10daa0 con 0x7f9ddc101d70 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9ddb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9ddb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60598/0 (socket says 172.21.15.153:60598) 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9ddb7fe700 1 -- 172.21.15.153:0/2711715467 learned_addr learned my addr 172.21.15.153:0/2711715467 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.519+0000 7f9ddbfff700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc101d70 0x7f9ddc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9ddb7fe700 1 -- 172.21.15.153:0/2711715467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc101d70 msgr2=0x7f9ddc113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9ddb7fe700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc101d70 0x7f9ddc113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9ddb7fe700 1 -- 172.21.15.153:0/2711715467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0005040 con 0x7f9ddc102710 2024-04-05T03:14:18.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9ddb7fe700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc002a80 tx=0x7f9dcc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dcc004e80 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dcc00f040 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc10dd80 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc10e340 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.520+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dcc00daa0 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.521+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc0fd4c0 con 0x7f9ddc102710 2024-04-05T03:14:18.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.522+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9dcc007cc0 con 0x7f9ddc102710 2024-04-05T03:14:18.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.522+0000 7f9dd8ff9700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.523+0000 7f9ddbfff700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.523+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9dcc08cac0 con 0x7f9ddc102710 2024-04-05T03:14:18.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.523+0000 7f9ddbfff700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004f90 tx=0x7f9dd0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.527+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9dcc059bf0 con 0x7f9ddc102710 2024-04-05T03:14:18.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.803+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9ddc04bed0 con 0x7f9ddc102710 2024-04-05T03:14:18.803 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:18 smithi153 ceph-mon[106102]: pgmap v259: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:18.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.804+0000 7f9dd8ff9700 1 -- 172.21.15.153:0/2711715467 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9dcc059780 con 0x7f9ddc102710 2024-04-05T03:14:18.803 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:14:18.804 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:14:18.805 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:14:18.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 msgr2=0x7f9dc4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004f90 tx=0x7f9dd0009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 msgr2=0x7f9ddc114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc002a80 tx=0x7f9dcc004630 comp rx=0 tx=0).stop 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 shutdown_connections 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc101d70 0x7f9ddc113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 --2- 172.21.15.153:0/2711715467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc102710 0x7f9ddc114020 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 >> 172.21.15.153:0/2711715467 conn(0x7f9ddc072bc0 msgr2=0x7f9ddc065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 shutdown_connections 2024-04-05T03:14:18.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.806+0000 7f9de2983700 1 -- 172.21.15.153:0/2711715467 wait complete. 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 -- 172.21.15.153:0/1770110524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 msgr2=0x7f6b68106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 --2- 172.21.15.153:0/1770110524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68106e40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c004970 tx=0x7f6b5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 -- 172.21.15.153:0/1770110524 shutdown_connections 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 --2- 172.21.15.153:0/1770110524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68106e40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 --2- 172.21.15.153:0/1770110524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 0x7f6b680fe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 -- 172.21.15.153:0/1770110524 >> 172.21.15.153:0/1770110524 conn(0x7f6b680f7870 msgr2=0x7f6b680f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:18.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 -- 172.21.15.153:0/1770110524 shutdown_connections 2024-04-05T03:14:18.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.907+0000 7f6b6e101700 1 -- 172.21.15.153:0/1770110524 wait complete. 2024-04-05T03:14:18.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 Processor -- start 2024-04-05T03:14:18.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 -- start start 2024-04-05T03:14:18.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 0x7f6b68113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681144e0 con 0x7f6b680fe800 2024-04-05T03:14:18.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b6e101700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6810dae0 con 0x7f6b680fde60 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b66ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.908+0000 7f6b66ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41386/0 (socket says 172.21.15.153:41386) 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b66ffd700 1 -- 172.21.15.153:0/2688189700 learned_addr learned my addr 172.21.15.153:0/2688189700 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b677fe700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 0x7f6b68113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b66ffd700 1 -- 172.21.15.153:0/2688189700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 msgr2=0x7f6b68113950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b66ffd700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 0x7f6b68113950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b66ffd700 1 -- 172.21.15.153:0/2688189700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c005040 con 0x7f6b680fe800 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b66ffd700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c004f50 tx=0x7f6b5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b5c00e040 con 0x7f6b680fe800 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b5c004290 con 0x7f6b680fe800 2024-04-05T03:14:18.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6810dd60 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b5c01c3f0 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.909+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6810e2a0 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.910+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b68066b30 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.911+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6b5c013420 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.911+0000 7f6b4ffff700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 0x7f6b50070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.911+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6b5c08e7e0 con 0x7f6b680fe800 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.911+0000 7f6b677fe700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 0x7f6b50070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:18.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.912+0000 7f6b677fe700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 0x7f6b50070150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b58004f40 tx=0x7f6b5800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:18.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:18.916+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6b5c05b910 con 0x7f6b680fe800 2024-04-05T03:14:19.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:18 smithi203 ceph-mon[95988]: pgmap v259: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:19.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.150+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6b68109f10 con 0x7f6b680fe800 2024-04-05T03:14:19.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.151+0000 7f6b4ffff700 1 -- 172.21.15.153:0/2688189700 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f6b5c009400 con 0x7f6b680fe800 2024-04-05T03:14:19.150 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:14:19.150 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:14:19.150 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:14:19.151 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:14:19.152 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 msgr2=0x7f6b50070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 0x7f6b50070150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b58004f40 tx=0x7f6b5800b040 comp rx=0 tx=0).stop 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 msgr2=0x7f6b68113e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c004f50 tx=0x7f6b5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 shutdown_connections 2024-04-05T03:14:19.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b5006dca0 0x7f6b50070150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b680fde60 0x7f6b68113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 --2- 172.21.15.153:0/2688189700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b680fe800 0x7f6b68113e90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 >> 172.21.15.153:0/2688189700 conn(0x7f6b680f7870 msgr2=0x7f6b681036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 shutdown_connections 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.153+0000 7f6b6e101700 1 -- 172.21.15.153:0/2688189700 wait complete. 2024-04-05T03:14:19.155 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 -- 172.21.15.153:0/1548787258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 msgr2=0x7f807c0fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 --2- 172.21.15.153:0/1548787258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c0fc650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8078004940 tx=0x7f8078009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 -- 172.21.15.153:0/1548787258 shutdown_connections 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 --2- 172.21.15.153:0/1548787258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c0fc650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 --2- 172.21.15.153:0/1548787258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 0x7f807c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 -- 172.21.15.153:0/1548787258 >> 172.21.15.153:0/1548787258 conn(0x7f807c071530 msgr2=0x7f807c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 -- 172.21.15.153:0/1548787258 shutdown_connections 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.255+0000 7f808370c700 1 -- 172.21.15.153:0/1548787258 wait complete. 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 Processor -- start 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 -- start start 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 0x7f807c113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c114500 con 0x7f807c102800 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f808370c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c10db00 con 0x7f807c101e60 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41412/0 (socket says 172.21.15.153:41412) 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 -- 172.21.15.153:0/1248635926 learned_addr learned my addr 172.21.15.153:0/1248635926 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f80814a8700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 0x7f807c113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 -- 172.21.15.153:0/1248635926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 msgr2=0x7f807c113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 0x7f807c113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.256+0000 7f8080ca7700 1 -- 172.21.15.153:0/1248635926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078005040 con 0x7f807c102800 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f8080ca7700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8078009100 tx=0x7f8078009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f807800e040 con 0x7f807c102800 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c10dd80 con 0x7f807c102800 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8078013420 con 0x7f807c102800 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f807801c610 con 0x7f807c102800 2024-04-05T03:14:19.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.257+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c10e340 con 0x7f807c102800 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.258+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c1001d0 con 0x7f807c102800 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.259+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8078013590 con 0x7f807c102800 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.259+0000 7f8071ffb700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 0x7f8068070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.259+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f807808e730 con 0x7f807c102800 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.259+0000 7f80814a8700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 0x7f8068070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.260+0000 7f80814a8700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 0x7f8068070150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f806c004cc0 tx=0x7f806c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:19.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.264+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f807805b900 con 0x7f807c102800 2024-04-05T03:14:19.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.477+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f807c10efc0 con 0x7f806806dca0 2024-04-05T03:14:19.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.478+0000 7f8071ffb700 1 -- 172.21.15.153:0/1248635926 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f807c10efc0 con 0x7f806806dca0 2024-04-05T03:14:19.477 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:14:19.477 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:14:19.477 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:14:19.477 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:14:19.478 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 msgr2=0x7f8068070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 0x7f8068070150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f806c004cc0 tx=0x7f806c00b040 comp rx=0 tx=0).stop 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 msgr2=0x7f807c113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8078009100 tx=0x7f8078009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 shutdown_connections 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f806806dca0 0x7f8068070150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f807c101e60 0x7f807c113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 --2- 172.21.15.153:0/1248635926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f807c102800 0x7f807c113eb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 >> 172.21.15.153:0/1248635926 conn(0x7f807c071530 msgr2=0x7f807c1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 shutdown_connections 2024-04-05T03:14:19.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.480+0000 7f808370c700 1 -- 172.21.15.153:0/1248635926 wait complete. 2024-04-05T03:14:19.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 -- 172.21.15.153:0/3760066162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c101d70 msgr2=0x7fea2c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 --2- 172.21.15.153:0/3760066162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c101d70 0x7fea2c102140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fea24004940 tx=0x7fea24009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:19.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 -- 172.21.15.153:0/3760066162 shutdown_connections 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 --2- 172.21.15.153:0/3760066162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c102710 0x7fea2c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 --2- 172.21.15.153:0/3760066162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c101d70 0x7fea2c102140 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 -- 172.21.15.153:0/3760066162 >> 172.21.15.153:0/3760066162 conn(0x7fea2c072bc0 msgr2=0x7fea2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 -- 172.21.15.153:0/3760066162 shutdown_connections 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.584+0000 7fea308d4700 1 -- 172.21.15.153:0/3760066162 wait complete. 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 Processor -- start 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 -- start start 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c101d70 0x7fea2c112a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1135b0 con 0x7fea2c102710 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea308d4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c10c9a0 con 0x7fea2c101d70 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea29d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea29d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41430/0 (socket says 172.21.15.153:41430) 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea29d9b700 1 -- 172.21.15.153:0/1027725921 learned_addr learned my addr 172.21.15.153:0/1027725921 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.585+0000 7fea2a59c700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c101d70 0x7fea2c112a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea29d9b700 1 -- 172.21.15.153:0/1027725921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c101d70 msgr2=0x7fea2c112a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea29d9b700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c101d70 0x7fea2c112a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea29d9b700 1 -- 172.21.15.153:0/1027725921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea24005040 con 0x7fea2c102710 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea29d9b700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fea20010d50 tx=0x7fea2000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea20007ab0 con 0x7fea2c102710 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea20007c10 con 0x7fea2c102710 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c10cc20 con 0x7fea2c102710 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea20012700 con 0x7fea2c102710 2024-04-05T03:14:19.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.586+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c10d160 con 0x7fea2c102710 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.587+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c0fd4c0 con 0x7fea2c102710 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.587+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fea20018030 con 0x7fea2c102710 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.588+0000 7fea1affd700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 0x7fea14070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.588+0000 7fea2a59c700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 0x7fea14070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.588+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fea20016070 con 0x7fea2c102710 2024-04-05T03:14:19.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.588+0000 7fea2a59c700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 0x7fea14070150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b0d0 tx=0x7fea24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:19.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.593+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fea2005c9c0 con 0x7fea2c102710 2024-04-05T03:14:19.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:19 smithi153 ceph-mon[106102]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:19 smithi153 ceph-mon[106102]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:19 smithi153 ceph-mon[106102]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:19 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2711715467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:14:19.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:19 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2688189700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:14:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.869+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fea2c108e70 con 0x7fea2c102710 2024-04-05T03:14:19.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.869+0000 7fea1affd700 1 -- 172.21.15.153:0/1027725921 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fea2005c550 con 0x7fea2c102710 2024-04-05T03:14:19.869 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:14:19.869 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:14:19.869 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:14:19.869 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:14:19.869 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:14:19.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.871+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 msgr2=0x7fea14070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.871+0000 7fea308d4700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 0x7fea14070150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b0d0 tx=0x7fea24009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.871+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 msgr2=0x7fea2c112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.871+0000 7fea308d4700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fea20010d50 tx=0x7fea2000b040 comp rx=0 tx=0).stop 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 shutdown_connections 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fea1406dca0 0x7fea14070150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2c101d70 0x7fea2c112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 --2- 172.21.15.153:0/1027725921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c102710 0x7fea2c112f60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 >> 172.21.15.153:0/1027725921 conn(0x7fea2c072bc0 msgr2=0x7fea2c0faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 shutdown_connections 2024-04-05T03:14:19.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:19.872+0000 7fea308d4700 1 -- 172.21.15.153:0/1027725921 wait complete. 2024-04-05T03:14:20.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:19 smithi203 ceph-mon[95988]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:20.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:19 smithi203 ceph-mon[95988]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:20.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:19 smithi203 ceph-mon[95988]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:20.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:19 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2711715467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:14:20.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:19 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2688189700' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:14:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:20 smithi203 ceph-mon[95988]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:20 smithi203 ceph-mon[95988]: pgmap v260: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:21.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:20 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1027725921' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:14:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:20 smithi153 ceph-mon[106102]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:20 smithi153 ceph-mon[106102]: pgmap v260: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:21.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:20 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1027725921' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:14:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:22 smithi203 ceph-mon[95988]: pgmap v261: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:22 smithi153 ceph-mon[106102]: pgmap v261: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:24.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:23 smithi203 ceph-mon[95988]: pgmap v262: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:24.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:23 smithi153 ceph-mon[106102]: pgmap v262: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:26 smithi203 ceph-mon[95988]: pgmap v263: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:27.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:26 smithi153 ceph-mon[106102]: pgmap v263: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:28.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:28 smithi203 ceph-mon[95988]: pgmap v264: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:29.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:28 smithi153 ceph-mon[106102]: pgmap v264: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:30.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:30 smithi153 ceph-mon[106102]: pgmap v265: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:31.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:30 smithi203 ceph-mon[95988]: pgmap v265: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:33.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:32 smithi203 ceph-mon[95988]: pgmap v266: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:32 smithi153 ceph-mon[106102]: pgmap v266: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:34 smithi203 ceph-mon[95988]: pgmap v267: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:34 smithi153 ceph-mon[106102]: pgmap v267: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:37.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:36 smithi203 ceph-mon[95988]: pgmap v268: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:36 smithi153 ceph-mon[106102]: pgmap v268: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:38 smithi203 ceph-mon[95988]: pgmap v269: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:38 smithi153 ceph-mon[106102]: pgmap v269: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:40.609 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:40 smithi153 ceph-mon[106102]: pgmap v270: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:40 smithi203 ceph-mon[95988]: pgmap v270: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:42 smithi203 ceph-mon[95988]: pgmap v271: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:42 smithi153 ceph-mon[106102]: pgmap v271: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:43.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:44.874 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:44 smithi153 ceph-mon[106102]: pgmap v272: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:44 smithi203 ceph-mon[95988]: pgmap v272: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:46 smithi203 ceph-mon[95988]: pgmap v273: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:46 smithi153 ceph-mon[106102]: pgmap v273: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:49.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:48 smithi203 ceph-mon[95988]: pgmap v274: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:48 smithi153 ceph-mon[106102]: pgmap v274: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:49.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 -- 172.21.15.153:0/856556409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4304102e00 msgr2=0x7f43041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 --2- 172.21.15.153:0/856556409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4304102e00 0x7f43041031d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42f4004940 tx=0x7f42f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 -- 172.21.15.153:0/856556409 shutdown_connections 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 --2- 172.21.15.153:0/856556409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041037a0 0x7f43040fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 --2- 172.21.15.153:0/856556409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4304102e00 0x7f43041031d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 -- 172.21.15.153:0/856556409 >> 172.21.15.153:0/856556409 conn(0x7f43040f89f0 msgr2=0x7f43040fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 -- 172.21.15.153:0/856556409 shutdown_connections 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.986+0000 7f4308a95700 1 -- 172.21.15.153:0/856556409 wait complete. 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 Processor -- start 2024-04-05T03:14:49.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 -- start start 2024-04-05T03:14:49.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 0x7f43040702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:49.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040721c0 con 0x7f43041037a0 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4308a95700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304070d40 con 0x7f4304102e00 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49048/0 (socket says 172.21.15.153:49048) 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 -- 172.21.15.153:0/1966844628 learned_addr learned my addr 172.21.15.153:0/1966844628 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f430259c700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 0x7f43040702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 -- 172.21.15.153:0/1966844628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 msgr2=0x7f43040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 0x7f43040702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 -- 172.21.15.153:0/1966844628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4005040 con 0x7f43041037a0 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f430259c700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 0x7f43040702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.987+0000 7f4301d9b700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f42f8002a50 tx=0x7f42f8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:49.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.988+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f8007cc0 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.988+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42f800f040 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.988+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4304071020 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.988+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42f800daf0 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.988+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430411a120 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.989+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4304100110 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.989+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f42f8009810 con 0x7f43041037a0 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.990+0000 7f42f2ffd700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 0x7f42ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:49.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.990+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f42f808d350 con 0x7f43041037a0 2024-04-05T03:14:49.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.990+0000 7f430259c700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 0x7f42ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:49.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.990+0000 7f430259c700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 0x7f42ec070150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42f4004560 tx=0x7f42f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:49.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:49.994+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f42f80568d0 con 0x7f43041037a0 2024-04-05T03:14:50.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.207+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43040717f0 con 0x7f42ec06dca0 2024-04-05T03:14:50.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.208+0000 7f42f2ffd700 1 -- 172.21.15.153:0/1966844628 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f43040717f0 con 0x7f42ec06dca0 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 msgr2=0x7f42ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 0x7f42ec070150 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42f4004560 tx=0x7f42f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 msgr2=0x7f4304070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f42f8002a50 tx=0x7f42f8004600 comp rx=0 tx=0).stop 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 shutdown_connections 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42ec06dca0 0x7f42ec070150 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4304102e00 0x7f43040702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 --2- 172.21.15.153:0/1966844628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43041037a0 0x7f4304070800 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 >> 172.21.15.153:0/1966844628 conn(0x7f43040f89f0 msgr2=0x7f43041073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 shutdown_connections 2024-04-05T03:14:50.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.210+0000 7f4308a95700 1 -- 172.21.15.153:0/1966844628 wait complete. 2024-04-05T03:14:50.220 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:14:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 -- 172.21.15.153:0/977349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 msgr2=0x7f6dc80fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/977349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc80fbfe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009ce0 tx=0x7f6dc0004d10 comp rx=0 tx=0).stop 2024-04-05T03:14:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 -- 172.21.15.153:0/977349579 shutdown_connections 2024-04-05T03:14:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/977349579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc8100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/977349579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc80fbfe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 -- 172.21.15.153:0/977349579 >> 172.21.15.153:0/977349579 conn(0x7f6dc80f7870 msgr2=0x7f6dc80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 -- 172.21.15.153:0/977349579 shutdown_connections 2024-04-05T03:14:50.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.313+0000 7f6dcfed6700 1 -- 172.21.15.153:0/977349579 wait complete. 2024-04-05T03:14:50.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 Processor -- start 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 -- start start 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc811a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc811adb0 con 0x7f6dc80fbc10 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcfed6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc8114390 con 0x7f6dc80fc5b0 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49068/0 (socket says 172.21.15.153:49068) 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 -- 172.21.15.153:0/3730237469 learned_addr learned my addr 172.21.15.153:0/3730237469 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcd471700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc811a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 -- 172.21.15.153:0/3730237469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 msgr2=0x7f6dc811a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc811a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcdc72700 1 -- 172.21.15.153:0/3730237469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009940 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.314+0000 7f6dcd471700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc811a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dcdc72700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0005010 tx=0x7f6dc0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc000e040 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc8114610 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc0008da0 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc001c640 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.315+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc8114b50 con 0x7f6dc80fbc10 2024-04-05T03:14:50.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.316+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc8066b50 con 0x7f6dc80fbc10 2024-04-05T03:14:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.316+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6dc0012410 con 0x7f6dc80fbc10 2024-04-05T03:14:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.317+0000 7f6dc67fc700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 0x7f6db0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.317+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6dc008f0d0 con 0x7f6dc80fbc10 2024-04-05T03:14:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.317+0000 7f6dcd471700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 0x7f6db0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.317+0000 7f6dcd471700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 0x7f6db0070150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009fd0 tx=0x7f6dbc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.321+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6dc005c180 con 0x7f6dc80fbc10 2024-04-05T03:14:50.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.536+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dc81158e0 con 0x7f6db006dca0 2024-04-05T03:14:50.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.538+0000 7f6dc67fc700 1 -- 172.21.15.153:0/3730237469 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6dc81158e0 con 0x7f6db006dca0 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 msgr2=0x7f6db0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 0x7f6db0070150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009fd0 tx=0x7f6dbc009ee0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 msgr2=0x7f6dc811a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0005010 tx=0x7f6dc0004d10 comp rx=0 tx=0).stop 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 shutdown_connections 2024-04-05T03:14:50.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6db006dca0 0x7f6db0070150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dc80fbc10 0x7f6dc811a220 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 --2- 172.21.15.153:0/3730237469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc80fc5b0 0x7f6dc811a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 >> 172.21.15.153:0/3730237469 conn(0x7f6dc80f7870 msgr2=0x7f6dc80f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 shutdown_connections 2024-04-05T03:14:50.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.540+0000 7f6dcfed6700 1 -- 172.21.15.153:0/3730237469 wait complete. 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1125778672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a81037a0 msgr2=0x7fd5a80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1125778672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a81037a0 0x7fd5a80fce10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd594004940 tx=0x7fd594009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1125778672 shutdown_connections 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1125778672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a81037a0 0x7fd5a80fce10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1125778672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a8102e00 0x7fd5a81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1125778672 >> 172.21.15.153:0/1125778672 conn(0x7fd5a80f89b0 msgr2=0x7fd5a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1125778672 shutdown_connections 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.642+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1125778672 wait complete. 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 Processor -- start 2024-04-05T03:14:50.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 -- start start 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a81037a0 0x7fd5a81140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8114710 con 0x7fd5a8102e00 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5af0d1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a810db70 con 0x7fd5a81037a0 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5ace6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5ace6d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49086/0 (socket says 172.21.15.153:49086) 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5ace6d700 1 -- 172.21.15.153:0/1954297456 learned_addr learned my addr 172.21.15.153:0/1954297456 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.643+0000 7fd5a7fff700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a81037a0 0x7fd5a81140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5ace6d700 1 -- 172.21.15.153:0/1954297456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a81037a0 msgr2=0x7fd5a81140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5ace6d700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a81037a0 0x7fd5a81140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5ace6d700 1 -- 172.21.15.153:0/1954297456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594005040 con 0x7fd5a8102e00 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5ace6d700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd59c002a50 tx=0x7fd59c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59c007cc0 con 0x7fd5a8102e00 2024-04-05T03:14:50.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd59c00f040 con 0x7fd5a8102e00 2024-04-05T03:14:50.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a810de50 con 0x7fd5a8102e00 2024-04-05T03:14:50.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59c00da30 con 0x7fd5a8102e00 2024-04-05T03:14:50.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.644+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a810e3e0 con 0x7fd5a8102e00 2024-04-05T03:14:50.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.645+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a81052d0 con 0x7fd5a8102e00 2024-04-05T03:14:50.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.646+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd59c00db90 con 0x7fd5a8102e00 2024-04-05T03:14:50.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.646+0000 7fd5a57fa700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 0x7fd598070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.646+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd59c08d310 con 0x7fd5a8102e00 2024-04-05T03:14:50.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.646+0000 7fd5a7fff700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 0x7fd598070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.646+0000 7fd5a7fff700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 0x7fd598070150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd594004890 tx=0x7fd594009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.651+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd59c056890 con 0x7fd5a8102e00 2024-04-05T03:14:50.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.855+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5a8109f50 con 0x7fd59806dca0 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.865+0000 7fd5a57fa700 1 -- 172.21.15.153:0/1954297456 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd5a8109f50 con 0x7fd59806dca0 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (9m) 2m ago 20m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (6m) 2m ago 21m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (6m) 2m ago 19m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:14:50.864 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (9m) 2m ago 20m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (3m) 2m ago 14m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (3m) 2m ago 15m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (2m) 2m ago 14m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (2m) 2m ago 15m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (10m) 2m ago 22m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (10m) 2m ago 19m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 2m ago 22m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (7m) 2m ago 19m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (10m) 2m ago 20m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (9m) 2m ago 19m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (6m) 2m ago 18m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 2m ago 18m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 2m ago 17m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (4m) 2m ago 17m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (4m) 2m ago 16m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (3m) 2m ago 16m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:14:50.865 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (9m) 2m ago 19m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:14:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 msgr2=0x7fd598070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 0x7fd598070150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd594004890 tx=0x7fd594009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 msgr2=0x7fd5a8113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd59c002a50 tx=0x7fd59c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 shutdown_connections 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd59806dca0 0x7fd598070150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5a8102e00 0x7fd5a8113b80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 --2- 172.21.15.153:0/1954297456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5a81037a0 0x7fd5a81140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 >> 172.21.15.153:0/1954297456 conn(0x7fd5a80f89b0 msgr2=0x7fd5a8102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 shutdown_connections 2024-04-05T03:14:50.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.867+0000 7fd5af0d1700 1 -- 172.21.15.153:0/1954297456 wait complete. 2024-04-05T03:14:50.885 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:50 smithi153 ceph-mon[106102]: pgmap v275: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 -- 172.21.15.153:0/4287026948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 msgr2=0x7f08281031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 --2- 172.21.15.153:0/4287026948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f08281031d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0818004940 tx=0x7f0818009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 -- 172.21.15.153:0/4287026948 shutdown_connections 2024-04-05T03:14:50.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 --2- 172.21.15.153:0/4287026948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 0x7f08280fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 --2- 172.21.15.153:0/4287026948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f08281031d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 -- 172.21.15.153:0/4287026948 >> 172.21.15.153:0/4287026948 conn(0x7f08280f89b0 msgr2=0x7f08280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 -- 172.21.15.153:0/4287026948 shutdown_connections 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.967+0000 7f083023b700 1 -- 172.21.15.153:0/4287026948 wait complete. 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 Processor -- start 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 -- start start 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 0x7f0828113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0828114520 con 0x7f0828102e00 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f083023b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082810db70 con 0x7f08281037a0 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49108/0 (socket says 172.21.15.153:49108) 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 -- 172.21.15.153:0/2079301314 learned_addr learned my addr 172.21.15.153:0/2079301314 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082d7d6700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 0x7f0828113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 -- 172.21.15.153:0/2079301314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 msgr2=0x7f0828113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 0x7f0828113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 -- 172.21.15.153:0/2079301314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0818005040 con 0x7f0828102e00 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.968+0000 7f082dfd7700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f081800b8b0 tx=0x7f0818009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.969+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081800e040 con 0x7f0828102e00 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.969+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0818004290 con 0x7f0828102e00 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.969+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082810ddf0 con 0x7f0828102e00 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.969+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081801c3f0 con 0x7f0828102e00 2024-04-05T03:14:50.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.969+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082810e3b0 con 0x7f0828102e00 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.970+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0828100110 con 0x7f0828102e00 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.970+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f081800baf0 con 0x7f0828102e00 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.970+0000 7f081e7fc700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 0x7f0814070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.971+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f081808e8d0 con 0x7f0828102e00 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.971+0000 7f082d7d6700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 0x7f0814070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:50.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.971+0000 7f082d7d6700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 0x7f0814070150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f082810ece0 tx=0x7f0824009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:50.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:50.975+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f081805ba00 con 0x7f0828102e00 2024-04-05T03:14:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:50 smithi203 ceph-mon[95988]: pgmap v275: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:51.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.252+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f082810f030 con 0x7f0828102e00 2024-04-05T03:14:51.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.252+0000 7f081e7fc700 1 -- 172.21.15.153:0/2079301314 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f081805b590 con 0x7f0828102e00 2024-04-05T03:14:51.251 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:14:51.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:14:51.253 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.254+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 msgr2=0x7f0814070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.254+0000 7f083023b700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 0x7f0814070150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f082810ece0 tx=0x7f0824009040 comp rx=0 tx=0).stop 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.254+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 msgr2=0x7f0828113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.254+0000 7f083023b700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f081800b8b0 tx=0x7f0818009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 shutdown_connections 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f081406dca0 0x7f0814070150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0828102e00 0x7f0828113990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 --2- 172.21.15.153:0/2079301314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08281037a0 0x7f0828113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 >> 172.21.15.153:0/2079301314 conn(0x7f08280f89b0 msgr2=0x7f08281073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:51.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 shutdown_connections 2024-04-05T03:14:51.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.255+0000 7f083023b700 1 -- 172.21.15.153:0/2079301314 wait complete. 2024-04-05T03:14:51.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 -- 172.21.15.153:0/3216482583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 msgr2=0x7f04a41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 --2- 172.21.15.153:0/3216482583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a41020f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f049c009ce0 tx=0x7f049c004d10 comp rx=0 tx=0).stop 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 -- 172.21.15.153:0/3216482583 shutdown_connections 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 --2- 172.21.15.153:0/3216482583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 --2- 172.21.15.153:0/3216482583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a41020f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 -- 172.21.15.153:0/3216482583 >> 172.21.15.153:0/3216482583 conn(0x7f04a40f78d0 msgr2=0x7f04a40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 -- 172.21.15.153:0/3216482583 shutdown_connections 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.356+0000 7f04ab19f700 1 -- 172.21.15.153:0/3216482583 wait complete. 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 Processor -- start 2024-04-05T03:14:51.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 -- start start 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a410f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a410f7a0 con 0x7f04a4101d20 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04ab19f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a410f8e0 con 0x7f04a41026c0 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49124/0 (socket says 172.21.15.153:49124) 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 -- 172.21.15.153:0/323774307 learned_addr learned my addr 172.21.15.153:0/323774307 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a3fff700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a410f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 -- 172.21.15.153:0/323774307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 msgr2=0x7f04a410f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a410f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a8f3b700 1 -- 172.21.15.153:0/323774307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c009940 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.357+0000 7f04a3fff700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a410f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04a8f3b700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f049c00ba50 tx=0x7f049c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f049c00e040 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a4117bd0 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f049c01cb20 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f049c012a70 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.358+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a4118160 con 0x7f04a4101d20 2024-04-05T03:14:51.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.359+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a40fd470 con 0x7f04a4101d20 2024-04-05T03:14:51.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.359+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f049c01cc90 con 0x7f04a4101d20 2024-04-05T03:14:51.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.360+0000 7f04a17fa700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 0x7f0490070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.360+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f049c08f930 con 0x7f04a4101d20 2024-04-05T03:14:51.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.360+0000 7f04a3fff700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 0x7f0490070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.360+0000 7f04a3fff700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 0x7f0490070150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0494004600 tx=0x7f049400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:51.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.364+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f049c058eb0 con 0x7f04a4101d20 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.604+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f04a4066660 con 0x7f04a4101d20 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.605+0000 7f04a17fa700 1 -- 172.21.15.153:0/323774307 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f049c05c570 con 0x7f04a4101d20 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:14:51.604 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:14:51.605 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:14:51.606 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:51.607 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 msgr2=0x7f0490070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 0x7f0490070150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0494004600 tx=0x7f049400b040 comp rx=0 tx=0).stop 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 msgr2=0x7f04a410ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f049c00ba50 tx=0x7f049c004d10 comp rx=0 tx=0).stop 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 shutdown_connections 2024-04-05T03:14:51.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f049006dca0 0x7f0490070150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04a4101d20 0x7f04a410ed20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 --2- 172.21.15.153:0/323774307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a41026c0 0x7f04a410f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 >> 172.21.15.153:0/323774307 conn(0x7f04a40f78d0 msgr2=0x7f04a40f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 shutdown_connections 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.607+0000 7f04ab19f700 1 -- 172.21.15.153:0/323774307 wait complete. 2024-04-05T03:14:51.609 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:14:51.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- 172.21.15.153:0/311768178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 msgr2=0x7fa8181022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/311768178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa8181022d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa808004940 tx=0x7fa808009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:51.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- 172.21.15.153:0/311768178 shutdown_connections 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/311768178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 0x7fa8180fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/311768178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa8181022d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- 172.21.15.153:0/311768178 >> 172.21.15.153:0/311768178 conn(0x7fa8180714c0 msgr2=0x7fa8180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- 172.21.15.153:0/311768178 shutdown_connections 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- 172.21.15.153:0/311768178 wait complete. 2024-04-05T03:14:51.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 Processor -- start 2024-04-05T03:14:51.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.710+0000 7fa81ea1a700 1 -- start start 2024-04-05T03:14:51.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa81ea1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 0x7fa8181128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa81ea1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa81ea1a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa818113430 con 0x7fa818101f00 2024-04-05T03:14:51.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa81ea1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81810ca80 con 0x7fa8180fc5e0 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49140/0 (socket says 172.21.15.153:49140) 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 -- 172.21.15.153:0/164578479 learned_addr learned my addr 172.21.15.153:0/164578479 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa817fff700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 0x7fa8181128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 -- 172.21.15.153:0/164578479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 msgr2=0x7fa8181128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 0x7fa8181128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 -- 172.21.15.153:0/164578479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808005040 con 0x7fa818101f00 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa8177fe700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00ead0 tx=0x7fa80c004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.711+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80c00cb90 con 0x7fa818101f00 2024-04-05T03:14:51.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.712+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa80c00ccf0 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.712+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80c010870 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.712+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81810cd00 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.712+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81810d240 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.713+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8180fd5f0 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.713+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa80c0109f0 con 0x7fa818101f00 2024-04-05T03:14:51.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.714+0000 7fa814ff9700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 0x7fa800070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:51.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.714+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa80c08fd80 con 0x7fa818101f00 2024-04-05T03:14:51.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.714+0000 7fa817fff700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 0x7fa800070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:51.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.714+0000 7fa817fff700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 0x7fa800070150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa808009fd0 tx=0x7fa808009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:51.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.718+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa80c05ceb0 con 0x7fa818101f00 2024-04-05T03:14:51.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.928+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa818112570 con 0x7fa80006dca0 2024-04-05T03:14:51.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:51 smithi153 ceph-mon[106102]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:51.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:51 smithi153 ceph-mon[106102]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:51.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:51 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2079301314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:14:51.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:51 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/323774307' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.930+0000 7fa814ff9700 1 -- 172.21.15.153:0/164578479 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa818112570 con 0x7fa80006dca0 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:14:51.929 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:14:51.930 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:14:51.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 msgr2=0x7fa800070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 0x7fa800070150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa808009fd0 tx=0x7fa808009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 msgr2=0x7fa818112de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00ead0 tx=0x7fa80c004500 comp rx=0 tx=0).stop 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 shutdown_connections 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa80006dca0 0x7fa800070150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8180fc5e0 0x7fa8181128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 --2- 172.21.15.153:0/164578479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa818101f00 0x7fa818112de0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.932+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 >> 172.21.15.153:0/164578479 conn(0x7fa8180714c0 msgr2=0x7fa8180f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.933+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 shutdown_connections 2024-04-05T03:14:51.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:51.933+0000 7fa81ea1a700 1 -- 172.21.15.153:0/164578479 wait complete. 2024-04-05T03:14:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- 172.21.15.153:0/2873617079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 msgr2=0x7fe53c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 --2- 172.21.15.153:0/2873617079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c1020b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe530004940 tx=0x7fe530009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- 172.21.15.153:0/2873617079 shutdown_connections 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 --2- 172.21.15.153:0/2873617079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 0x7fe53c0fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 --2- 172.21.15.153:0/2873617079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c1020b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- 172.21.15.153:0/2873617079 >> 172.21.15.153:0/2873617079 conn(0x7fe53c0f7870 msgr2=0x7fe53c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- 172.21.15.153:0/2873617079 shutdown_connections 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- 172.21.15.153:0/2873617079 wait complete. 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 Processor -- start 2024-04-05T03:14:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.035+0000 7fe5430a6700 1 -- start start 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe5430a6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe5430a6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 0x7fe53c112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe5430a6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c1133f0 con 0x7fe53c101ce0 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe5430a6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c10c9f0 con 0x7fe53c102680 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49148/0 (socket says 172.21.15.153:49148) 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 -- 172.21.15.153:0/778793137 learned_addr learned my addr 172.21.15.153:0/778793137 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe53bfff700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 0x7fe53c112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:52.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 -- 172.21.15.153:0/778793137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 msgr2=0x7fe53c112da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 0x7fe53c112da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 -- 172.21.15.153:0/778793137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530005040 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe540e42700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b3b0 tx=0x7fe530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.036+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53000e040 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.037+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe53000bbc0 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.037+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c10cc70 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.037+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53001c3f0 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.037+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c10d230 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.038+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c0fd430 con 0x7fe53c101ce0 2024-04-05T03:14:52.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.038+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe53000bd30 con 0x7fe53c101ce0 2024-04-05T03:14:52.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.038+0000 7fe5397fa700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 0x7fe52c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:14:52.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.038+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe53008e830 con 0x7fe53c101ce0 2024-04-05T03:14:52.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.038+0000 7fe53bfff700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 0x7fe52c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:14:52.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.039+0000 7fe53bfff700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 0x7fe52c070150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe53c10dbb0 tx=0x7fe528009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:14:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.043+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe53005b8e0 con 0x7fe53c101ce0 2024-04-05T03:14:52.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:51 smithi203 ceph-mon[95988]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:52.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:51 smithi203 ceph-mon[95988]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:52.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:51 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2079301314' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:14:52.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:51 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/323774307' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:14:52.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.317+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe53c108de0 con 0x7fe53c101ce0 2024-04-05T03:14:52.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.317+0000 7fe5397fa700 1 -- 172.21.15.153:0/778793137 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe53005b470 con 0x7fe53c101ce0 2024-04-05T03:14:52.317 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:14:52.317 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:14:52.317 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:14:52.317 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:14:52.317 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 msgr2=0x7fe52c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 0x7fe52c070150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe53c10dbb0 tx=0x7fe528009ee0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 msgr2=0x7fe53c112860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b3b0 tx=0x7fe530009e30 comp rx=0 tx=0).stop 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 shutdown_connections 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe52c06dca0 0x7fe52c070150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c101ce0 0x7fe53c112860 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 --2- 172.21.15.153:0/778793137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe53c102680 0x7fe53c112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 >> 172.21.15.153:0/778793137 conn(0x7fe53c0f7870 msgr2=0x7fe53c0f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:14:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 shutdown_connections 2024-04-05T03:14:52.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:14:52.320+0000 7fe5430a6700 1 -- 172.21.15.153:0/778793137 wait complete. 2024-04-05T03:14:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:52 smithi203 ceph-mon[95988]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:52 smithi203 ceph-mon[95988]: pgmap v276: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:53.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:52 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/778793137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:14:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:52 smithi153 ceph-mon[106102]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:52 smithi153 ceph-mon[106102]: pgmap v276: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:53.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:52 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/778793137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:14:54.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:53 smithi203 ceph-mon[95988]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:54.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:53 smithi153 ceph-mon[106102]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:14:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:54 smithi203 ceph-mon[95988]: pgmap v277: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:54 smithi153 ceph-mon[106102]: pgmap v277: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:56 smithi203 ceph-mon[95988]: pgmap v278: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:56 smithi153 ceph-mon[106102]: pgmap v278: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:14:58.900 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:14:58 smithi203 ceph-mon[95988]: pgmap v279: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:14:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:14:58 smithi153 ceph-mon[106102]: pgmap v279: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:00 smithi203 ceph-mon[95988]: pgmap v280: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:01.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:00 smithi153 ceph-mon[106102]: pgmap v280: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:03.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:02 smithi203 ceph-mon[95988]: pgmap v281: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:02 smithi153 ceph-mon[106102]: pgmap v281: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:04 smithi203 ceph-mon[95988]: pgmap v282: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:04 smithi153 ceph-mon[106102]: pgmap v282: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:06 smithi203 ceph-mon[95988]: pgmap v283: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:06 smithi153 ceph-mon[106102]: pgmap v283: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:08 smithi203 ceph-mon[95988]: pgmap v284: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:08 smithi153 ceph-mon[106102]: pgmap v284: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:11.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:10 smithi203 ceph-mon[95988]: pgmap v285: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:10 smithi153 ceph-mon[106102]: pgmap v285: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:12 smithi203 ceph-mon[95988]: pgmap v286: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:12 smithi153 ceph-mon[106102]: pgmap v286: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:13 smithi203 ceph-mon[95988]: pgmap v287: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:14.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:13 smithi153 ceph-mon[106102]: pgmap v287: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:16 smithi203 ceph-mon[95988]: pgmap v288: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:16 smithi153 ceph-mon[106102]: pgmap v288: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:18 smithi203 ceph-mon[95988]: pgmap v289: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:18 smithi153 ceph-mon[106102]: pgmap v289: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:20.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:20 smithi153 ceph-mon[106102]: pgmap v290: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:20 smithi203 ceph-mon[95988]: pgmap v290: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- 172.21.15.153:0/2305640199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4103780 msgr2=0x7f09b40fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2305640199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4103780 0x7f09b40fcdf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f09a4004940 tx=0x7f09a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- 172.21.15.153:0/2305640199 shutdown_connections 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2305640199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4103780 0x7f09b40fcdf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2305640199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4102de0 0x7f09b41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- 172.21.15.153:0/2305640199 >> 172.21.15.153:0/2305640199 conn(0x7f09b40f8950 msgr2=0x7f09b40fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- 172.21.15.153:0/2305640199 shutdown_connections 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- 172.21.15.153:0/2305640199 wait complete. 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 Processor -- start 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.434+0000 7f09b8acd700 1 -- start start 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b8acd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b8acd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4103780 0x7f09b4116010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b8acd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4116660 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b8acd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b410fc80 con 0x7f09b4103780 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56462/0 (socket says 172.21.15.153:56462) 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 -- 172.21.15.153:0/2452379677 learned_addr learned my addr 172.21.15.153:0/2452379677 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b1d9b700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4103780 0x7f09b4116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 -- 172.21.15.153:0/2452379677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4103780 msgr2=0x7f09b4116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4103780 0x7f09b4116010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 -- 172.21.15.153:0/2452379677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4005040 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b259c700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09a8002a80 tx=0x7f09a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a8007cc0 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b410ff60 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09a800f040 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.435+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a80096f0 con 0x7f09b4102de0 2024-04-05T03:15:22.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.436+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b4110520 con 0x7f09b4102de0 2024-04-05T03:15:22.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.437+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b40fe530 con 0x7f09b4102de0 2024-04-05T03:15:22.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.437+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f09a800d690 con 0x7f09b4102de0 2024-04-05T03:15:22.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.438+0000 7f09a2ffd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 0x7f099c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.438+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f09a808ccf0 con 0x7f09b4102de0 2024-04-05T03:15:22.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.438+0000 7f09b1d9b700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 0x7f099c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.438+0000 7f09b1d9b700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 0x7f099c070150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b3b0 tx=0x7f09a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:22.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.443+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f09a8056270 con 0x7f09b4102de0 2024-04-05T03:15:22.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.661+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09b4111160 con 0x7f099c06dca0 2024-04-05T03:15:22.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.662+0000 7f09a2ffd700 1 -- 172.21.15.153:0/2452379677 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f09b4111160 con 0x7f099c06dca0 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 msgr2=0x7f099c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 0x7f099c070150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b3b0 tx=0x7f09a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 msgr2=0x7f09b4115ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09a8002a80 tx=0x7f09a80046b0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 shutdown_connections 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f099c06dca0 0x7f099c070150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09b4102de0 0x7f09b4115ad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 --2- 172.21.15.153:0/2452379677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b4103780 0x7f09b4116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 >> 172.21.15.153:0/2452379677 conn(0x7f09b40f8950 msgr2=0x7f09b4071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:22.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 shutdown_connections 2024-04-05T03:15:22.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.664+0000 7f09b8acd700 1 -- 172.21.15.153:0/2452379677 wait complete. 2024-04-05T03:15:22.673 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- 172.21.15.153:0/3244029456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 msgr2=0x7f50d40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 --2- 172.21.15.153:0/3244029456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d40fce30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50c4004940 tx=0x7f50c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- 172.21.15.153:0/3244029456 shutdown_connections 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 --2- 172.21.15.153:0/3244029456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d40fce30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 --2- 172.21.15.153:0/3244029456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 0x7f50d41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- 172.21.15.153:0/3244029456 >> 172.21.15.153:0/3244029456 conn(0x7f50d40f89b0 msgr2=0x7f50d40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:22.766 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:22 smithi153 ceph-mon[106102]: pgmap v291: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:22.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- 172.21.15.153:0/3244029456 shutdown_connections 2024-04-05T03:15:22.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- 172.21.15.153:0/3244029456 wait complete. 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 Processor -- start 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.766+0000 7f50daab8700 1 -- start start 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50daab8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 0x7f50d40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50daab8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50daab8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4072210 con 0x7f50d41037c0 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50daab8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4070d40 con 0x7f50d4102e20 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56486/0 (socket says 172.21.15.153:56486) 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 -- 172.21.15.153:0/2931988643 learned_addr learned my addr 172.21.15.153:0/2931988643 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50d8854700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 0x7f50d40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 -- 172.21.15.153:0/2931988643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 msgr2=0x7f50d40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 0x7f50d40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 -- 172.21.15.153:0/2931988643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4005040 con 0x7f50d41037c0 2024-04-05T03:15:22.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.767+0000 7f50cbfff700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009100 tx=0x7f50c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:22.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.768+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c400e040 con 0x7f50d41037c0 2024-04-05T03:15:22.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.768+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d4070fc0 con 0x7f50d41037c0 2024-04-05T03:15:22.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.768+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50c4009af0 con 0x7f50d41037c0 2024-04-05T03:15:22.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.768+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c401c3f0 con 0x7f50d41037c0 2024-04-05T03:15:22.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.768+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d411a0f0 con 0x7f50d41037c0 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.769+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d41052f0 con 0x7f50d41037c0 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.769+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f50c400baf0 con 0x7f50d41037c0 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.769+0000 7f50c97fa700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 0x7f50bc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.770+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f50c408e790 con 0x7f50d41037c0 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.770+0000 7f50d8854700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 0x7f50bc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:22.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.770+0000 7f50d8854700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 0x7f50bc070150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50d0004eb0 tx=0x7f50d000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:22.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.774+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f50c4057d90 con 0x7f50d41037c0 2024-04-05T03:15:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.986+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50d4109f00 con 0x7f50bc06dca0 2024-04-05T03:15:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.987+0000 7f50c97fa700 1 -- 172.21.15.153:0/2931988643 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f50d4109f00 con 0x7f50bc06dca0 2024-04-05T03:15:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 msgr2=0x7f50bc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 0x7f50bc070150 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50d0004eb0 tx=0x7f50d000b040 comp rx=0 tx=0).stop 2024-04-05T03:15:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 msgr2=0x7f50d4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009100 tx=0x7f50c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 shutdown_connections 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f50bc06dca0 0x7f50bc070150 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50d4102e20 0x7f50d40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 --2- 172.21.15.153:0/2931988643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d41037c0 0x7f50d4070800 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 >> 172.21.15.153:0/2931988643 conn(0x7f50d40f89b0 msgr2=0x7f50d41021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 shutdown_connections 2024-04-05T03:15:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:22.989+0000 7f50daab8700 1 -- 172.21.15.153:0/2931988643 wait complete. 2024-04-05T03:15:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:22 smithi203 ceph-mon[95988]: pgmap v291: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 -- 172.21.15.153:0/1438519887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c102fd0 msgr2=0x7f597c1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 --2- 172.21.15.153:0/1438519887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c102fd0 0x7f597c1033a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f596c004940 tx=0x7f596c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 -- 172.21.15.153:0/1438519887 shutdown_connections 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 --2- 172.21.15.153:0/1438519887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c0fb540 0x7f597c0fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 --2- 172.21.15.153:0/1438519887 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c102fd0 0x7f597c1033a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 -- 172.21.15.153:0/1438519887 >> 172.21.15.153:0/1438519887 conn(0x7f597c071530 msgr2=0x7f597c071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:23.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.090+0000 7f59825f9700 1 -- 172.21.15.153:0/1438519887 shutdown_connections 2024-04-05T03:15:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 -- 172.21.15.153:0/1438519887 wait complete. 2024-04-05T03:15:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 Processor -- start 2024-04-05T03:15:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 -- start start 2024-04-05T03:15:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c102fd0 0x7f597c0fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c0fe670 con 0x7f597c0fb540 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f59825f9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c0fd1a0 con 0x7f597c102fd0 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.091+0000 7f597bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56512/0 (socket says 172.21.15.153:56512) 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 -- 172.21.15.153:0/2934574981 learned_addr learned my addr 172.21.15.153:0/2934574981 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597b7fe700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c102fd0 0x7f597c0fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 -- 172.21.15.153:0/2934574981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c102fd0 msgr2=0x7f597c0fcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c102fd0 0x7f597c0fcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 -- 172.21.15.153:0/2934574981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c005040 con 0x7f597c0fb540 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f597bfff700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f596c009150 tx=0x7f596c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c021070 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c0fd420 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f596c013c00 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c00e070 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.092+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c119ff0 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.093+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597c100270 con 0x7f597c0fb540 2024-04-05T03:15:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.094+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f596c01c400 con 0x7f597c0fb540 2024-04-05T03:15:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.094+0000 7f5978ff9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 0x7f5964070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.094+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f596c08e950 con 0x7f597c0fb540 2024-04-05T03:15:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.094+0000 7f597b7fe700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 0x7f5964070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.094+0000 7f597b7fe700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 0x7f5964070150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5970004600 tx=0x7f597000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.099+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f596c057ff0 con 0x7f597c0fb540 2024-04-05T03:15:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.302+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f597c0fe800 con 0x7f596406dca0 2024-04-05T03:15:23.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.312+0000 7f5978ff9700 1 -- 172.21.15.153:0/2934574981 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f597c0fe800 con 0x7f596406dca0 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (9m) 3m ago 21m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (7m) 3m ago 22m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (7m) 2m ago 20m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (9m) 3m ago 20m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (3m) 3m ago 15m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (3m) 3m ago 15m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (3m) 2m ago 15m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (3m) 2m ago 15m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (11m) 3m ago 23m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (10m) 2m ago 19m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 3m ago 23m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (8m) 2m ago 19m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:15:23.313 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (10m) 3m ago 21m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (10m) 2m ago 19m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (7m) 3m ago 19m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (6m) 3m ago 18m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (5m) 3m ago 18m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (5m) 2m ago 17m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (4m) 2m ago 17m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (4m) 2m ago 17m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:15:23.314 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (10m) 2m ago 19m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 msgr2=0x7f5964070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 0x7f5964070150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5970004600 tx=0x7f597000b040 comp rx=0 tx=0).stop 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 msgr2=0x7f597c0fc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f596c009150 tx=0x7f596c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 shutdown_connections 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f596406dca0 0x7f5964070150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f597c0fb540 0x7f597c0fc720 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 --2- 172.21.15.153:0/2934574981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f597c102fd0 0x7f597c0fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.314+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 >> 172.21.15.153:0/2934574981 conn(0x7f597c071530 msgr2=0x7f597c0fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.315+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 shutdown_connections 2024-04-05T03:15:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.315+0000 7f59825f9700 1 -- 172.21.15.153:0/2934574981 wait complete. 2024-04-05T03:15:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.413+0000 7faef0856700 1 -- 172.21.15.153:0/1058618180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fde10 msgr2=0x7faeec0fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.413+0000 7faef0856700 1 --2- 172.21.15.153:0/1058618180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fde10 0x7faeec0fe1e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faee0004940 tx=0x7faee0009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- 172.21.15.153:0/1058618180 shutdown_connections 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 --2- 172.21.15.153:0/1058618180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fe7b0 0x7faeec106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 --2- 172.21.15.153:0/1058618180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fde10 0x7faeec0fe1e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- 172.21.15.153:0/1058618180 >> 172.21.15.153:0/1058618180 conn(0x7faeec0f7870 msgr2=0x7faeec0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- 172.21.15.153:0/1058618180 shutdown_connections 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- 172.21.15.153:0/1058618180 wait complete. 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 Processor -- start 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- start start 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fde10 0x7faeec0fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0fdc20 con 0x7faeec0fe7b0 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faef0856700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0fc750 con 0x7faeec0fde10 2024-04-05T03:15:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faee9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faee9d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56542/0 (socket says 172.21.15.153:56542) 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.414+0000 7faee9d9b700 1 -- 172.21.15.153:0/1007618741 learned_addr learned my addr 172.21.15.153:0/1007618741 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faeea59c700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fde10 0x7faeec0fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faee9d9b700 1 -- 172.21.15.153:0/1007618741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fde10 msgr2=0x7faeec0fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faee9d9b700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fde10 0x7faeec0fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faee9d9b700 1 -- 172.21.15.153:0/1007618741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0005040 con 0x7faeec0fe7b0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faee9d9b700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faedc00eb40 tx=0x7faedc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faedc007cc0 con 0x7faeec0fe7b0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faedc00f040 con 0x7faeec0fe7b0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faedc00ca10 con 0x7faeec0fe7b0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec0fc9d0 con 0x7faeec0fe7b0 2024-04-05T03:15:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.415+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec11a090 con 0x7faeec0fe7b0 2024-04-05T03:15:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.416+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec066b10 con 0x7faeec0fe7b0 2024-04-05T03:15:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.417+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7faedc00cc80 con 0x7faeec0fe7b0 2024-04-05T03:15:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.417+0000 7faedaffd700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 0x7faed4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.417+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faedc08d140 con 0x7faeec0fe7b0 2024-04-05T03:15:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.417+0000 7faeea59c700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 0x7faed4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.418+0000 7faeea59c700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 0x7faed4070150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faee00045a0 tx=0x7faee0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.422+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faedc056770 con 0x7faeec0fe7b0 2024-04-05T03:15:23.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.699+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faeec11a280 con 0x7faeec0fe7b0 2024-04-05T03:15:23.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:23 smithi153 ceph-mon[106102]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:23.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.700+0000 7faedaffd700 1 -- 172.21.15.153:0/1007618741 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7faedc059e30 con 0x7faeec0fe7b0 2024-04-05T03:15:23.700 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:15:23.700 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:15:23.700 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:15:23.701 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:15:23.702 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:15:23.702 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:15:23.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 msgr2=0x7faed4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 0x7faed4070150 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faee00045a0 tx=0x7faee0009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 msgr2=0x7faeec0fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faedc00eb40 tx=0x7faedc00b040 comp rx=0 tx=0).stop 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 shutdown_connections 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faed406dca0 0x7faed4070150 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faeec0fde10 0x7faeec0fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 --2- 172.21.15.153:0/1007618741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faeec0fe7b0 0x7faeec0fc210 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 >> 172.21.15.153:0/1007618741 conn(0x7faeec0f7870 msgr2=0x7faeec0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 shutdown_connections 2024-04-05T03:15:23.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.702+0000 7faef0856700 1 -- 172.21.15.153:0/1007618741 wait complete. 2024-04-05T03:15:23.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 -- 172.21.15.153:0/3324951799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 msgr2=0x7faf2c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 --2- 172.21.15.153:0/3324951799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c103340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf28004940 tx=0x7faf28009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 -- 172.21.15.153:0/3324951799 shutdown_connections 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 --2- 172.21.15.153:0/3324951799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 0x7faf2c0fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 --2- 172.21.15.153:0/3324951799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c103340 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.803+0000 7faf334ba700 1 -- 172.21.15.153:0/3324951799 >> 172.21.15.153:0/3324951799 conn(0x7faf2c072310 msgr2=0x7faf2c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 -- 172.21.15.153:0/3324951799 shutdown_connections 2024-04-05T03:15:23.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 -- 172.21.15.153:0/3324951799 wait complete. 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 Processor -- start 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 -- start start 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 0x7faf2c115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1166a0 con 0x7faf2c102f70 2024-04-05T03:15:23.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf334ba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c10fc80 con 0x7faf2c0fb3b0 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf30a55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf30a55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56556/0 (socket says 172.21.15.153:56556) 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf30a55700 1 -- 172.21.15.153:0/738582061 learned_addr learned my addr 172.21.15.153:0/738582061 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.804+0000 7faf31256700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 0x7faf2c115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf30a55700 1 -- 172.21.15.153:0/738582061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 msgr2=0x7faf2c115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf30a55700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 0x7faf2c115b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf30a55700 1 -- 172.21.15.153:0/738582061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28005040 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf30a55700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00c830 tx=0x7faf1c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf1c0106f0 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf1c010d20 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf1c0113a0 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c10ff60 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.805+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c110520 con 0x7faf2c102f70 2024-04-05T03:15:23.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.806+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c101160 con 0x7faf2c102f70 2024-04-05T03:15:23.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.807+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7faf1c012400 con 0x7faf2c102f70 2024-04-05T03:15:23.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.807+0000 7faf21ffb700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 0x7faf18070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:23.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.807+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faf1c08d4d0 con 0x7faf2c102f70 2024-04-05T03:15:23.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.807+0000 7faf31256700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 0x7faf18070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:23.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.807+0000 7faf31256700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 0x7faf18070150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faf28004f90 tx=0x7faf28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:23.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:23.812+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faf1c05a600 con 0x7faf2c102f70 2024-04-05T03:15:24.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:23 smithi203 ceph-mon[95988]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:24.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.051+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faf2c111160 con 0x7faf2c102f70 2024-04-05T03:15:24.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.052+0000 7faf21ffb700 1 -- 172.21.15.153:0/738582061 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7faf1c05a190 con 0x7faf2c102f70 2024-04-05T03:15:24.052 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:15:24.052 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:15:24.053 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:15:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:15:24.055 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:24.056 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:24.057 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 msgr2=0x7faf18070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 0x7faf18070150 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faf28004f90 tx=0x7faf28009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 msgr2=0x7faf2c116050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00c830 tx=0x7faf1c004470 comp rx=0 tx=0).stop 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 shutdown_connections 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faf1806dca0 0x7faf18070150 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf2c0fb3b0 0x7faf2c115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 --2- 172.21.15.153:0/738582061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf2c102f70 0x7faf2c116050 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 >> 172.21.15.153:0/738582061 conn(0x7faf2c072310 msgr2=0x7faf2c0f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 shutdown_connections 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.055+0000 7faf334ba700 1 -- 172.21.15.153:0/738582061 wait complete. 2024-04-05T03:15:24.060 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/2032245613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 msgr2=0x7fd7a4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/2032245613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4103320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd794004940 tx=0x7fd794009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/2032245613 shutdown_connections 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/2032245613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 0x7fd7a40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/2032245613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4103320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/2032245613 >> 172.21.15.153:0/2032245613 conn(0x7fd7a40714c0 msgr2=0x7fd7a40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/2032245613 shutdown_connections 2024-04-05T03:15:24.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/2032245613 wait complete. 2024-04-05T03:15:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 Processor -- start 2024-04-05T03:15:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- start start 2024-04-05T03:15:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 0x7fd7a41139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4114530 con 0x7fd7a4102f50 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.158+0000 7fd7a8a4d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a410db30 con 0x7fd7a40fb4d0 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56574/0 (socket says 172.21.15.153:56574) 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 -- 172.21.15.153:0/1983183638 learned_addr learned my addr 172.21.15.153:0/1983183638 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a259c700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 0x7fd7a41139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 -- 172.21.15.153:0/1983183638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 msgr2=0x7fd7a41139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 0x7fd7a41139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 -- 172.21.15.153:0/1983183638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794005040 con 0x7fd7a4102f50 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a1d9b700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd798002a50 tx=0x7fd7980046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd798007cc0 con 0x7fd7a4102f50 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd79800f040 con 0x7fd7a4102f50 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a410de10 con 0x7fd7a4102f50 2024-04-05T03:15:24.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd79800da30 con 0x7fd7a4102f50 2024-04-05T03:15:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.159+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a410e350 con 0x7fd7a4102f50 2024-04-05T03:15:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.160+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a4100200 con 0x7fd7a4102f50 2024-04-05T03:15:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.161+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd79800db90 con 0x7fd7a4102f50 2024-04-05T03:15:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.161+0000 7fd792ffd700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 0x7fd78c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.161+0000 7fd7a259c700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 0x7fd78c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.161+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd79808d310 con 0x7fd7a4102f50 2024-04-05T03:15:24.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.162+0000 7fd7a259c700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 0x7fd78c070150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd794004890 tx=0x7fd794009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:24.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.166+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd79805a3c0 con 0x7fd7a4102f50 2024-04-05T03:15:24.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.377+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7a4113670 con 0x7fd78c06dca0 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.378+0000 7fd792ffd700 1 -- 172.21.15.153:0/1983183638 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd7a4113670 con 0x7fd78c06dca0 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:15:24.378 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:15:24.379 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:15:24.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 msgr2=0x7fd78c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 0x7fd78c070150 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd794004890 tx=0x7fd794009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:24.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 msgr2=0x7fd7a4113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd798002a50 tx=0x7fd7980046b0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 shutdown_connections 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd78c06dca0 0x7fd78c070150 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7a40fb4d0 0x7fd7a41139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 --2- 172.21.15.153:0/1983183638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7a4102f50 0x7fd7a4113ee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 >> 172.21.15.153:0/1983183638 conn(0x7fd7a40714c0 msgr2=0x7fd7a40fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 shutdown_connections 2024-04-05T03:15:24.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.380+0000 7fd7a8a4d700 1 -- 172.21.15.153:0/1983183638 wait complete. 2024-04-05T03:15:24.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1588646452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fde10 msgr2=0x7fa4280fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1588646452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fde10 0x7fa4280fe1e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa418004940 tx=0x7fa418009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1588646452 shutdown_connections 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1588646452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fe7b0 0x7fa428106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1588646452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fde10 0x7fa4280fe1e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1588646452 >> 172.21.15.153:0/1588646452 conn(0x7fa4280f7870 msgr2=0x7fa4280f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1588646452 shutdown_connections 2024-04-05T03:15:24.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.480+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1588646452 wait complete. 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 Processor -- start 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 -- start start 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fde10 0x7fa428119f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42811abb0 con 0x7fa4280fe7b0 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42fa3e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa428114100 con 0x7fa4280fde10 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42cfd9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42cfd9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56586/0 (socket says 172.21.15.153:56586) 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42cfd9700 1 -- 172.21.15.153:0/1372203135 learned_addr learned my addr 172.21.15.153:0/1372203135 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:24.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.481+0000 7fa42d7da700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fde10 0x7fa428119f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42cfd9700 1 -- 172.21.15.153:0/1372203135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fde10 msgr2=0x7fa428119f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42cfd9700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fde10 0x7fa428119f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42cfd9700 1 -- 172.21.15.153:0/1372203135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa418005040 con 0x7fa4280fe7b0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42cfd9700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b790 tx=0x7fa424009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa42400f6f0 con 0x7fa4280fe7b0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa428114380 con 0x7fa4280fe7b0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa42400fd20 con 0x7fa4280fe7b0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4240103a0 con 0x7fa4280fe7b0 2024-04-05T03:15:24.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.482+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4281148c0 con 0x7fa4280fe7b0 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.483+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa428102530 con 0x7fa4280fe7b0 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.484+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa424011400 con 0x7fa4280fe7b0 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.484+0000 7fa41dffb700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 0x7fa414070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.484+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa42408d380 con 0x7fa4280fe7b0 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.484+0000 7fa42d7da700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 0x7fa414070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:24.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.484+0000 7fa42d7da700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 0x7fa414070150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa418004f90 tx=0x7fa418009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:24.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.489+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa42405a4b0 con 0x7fa4280fe7b0 2024-04-05T03:15:24.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.764+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa428109f20 con 0x7fa4280fe7b0 2024-04-05T03:15:24.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:24 smithi153 ceph-mon[106102]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:24.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:24 smithi153 ceph-mon[106102]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:24.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:24 smithi153 ceph-mon[106102]: pgmap v292: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:24.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:24 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1007618741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:15:24.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:24 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/738582061' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.765+0000 7fa41dffb700 1 -- 172.21.15.153:0/1372203135 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa42405a040 con 0x7fa4280fe7b0 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:15:24.766 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 msgr2=0x7fa414070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 0x7fa414070150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa418004f90 tx=0x7fa418009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 msgr2=0x7fa42811a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b790 tx=0x7fa424009250 comp rx=0 tx=0).stop 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 shutdown_connections 2024-04-05T03:15:24.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa41406dca0 0x7fa414070150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4280fde10 0x7fa428119f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 --2- 172.21.15.153:0/1372203135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4280fe7b0 0x7fa42811a4d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:24.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 >> 172.21.15.153:0/1372203135 conn(0x7fa4280f7870 msgr2=0x7fa4280fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:24.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 shutdown_connections 2024-04-05T03:15:24.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:24.767+0000 7fa42fa3e700 1 -- 172.21.15.153:0/1372203135 wait complete. 2024-04-05T03:15:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:24 smithi203 ceph-mon[95988]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:24 smithi203 ceph-mon[95988]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:24 smithi203 ceph-mon[95988]: pgmap v292: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:24 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1007618741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:15:25.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:24 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/738582061' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:15:26.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:25 smithi203 ceph-mon[95988]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:26.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:25 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1372203135' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:15:26.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:25 smithi153 ceph-mon[106102]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:26.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:25 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1372203135' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:15:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:26 smithi203 ceph-mon[95988]: pgmap v293: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:26 smithi153 ceph-mon[106102]: pgmap v293: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:28 smithi203 ceph-mon[95988]: pgmap v294: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:28 smithi153 ceph-mon[106102]: pgmap v294: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:30.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:30 smithi153 ceph-mon[106102]: pgmap v295: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:30 smithi203 ceph-mon[95988]: pgmap v295: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:32 smithi203 ceph-mon[95988]: pgmap v296: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:32 smithi153 ceph-mon[106102]: pgmap v296: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:34 smithi203 ceph-mon[95988]: pgmap v297: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:34 smithi153 ceph-mon[106102]: pgmap v297: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:36 smithi203 ceph-mon[95988]: pgmap v298: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:36 smithi153 ceph-mon[106102]: pgmap v298: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:39.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:38 smithi203 ceph-mon[95988]: pgmap v299: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:38 smithi153 ceph-mon[106102]: pgmap v299: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:40 smithi203 ceph-mon[95988]: pgmap v300: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:40 smithi153 ceph-mon[106102]: pgmap v300: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:42 smithi203 ceph-mon[95988]: pgmap v301: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:42 smithi153 ceph-mon[106102]: pgmap v301: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:44 smithi203 ceph-mon[95988]: pgmap v302: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:44 smithi153 ceph-mon[106102]: pgmap v302: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:46 smithi203 ceph-mon[95988]: pgmap v303: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:46 smithi153 ceph-mon[106102]: pgmap v303: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:48 smithi203 ceph-mon[95988]: pgmap v304: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:48 smithi153 ceph-mon[106102]: pgmap v304: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:50 smithi203 ceph-mon[95988]: pgmap v305: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:50 smithi153 ceph-mon[106102]: pgmap v305: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:52 smithi203 ceph-mon[95988]: pgmap v306: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:52 smithi153 ceph-mon[106102]: pgmap v306: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 -- 172.21.15.153:0/303758481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 msgr2=0x7f70d4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 --2- 172.21.15.153:0/303758481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d4102140 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f70c4004940 tx=0x7f70c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 -- 172.21.15.153:0/303758481 shutdown_connections 2024-04-05T03:15:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 --2- 172.21.15.153:0/303758481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 0x7f70d40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:54.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 --2- 172.21.15.153:0/303758481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d4102140 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 -- 172.21.15.153:0/303758481 >> 172.21.15.153:0/303758481 conn(0x7f70d4072bc0 msgr2=0x7f70d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 -- 172.21.15.153:0/303758481 shutdown_connections 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.883+0000 7f70d9d72700 1 -- 172.21.15.153:0/303758481 wait complete. 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 Processor -- start 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 -- start start 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 0x7f70d410ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d410d350 con 0x7f70d4101d70 2024-04-05T03:15:54.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d9d72700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d410d4c0 con 0x7f70d4102710 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46980/0 (socket says 172.21.15.153:46980) 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70d37fe700 1 -- 172.21.15.153:0/2421185310 learned_addr learned my addr 172.21.15.153:0/2421185310 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.884+0000 7f70caffd700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 0x7f70d410ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d37fe700 1 -- 172.21.15.153:0/2421185310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 msgr2=0x7f70d410ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d37fe700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 0x7f70d410ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d37fe700 1 -- 172.21.15.153:0/2421185310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4005040 con 0x7f70d4101d70 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d37fe700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f70c4009380 tx=0x7f70c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c4021070 con 0x7f70d4101d70 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70c4013cd0 con 0x7f70d4101d70 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d410d740 con 0x7f70d4101d70 2024-04-05T03:15:54.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70c400e070 con 0x7f70d4101d70 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.885+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d411a240 con 0x7f70d4101d70 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.886+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d40ff080 con 0x7f70d4101d70 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.887+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f70c401ca40 con 0x7f70d4101d70 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.887+0000 7f70d0ff9700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 0x7f70c0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.887+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f70c408e830 con 0x7f70d4101d70 2024-04-05T03:15:54.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.887+0000 7f70caffd700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 0x7f70c0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:54.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.888+0000 7f70caffd700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 0x7f70c0070150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70d4066000 tx=0x7f70bc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:54.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:54.892+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f70c405b960 con 0x7f70d4101d70 2024-04-05T03:15:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:54 smithi203 ceph-mon[95988]: pgmap v307: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:54 smithi153 ceph-mon[106102]: pgmap v307: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:55.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.107+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f70d410d740 con 0x7f70c006dca0 2024-04-05T03:15:55.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.108+0000 7f70d0ff9700 1 -- 172.21.15.153:0/2421185310 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f70d410d740 con 0x7f70c006dca0 2024-04-05T03:15:55.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 msgr2=0x7f70c0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 0x7f70c0070150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70d4066000 tx=0x7f70bc00b040 comp rx=0 tx=0).stop 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 msgr2=0x7f70d410c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f70c4009380 tx=0x7f70c4009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 shutdown_connections 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f70c006dca0 0x7f70c0070150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70d4101d70 0x7f70d410c8d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 --2- 172.21.15.153:0/2421185310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70d4102710 0x7f70d410ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 >> 172.21.15.153:0/2421185310 conn(0x7f70d4072bc0 msgr2=0x7f70d40faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 shutdown_connections 2024-04-05T03:15:55.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.110+0000 7f70d9d72700 1 -- 172.21.15.153:0/2421185310 wait complete. 2024-04-05T03:15:55.120 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:15:55.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- 172.21.15.153:0/1138493710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070340 msgr2=0x7f9eb4070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1138493710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070340 0x7f9eb4070710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009ce0 tx=0x7f9eac004d10 comp rx=0 tx=0).stop 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- 172.21.15.153:0/1138493710 shutdown_connections 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1138493710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070c50 0x7f9eb4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1138493710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070340 0x7f9eb4070710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- 172.21.15.153:0/1138493710 >> 172.21.15.153:0/1138493710 conn(0x7f9eb40f89f0 msgr2=0x7f9eb40fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- 172.21.15.153:0/1138493710 shutdown_connections 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- 172.21.15.153:0/1138493710 wait complete. 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 Processor -- start 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.213+0000 7f9ebb855700 1 -- start start 2024-04-05T03:15:55.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9ebb855700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070340 0x7f9eb4115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9ebb855700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9ebb855700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb4116980 con 0x7f9eb4070c50 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9ebb855700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb410fed0 con 0x7f9eb4070340 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47000/0 (socket says 172.21.15.153:47000) 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 -- 172.21.15.153:0/1917544424 learned_addr learned my addr 172.21.15.153:0/1917544424 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb95f1700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070340 0x7f9eb4115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 -- 172.21.15.153:0/1917544424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070340 msgr2=0x7f9eb4115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070340 0x7f9eb4115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 -- 172.21.15.153:0/1917544424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac009940 con 0x7f9eb4070c50 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb8df0700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800b790 tx=0x7f9ea8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea800f770 con 0x7f9eb4070c50 2024-04-05T03:15:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb4110150 con 0x7f9eb4070c50 2024-04-05T03:15:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ea800fda0 con 0x7f9eb4070c50 2024-04-05T03:15:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.214+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea800d310 con 0x7f9eb4070c50 2024-04-05T03:15:55.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.215+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb4110690 con 0x7f9eb4070c50 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.216+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb4071a80 con 0x7f9eb4070c50 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.216+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9ea8010420 con 0x7f9eb4070c50 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.216+0000 7f9eb1ffb700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 0x7f9e9c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.216+0000 7f9eb95f1700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 0x7f9e9c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.216+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9ea808d5e0 con 0x7f9eb4070c50 2024-04-05T03:15:55.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.217+0000 7f9eb95f1700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 0x7f9e9c070150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00b740 tx=0x7f9eac004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.221+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9ea805a710 con 0x7f9eb4070c50 2024-04-05T03:15:55.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.433+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eb410c1f0 con 0x7f9e9c06dca0 2024-04-05T03:15:55.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.434+0000 7f9eb1ffb700 1 -- 172.21.15.153:0/1917544424 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9eb410c1f0 con 0x7f9e9c06dca0 2024-04-05T03:15:55.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 msgr2=0x7f9e9c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 0x7f9e9c070150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00b740 tx=0x7f9eac004d10 comp rx=0 tx=0).stop 2024-04-05T03:15:55.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 msgr2=0x7f9eb41162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800b790 tx=0x7f9ea8009250 comp rx=0 tx=0).stop 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 shutdown_connections 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9e9c06dca0 0x7f9e9c070150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9eb4070340 0x7f9eb4115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 --2- 172.21.15.153:0/1917544424 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9eb4070c50 0x7f9eb41162a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 >> 172.21.15.153:0/1917544424 conn(0x7f9eb40f89f0 msgr2=0x7f9eb40f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 shutdown_connections 2024-04-05T03:15:55.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.436+0000 7f9ebb855700 1 -- 172.21.15.153:0/1917544424 wait complete. 2024-04-05T03:15:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 -- 172.21.15.153:0/2963496263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 msgr2=0x7f6130102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 --2- 172.21.15.153:0/2963496263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f6130102140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6124004940 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:55.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 -- 172.21.15.153:0/2963496263 shutdown_connections 2024-04-05T03:15:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 --2- 172.21.15.153:0/2963496263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 0x7f61300fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 --2- 172.21.15.153:0/2963496263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f6130102140 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 -- 172.21.15.153:0/2963496263 >> 172.21.15.153:0/2963496263 conn(0x7f6130072bc0 msgr2=0x7f6130072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 -- 172.21.15.153:0/2963496263 shutdown_connections 2024-04-05T03:15:55.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.539+0000 7f61348cd700 1 -- 172.21.15.153:0/2963496263 wait complete. 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 Processor -- start 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 -- start start 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 0x7f613010f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613010f700 con 0x7f6130101d70 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f61348cd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613010f840 con 0x7f6130102710 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47018/0 (socket says 172.21.15.153:47018) 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 -- 172.21.15.153:0/2253606745 learned_addr learned my addr 172.21.15.153:0/2253606745 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612dd9b700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 0x7f613010f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 -- 172.21.15.153:0/2253606745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 msgr2=0x7f613010f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 0x7f613010f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.540+0000 7f612e59c700 1 -- 172.21.15.153:0/2253606745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124005040 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f612e59c700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f612400b3b0 tx=0x7f6124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612400e040 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6130117b30 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61301180c0 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f612400bbc0 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.541+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f612401c3f0 con 0x7f6130101d70 2024-04-05T03:15:55.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.542+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61100052e0 con 0x7f6130101d70 2024-04-05T03:15:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.543+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f612400bd30 con 0x7f6130101d70 2024-04-05T03:15:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.543+0000 7f611effd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 0x7f6118070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.543+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f612408e700 con 0x7f6130101d70 2024-04-05T03:15:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.543+0000 7f612dd9b700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 0x7f6118070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.543+0000 7f612dd9b700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 0x7f6118070150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6130066020 tx=0x7f6120009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.549+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6124057d00 con 0x7f6130101d70 2024-04-05T03:15:55.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.752+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6110000bf0 con 0x7f611806dca0 2024-04-05T03:15:55.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.762+0000 7f611effd700 1 -- 172.21.15.153:0/2253606745 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f6110000bf0 con 0x7f611806dca0 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (10m) 4m ago 21m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 4m ago 22m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (7m) 3m ago 20m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (10m) 4m ago 21m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (4m) 4m ago 16m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (4m) 4m ago 16m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (3m) 3m ago 16m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (3m) 3m ago 16m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (12m) 4m ago 23m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:15:55.762 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (11m) 3m ago 20m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 4m ago 24m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (8m) 3m ago 20m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (11m) 4m ago 21m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (11m) 3m ago 20m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (7m) 4m ago 19m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (7m) 4m ago 19m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 4m ago 18m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (5m) 3m ago 18m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (5m) 3m ago 18m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (4m) 3m ago 17m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:15:55.763 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (10m) 3m ago 20m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:15:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 msgr2=0x7f6118070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 0x7f6118070150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6130066020 tx=0x7f6120009040 comp rx=0 tx=0).stop 2024-04-05T03:15:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 msgr2=0x7f613010ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f612400b3b0 tx=0x7f6124009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:55.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 shutdown_connections 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f611806dca0 0x7f6118070150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6130101d70 0x7f613010ec80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 --2- 172.21.15.153:0/2253606745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6130102710 0x7f613010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 >> 172.21.15.153:0/2253606745 conn(0x7f6130072bc0 msgr2=0x7f61300fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 shutdown_connections 2024-04-05T03:15:55.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.765+0000 7f61348cd700 1 -- 172.21.15.153:0/2253606745 wait complete. 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.864+0000 7f8729302700 1 -- 172.21.15.153:0/2014038690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87240fc4c0 msgr2=0x7f87240fc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.864+0000 7f8729302700 1 --2- 172.21.15.153:0/2014038690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87240fc4c0 0x7f87240fc930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f870c004940 tx=0x7f870c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 -- 172.21.15.153:0/2014038690 shutdown_connections 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 --2- 172.21.15.153:0/2014038690 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87240fc4c0 0x7f87240fc930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 --2- 172.21.15.153:0/2014038690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8724101db0 0x7f8724102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 -- 172.21.15.153:0/2014038690 >> 172.21.15.153:0/2014038690 conn(0x7f8724072310 msgr2=0x7f8724072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:55.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 -- 172.21.15.153:0/2014038690 shutdown_connections 2024-04-05T03:15:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 -- 172.21.15.153:0/2014038690 wait complete. 2024-04-05T03:15:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 Processor -- start 2024-04-05T03:15:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.865+0000 7f8729302700 1 -- start start 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87240fc4c0 0x7f872410eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872410f620 con 0x7f8724101db0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872410f760 con 0x7f87240fc4c0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47024/0 (socket says 172.21.15.153:47024) 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 -- 172.21.15.153:0/2822271491 learned_addr learned my addr 172.21.15.153:0/2822271491 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8722ffd700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87240fc4c0 0x7f872410eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 -- 172.21.15.153:0/2822271491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87240fc4c0 msgr2=0x7f872410eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87240fc4c0 0x7f872410eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 -- 172.21.15.153:0/2822271491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c005040 con 0x7f8724101db0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f87227fc700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b3b0 tx=0x7f870c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f870c00e040 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872410f9e0 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f870c013420 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f870c01c630 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.866+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8724117e90 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.867+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87240ff090 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.868+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f870c01c8b0 con 0x7f8724101db0 2024-04-05T03:15:55.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.868+0000 7f871b7fe700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 0x7f8710070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:55.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.868+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f870c08e720 con 0x7f8724101db0 2024-04-05T03:15:55.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.868+0000 7f8722ffd700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 0x7f8710070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:55.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.869+0000 7f8722ffd700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 0x7f8710070150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f871c004fd0 tx=0x7f871c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:55.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:55.873+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f870c057d20 con 0x7f8724101db0 2024-04-05T03:15:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.150+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f87240fd4d0 con 0x7f8724101db0 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.151+0000 7f871b7fe700 1 -- 172.21.15.153:0/2822271491 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f870c05b3e0 con 0x7f8724101db0 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:56.150 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:15:56.151 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 msgr2=0x7f8710070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 0x7f8710070150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f871c004fd0 tx=0x7f871c009040 comp rx=0 tx=0).stop 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 msgr2=0x7f872410f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b3b0 tx=0x7f870c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 shutdown_connections 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f871006dca0 0x7f8710070150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87240fc4c0 0x7f872410eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 --2- 172.21.15.153:0/2822271491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8724101db0 0x7f872410f0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 >> 172.21.15.153:0/2822271491 conn(0x7f8724072310 msgr2=0x7f87241062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 shutdown_connections 2024-04-05T03:15:56.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.153+0000 7f8729302700 1 -- 172.21.15.153:0/2822271491 wait complete. 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 -- 172.21.15.153:0/2015153643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 msgr2=0x7fb1440fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2015153643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1440fbd20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb134004940 tx=0x7fb134009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 -- 172.21.15.153:0/2015153643 shutdown_connections 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2015153643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1440fbd20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2015153643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 0x7fb1441020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 -- 172.21.15.153:0/2015153643 >> 172.21.15.153:0/2015153643 conn(0x7fb1440f7880 msgr2=0x7fb1440f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 -- 172.21.15.153:0/2015153643 shutdown_connections 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.253+0000 7fb148ab8700 1 -- 172.21.15.153:0/2015153643 wait complete. 2024-04-05T03:15:56.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 Processor -- start 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 -- start start 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 0x7fb144112db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144113940 con 0x7fb1441026b0 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb148ab8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14410cd10 con 0x7fb144101d10 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47034/0 (socket says 172.21.15.153:47034) 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 -- 172.21.15.153:0/2178635745 learned_addr learned my addr 172.21.15.153:0/2178635745 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:56.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb14259c700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 0x7fb144112db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 -- 172.21.15.153:0/2178635745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 msgr2=0x7fb144112db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 0x7fb144112db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 -- 172.21.15.153:0/2178635745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb134005040 con 0x7fb1441026b0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.254+0000 7fb141d9b700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb134009100 tx=0x7fb134009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.255+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13400e040 con 0x7fb1441026b0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.255+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb134009af0 con 0x7fb1441026b0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.255+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14410cf10 con 0x7fb1441026b0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.255+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13401c3f0 con 0x7fb1441026b0 2024-04-05T03:15:56.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.255+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14410d4d0 con 0x7fb1441026b0 2024-04-05T03:15:56.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.256+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1440fd460 con 0x7fb1441026b0 2024-04-05T03:15:56.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.256+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb13400baf0 con 0x7fb1441026b0 2024-04-05T03:15:56.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.256+0000 7fb132ffd700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 0x7fb12c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.257+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb13408e700 con 0x7fb1441026b0 2024-04-05T03:15:56.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.257+0000 7fb14259c700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 0x7fb12c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.257+0000 7fb14259c700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 0x7fb12c070150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb138004eb0 tx=0x7fb13800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.261+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb13405b8d0 con 0x7fb1441026b0 2024-04-05T03:15:56.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.501+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb14405f5f0 con 0x7fb1441026b0 2024-04-05T03:15:56.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.502+0000 7fb132ffd700 1 -- 172.21.15.153:0/2178635745 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fb13405b460 con 0x7fb1441026b0 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:15:56.501 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:15:56.502 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:56.503 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:56.504 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 msgr2=0x7fb12c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 0x7fb12c070150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb138004eb0 tx=0x7fb13800b040 comp rx=0 tx=0).stop 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 msgr2=0x7fb1441132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb134009100 tx=0x7fb134009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 shutdown_connections 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb12c06dca0 0x7fb12c070150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb144101d10 0x7fb144112db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 --2- 172.21.15.153:0/2178635745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1441026b0 0x7fb1441132f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 >> 172.21.15.153:0/2178635745 conn(0x7fb1440f7880 msgr2=0x7fb1440f8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 shutdown_connections 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.505+0000 7fb148ab8700 1 -- 172.21.15.153:0/2178635745 wait complete. 2024-04-05T03:15:56.505 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:15:56.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.608+0000 7f155e654700 1 -- 172.21.15.153:0/2571102625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 msgr2=0x7f15580fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.608+0000 7f155e654700 1 --2- 172.21.15.153:0/2571102625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f15580fce10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f154c004940 tx=0x7f154c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 -- 172.21.15.153:0/2571102625 shutdown_connections 2024-04-05T03:15:56.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 --2- 172.21.15.153:0/2571102625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f15580fce10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 --2- 172.21.15.153:0/2571102625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 0x7f15581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 -- 172.21.15.153:0/2571102625 >> 172.21.15.153:0/2571102625 conn(0x7f15580f89b0 msgr2=0x7f15580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 -- 172.21.15.153:0/2571102625 shutdown_connections 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 -- 172.21.15.153:0/2571102625 wait complete. 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 Processor -- start 2024-04-05T03:15:56.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 -- start start 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 0x7f1558113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.609+0000 7f155e654700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f155e654700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15581145b0 con 0x7f15581037a0 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f155e654700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155810db40 con 0x7f1558102e00 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47048/0 (socket says 172.21.15.153:47048) 2024-04-05T03:15:56.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 -- 172.21.15.153:0/2503289838 learned_addr learned my addr 172.21.15.153:0/2503289838 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f1557fff700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 0x7f1558113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 -- 172.21.15.153:0/2503289838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 msgr2=0x7f1558113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 0x7f1558113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 -- 172.21.15.153:0/2503289838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c005040 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f15577fe700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f154c009100 tx=0x7f154c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f154c00e040 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f154c013420 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155810ddc0 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f154c01c630 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.610+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155810e380 con 0x7f15581037a0 2024-04-05T03:15:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.611+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1558100110 con 0x7f15581037a0 2024-04-05T03:15:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.612+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f154c013590 con 0x7f15581037a0 2024-04-05T03:15:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.612+0000 7f1554ff9700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 0x7f1540070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.612+0000 7f1557fff700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 0x7f1540070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.612+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f154c08e690 con 0x7f15581037a0 2024-04-05T03:15:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.613+0000 7f1557fff700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 0x7f1540070150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.617+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f154c057d30 con 0x7f15581037a0 2024-04-05T03:15:56.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:56 smithi153 ceph-mon[106102]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:56.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:56 smithi153 ceph-mon[106102]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:56.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:56 smithi153 ceph-mon[106102]: pgmap v308: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:56 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2822271491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:15:56.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:56 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2178635745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:15:56.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.832+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1558109f50 con 0x7f154006dca0 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.833+0000 7f1554ff9700 1 -- 172.21.15.153:0/2503289838 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1558109f50 con 0x7f154006dca0 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:15:56.832 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:15:56.833 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:15:56.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 msgr2=0x7f1540070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 0x7f1540070150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548008040 comp rx=0 tx=0).stop 2024-04-05T03:15:56.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 msgr2=0x7f1558113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f154c009100 tx=0x7f154c009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 shutdown_connections 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f154006dca0 0x7f1540070150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1558102e00 0x7f1558113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 --2- 172.21.15.153:0/2503289838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f15581037a0 0x7f1558113ed0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 >> 172.21.15.153:0/2503289838 conn(0x7f15580f89b0 msgr2=0x7f15580fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 shutdown_connections 2024-04-05T03:15:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.835+0000 7f155e654700 1 -- 172.21.15.153:0/2503289838 wait complete. 2024-04-05T03:15:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 -- 172.21.15.153:0/1350800162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 msgr2=0x7fe6681020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 --2- 172.21.15.153:0/1350800162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe6681020e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe658004940 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 -- 172.21.15.153:0/1350800162 shutdown_connections 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 --2- 172.21.15.153:0/1350800162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 0x7fe6680fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 --2- 172.21.15.153:0/1350800162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe6681020e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 -- 172.21.15.153:0/1350800162 >> 172.21.15.153:0/1350800162 conn(0x7fe6680f7880 msgr2=0x7fe6680f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 -- 172.21.15.153:0/1350800162 shutdown_connections 2024-04-05T03:15:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.936+0000 7fe670119700 1 -- 172.21.15.153:0/1350800162 wait complete. 2024-04-05T03:15:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 Processor -- start 2024-04-05T03:15:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 -- start start 2024-04-05T03:15:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 0x7fe6681164e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe668116b30 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe670119700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810ff40 con 0x7fe6681026b0 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe66deb5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe66deb5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47078/0 (socket says 172.21.15.153:47078) 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe66deb5700 1 -- 172.21.15.153:0/3604930470 learned_addr learned my addr 172.21.15.153:0/3604930470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.937+0000 7fe66d6b4700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 0x7fe6681164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe66deb5700 1 -- 172.21.15.153:0/3604930470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 msgr2=0x7fe6681164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe66deb5700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 0x7fe6681164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe66deb5700 1 -- 172.21.15.153:0/3604930470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658005040 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe66deb5700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe658009b30 tx=0x7fe658009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe658021070 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe658013c00 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6681101c0 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65800e070 con 0x7fe668101d10 2024-04-05T03:15:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.938+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681106a0 con 0x7fe668101d10 2024-04-05T03:15:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.939+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6680fd460 con 0x7fe668101d10 2024-04-05T03:15:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.939+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe65801cad0 con 0x7fe668101d10 2024-04-05T03:15:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.940+0000 7fe65e7fc700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 0x7fe654070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:15:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.940+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe65808e9d0 con 0x7fe668101d10 2024-04-05T03:15:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.940+0000 7fe66d6b4700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 0x7fe654070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:15:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.940+0000 7fe66d6b4700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 0x7fe654070150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe668111880 tx=0x7fe66400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:15:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:56.944+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe658057fd0 con 0x7fe668101d10 2024-04-05T03:15:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:56 smithi203 ceph-mon[95988]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:56 smithi203 ceph-mon[95988]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:56 smithi203 ceph-mon[95988]: pgmap v308: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:56 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2822271491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:15:57.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:56 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2178635745' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:15:57.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.218+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe66804bed0 con 0x7fe668101d10 2024-04-05T03:15:57.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.219+0000 7fe65e7fc700 1 -- 172.21.15.153:0/3604930470 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe658004490 con 0x7fe668101d10 2024-04-05T03:15:57.218 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:15:57.218 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:15:57.218 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:15:57.219 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:15:57.219 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 msgr2=0x7fe654070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 0x7fe654070150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe668111880 tx=0x7fe66400b040 comp rx=0 tx=0).stop 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 msgr2=0x7fe668115fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe658009b30 tx=0x7fe658009e30 comp rx=0 tx=0).stop 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 shutdown_connections 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe65406dca0 0x7fe654070150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe668101d10 0x7fe668115fa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:57.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 --2- 172.21.15.153:0/3604930470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6681026b0 0x7fe6681164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:15:57.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 >> 172.21.15.153:0/3604930470 conn(0x7fe6680f7880 msgr2=0x7fe6680f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:15:57.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 shutdown_connections 2024-04-05T03:15:57.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:15:57.221+0000 7fe670119700 1 -- 172.21.15.153:0/3604930470 wait complete. 2024-04-05T03:15:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:57 smithi203 ceph-mon[95988]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:57 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3604930470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:15:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:57 smithi153 ceph-mon[106102]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:57 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3604930470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:15:58.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:15:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:58 smithi203 ceph-mon[95988]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:59.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:58 smithi203 ceph-mon[95988]: pgmap v309: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:15:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:58 smithi153 ceph-mon[106102]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:15:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:58 smithi153 ceph-mon[106102]: pgmap v309: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:00.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:15:59 smithi203 ceph-mon[95988]: pgmap v310: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:00.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:15:59 smithi153 ceph-mon[106102]: pgmap v310: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:02 smithi203 ceph-mon[95988]: pgmap v311: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:02 smithi153 ceph-mon[106102]: pgmap v311: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:04 smithi203 ceph-mon[95988]: pgmap v312: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:04 smithi153 ceph-mon[106102]: pgmap v312: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:06 smithi203 ceph-mon[95988]: pgmap v313: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:06 smithi153 ceph-mon[106102]: pgmap v313: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:08 smithi203 ceph-mon[95988]: pgmap v314: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:08 smithi153 ceph-mon[106102]: pgmap v314: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:10.885 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:10 smithi153 ceph-mon[106102]: pgmap v315: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:10 smithi203 ceph-mon[95988]: pgmap v315: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:12 smithi203 ceph-mon[95988]: pgmap v316: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:13.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:12 smithi153 ceph-mon[106102]: pgmap v316: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:14 smithi203 ceph-mon[95988]: pgmap v317: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:14 smithi153 ceph-mon[106102]: pgmap v317: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:16 smithi203 ceph-mon[95988]: pgmap v318: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:16 smithi153 ceph-mon[106102]: pgmap v318: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:18 smithi203 ceph-mon[95988]: pgmap v319: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:18 smithi153 ceph-mon[106102]: pgmap v319: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:20 smithi203 ceph-mon[95988]: pgmap v320: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:20 smithi153 ceph-mon[106102]: pgmap v320: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:23.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:22 smithi203 ceph-mon[95988]: pgmap v321: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:22 smithi153 ceph-mon[106102]: pgmap v321: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:24 smithi203 ceph-mon[95988]: pgmap v322: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:24 smithi153 ceph-mon[106102]: pgmap v322: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:26 smithi203 ceph-mon[95988]: pgmap v323: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:26 smithi153 ceph-mon[106102]: pgmap v323: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.334+0000 7fbecdfdc700 1 -- 172.21.15.153:0/1131727621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec81026a0 msgr2=0x7fbec80fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.334+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/1131727621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec81026a0 0x7fbec80fbd10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbebc004940 tx=0x7fbebc009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 -- 172.21.15.153:0/1131727621 shutdown_connections 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/1131727621 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec81026a0 0x7fbec80fbd10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/1131727621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec8101d00 0x7fbec81020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 -- 172.21.15.153:0/1131727621 >> 172.21.15.153:0/1131727621 conn(0x7fbec80f7850 msgr2=0x7fbec80f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 -- 172.21.15.153:0/1131727621 shutdown_connections 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 -- 172.21.15.153:0/1131727621 wait complete. 2024-04-05T03:16:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 Processor -- start 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.335+0000 7fbecdfdc700 1 -- start start 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbecdfdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbecdfdc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec81026a0 0x7fbec8113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbecdfdc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec8114500 con 0x7fbec8101d00 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbecdfdc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec810db20 con 0x7fbec81026a0 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57152/0 (socket says 172.21.15.153:57152) 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 -- 172.21.15.153:0/887954847 learned_addr learned my addr 172.21.15.153:0/887954847 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec6ffd700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec81026a0 0x7fbec8113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 -- 172.21.15.153:0/887954847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec81026a0 msgr2=0x7fbec8113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec81026a0 0x7fbec8113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 -- 172.21.15.153:0/887954847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc005040 con 0x7fbec8101d00 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbec77fe700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8002a50 tx=0x7fbeb80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb8007cc0 con 0x7fbec8101d00 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.336+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec810de00 con 0x7fbec8101d00 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.337+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbeb800f040 con 0x7fbec8101d00 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.337+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb800da30 con 0x7fbec8101d00 2024-04-05T03:16:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.337+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec810e390 con 0x7fbec8101d00 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.338+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec80fd450 con 0x7fbec8101d00 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.338+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbeb800db90 con 0x7fbec8101d00 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.339+0000 7fbeaffff700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 0x7fbeb0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.339+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbeb808d310 con 0x7fbec8101d00 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.339+0000 7fbec6ffd700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 0x7fbeb0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.339+0000 7fbec6ffd700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 0x7fbeb0070150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbebc004890 tx=0x7fbebc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.344+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbeb8056890 con 0x7fbec8101d00 2024-04-05T03:16:27.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.560+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbec810f030 con 0x7fbeb006dca0 2024-04-05T03:16:27.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.561+0000 7fbeaffff700 1 -- 172.21.15.153:0/887954847 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbec810f030 con 0x7fbeb006dca0 2024-04-05T03:16:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 msgr2=0x7fbeb0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 0x7fbeb0070150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbebc004890 tx=0x7fbebc009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 msgr2=0x7fbec8113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8002a50 tx=0x7fbeb80046b0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 shutdown_connections 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb006dca0 0x7fbeb0070150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbec8101d00 0x7fbec8113970 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 --2- 172.21.15.153:0/887954847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec81026a0 0x7fbec8113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 >> 172.21.15.153:0/887954847 conn(0x7fbec80f7850 msgr2=0x7fbec80f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 shutdown_connections 2024-04-05T03:16:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.564+0000 7fbecdfdc700 1 -- 172.21.15.153:0/887954847 wait complete. 2024-04-05T03:16:27.574 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.667+0000 7ff2ed817700 1 -- 172.21.15.153:0/1462343483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e8101d00 msgr2=0x7ff2e81020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.667+0000 7ff2ed817700 1 --2- 172.21.15.153:0/1462343483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e8101d00 0x7ff2e81020d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8004940 tx=0x7ff2d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- 172.21.15.153:0/1462343483 shutdown_connections 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 --2- 172.21.15.153:0/1462343483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e81026a0 0x7ff2e80fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 --2- 172.21.15.153:0/1462343483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e8101d00 0x7ff2e81020d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- 172.21.15.153:0/1462343483 >> 172.21.15.153:0/1462343483 conn(0x7ff2e80f78d0 msgr2=0x7ff2e80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- 172.21.15.153:0/1462343483 shutdown_connections 2024-04-05T03:16:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- 172.21.15.153:0/1462343483 wait complete. 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 Processor -- start 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- start start 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e8101d00 0x7ff2e8113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81147c0 con 0x7ff2e81026a0 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.668+0000 7ff2ed817700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e810db20 con 0x7ff2e8101d00 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57178/0 (socket says 172.21.15.153:57178) 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 -- 172.21.15.153:0/3981424994 learned_addr learned my addr 172.21.15.153:0/3981424994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2e6ffd700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e8101d00 0x7ff2e8113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 -- 172.21.15.153:0/3981424994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e8101d00 msgr2=0x7ff2e8113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e8101d00 0x7ff2e8113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 -- 172.21.15.153:0/3981424994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8005040 con 0x7ff2e81026a0 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2de7fc700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000b790 tx=0x7ff2d0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:27.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d000f6f0 con 0x7ff2e81026a0 2024-04-05T03:16:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2d000fd20 con 0x7ff2e81026a0 2024-04-05T03:16:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e810de00 con 0x7ff2e81026a0 2024-04-05T03:16:27.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2d00103a0 con 0x7ff2e81026a0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.669+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e810e3c0 con 0x7ff2e81026a0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.670+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e80fd450 con 0x7ff2e81026a0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.671+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff2d0011400 con 0x7ff2e81026a0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.671+0000 7ff2dffff700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 0x7ff2d4070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.671+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff2d008d420 con 0x7ff2e81026a0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.671+0000 7ff2e6ffd700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 0x7ff2d4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.672+0000 7ff2e6ffd700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 0x7ff2d4070160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800b3b0 tx=0x7ff2d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:27.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.677+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff2d005a4d0 con 0x7ff2e81026a0 2024-04-05T03:16:27.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.889+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2e8108e70 con 0x7ff2d406dcb0 2024-04-05T03:16:27.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.890+0000 7ff2dffff700 1 -- 172.21.15.153:0/3981424994 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff2e8108e70 con 0x7ff2d406dcb0 2024-04-05T03:16:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 msgr2=0x7ff2d4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 0x7ff2d4070160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800b3b0 tx=0x7ff2d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 msgr2=0x7ff2e81140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000b790 tx=0x7ff2d0009220 comp rx=0 tx=0).stop 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 shutdown_connections 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff2d406dcb0 0x7ff2d4070160 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2e8101d00 0x7ff2e8113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 --2- 172.21.15.153:0/3981424994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2e81026a0 0x7ff2e81140e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 >> 172.21.15.153:0/3981424994 conn(0x7ff2e80f78d0 msgr2=0x7ff2e8065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 shutdown_connections 2024-04-05T03:16:27.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.893+0000 7ff2ed817700 1 -- 172.21.15.153:0/3981424994 wait complete. 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 -- 172.21.15.153:0/3436827796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fde70 msgr2=0x7f7f580fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/3436827796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fde70 0x7f7f580fe240 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c004940 tx=0x7f7f4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 -- 172.21.15.153:0/3436827796 shutdown_connections 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/3436827796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fe810 0x7f7f58106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/3436827796 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fde70 0x7f7f580fe240 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 -- 172.21.15.153:0/3436827796 >> 172.21.15.153:0/3436827796 conn(0x7f7f580f7880 msgr2=0x7f7f580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 -- 172.21.15.153:0/3436827796 shutdown_connections 2024-04-05T03:16:27.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.993+0000 7f7f5ef07700 1 -- 172.21.15.153:0/3436827796 wait complete. 2024-04-05T03:16:27.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 Processor -- start 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 -- start start 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fde70 0x7f7f58115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58116740 con 0x7f7f580fe810 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f5ef07700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5810fd60 con 0x7f7f580fde70 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.994+0000 7f7f57fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57190/0 (socket says 172.21.15.153:57190) 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 -- 172.21.15.153:0/1747552152 learned_addr learned my addr 172.21.15.153:0/1747552152 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f5cca3700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fde70 0x7f7f58115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 -- 172.21.15.153:0/1747552152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fde70 msgr2=0x7f7f58115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fde70 0x7f7f58115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 -- 172.21.15.153:0/1747552152 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c005040 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f57fff700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400c8c0 tx=0x7f7f440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f4400a760 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f4400ad90 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f58110040 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f440113a0 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.995+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f58110580 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.996+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f58066b30 con 0x7f7f580fe810 2024-04-05T03:16:27.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.997+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7f44012400 con 0x7f7f580fe810 2024-04-05T03:16:27.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.997+0000 7f7f557fa700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 0x7f7f48070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:27.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.997+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f4408d620 con 0x7f7f580fe810 2024-04-05T03:16:27.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.997+0000 7f7f5cca3700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 0x7f7f48070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:27.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:27.998+0000 7f7f5cca3700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 0x7f7f48070150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c004f90 tx=0x7f7f4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:28.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.002+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7f44056b80 con 0x7f7f580fe810 2024-04-05T03:16:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:28.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.204+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f58111240 con 0x7f7f4806dca0 2024-04-05T03:16:28.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.214+0000 7f7f557fa700 1 -- 172.21.15.153:0/1747552152 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f7f58111240 con 0x7f7f4806dca0 2024-04-05T03:16:28.214 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (11m) 4m ago 22m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (8m) 4m ago 23m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (8m) 4m ago 21m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (10m) 4m ago 22m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (4m) 4m ago 16m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (4m) 4m ago 16m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (4m) 4m ago 16m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (4m) 4m ago 16m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (12m) 4m ago 24m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (11m) 4m ago 20m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (9m) 4m ago 24m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (9m) 4m ago 20m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (11m) 4m ago 22m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (11m) 4m ago 21m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (8m) 4m ago 20m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:16:28.215 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (7m) 4m ago 19m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:16:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (6m) 4m ago 19m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:16:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (6m) 4m ago 18m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:16:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (5m) 4m ago 18m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:16:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (5m) 4m ago 18m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:16:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (11m) 4m ago 20m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:16:28.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 msgr2=0x7f7f48070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 0x7f7f48070150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c004f90 tx=0x7f7f4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:28.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 msgr2=0x7f7f581160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7f4400c8c0 tx=0x7f7f440044c0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 shutdown_connections 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f4806dca0 0x7f7f48070150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f580fde70 0x7f7f58115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 --2- 172.21.15.153:0/1747552152 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f580fe810 0x7f7f581160f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 >> 172.21.15.153:0/1747552152 conn(0x7f7f580f7880 msgr2=0x7f7f580f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 shutdown_connections 2024-04-05T03:16:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.217+0000 7f7f5ef07700 1 -- 172.21.15.153:0/1747552152 wait complete. 2024-04-05T03:16:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 -- 172.21.15.153:0/4105663958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fde90 msgr2=0x7f0ea40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 --2- 172.21.15.153:0/4105663958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fe260 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0e94004940 tx=0x7f0e94009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:28.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 -- 172.21.15.153:0/4105663958 shutdown_connections 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 --2- 172.21.15.153:0/4105663958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fe830 0x7f0ea4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 --2- 172.21.15.153:0/4105663958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fe260 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 -- 172.21.15.153:0/4105663958 >> 172.21.15.153:0/4105663958 conn(0x7f0ea4072bc0 msgr2=0x7f0ea4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 -- 172.21.15.153:0/4105663958 shutdown_connections 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 -- 172.21.15.153:0/4105663958 wait complete. 2024-04-05T03:16:28.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.317+0000 7f0ea9229700 1 Processor -- start 2024-04-05T03:16:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea9229700 1 -- start start 2024-04-05T03:16:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea9229700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea9229700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea9229700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40fc820 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea9229700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40fc960 con 0x7f0ea40fde90 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57210/0 (socket says 172.21.15.153:57210) 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 -- 172.21.15.153:0/262200370 learned_addr learned my addr 172.21.15.153:0/262200370 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0ea2d9d700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 -- 172.21.15.153:0/262200370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fde90 msgr2=0x7f0ea40fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.318+0000 7f0e9a59c700 1 -- 172.21.15.153:0/262200370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e94005040 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0e9a59c700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c002a50 tx=0x7f0e8c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c007cc0 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8c00f040 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea4119ed0 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c00da30 con 0x7f0ea40fe830 2024-04-05T03:16:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.319+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea411a2d0 con 0x7f0ea40fe830 2024-04-05T03:16:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.320+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea4065e30 con 0x7f0ea40fe830 2024-04-05T03:16:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.321+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f0e8c00db90 con 0x7f0ea40fe830 2024-04-05T03:16:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.321+0000 7f0e9bfff700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 0x7f0e90070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.322+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e8c08d230 con 0x7f0ea40fe830 2024-04-05T03:16:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.322+0000 7f0ea2d9d700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 0x7f0e90070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.322+0000 7f0ea2d9d700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 0x7f0e90070160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400b3b0 tx=0x7f0e94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:28.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.326+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0e8c056830 con 0x7f0ea40fe830 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.604+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0ea404bf60 con 0x7f0ea40fe830 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.605+0000 7f0e9bfff700 1 -- 172.21.15.153:0/262200370 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0e8c059ef0 con 0x7f0ea40fe830 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:16:28.605 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:16:28.606 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:16:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 msgr2=0x7f0e90070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 0x7f0e90070160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400b3b0 tx=0x7f0e94009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 msgr2=0x7f0ea40fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c002a50 tx=0x7f0e8c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 shutdown_connections 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e9006dcb0 0x7f0e90070160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ea40fde90 0x7f0ea40fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 --2- 172.21.15.153:0/262200370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ea40fe830 0x7f0ea40fc290 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 >> 172.21.15.153:0/262200370 conn(0x7f0ea4072bc0 msgr2=0x7f0ea40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 shutdown_connections 2024-04-05T03:16:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.607+0000 7f0ea9229700 1 -- 172.21.15.153:0/262200370 wait complete. 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 -- 172.21.15.153:0/2982186595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa278101d20 msgr2=0x7fa2781020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 --2- 172.21.15.153:0/2982186595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa278101d20 0x7fa2781020f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa274004940 tx=0x7fa274009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 -- 172.21.15.153:0/2982186595 shutdown_connections 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 --2- 172.21.15.153:0/2982186595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2781026c0 0x7fa2780fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 --2- 172.21.15.153:0/2982186595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa278101d20 0x7fa2781020f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.709+0000 7fa27f347700 1 -- 172.21.15.153:0/2982186595 >> 172.21.15.153:0/2982186595 conn(0x7fa2780f7870 msgr2=0x7fa2780f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 -- 172.21.15.153:0/2982186595 shutdown_connections 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 -- 172.21.15.153:0/2982186595 wait complete. 2024-04-05T03:16:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 Processor -- start 2024-04-05T03:16:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 -- start start 2024-04-05T03:16:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 0x7fa278113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2781144f0 con 0x7fa2781026c0 2024-04-05T03:16:28.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27f347700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27810db40 con 0x7fa278101d20 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27c8e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27c8e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43690/0 (socket says 172.21.15.153:43690) 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.710+0000 7fa27c8e2700 1 -- 172.21.15.153:0/3064571122 learned_addr learned my addr 172.21.15.153:0/3064571122 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27c8e2700 1 -- 172.21.15.153:0/3064571122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 msgr2=0x7fa278113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27d0e3700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 0x7fa278113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27c8e2700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 0x7fa278113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27c8e2700 1 -- 172.21.15.153:0/3064571122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274005040 con 0x7fa2781026c0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27d0e3700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 0x7fa278113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27c8e2700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa26800c800 tx=0x7fa268004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa26800a760 con 0x7fa2781026c0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa26800ad90 con 0x7fa2781026c0 2024-04-05T03:16:28.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27810de20 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2680113a0 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.711+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27810e3e0 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.712+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2780fd470 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.712+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa268012400 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.713+0000 7fa26dffb700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 0x7fa264070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.713+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa26808d4b0 con 0x7fa2781026c0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.713+0000 7fa27d0e3700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 0x7fa264070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:28.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.713+0000 7fa27d0e3700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 0x7fa264070150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa274004560 tx=0x7fa274009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:28.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.718+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa268056b50 con 0x7fa2781026c0 2024-04-05T03:16:28.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.957+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa278108e20 con 0x7fa2781026c0 2024-04-05T03:16:28.956 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:28 smithi153 ceph-mon[106102]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:28.957 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:28 smithi153 ceph-mon[106102]: pgmap v324: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:28.957 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:28 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/262200370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.958+0000 7fa26dffb700 1 -- 172.21.15.153:0/3064571122 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fa26805a210 con 0x7fa2781026c0 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:16:28.957 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:16:28.958 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:16:28.959 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:16:28.960 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 msgr2=0x7fa264070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 0x7fa264070150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa274004560 tx=0x7fa274009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 msgr2=0x7fa278113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa26800c800 tx=0x7fa268004470 comp rx=0 tx=0).stop 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 shutdown_connections 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa26406dca0 0x7fa264070150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa278101d20 0x7fa278113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 --2- 172.21.15.153:0/3064571122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2781026c0 0x7fa278113f40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:28.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 >> 172.21.15.153:0/3064571122 conn(0x7fa2780f7870 msgr2=0x7fa2780662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:28.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 shutdown_connections 2024-04-05T03:16:28.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:28.961+0000 7fa27f347700 1 -- 172.21.15.153:0/3064571122 wait complete. 2024-04-05T03:16:28.962 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:16:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:28 smithi203 ceph-mon[95988]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:28 smithi203 ceph-mon[95988]: pgmap v324: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:28 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/262200370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 -- 172.21.15.153:0/764521810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 msgr2=0x7f5e04102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 --2- 172.21.15.153:0/764521810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04102140 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5df4004940 tx=0x7f5df4009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 -- 172.21.15.153:0/764521810 shutdown_connections 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 --2- 172.21.15.153:0/764521810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e040fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 --2- 172.21.15.153:0/764521810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04102140 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 -- 172.21.15.153:0/764521810 >> 172.21.15.153:0/764521810 conn(0x7f5e04072bc0 msgr2=0x7f5e04072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:29.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.064+0000 7f5e0a337700 1 -- 172.21.15.153:0/764521810 shutdown_connections 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 -- 172.21.15.153:0/764521810 wait complete. 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 Processor -- start 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 -- start start 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e04114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04114680 con 0x7f5e04101d70 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e0a337700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0410da70 con 0x7f5e04102710 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e03fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e03fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43704/0 (socket says 172.21.15.153:43704) 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.065+0000 7f5e03fff700 1 -- 172.21.15.153:0/1749285762 learned_addr learned my addr 172.21.15.153:0/1749285762 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e037fe700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e04114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e03fff700 1 -- 172.21.15.153:0/1749285762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 msgr2=0x7f5e04114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e03fff700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e04114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e03fff700 1 -- 172.21.15.153:0/1749285762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4005040 con 0x7f5e04101d70 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e037fe700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e04114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e03fff700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009150 tx=0x7f5df4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5df4021070 con 0x7f5e04101d70 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5df4013c00 con 0x7f5e04101d70 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0410dcf0 con 0x7f5e04101d70 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5df400e070 con 0x7f5e04101d70 2024-04-05T03:16:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.066+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0410e1d0 con 0x7f5e04101d70 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.067+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e040fd4c0 con 0x7f5e04101d70 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.068+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5df401c400 con 0x7f5e04101d70 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.068+0000 7f5e00ff9700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 0x7f5dec070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.068+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5df408e960 con 0x7f5e04101d70 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.068+0000 7f5e037fe700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 0x7f5dec070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.069+0000 7f5e037fe700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 0x7f5dec070160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5df8004600 tx=0x7f5df800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:29.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.073+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5df405bb30 con 0x7f5e04101d70 2024-04-05T03:16:29.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.287+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e04108e70 con 0x7f5dec06dcb0 2024-04-05T03:16:29.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.288+0000 7f5e00ff9700 1 -- 172.21.15.153:0/1749285762 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5e04108e70 con 0x7f5dec06dcb0 2024-04-05T03:16:29.287 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:16:29.287 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:16:29.287 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:16:29.288 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.290+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 msgr2=0x7f5dec070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.290+0000 7f5e0a337700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 0x7f5dec070160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5df8004600 tx=0x7f5df800b040 comp rx=0 tx=0).stop 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.290+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 msgr2=0x7f5e04113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.290+0000 7f5e0a337700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009150 tx=0x7f5df4009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 shutdown_connections 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5dec06dcb0 0x7f5dec070160 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e04101d70 0x7f5e04113af0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 --2- 172.21.15.153:0/1749285762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e04102710 0x7f5e04114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 >> 172.21.15.153:0/1749285762 conn(0x7f5e04072bc0 msgr2=0x7f5e040658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 shutdown_connections 2024-04-05T03:16:29.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.291+0000 7f5e0a337700 1 -- 172.21.15.153:0/1749285762 wait complete. 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 -- 172.21.15.153:0/1882893812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93181026c0 msgr2=0x7f93180fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 --2- 172.21.15.153:0/1882893812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93181026c0 0x7f93180fbd30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9314004940 tx=0x7f9314009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 -- 172.21.15.153:0/1882893812 shutdown_connections 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 --2- 172.21.15.153:0/1882893812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93181026c0 0x7f93180fbd30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 --2- 172.21.15.153:0/1882893812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318101d20 0x7f93181020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 -- 172.21.15.153:0/1882893812 >> 172.21.15.153:0/1882893812 conn(0x7f93180f7870 msgr2=0x7f93180f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 -- 172.21.15.153:0/1882893812 shutdown_connections 2024-04-05T03:16:29.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.391+0000 7f931fd70700 1 -- 172.21.15.153:0/1882893812 wait complete. 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 Processor -- start 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 -- start start 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93181026c0 0x7f93181160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93181167d0 con 0x7f9318101d20 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931fd70700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931810fd20 con 0x7f93181026c0 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43724/0 (socket says 172.21.15.153:43724) 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 -- 172.21.15.153:0/2498888815 learned_addr learned my addr 172.21.15.153:0/2498888815 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931d30b700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93181026c0 0x7f93181160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 -- 172.21.15.153:0/2498888815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93181026c0 msgr2=0x7f93181160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93181026c0 0x7f93181160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.392+0000 7f931db0c700 1 -- 172.21.15.153:0/2498888815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314005040 con 0x7f9318101d20 2024-04-05T03:16:29.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f931db0c700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f930800b790 tx=0x7f9308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f930800f770 con 0x7f9318101d20 2024-04-05T03:16:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f930800fda0 con 0x7f9318101d20 2024-04-05T03:16:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931810ffa0 con 0x7f9318101d20 2024-04-05T03:16:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f930800d310 con 0x7f9318101d20 2024-04-05T03:16:29.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.393+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9318110560 con 0x7f9318101d20 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.394+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93180fd470 con 0x7f9318101d20 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.394+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9308010420 con 0x7f9318101d20 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.395+0000 7f930e7fc700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 0x7f9304070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.395+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f930808d4a0 con 0x7f9318101d20 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.395+0000 7f931d30b700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 0x7f9304070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:29.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.395+0000 7f931d30b700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 0x7f9304070150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f931400b020 tx=0x7f9314009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:29.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.399+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f930805a670 con 0x7f9318101d20 2024-04-05T03:16:29.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.679+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9318108e70 con 0x7f9318101d20 2024-04-05T03:16:29.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.679+0000 7f930e7fc700 1 -- 172.21.15.153:0/2498888815 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f930805a200 con 0x7f9318101d20 2024-04-05T03:16:29.679 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:16:29.679 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:16:29.679 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:16:29.679 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:16:29.679 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 msgr2=0x7f9304070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 0x7f9304070150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f931400b020 tx=0x7f9314009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 msgr2=0x7f9318115bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f930800b790 tx=0x7f9308009250 comp rx=0 tx=0).stop 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 shutdown_connections 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f930406dca0 0x7f9304070150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9318101d20 0x7f9318115bb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 --2- 172.21.15.153:0/2498888815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93181026c0 0x7f93181160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 >> 172.21.15.153:0/2498888815 conn(0x7f93180f7870 msgr2=0x7f93180662c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:29.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 shutdown_connections 2024-04-05T03:16:29.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:29.682+0000 7f931fd70700 1 -- 172.21.15.153:0/2498888815 wait complete. 2024-04-05T03:16:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:29 smithi203 ceph-mon[95988]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:30.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:29 smithi203 ceph-mon[95988]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:30.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:29 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3064571122' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:16:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:29 smithi153 ceph-mon[106102]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:29 smithi153 ceph-mon[106102]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:29 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3064571122' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:16:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:30 smithi203 ceph-mon[95988]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:30 smithi203 ceph-mon[95988]: pgmap v325: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:31.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:30 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2498888815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:16:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:30 smithi153 ceph-mon[106102]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:16:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:30 smithi153 ceph-mon[106102]: pgmap v325: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:30 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2498888815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:16:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:32 smithi203 ceph-mon[95988]: pgmap v326: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:32 smithi153 ceph-mon[106102]: pgmap v326: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:34 smithi203 ceph-mon[95988]: pgmap v327: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:34 smithi153 ceph-mon[106102]: pgmap v327: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:36 smithi203 ceph-mon[95988]: pgmap v328: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:36 smithi153 ceph-mon[106102]: pgmap v328: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:38 smithi203 ceph-mon[95988]: pgmap v329: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:38 smithi153 ceph-mon[106102]: pgmap v329: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:40 smithi203 ceph-mon[95988]: pgmap v330: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:40 smithi153 ceph-mon[106102]: pgmap v330: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:42 smithi203 ceph-mon[95988]: pgmap v331: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:42 smithi153 ceph-mon[106102]: pgmap v331: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:44 smithi203 ceph-mon[95988]: pgmap v332: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:44 smithi153 ceph-mon[106102]: pgmap v332: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:46 smithi203 ceph-mon[95988]: pgmap v333: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:46 smithi153 ceph-mon[106102]: pgmap v333: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:48.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:47 smithi203 ceph-mon[95988]: pgmap v334: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:48.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:47 smithi153 ceph-mon[106102]: pgmap v334: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:50.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:50 smithi153 ceph-mon[106102]: pgmap v335: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:50 smithi203 ceph-mon[95988]: pgmap v335: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:52 smithi203 ceph-mon[95988]: pgmap v336: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:52 smithi153 ceph-mon[106102]: pgmap v336: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:54 smithi203 ceph-mon[95988]: pgmap v337: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:54 smithi153 ceph-mon[106102]: pgmap v337: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:56 smithi203 ceph-mon[95988]: pgmap v338: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:56 smithi153 ceph-mon[106102]: pgmap v338: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:16:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:16:58 smithi203 ceph-mon[95988]: pgmap v339: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:16:58 smithi153 ceph-mon[106102]: pgmap v339: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:16:59.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 -- 172.21.15.153:0/2064184767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f8101d20 msgr2=0x7f42f81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 --2- 172.21.15.153:0/2064184767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f8101d20 0x7f42f81020f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f42e8004940 tx=0x7f42e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 -- 172.21.15.153:0/2064184767 shutdown_connections 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 --2- 172.21.15.153:0/2064184767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f81026c0 0x7f42f80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 --2- 172.21.15.153:0/2064184767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f8101d20 0x7f42f81020f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 -- 172.21.15.153:0/2064184767 >> 172.21.15.153:0/2064184767 conn(0x7f42f80f7870 msgr2=0x7f42f80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.796+0000 7f42fd101700 1 -- 172.21.15.153:0/2064184767 shutdown_connections 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 -- 172.21.15.153:0/2064184767 wait complete. 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 Processor -- start 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 -- start start 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f8101d20 0x7f42f8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8104ad0 con 0x7f42f81026c0 2024-04-05T03:16:59.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.797+0000 7f42fd101700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8103600 con 0x7f42f8101d20 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43626/0 (socket says 172.21.15.153:43626) 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 -- 172.21.15.153:0/3066400381 learned_addr learned my addr 172.21.15.153:0/3066400381 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f6d9d700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f8101d20 0x7f42f8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 -- 172.21.15.153:0/3066400381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f8101d20 msgr2=0x7f42f8102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f8101d20 0x7f42f8102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:16:59.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 -- 172.21.15.153:0/3066400381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8005040 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42f659c700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c800 tx=0x7f42ec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec00a760 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f81038e0 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42ec00ad90 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec0113a0 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.798+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f811a180 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.799+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f80fd470 con 0x7f42f81026c0 2024-04-05T03:16:59.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.800+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f42ec012400 con 0x7f42f81026c0 2024-04-05T03:16:59.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.800+0000 7f42df7fe700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 0x7f42e0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:16:59.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.800+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f42ec08d5a0 con 0x7f42f81026c0 2024-04-05T03:16:59.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.800+0000 7f42f6d9d700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 0x7f42e0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:16:59.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.801+0000 7f42f6d9d700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 0x7f42e0070150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f42e8004f90 tx=0x7f42e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:16:59.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:16:59.805+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f42ec056ba0 con 0x7f42f81026c0 2024-04-05T03:17:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.017+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f42f80f9640 con 0x7f42e006dca0 2024-04-05T03:17:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.018+0000 7f42df7fe700 1 -- 172.21.15.153:0/3066400381 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f42f80f9640 con 0x7f42e006dca0 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.020+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 msgr2=0x7f42e0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.020+0000 7f42fd101700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 0x7f42e0070150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f42e8004f90 tx=0x7f42e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 msgr2=0x7f42f81030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c800 tx=0x7f42ec004470 comp rx=0 tx=0).stop 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 shutdown_connections 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f42e006dca0 0x7f42e0070150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42f8101d20 0x7f42f8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 --2- 172.21.15.153:0/3066400381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42f81026c0 0x7f42f81030c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 >> 172.21.15.153:0/3066400381 conn(0x7f42f80f7870 msgr2=0x7f42f80f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 shutdown_connections 2024-04-05T03:17:00.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.021+0000 7f42fd101700 1 -- 172.21.15.153:0/3066400381 wait complete. 2024-04-05T03:17:00.030 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 -- 172.21.15.153:0/738866268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fddd0 msgr2=0x7fe0e40fe1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/738866268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fddd0 0x7fe0e40fe1a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8004940 tx=0x7fe0d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 -- 172.21.15.153:0/738866268 shutdown_connections 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/738866268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fe770 0x7fe0e4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/738866268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fddd0 0x7fe0e40fe1a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.123+0000 7fe0eaf03700 1 -- 172.21.15.153:0/738866268 >> 172.21.15.153:0/738866268 conn(0x7fe0e40f7890 msgr2=0x7fe0e40f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 -- 172.21.15.153:0/738866268 shutdown_connections 2024-04-05T03:17:00.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 -- 172.21.15.153:0/738866268 wait complete. 2024-04-05T03:17:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 Processor -- start 2024-04-05T03:17:00.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 -- start start 2024-04-05T03:17:00.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fddd0 0x7fe0e4115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e41166b0 con 0x7fe0e40fe770 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.124+0000 7fe0eaf03700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e410fc90 con 0x7fe0e40fddd0 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43650/0 (socket says 172.21.15.153:43650) 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 -- 172.21.15.153:0/3066372348 learned_addr learned my addr 172.21.15.153:0/3066372348 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e8c9f700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fddd0 0x7fe0e4115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 -- 172.21.15.153:0/3066372348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fddd0 msgr2=0x7fe0e4115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fddd0 0x7fe0e4115b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 -- 172.21.15.153:0/3066372348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8005040 con 0x7fe0e40fe770 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e3fff700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000eb70 tx=0x7fe0d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0d0007cc0 con 0x7fe0e40fe770 2024-04-05T03:17:00.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0d000f040 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e410ff10 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0d0009640 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.125+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e4110450 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.126+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4066b50 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.127+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe0d00097a0 con 0x7fe0e40fe770 2024-04-05T03:17:00.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.127+0000 7fe0e17fa700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 0x7fe0d4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.127+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe0d008d1d0 con 0x7fe0e40fe770 2024-04-05T03:17:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.127+0000 7fe0e8c9f700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 0x7fe0d4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.127+0000 7fe0e8c9f700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 0x7fe0d4070150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800b0d0 tx=0x7fe0d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.132+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe0d005a280 con 0x7fe0e40fe770 2024-04-05T03:17:00.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.344+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0e4109ef0 con 0x7fe0d406dca0 2024-04-05T03:17:00.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.346+0000 7fe0e17fa700 1 -- 172.21.15.153:0/3066372348 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe0e4109ef0 con 0x7fe0d406dca0 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 msgr2=0x7fe0d4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 0x7fe0d4070150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800b0d0 tx=0x7fe0d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 msgr2=0x7fe0e4116060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000eb70 tx=0x7fe0d000b040 comp rx=0 tx=0).stop 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 shutdown_connections 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0d406dca0 0x7fe0d4070150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0e40fddd0 0x7fe0e4115b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 --2- 172.21.15.153:0/3066372348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0e40fe770 0x7fe0e4116060 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 >> 172.21.15.153:0/3066372348 conn(0x7fe0e40f7890 msgr2=0x7fe0e40fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 shutdown_connections 2024-04-05T03:17:00.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.348+0000 7fe0eaf03700 1 -- 172.21.15.153:0/3066372348 wait complete. 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 -- 172.21.15.153:0/1723960504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c102e00 msgr2=0x7fa23c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 --2- 172.21.15.153:0/1723960504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c102e00 0x7fa23c1031d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004940 tx=0x7fa22c009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 -- 172.21.15.153:0/1723960504 shutdown_connections 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 --2- 172.21.15.153:0/1723960504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c1037a0 0x7fa23c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 --2- 172.21.15.153:0/1723960504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c102e00 0x7fa23c1031d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 -- 172.21.15.153:0/1723960504 >> 172.21.15.153:0/1723960504 conn(0x7fa23c0f89b0 msgr2=0x7fa23c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 -- 172.21.15.153:0/1723960504 shutdown_connections 2024-04-05T03:17:00.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.449+0000 7fa240990700 1 -- 172.21.15.153:0/1723960504 wait complete. 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 Processor -- start 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 -- start start 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c102e00 0x7fa23c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c114520 con 0x7fa23c1037a0 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa240990700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c10db70 con 0x7fa23c102e00 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43660/0 (socket says 172.21.15.153:43660) 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 -- 172.21.15.153:0/1642338222 learned_addr learned my addr 172.21.15.153:0/1642338222 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa23a59c700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c102e00 0x7fa23c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 -- 172.21.15.153:0/1642338222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c102e00 msgr2=0x7fa23c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c102e00 0x7fa23c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.450+0000 7fa239d9b700 1 -- 172.21.15.153:0/1642338222 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c005040 con 0x7fa23c1037a0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa239d9b700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c890 tx=0x7fa2300044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa23000a700 con 0x7fa23c1037a0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa23000ad30 con 0x7fa23c1037a0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23c10de50 con 0x7fa23c1037a0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2300113a0 con 0x7fa23c1037a0 2024-04-05T03:17:00.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.451+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c10e410 con 0x7fa23c1037a0 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.452+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23c100110 con 0x7fa23c1037a0 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.452+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa230012400 con 0x7fa23c1037a0 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.453+0000 7fa22affd700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 0x7fa224070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.453+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa23008d600 con 0x7fa23c1037a0 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.453+0000 7fa23a59c700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 0x7fa224070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.453+0000 7fa23a59c700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 0x7fa224070150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004f90 tx=0x7fa22c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.457+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa23005a730 con 0x7fa23c1037a0 2024-04-05T03:17:00.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.668+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa23c10efe0 con 0x7fa22406dca0 2024-04-05T03:17:00.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.678+0000 7fa22affd700 1 -- 172.21.15.153:0/1642338222 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa23c10efe0 con 0x7fa22406dca0 2024-04-05T03:17:00.677 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (11m) 5m ago 22m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (9m) 5m ago 23m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (9m) 4m ago 21m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (11m) 5m ago 22m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (5m) 5m ago 17m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (5m) 5m ago 17m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (4m) 4m ago 17m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (4m) 4m ago 17m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (13m) 5m ago 25m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (12m) 4m ago 21m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (10m) 5m ago 25m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (9m) 4m ago 21m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (12m) 5m ago 22m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:17:00.678 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (12m) 4m ago 21m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (8m) 5m ago 20m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (8m) 5m ago 20m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (7m) 5m ago 19m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (6m) 4m ago 19m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (6m) 4m ago 19m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (5m) 4m ago 18m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:17:00.679 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (11m) 4m ago 21m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 msgr2=0x7fa224070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 0x7fa224070150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa22c004f90 tx=0x7fa22c009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 msgr2=0x7fa23c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c890 tx=0x7fa2300044c0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 shutdown_connections 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa22406dca0 0x7fa224070150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa23c102e00 0x7fa23c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 --2- 172.21.15.153:0/1642338222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa23c1037a0 0x7fa23c113ed0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.680+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 >> 172.21.15.153:0/1642338222 conn(0x7fa23c0f89b0 msgr2=0x7fa23c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.681+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 shutdown_connections 2024-04-05T03:17:00.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.681+0000 7fa240990700 1 -- 172.21.15.153:0/1642338222 wait complete. 2024-04-05T03:17:00.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 -- 172.21.15.153:0/1068063318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 msgr2=0x7ff18c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 --2- 172.21.15.153:0/1068063318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c1020f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff17c004940 tx=0x7ff17c009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:00.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 -- 172.21.15.153:0/1068063318 shutdown_connections 2024-04-05T03:17:00.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 --2- 172.21.15.153:0/1068063318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 0x7ff18c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 --2- 172.21.15.153:0/1068063318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c1020f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.780+0000 7ff1925cb700 1 -- 172.21.15.153:0/1068063318 >> 172.21.15.153:0/1068063318 conn(0x7ff18c0f7870 msgr2=0x7ff18c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:00.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:00 smithi153 ceph-mon[106102]: pgmap v340: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 -- 172.21.15.153:0/1068063318 shutdown_connections 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 -- 172.21.15.153:0/1068063318 wait complete. 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 Processor -- start 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 -- start start 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 0x7ff18c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c114520 con 0x7ff18c101d20 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.781+0000 7ff1925cb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c10db20 con 0x7ff18c1026c0 2024-04-05T03:17:00.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43684/0 (socket says 172.21.15.153:43684) 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 -- 172.21.15.153:0/248102585 learned_addr learned my addr 172.21.15.153:0/248102585 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18b7fe700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 0x7ff18c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 -- 172.21.15.153:0/248102585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 msgr2=0x7ff18c113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 0x7ff18c113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 -- 172.21.15.153:0/248102585 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c005040 con 0x7ff18c101d20 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff18bfff700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00b3b0 tx=0x7ff17c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff17c00e040 con 0x7ff18c101d20 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff17c013420 con 0x7ff18c101d20 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c10dda0 con 0x7ff18c101d20 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.782+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff17c01c630 con 0x7ff18c101d20 2024-04-05T03:17:00.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.783+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c10e360 con 0x7ff18c101d20 2024-04-05T03:17:00.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.784+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c0fd470 con 0x7ff18c101d20 2024-04-05T03:17:00.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.784+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff17c013590 con 0x7ff18c101d20 2024-04-05T03:17:00.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.784+0000 7ff188ff9700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 0x7ff174070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:00.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.784+0000 7ff18b7fe700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 0x7ff174070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.784+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff17c08e680 con 0x7ff18c101d20 2024-04-05T03:17:00.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.785+0000 7ff18b7fe700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 0x7ff174070150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff18c10ece0 tx=0x7ff18000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:00.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:00.789+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff17c05b850 con 0x7ff18c101d20 2024-04-05T03:17:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:00 smithi203 ceph-mon[95988]: pgmap v340: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:01.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.065+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff18c108e20 con 0x7ff18c101d20 2024-04-05T03:17:01.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.065+0000 7ff188ff9700 1 -- 172.21.15.153:0/248102585 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff17c05b3e0 con 0x7ff18c101d20 2024-04-05T03:17:01.064 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:17:01.065 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:17:01.066 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:17:01.066 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 msgr2=0x7ff174070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 0x7ff174070150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff18c10ece0 tx=0x7ff18000b040 comp rx=0 tx=0).stop 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 msgr2=0x7ff18c113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00b3b0 tx=0x7ff17c009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 shutdown_connections 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff17406dca0 0x7ff174070150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff18c101d20 0x7ff18c113990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 --2- 172.21.15.153:0/248102585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff18c1026c0 0x7ff18c113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 >> 172.21.15.153:0/248102585 conn(0x7ff18c0f7870 msgr2=0x7ff18c066200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 shutdown_connections 2024-04-05T03:17:01.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.068+0000 7ff1925cb700 1 -- 172.21.15.153:0/248102585 wait complete. 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 -- 172.21.15.153:0/4122417963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db0102e00 msgr2=0x7f4db01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 --2- 172.21.15.153:0/4122417963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db0102e00 0x7f4db01031d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4da0004940 tx=0x7f4da0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 -- 172.21.15.153:0/4122417963 shutdown_connections 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 --2- 172.21.15.153:0/4122417963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db01037a0 0x7f4db00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 --2- 172.21.15.153:0/4122417963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db0102e00 0x7f4db01031d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.169+0000 7f4db6860700 1 -- 172.21.15.153:0/4122417963 >> 172.21.15.153:0/4122417963 conn(0x7f4db00f89b0 msgr2=0x7f4db00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 -- 172.21.15.153:0/4122417963 shutdown_connections 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 -- 172.21.15.153:0/4122417963 wait complete. 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 Processor -- start 2024-04-05T03:17:01.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 -- start start 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db0102e00 0x7f4db0113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db01146f0 con 0x7f4db01037a0 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.170+0000 7f4db6860700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db010db30 con 0x7f4db0102e00 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43702/0 (socket says 172.21.15.153:43702) 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 -- 172.21.15.153:0/188857668 learned_addr learned my addr 172.21.15.153:0/188857668 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daffff700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db0102e00 0x7f4db0113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 -- 172.21.15.153:0/188857668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db0102e00 msgr2=0x7f4db0113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db0102e00 0x7f4db0113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 -- 172.21.15.153:0/188857668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0005040 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4daf7fe700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4da4002a50 tx=0x7f4da4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da4007cc0 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4da400f040 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da400daf0 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db010de10 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.171+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db010e3d0 con 0x7f4db01037a0 2024-04-05T03:17:01.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.172+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db0100110 con 0x7f4db01037a0 2024-04-05T03:17:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.173+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4da4009810 con 0x7f4db01037a0 2024-04-05T03:17:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.173+0000 7f4dacff9700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 0x7f4d98070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.173+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4da408d360 con 0x7f4db01037a0 2024-04-05T03:17:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.173+0000 7f4daffff700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 0x7f4d98070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.174+0000 7f4daffff700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 0x7f4d98070150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4da0004f90 tx=0x7f4da0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.178+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4da405a410 con 0x7f4db01037a0 2024-04-05T03:17:01.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.421+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4db00fe550 con 0x7f4db01037a0 2024-04-05T03:17:01.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.421+0000 7f4dacff9700 1 -- 172.21.15.153:0/188857668 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f4da4059fa0 con 0x7f4db01037a0 2024-04-05T03:17:01.420 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:17:01.421 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:17:01.422 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:01.423 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:01.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 msgr2=0x7f4d98070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 0x7f4d98070150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4da0004f90 tx=0x7f4da0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 msgr2=0x7f4db01140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4da4002a50 tx=0x7f4da4004600 comp rx=0 tx=0).stop 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 shutdown_connections 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9806dca0 0x7f4d98070150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4db0102e00 0x7f4db0113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 --2- 172.21.15.153:0/188857668 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4db01037a0 0x7f4db01140a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 >> 172.21.15.153:0/188857668 conn(0x7f4db00f89b0 msgr2=0x7f4db0107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.424+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 shutdown_connections 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.425+0000 7f4db6860700 1 -- 172.21.15.153:0/188857668 wait complete. 2024-04-05T03:17:01.425 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 -- 172.21.15.153:0/3648112799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 msgr2=0x7f13680fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 --2- 172.21.15.153:0/3648112799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f13680fbd30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1358004940 tx=0x7f1358009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 -- 172.21.15.153:0/3648112799 shutdown_connections 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 --2- 172.21.15.153:0/3648112799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f13680fbd30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 --2- 172.21.15.153:0/3648112799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f13681020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 -- 172.21.15.153:0/3648112799 >> 172.21.15.153:0/3648112799 conn(0x7f13680f7870 msgr2=0x7f13680f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 -- 172.21.15.153:0/3648112799 shutdown_connections 2024-04-05T03:17:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 -- 172.21.15.153:0/3648112799 wait complete. 2024-04-05T03:17:01.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.528+0000 7f136e371700 1 Processor -- start 2024-04-05T03:17:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f136e371700 1 -- start start 2024-04-05T03:17:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f136e371700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f136810eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f136e371700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f136e371700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810f5c0 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f136e371700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136810f700 con 0x7f1368101d20 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43730/0 (socket says 172.21.15.153:43730) 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 -- 172.21.15.153:0/3870913014 learned_addr learned my addr 172.21.15.153:0/3870913014 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f1367fff700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f136810eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 -- 172.21.15.153:0/3870913014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 msgr2=0x7f136810eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f136810eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 -- 172.21.15.153:0/3870913014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1358005040 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f1367fff700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f136810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f13677fe700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f135800b3b0 tx=0x7f1358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.529+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f135800e040 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.530+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1358013420 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.530+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136810f980 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.530+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f135801c630 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.530+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1368117f00 con 0x7f13681026c0 2024-04-05T03:17:01.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.531+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13680fd470 con 0x7f13681026c0 2024-04-05T03:17:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.531+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1358013590 con 0x7f13681026c0 2024-04-05T03:17:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.532+0000 7f1364ff9700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 0x7f1350070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.532+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f135808e790 con 0x7f13681026c0 2024-04-05T03:17:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.532+0000 7f1367fff700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 0x7f1350070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.532+0000 7f1367fff700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 0x7f1350070150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f135c009fd0 tx=0x7f135c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.536+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1358057d90 con 0x7f13681026c0 2024-04-05T03:17:01.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.752+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1368108e70 con 0x7f135006dca0 2024-04-05T03:17:01.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:01 smithi153 ceph-mon[106102]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:01 smithi153 ceph-mon[106102]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:01 smithi153 ceph-mon[106102]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:01 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/248102585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:17:01.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:01 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/188857668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:17:01.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.754+0000 7f1364ff9700 1 -- 172.21.15.153:0/3870913014 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1368108e70 con 0x7f135006dca0 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:17:01.753 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:17:01.754 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:17:01.754 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:17:01.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 msgr2=0x7f1350070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 0x7f1350070150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f135c009fd0 tx=0x7f135c009ee0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 msgr2=0x7f136810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f135800b3b0 tx=0x7f1358009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 shutdown_connections 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f135006dca0 0x7f1350070150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1368101d20 0x7f136810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 --2- 172.21.15.153:0/3870913014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13681026c0 0x7f136810f080 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 >> 172.21.15.153:0/3870913014 conn(0x7f13680f7870 msgr2=0x7f13680f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.756+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 shutdown_connections 2024-04-05T03:17:01.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.757+0000 7f136e371700 1 -- 172.21.15.153:0/3870913014 wait complete. 2024-04-05T03:17:01.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 -- 172.21.15.153:0/326157069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 msgr2=0x7fc3780fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 --2- 172.21.15.153:0/326157069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fb990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc368004940 tx=0x7fc368009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 -- 172.21.15.153:0/326157069 shutdown_connections 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 --2- 172.21.15.153:0/326157069 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fb990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 --2- 172.21.15.153:0/326157069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 0x7fc3781033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 -- 172.21.15.153:0/326157069 >> 172.21.15.153:0/326157069 conn(0x7fc378071530 msgr2=0x7fc378071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 -- 172.21.15.153:0/326157069 shutdown_connections 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.857+0000 7fc37e82a700 1 -- 172.21.15.153:0/326157069 wait complete. 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 Processor -- start 2024-04-05T03:17:01.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 -- start start 2024-04-05T03:17:01.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 0x7fc3780fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780fe700 con 0x7fc3780fb540 2024-04-05T03:17:01.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc37e82a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780fd1a0 con 0x7fc378102fd0 2024-04-05T03:17:01.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc377fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc377fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43740/0 (socket says 172.21.15.153:43740) 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc377fff700 1 -- 172.21.15.153:0/2180294084 learned_addr learned my addr 172.21.15.153:0/2180294084 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc3777fe700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 0x7fc3780fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc377fff700 1 -- 172.21.15.153:0/2180294084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 msgr2=0x7fc3780fcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.858+0000 7fc377fff700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 0x7fc3780fcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc377fff700 1 -- 172.21.15.153:0/2180294084 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368005040 con 0x7fc3780fb540 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc377fff700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b580 tx=0x7fc36c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36c00f6f0 con 0x7fc3780fb540 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc36c00fd20 con 0x7fc3780fb540 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3780fd480 con 0x7fc3780fb540 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc36c0103a0 con 0x7fc3780fb540 2024-04-05T03:17:01.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.859+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37811a250 con 0x7fc3780fb540 2024-04-05T03:17:01.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.860+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc378100270 con 0x7fc3780fb540 2024-04-05T03:17:01.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.860+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc36c00f850 con 0x7fc3780fb540 2024-04-05T03:17:01.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.861+0000 7fc374ff9700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 0x7fc360070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:01.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.861+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc36c08d0d0 con 0x7fc3780fb540 2024-04-05T03:17:01.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.861+0000 7fc3777fe700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 0x7fc360070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:01.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.861+0000 7fc3777fe700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 0x7fc360070150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3780fde50 tx=0x7fc368009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:01.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:01.865+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc36c056770 con 0x7fc3780fb540 2024-04-05T03:17:01.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:01 smithi203 ceph-mon[95988]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:01 smithi203 ceph-mon[95988]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:01 smithi203 ceph-mon[95988]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:01.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:01 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/248102585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:17:01.947 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:01 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/188857668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:17:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.143+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc37810a0e0 con 0x7fc3780fb540 2024-04-05T03:17:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.143+0000 7fc374ff9700 1 -- 172.21.15.153:0/2180294084 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc36c00fb00 con 0x7fc3780fb540 2024-04-05T03:17:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:17:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:17:02.143 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:17:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:17:02.143 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:17:02.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 msgr2=0x7fc360070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:02.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 0x7fc360070150 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3780fde50 tx=0x7fc368009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:02.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 msgr2=0x7fc3780fc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b580 tx=0x7fc36c0091a0 comp rx=0 tx=0).stop 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 shutdown_connections 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc36006dca0 0x7fc360070150 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3780fb540 0x7fc3780fc720 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 --2- 172.21.15.153:0/2180294084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc378102fd0 0x7fc3780fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.145+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 >> 172.21.15.153:0/2180294084 conn(0x7fc378071530 msgr2=0x7fc3780fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.146+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 shutdown_connections 2024-04-05T03:17:02.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:02.146+0000 7fc37e82a700 1 -- 172.21.15.153:0/2180294084 wait complete. 2024-04-05T03:17:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:02 smithi203 ceph-mon[95988]: pgmap v341: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:03.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:02 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2180294084' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:17:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:02 smithi153 ceph-mon[106102]: pgmap v341: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:02 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2180294084' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:17:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:03 smithi203 ceph-mon[95988]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:04.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:03 smithi153 ceph-mon[106102]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:04 smithi203 ceph-mon[95988]: pgmap v342: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:04 smithi153 ceph-mon[106102]: pgmap v342: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:06 smithi203 ceph-mon[95988]: pgmap v343: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:06 smithi153 ceph-mon[106102]: pgmap v343: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:08 smithi203 ceph-mon[95988]: pgmap v344: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:08 smithi153 ceph-mon[106102]: pgmap v344: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:10 smithi203 ceph-mon[95988]: pgmap v345: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:10 smithi153 ceph-mon[106102]: pgmap v345: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:12 smithi203 ceph-mon[95988]: pgmap v346: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:12 smithi153 ceph-mon[106102]: pgmap v346: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:14 smithi203 ceph-mon[95988]: pgmap v347: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:14 smithi153 ceph-mon[106102]: pgmap v347: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:16 smithi203 ceph-mon[95988]: pgmap v348: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:16 smithi153 ceph-mon[106102]: pgmap v348: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:18 smithi203 ceph-mon[95988]: pgmap v349: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:18 smithi153 ceph-mon[106102]: pgmap v349: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:20 smithi203 ceph-mon[95988]: pgmap v350: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:20 smithi153 ceph-mon[106102]: pgmap v350: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:22 smithi203 ceph-mon[95988]: pgmap v351: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:22 smithi153 ceph-mon[106102]: pgmap v351: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:24 smithi203 ceph-mon[95988]: pgmap v352: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:24 smithi153 ceph-mon[106102]: pgmap v352: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:26.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:25 smithi203 ceph-mon[95988]: pgmap v353: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:26.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:25 smithi153 ceph-mon[106102]: pgmap v353: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:27.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:28 smithi153 ceph-mon[106102]: pgmap v354: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:28 smithi203 ceph-mon[95988]: pgmap v354: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:30.885 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:30 smithi153 ceph-mon[106102]: pgmap v355: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:30 smithi203 ceph-mon[95988]: pgmap v355: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 -- 172.21.15.153:0/2420153224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe58102e00 msgr2=0x7efe581031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 --2- 172.21.15.153:0/2420153224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe58102e00 0x7efe581031d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe48004940 tx=0x7efe48009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 -- 172.21.15.153:0/2420153224 shutdown_connections 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 --2- 172.21.15.153:0/2420153224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe581037a0 0x7efe580fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 --2- 172.21.15.153:0/2420153224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe58102e00 0x7efe581031d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 -- 172.21.15.153:0/2420153224 >> 172.21.15.153:0/2420153224 conn(0x7efe580f89b0 msgr2=0x7efe580fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:32.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 -- 172.21.15.153:0/2420153224 shutdown_connections 2024-04-05T03:17:32.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.257+0000 7efe5f72a700 1 -- 172.21.15.153:0/2420153224 wait complete. 2024-04-05T03:17:32.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 Processor -- start 2024-04-05T03:17:32.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 -- start start 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe58102e00 0x7efe58113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58114520 con 0x7efe581037a0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5f72a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5810db70 con 0x7efe58102e00 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44762/0 (socket says 172.21.15.153:44762) 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 -- 172.21.15.153:0/3878126617 learned_addr learned my addr 172.21.15.153:0/3878126617 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 -- 172.21.15.153:0/3878126617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe58102e00 msgr2=0x7efe58113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5d4c6700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe58102e00 0x7efe58113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe58102e00 0x7efe58113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 -- 172.21.15.153:0/3878126617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe48005040 con 0x7efe581037a0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.258+0000 7efe5ccc5700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe54002a50 tx=0x7efe54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.259+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe54007cc0 con 0x7efe581037a0 2024-04-05T03:17:32.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.259+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5810de50 con 0x7efe581037a0 2024-04-05T03:17:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.259+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe5400f040 con 0x7efe581037a0 2024-04-05T03:17:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.259+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe5400da40 con 0x7efe581037a0 2024-04-05T03:17:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.259+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5810e410 con 0x7efe581037a0 2024-04-05T03:17:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.260+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe58100110 con 0x7efe581037a0 2024-04-05T03:17:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.260+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efe540097c0 con 0x7efe581037a0 2024-04-05T03:17:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.261+0000 7efe4dffb700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 0x7efe44070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.261+0000 7efe5d4c6700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 0x7efe44070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.261+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe5408d2d0 con 0x7efe581037a0 2024-04-05T03:17:32.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.261+0000 7efe5d4c6700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 0x7efe44070150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe48004f90 tx=0x7efe48009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.266+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efe5405a170 con 0x7efe581037a0 2024-04-05T03:17:32.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.479+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe58113660 con 0x7efe4406dca0 2024-04-05T03:17:32.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.481+0000 7efe4dffb700 1 -- 172.21.15.153:0/3878126617 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efe58113660 con 0x7efe4406dca0 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 msgr2=0x7efe44070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 0x7efe44070150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efe48004f90 tx=0x7efe48009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 msgr2=0x7efe58113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efe54002a50 tx=0x7efe54004600 comp rx=0 tx=0).stop 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 shutdown_connections 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe4406dca0 0x7efe44070150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe58102e00 0x7efe58113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 --2- 172.21.15.153:0/3878126617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe581037a0 0x7efe58113ed0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 >> 172.21.15.153:0/3878126617 conn(0x7efe580f89b0 msgr2=0x7efe581073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:32.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 shutdown_connections 2024-04-05T03:17:32.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.483+0000 7efe5f72a700 1 -- 172.21.15.153:0/3878126617 wait complete. 2024-04-05T03:17:32.492 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:17:32.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.584+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/863437591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fe850 msgr2=0x7f1e64106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.584+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/863437591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fe850 0x7f1e64106f20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1e54004970 tx=0x7f1e54009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:32.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/863437591 shutdown_connections 2024-04-05T03:17:32.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/863437591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fe850 0x7f1e64106f20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/863437591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fdeb0 0x7f1e640fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/863437591 >> 172.21.15.153:0/863437591 conn(0x7f1e64072bc0 msgr2=0x7f1e64072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/863437591 shutdown_connections 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/863437591 wait complete. 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 Processor -- start 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- start start 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fe850 0x7f1e64114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e64114770 con 0x7f1e640fdeb0 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.585+0000 7f1e6bc1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6410dad0 con 0x7f1e640fe850 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44782/0 (socket says 172.21.15.153:44782) 2024-04-05T03:17:32.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 -- 172.21.15.153:0/831619489 learned_addr learned my addr 172.21.15.153:0/831619489 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e691b5700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fe850 0x7f1e64114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 -- 172.21.15.153:0/831619489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fe850 msgr2=0x7f1e64114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fe850 0x7f1e64114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 -- 172.21.15.153:0/831619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e54005040 con 0x7f1e640fdeb0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e699b6700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000b790 tx=0x7f1e60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e6000f770 con 0x7f1e640fdeb0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6410ddb0 con 0x7f1e640fdeb0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e6000fda0 con 0x7f1e640fdeb0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e6000d310 con 0x7f1e640fdeb0 2024-04-05T03:17:32.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.586+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6410e2c0 con 0x7f1e640fdeb0 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.587+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e64065e10 con 0x7f1e640fdeb0 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.588+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1e60010420 con 0x7f1e640fdeb0 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.588+0000 7f1e5a7fc700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 0x7f1e50070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.588+0000 7f1e691b5700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 0x7f1e50070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.588+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e6008d5a0 con 0x7f1e640fdeb0 2024-04-05T03:17:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.589+0000 7f1e691b5700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 0x7f1e50070150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e54004f50 tx=0x7f1e54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.593+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1e6005a6d0 con 0x7f1e640fdeb0 2024-04-05T03:17:32.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.814+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e6410a050 con 0x7f1e5006dca0 2024-04-05T03:17:32.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.815+0000 7f1e5a7fc700 1 -- 172.21.15.153:0/831619489 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1e6410a050 con 0x7f1e5006dca0 2024-04-05T03:17:32.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 msgr2=0x7f1e50070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 0x7f1e50070150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e54004f50 tx=0x7f1e54009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 msgr2=0x7f1e64113b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000b790 tx=0x7f1e60009250 comp rx=0 tx=0).stop 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 shutdown_connections 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1e5006dca0 0x7f1e50070150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e640fdeb0 0x7f1e64113b50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 --2- 172.21.15.153:0/831619489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e640fe850 0x7f1e64114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.817+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 >> 172.21.15.153:0/831619489 conn(0x7f1e64072bc0 msgr2=0x7f1e64103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.818+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 shutdown_connections 2024-04-05T03:17:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.818+0000 7f1e6bc1a700 1 -- 172.21.15.153:0/831619489 wait complete. 2024-04-05T03:17:32.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- 172.21.15.153:0/3410284701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7801026c0 msgr2=0x7fb7800fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 --2- 172.21.15.153:0/3410284701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7801026c0 0x7fb7800fbd30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb770004940 tx=0x7fb770009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:32.917 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:32 smithi153 ceph-mon[106102]: pgmap v356: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- 172.21.15.153:0/3410284701 shutdown_connections 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 --2- 172.21.15.153:0/3410284701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7801026c0 0x7fb7800fbd30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 --2- 172.21.15.153:0/3410284701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb780101d20 0x7fb7801020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- 172.21.15.153:0/3410284701 >> 172.21.15.153:0/3410284701 conn(0x7fb7800f78b0 msgr2=0x7fb7800f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- 172.21.15.153:0/3410284701 shutdown_connections 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- 172.21.15.153:0/3410284701 wait complete. 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 Processor -- start 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.918+0000 7fb78857b700 1 -- start start 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb78857b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb78857b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7801026c0 0x7fb78010f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb78857b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78010f620 con 0x7fb780101d20 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb78857b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78010f760 con 0x7fb7801026c0 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44786/0 (socket says 172.21.15.153:44786) 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 -- 172.21.15.153:0/19502568 learned_addr learned my addr 172.21.15.153:0/19502568 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb785b16700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7801026c0 0x7fb78010f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 -- 172.21.15.153:0/19502568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7801026c0 msgr2=0x7fb78010f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7801026c0 0x7fb78010f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 -- 172.21.15.153:0/19502568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770005040 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb786317700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00b790 tx=0x7fb77c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c00f6f0 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.919+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb77c00fd20 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.920+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb780117c10 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.920+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77c0103a0 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.920+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb780117fb0 con 0x7fb780101d20 2024-04-05T03:17:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.921+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7800fd470 con 0x7fb780101d20 2024-04-05T03:17:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.921+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb77c010500 con 0x7fb780101d20 2024-04-05T03:17:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.922+0000 7fb776ffd700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 0x7fb76c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.922+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb77c08d350 con 0x7fb780101d20 2024-04-05T03:17:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.922+0000 7fb785b16700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 0x7fb76c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:32.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.922+0000 7fb785b16700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 0x7fb76c070150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb780066a80 tx=0x7fb770009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:32.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:32.926+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb77c056950 con 0x7fb780101d20 2024-04-05T03:17:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:32 smithi203 ceph-mon[95988]: pgmap v356: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:33.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.129+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7800668a0 con 0x7fb76c06dca0 2024-04-05T03:17:33.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.140+0000 7fb776ffd700 1 -- 172.21.15.153:0/19502568 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb7800668a0 con 0x7fb76c06dca0 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (12m) 5m ago 23m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (9m) 5m ago 24m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (9m) 5m ago 22m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (11m) 5m ago 23m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (5m) 5m ago 17m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (6m) 5m ago 17m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (5m) 5m ago 17m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (5m) 5m ago 17m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (13m) 5m ago 25m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:17:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (12m) 5m ago 22m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (10m) 5m ago 25m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (10m) 5m ago 21m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (12m) 5m ago 23m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (12m) 5m ago 22m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (9m) 5m ago 21m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (8m) 5m ago 20m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (8m) 5m ago 20m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (7m) 5m ago 20m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (6m) 5m ago 19m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (6m) 5m ago 19m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:17:33.140 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (12m) 5m ago 21m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:17:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 msgr2=0x7fb76c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 0x7fb76c070150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb780066a80 tx=0x7fb770009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 msgr2=0x7fb78010eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00b790 tx=0x7fb77c009250 comp rx=0 tx=0).stop 2024-04-05T03:17:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 shutdown_connections 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb76c06dca0 0x7fb76c070150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb780101d20 0x7fb78010eba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 --2- 172.21.15.153:0/19502568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7801026c0 0x7fb78010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 >> 172.21.15.153:0/19502568 conn(0x7fb7800f78b0 msgr2=0x7fb7800f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 shutdown_connections 2024-04-05T03:17:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.142+0000 7fb78857b700 1 -- 172.21.15.153:0/19502568 wait complete. 2024-04-05T03:17:33.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/347764064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 msgr2=0x7fe8a4106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/347764064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a4106dc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe894004940 tx=0x7fe894009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/347764064 shutdown_connections 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/347764064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a4106dc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/347764064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 0x7fe8a40fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/347764064 >> 172.21.15.153:0/347764064 conn(0x7fe8a40f7890 msgr2=0x7fe8a40f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/347764064 shutdown_connections 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.242+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/347764064 wait complete. 2024-04-05T03:17:33.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 Processor -- start 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 -- start start 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 0x7fe8a40fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40fdb60 con 0x7fe8a40fe780 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8aa9ca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40fc690 con 0x7fe8a40fdde0 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8a37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8a37fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44800/0 (socket says 172.21.15.153:44800) 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.243+0000 7fe8a37fe700 1 -- 172.21.15.153:0/1822241991 learned_addr learned my addr 172.21.15.153:0/1822241991 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a3fff700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 0x7fe8a40fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a37fe700 1 -- 172.21.15.153:0/1822241991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 msgr2=0x7fe8a40fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a37fe700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 0x7fe8a40fbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a37fe700 1 -- 172.21.15.153:0/1822241991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894005040 con 0x7fe8a40fe780 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a37fe700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe894004890 tx=0x7fe894009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89400e040 con 0x7fe8a40fe780 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe894009af0 con 0x7fe8a40fe780 2024-04-05T03:17:33.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89401c3f0 con 0x7fe8a40fe780 2024-04-05T03:17:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a40fc910 con 0x7fe8a40fe780 2024-04-05T03:17:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.244+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a4119f80 con 0x7fe8a40fe780 2024-04-05T03:17:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.245+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4066bd0 con 0x7fe8a40fe780 2024-04-05T03:17:33.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.246+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe89400baf0 con 0x7fe8a40fe780 2024-04-05T03:17:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.247+0000 7fe8a0ff9700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 0x7fe88c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.247+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe89408e730 con 0x7fe8a40fe780 2024-04-05T03:17:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.247+0000 7fe8a3fff700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 0x7fe88c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.247+0000 7fe8a3fff700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 0x7fe88c070150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe898009fd0 tx=0x7fe89800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.251+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe894057d30 con 0x7fe8a40fe780 2024-04-05T03:17:33.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.530+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe8a4109e70 con 0x7fe8a40fe780 2024-04-05T03:17:33.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.531+0000 7fe8a0ff9700 1 -- 172.21.15.153:0/1822241991 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe89405b3f0 con 0x7fe8a40fe780 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:17:33.530 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:17:33.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:17:33.531 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:17:33.531 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:17:33.531 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:17:33.531 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 msgr2=0x7fe88c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 0x7fe88c070150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe898009fd0 tx=0x7fe89800b040 comp rx=0 tx=0).stop 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 msgr2=0x7fe8a40fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe894004890 tx=0x7fe894009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 shutdown_connections 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe88c06dca0 0x7fe88c070150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8a40fdde0 0x7fe8a40fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 --2- 172.21.15.153:0/1822241991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8a40fe780 0x7fe8a40fc150 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 >> 172.21.15.153:0/1822241991 conn(0x7fe8a40f7890 msgr2=0x7fe8a40f8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 shutdown_connections 2024-04-05T03:17:33.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.533+0000 7fe8aa9ca700 1 -- 172.21.15.153:0/1822241991 wait complete. 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 -- 172.21.15.153:0/2015279477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fe850 msgr2=0x7f60d0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/2015279477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fe850 0x7f60d0106f20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f60c0004970 tx=0x7f60c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 -- 172.21.15.153:0/2015279477 shutdown_connections 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/2015279477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fe850 0x7f60d0106f20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/2015279477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 -- 172.21.15.153:0/2015279477 >> 172.21.15.153:0/2015279477 conn(0x7f60d0072bc0 msgr2=0x7f60d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 -- 172.21.15.153:0/2015279477 shutdown_connections 2024-04-05T03:17:33.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.635+0000 7f60d7b1a700 1 -- 172.21.15.153:0/2015279477 wait complete. 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 Processor -- start 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 -- start start 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fe850 0x7f60d00fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00fc8d0 con 0x7f60d00fdeb0 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d7b1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00fca10 con 0x7f60d00fe850 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44822/0 (socket says 172.21.15.153:44822) 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 -- 172.21.15.153:0/424220838 learned_addr learned my addr 172.21.15.153:0/424220838 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:33.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d50b5700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fe850 0x7f60d00fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 -- 172.21.15.153:0/424220838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fe850 msgr2=0x7f60d00fc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fe850 0x7f60d00fc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 -- 172.21.15.153:0/424220838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c0005040 con 0x7f60d00fdeb0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.636+0000 7f60d58b6700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00b790 tx=0x7f60cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc00f6f0 con 0x7f60d00fdeb0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d0119e50 con 0x7f60d00fdeb0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60cc00fd20 con 0x7f60d00fdeb0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60cc0103a0 con 0x7f60d00fdeb0 2024-04-05T03:17:33.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d011a2b0 con 0x7f60d00fdeb0 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.637+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d0065e10 con 0x7f60d00fdeb0 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.638+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f60cc010500 con 0x7f60d00fdeb0 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.638+0000 7f60c67fc700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 0x7f60bc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.638+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f60cc08d350 con 0x7f60d00fdeb0 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.638+0000 7f60d50b5700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 0x7f60bc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.639+0000 7f60d50b5700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 0x7f60bc070150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60c0004f50 tx=0x7f60c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.643+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f60cc056950 con 0x7f60d00fdeb0 2024-04-05T03:17:33.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:33 smithi153 ceph-mon[106102]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:33.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:33 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1822241991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:17:33.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.883+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f60d00fd460 con 0x7f60d00fdeb0 2024-04-05T03:17:33.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.883+0000 7f60c67fc700 1 -- 172.21.15.153:0/424220838 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f60cc05a010 con 0x7f60d00fdeb0 2024-04-05T03:17:33.882 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:17:33.882 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:17:33.883 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:17:33.884 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:33.885 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 msgr2=0x7f60bc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 0x7f60bc070150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60c0004f50 tx=0x7f60c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 msgr2=0x7f60d00fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00b790 tx=0x7f60cc009250 comp rx=0 tx=0).stop 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 shutdown_connections 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f60bc06dca0 0x7f60bc070150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d00fdeb0 0x7f60d00fbd70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 --2- 172.21.15.153:0/424220838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d00fe850 0x7f60d00fc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 >> 172.21.15.153:0/424220838 conn(0x7f60d0072bc0 msgr2=0x7f60d0103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.885+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 shutdown_connections 2024-04-05T03:17:33.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.886+0000 7f60d7b1a700 1 -- 172.21.15.153:0/424220838 wait complete. 2024-04-05T03:17:33.887 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- 172.21.15.153:0/2389357499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 msgr2=0x7f68ec0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 --2- 172.21.15.153:0/2389357499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec0fcdf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f68e0004940 tx=0x7f68e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- 172.21.15.153:0/2389357499 shutdown_connections 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 --2- 172.21.15.153:0/2389357499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec0fcdf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 --2- 172.21.15.153:0/2389357499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 0x7f68ec1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- 172.21.15.153:0/2389357499 >> 172.21.15.153:0/2389357499 conn(0x7f68ec0f8950 msgr2=0x7f68ec0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- 172.21.15.153:0/2389357499 shutdown_connections 2024-04-05T03:17:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- 172.21.15.153:0/2389357499 wait complete. 2024-04-05T03:17:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 Processor -- start 2024-04-05T03:17:33.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.988+0000 7f68f23d6700 1 -- start start 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 0x7f68ec113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec114500 con 0x7f68ec103780 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec10db00 con 0x7f68ec102de0 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44832/0 (socket says 172.21.15.153:44832) 2024-04-05T03:17:33.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 -- 172.21.15.153:0/3444988583 learned_addr learned my addr 172.21.15.153:0/3444988583 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68ebfff700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 0x7f68ec113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 -- 172.21.15.153:0/3444988583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 msgr2=0x7f68ec113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 0x7f68ec113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 -- 172.21.15.153:0/3444988583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0005040 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68eb7fe700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009100 tx=0x7f68e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68e000e040 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68e0009af0 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec10dd80 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68e001c3f0 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.989+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec10e340 con 0x7f68ec103780 2024-04-05T03:17:33.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.990+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec0fe530 con 0x7f68ec103780 2024-04-05T03:17:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.991+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f68e000baf0 con 0x7f68ec103780 2024-04-05T03:17:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.991+0000 7f68e8ff9700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 0x7f68d4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.991+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f68e008e7d0 con 0x7f68ec103780 2024-04-05T03:17:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.991+0000 7f68ebfff700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 0x7f68d4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:33.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.992+0000 7f68ebfff700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 0x7f68d4070150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004eb0 tx=0x7f68dc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:33.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:33.996+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f68e005b900 con 0x7f68ec103780 2024-04-05T03:17:34.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:33 smithi203 ceph-mon[95988]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:34.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:33 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1822241991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:17:34.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.209+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68ec109ec0 con 0x7f68d406dca0 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.210+0000 7f68e8ff9700 1 -- 172.21.15.153:0/3444988583 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68ec109ec0 con 0x7f68d406dca0 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:17:34.209 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:17:34.210 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 msgr2=0x7f68d4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 0x7f68d4070150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004eb0 tx=0x7f68dc00b040 comp rx=0 tx=0).stop 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 msgr2=0x7f68ec113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009100 tx=0x7f68e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 shutdown_connections 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f68d406dca0 0x7f68d4070150 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68ec102de0 0x7f68ec113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 --2- 172.21.15.153:0/3444988583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f68ec103780 0x7f68ec113eb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 >> 172.21.15.153:0/3444988583 conn(0x7f68ec0f8950 msgr2=0x7f68ec0f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 shutdown_connections 2024-04-05T03:17:34.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.213+0000 7f68f23d6700 1 -- 172.21.15.153:0/3444988583 wait complete. 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 -- 172.21.15.153:0/37962597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 msgr2=0x7fe4880fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 --2- 172.21.15.153:0/37962597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe4880fbd30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe470004940 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 -- 172.21.15.153:0/37962597 shutdown_connections 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 --2- 172.21.15.153:0/37962597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe4880fbd30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 --2- 172.21.15.153:0/37962597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe4881020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 -- 172.21.15.153:0/37962597 >> 172.21.15.153:0/37962597 conn(0x7fe4880f78d0 msgr2=0x7fe4880f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:34.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 -- 172.21.15.153:0/37962597 shutdown_connections 2024-04-05T03:17:34.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.314+0000 7fe48e025700 1 -- 172.21.15.153:0/37962597 wait complete. 2024-04-05T03:17:34.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 Processor -- start 2024-04-05T03:17:34.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 -- start start 2024-04-05T03:17:34.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe488112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:34.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:34.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4881136b0 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe48e025700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48810caa0 con 0x7fe488101d20 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe486ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe486ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44850/0 (socket says 172.21.15.153:44850) 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe486ffd700 1 -- 172.21.15.153:0/3825724906 learned_addr learned my addr 172.21.15.153:0/3825724906 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.315+0000 7fe4877fe700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe488112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe486ffd700 1 -- 172.21.15.153:0/3825724906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 msgr2=0x7fe488112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe486ffd700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe488112b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe486ffd700 1 -- 172.21.15.153:0/3825724906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470005040 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe4877fe700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe488112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe486ffd700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe470009100 tx=0x7fe470009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47000e040 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48810cd20 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe470009af0 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47001a630 con 0x7fe4881026c0 2024-04-05T03:17:34.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.316+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48810d2e0 con 0x7fe4881026c0 2024-04-05T03:17:34.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.317+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4880fd470 con 0x7fe4881026c0 2024-04-05T03:17:34.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.318+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe470020020 con 0x7fe4881026c0 2024-04-05T03:17:34.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.318+0000 7fe47ffff700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 0x7fe474070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:17:34.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.318+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe470090280 con 0x7fe4881026c0 2024-04-05T03:17:34.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.318+0000 7fe4877fe700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 0x7fe474070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:17:34.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.319+0000 7fe4877fe700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 0x7fe474070150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe480007d40 tx=0x7fe480009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:17:34.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.323+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe47005d3b0 con 0x7fe4881026c0 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.600+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe48805f5f0 con 0x7fe4881026c0 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.600+0000 7fe47ffff700 1 -- 172.21.15.153:0/3825724906 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe47005cf40 con 0x7fe4881026c0 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:17:34.599 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:17:34.600 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:17:34.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 msgr2=0x7fe474070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 0x7fe474070150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe480007d40 tx=0x7fe480009da0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 msgr2=0x7fe488113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe470009100 tx=0x7fe470009e30 comp rx=0 tx=0).stop 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 shutdown_connections 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe47406dca0 0x7fe474070150 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe488101d20 0x7fe488112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 --2- 172.21.15.153:0/3825724906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4881026c0 0x7fe488113060 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.602+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 >> 172.21.15.153:0/3825724906 conn(0x7fe4880f78d0 msgr2=0x7fe4880f88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.603+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 shutdown_connections 2024-04-05T03:17:34.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:17:34.603+0000 7fe48e025700 1 -- 172.21.15.153:0/3825724906 wait complete. 2024-04-05T03:17:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:34 smithi203 ceph-mon[95988]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:34 smithi203 ceph-mon[95988]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:34 smithi203 ceph-mon[95988]: pgmap v357: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:34 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/424220838' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:17:35.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:34 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3825724906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:17:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:34 smithi153 ceph-mon[106102]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:34 smithi153 ceph-mon[106102]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:34 smithi153 ceph-mon[106102]: pgmap v357: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:34 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/424220838' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:17:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:34 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3825724906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:17:36.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:35 smithi203 ceph-mon[95988]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:36.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:35 smithi153 ceph-mon[106102]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:17:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:36 smithi203 ceph-mon[95988]: pgmap v358: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:36 smithi153 ceph-mon[106102]: pgmap v358: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:38 smithi203 ceph-mon[95988]: pgmap v359: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:38 smithi153 ceph-mon[106102]: pgmap v359: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:41.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:40 smithi203 ceph-mon[95988]: pgmap v360: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:40 smithi153 ceph-mon[106102]: pgmap v360: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:42 smithi203 ceph-mon[95988]: pgmap v361: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:43.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:42 smithi153 ceph-mon[106102]: pgmap v361: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:44 smithi203 ceph-mon[95988]: pgmap v362: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:44 smithi153 ceph-mon[106102]: pgmap v362: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:46 smithi203 ceph-mon[95988]: pgmap v363: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:46 smithi153 ceph-mon[106102]: pgmap v363: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:48 smithi203 ceph-mon[95988]: pgmap v364: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:48 smithi153 ceph-mon[106102]: pgmap v364: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:50 smithi203 ceph-mon[95988]: pgmap v365: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:50 smithi153 ceph-mon[106102]: pgmap v365: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:52 smithi203 ceph-mon[95988]: pgmap v366: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:52 smithi153 ceph-mon[106102]: pgmap v366: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:54 smithi203 ceph-mon[95988]: pgmap v367: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:54 smithi153 ceph-mon[106102]: pgmap v367: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:56 smithi203 ceph-mon[95988]: pgmap v368: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:56 smithi153 ceph-mon[106102]: pgmap v368: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:17:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:58 smithi203 ceph-mon[95988]: pgmap v369: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:17:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:58 smithi153 ceph-mon[106102]: pgmap v369: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:00.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:17:59 smithi203 ceph-mon[95988]: pgmap v370: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:00.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:17:59 smithi153 ceph-mon[106102]: pgmap v370: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:02 smithi203 ceph-mon[95988]: pgmap v371: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:02 smithi153 ceph-mon[106102]: pgmap v371: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:04.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 -- 172.21.15.153:0/369911931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a4101d20 msgr2=0x7f78a41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:04.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 --2- 172.21.15.153:0/369911931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a4101d20 0x7f78a41020f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7894009ce0 tx=0x7f7894004d10 comp rx=0 tx=0).stop 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 -- 172.21.15.153:0/369911931 shutdown_connections 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 --2- 172.21.15.153:0/369911931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a41026c0 0x7f78a40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 --2- 172.21.15.153:0/369911931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a4101d20 0x7f78a41020f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 -- 172.21.15.153:0/369911931 >> 172.21.15.153:0/369911931 conn(0x7f78a40f7850 msgr2=0x7f78a40f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 -- 172.21.15.153:0/369911931 shutdown_connections 2024-04-05T03:18:04.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.718+0000 7f78a90c1700 1 -- 172.21.15.153:0/369911931 wait complete. 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 Processor -- start 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 -- start start 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a4101d20 0x7f78a4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4114680 con 0x7f78a41026c0 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a90c1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a410dc60 con 0x7f78a4101d20 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52558/0 (socket says 172.21.15.153:52558) 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.719+0000 7f78a259c700 1 -- 172.21.15.153:0/2734627082 learned_addr learned my addr 172.21.15.153:0/2734627082 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:04.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a2d9d700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a4101d20 0x7f78a4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a259c700 1 -- 172.21.15.153:0/2734627082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a4101d20 msgr2=0x7f78a4113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a259c700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a4101d20 0x7f78a4113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a259c700 1 -- 172.21.15.153:0/2734627082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894009940 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a259c700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c800 tx=0x7f788c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f788c00a760 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f788c00ad90 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a410df40 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f788c0113a0 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.720+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a410e500 con 0x7f78a41026c0 2024-04-05T03:18:04.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.721+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a40fd470 con 0x7f78a41026c0 2024-04-05T03:18:04.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.722+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f788c012400 con 0x7f78a41026c0 2024-04-05T03:18:04.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.722+0000 7f789b7fe700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 0x7f7890070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:04.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.722+0000 7f78a2d9d700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 0x7f7890070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:04.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.722+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f788c08d660 con 0x7f78a41026c0 2024-04-05T03:18:04.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.723+0000 7f78a2d9d700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 0x7f7890070150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f789400b740 tx=0x7f7894004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:04.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.727+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f788c05a6f0 con 0x7f78a41026c0 2024-04-05T03:18:04.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.942+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78a4108e20 con 0x7f789006dca0 2024-04-05T03:18:04.941 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:04 smithi153 ceph-mon[106102]: pgmap v372: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:04.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.943+0000 7f789b7fe700 1 -- 172.21.15.153:0/2734627082 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f78a4108e20 con 0x7f789006dca0 2024-04-05T03:18:04.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 msgr2=0x7f7890070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:04.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 0x7f7890070150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f789400b740 tx=0x7f7894004d10 comp rx=0 tx=0).stop 2024-04-05T03:18:04.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 msgr2=0x7f78a4114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:04.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c800 tx=0x7f788c004470 comp rx=0 tx=0).stop 2024-04-05T03:18:04.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 shutdown_connections 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f789006dca0 0x7f7890070150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78a4101d20 0x7f78a4113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 --2- 172.21.15.153:0/2734627082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78a41026c0 0x7f78a4114030 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.945+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 >> 172.21.15.153:0/2734627082 conn(0x7f78a40f7850 msgr2=0x7f78a40f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.946+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 shutdown_connections 2024-04-05T03:18:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:04.946+0000 7f78a90c1700 1 -- 172.21.15.153:0/2734627082 wait complete. 2024-04-05T03:18:04.955 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:18:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:04 smithi203 ceph-mon[95988]: pgmap v372: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:05.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.048+0000 7f147cd86700 1 -- 172.21.15.153:0/532719292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 msgr2=0x7f14780fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.048+0000 7f147cd86700 1 --2- 172.21.15.153:0/532719292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f14780fbd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f146c004940 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.048+0000 7f147cd86700 1 -- 172.21.15.153:0/532719292 shutdown_connections 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.048+0000 7f147cd86700 1 --2- 172.21.15.153:0/532719292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f14780fbd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.048+0000 7f147cd86700 1 --2- 172.21.15.153:0/532719292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 0x7f14781020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- 172.21.15.153:0/532719292 >> 172.21.15.153:0/532719292 conn(0x7f14780f7880 msgr2=0x7f14780f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- 172.21.15.153:0/532719292 shutdown_connections 2024-04-05T03:18:05.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- 172.21.15.153:0/532719292 wait complete. 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 Processor -- start 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- start start 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 0x7f1478112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781138b0 con 0x7f14781026b0 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f147cd86700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147810cc80 con 0x7f1478101d10 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.049+0000 7f1475d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52580/0 (socket says 172.21.15.153:52580) 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 -- 172.21.15.153:0/3683185217 learned_addr learned my addr 172.21.15.153:0/3683185217 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f147659c700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 0x7f1478112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 -- 172.21.15.153:0/3683185217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 msgr2=0x7f1478112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 0x7f1478112d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 -- 172.21.15.153:0/3683185217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c005040 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1475d9b700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f146c000c00 tx=0x7f146c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c021070 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f146c013c00 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147810cf00 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c00e070 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.050+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147810d3e0 con 0x7f14781026b0 2024-04-05T03:18:05.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.051+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14780fd460 con 0x7f14781026b0 2024-04-05T03:18:05.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.052+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f146c01ca60 con 0x7f14781026b0 2024-04-05T03:18:05.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.052+0000 7f1466ffd700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 0x7f1460070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.052+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f146c08e8d0 con 0x7f14781026b0 2024-04-05T03:18:05.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.052+0000 7f147659c700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 0x7f1460070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.052+0000 7f147659c700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 0x7f1460070150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1468009b80 tx=0x7f146800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.057+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f146c05baa0 con 0x7f14781026b0 2024-04-05T03:18:05.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.269+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1478108e60 con 0x7f146006dca0 2024-04-05T03:18:05.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.271+0000 7f1466ffd700 1 -- 172.21.15.153:0/3683185217 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1478108e60 con 0x7f146006dca0 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 msgr2=0x7f1460070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 0x7f1460070150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1468009b80 tx=0x7f146800b040 comp rx=0 tx=0).stop 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 msgr2=0x7f1478113260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f146c000c00 tx=0x7f146c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 shutdown_connections 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f146006dca0 0x7f1460070150 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1478101d10 0x7f1478112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 --2- 172.21.15.153:0/3683185217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14781026b0 0x7f1478113260 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 >> 172.21.15.153:0/3683185217 conn(0x7f14780f7880 msgr2=0x7f14780f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 shutdown_connections 2024-04-05T03:18:05.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.273+0000 7f147cd86700 1 -- 172.21.15.153:0/3683185217 wait complete. 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 -- 172.21.15.153:0/1736768865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9341037b0 msgr2=0x7fa9340fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 --2- 172.21.15.153:0/1736768865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9341037b0 0x7fa9340fce20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa91c004940 tx=0x7fa91c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 -- 172.21.15.153:0/1736768865 shutdown_connections 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 --2- 172.21.15.153:0/1736768865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9341037b0 0x7fa9340fce20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 --2- 172.21.15.153:0/1736768865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa934102e10 0x7fa9341031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 -- 172.21.15.153:0/1736768865 >> 172.21.15.153:0/1736768865 conn(0x7fa9340f89a0 msgr2=0x7fa9340fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 -- 172.21.15.153:0/1736768865 shutdown_connections 2024-04-05T03:18:05.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.374+0000 7fa939ec8700 1 -- 172.21.15.153:0/1736768865 wait complete. 2024-04-05T03:18:05.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 Processor -- start 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 -- start start 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 0x7fa934070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa934070d90 con 0x7fa934102e10 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa939ec8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa934070ed0 con 0x7fa9341037b0 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52596/0 (socket says 172.21.15.153:52596) 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 -- 172.21.15.153:0/2803360649 learned_addr learned my addr 172.21.15.153:0/2803360649 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa92affd700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 0x7fa934070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 -- 172.21.15.153:0/2803360649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 msgr2=0x7fa934070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 0x7fa934070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa9337fe700 1 -- 172.21.15.153:0/2803360649 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c005040 con 0x7fa934102e10 2024-04-05T03:18:05.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.375+0000 7fa92affd700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 0x7fa934070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa9337fe700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa924002a50 tx=0x7fa9240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa924007cc0 con 0x7fa934102e10 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa92400f040 con 0x7fa934102e10 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa934119ea0 con 0x7fa934102e10 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa924009640 con 0x7fa934102e10 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.376+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93411a320 con 0x7fa934102e10 2024-04-05T03:18:05.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.377+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9341052e0 con 0x7fa934102e10 2024-04-05T03:18:05.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.377+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa9240097a0 con 0x7fa934102e10 2024-04-05T03:18:05.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.378+0000 7fa930ff9700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 0x7fa920070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.378+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa92408d280 con 0x7fa934102e10 2024-04-05T03:18:05.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.378+0000 7fa92affd700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 0x7fa920070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.378+0000 7fa92affd700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 0x7fa920070150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000c00 tx=0x7fa91c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.382+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa924056880 con 0x7fa934102e10 2024-04-05T03:18:05.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.588+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa93404bed0 con 0x7fa92006dca0 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.598+0000 7fa930ff9700 1 -- 172.21.15.153:0/2803360649 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa93404bed0 con 0x7fa92006dca0 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (12m) 6m ago 23m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (10m) 6m ago 24m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (10m) 5m ago 22m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (12m) 6m ago 23m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:18:05.597 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (6m) 6m ago 18m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (6m) 6m ago 18m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (5m) 5m ago 18m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (6m) 5m ago 18m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (14m) 6m ago 26m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (13m) 5m ago 22m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (11m) 6m ago 26m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (10m) 5m ago 22m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:18:05.598 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (13m) 6m ago 23m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (13m) 5m ago 22m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (9m) 6m ago 21m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (9m) 6m ago 21m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (8m) 6m ago 20m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (8m) 5m ago 20m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (7m) 5m ago 20m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (6m) 5m ago 19m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:18:05.599 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (12m) 5m ago 22m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 msgr2=0x7fa920070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 0x7fa920070150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000c00 tx=0x7fa91c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 msgr2=0x7fa9340702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa924002a50 tx=0x7fa9240046b0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 shutdown_connections 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa92006dca0 0x7fa920070150 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa934102e10 0x7fa9340702c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 --2- 172.21.15.153:0/2803360649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341037b0 0x7fa934070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 >> 172.21.15.153:0/2803360649 conn(0x7fa9340f89a0 msgr2=0x7fa934102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 shutdown_connections 2024-04-05T03:18:05.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.600+0000 7fa939ec8700 1 -- 172.21.15.153:0/2803360649 wait complete. 2024-04-05T03:18:05.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- 172.21.15.153:0/1826296864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 msgr2=0x7f2a7c0fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 --2- 172.21.15.153:0/1826296864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fb820 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2a64004940 tx=0x7f2a64009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- 172.21.15.153:0/1826296864 shutdown_connections 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 --2- 172.21.15.153:0/1826296864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fb820 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 --2- 172.21.15.153:0/1826296864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- 172.21.15.153:0/1826296864 >> 172.21.15.153:0/1826296864 conn(0x7f2a7c072310 msgr2=0x7f2a7c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- 172.21.15.153:0/1826296864 shutdown_connections 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- 172.21.15.153:0/1826296864 wait complete. 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 Processor -- start 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.702+0000 7f2a80c64700 1 -- start start 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c0fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c0fe520 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c0fd010 con 0x7f2a7c102f70 2024-04-05T03:18:05.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52618/0 (socket says 172.21.15.153:52618) 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 -- 172.21.15.153:0/122968199 learned_addr learned my addr 172.21.15.153:0/122968199 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a71d9b700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c0fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 -- 172.21.15.153:0/122968199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 msgr2=0x7f2a7c0fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c0fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 -- 172.21.15.153:0/122968199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64005040 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a71d9b700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c0fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a7a59c700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00b790 tx=0x7f2a6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c00f770 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a6c00fda0 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a6c00d310 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c0fd2f0 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.703+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c11a180 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.704+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c1053b0 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.705+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2a6c010420 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.705+0000 7f2a737fe700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 0x7f2a68070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.705+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2a6c08d5b0 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.705+0000 7f2a71d9b700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 0x7f2a68070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:05.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.706+0000 7f2a71d9b700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 0x7f2a68070150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2a64004f90 tx=0x7f2a64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:05.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.710+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2a6c056bb0 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.989+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2a7c04c710 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.990+0000 7f2a737fe700 1 -- 172.21.15.153:0/122968199 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2a6c05a270 con 0x7f2a7c0fb3b0 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:18:05.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:18:05.990 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:18:05.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 msgr2=0x7f2a68070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 0x7f2a68070150 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2a64004f90 tx=0x7f2a64009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:05.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 msgr2=0x7f2a7c0fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:05.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00b790 tx=0x7f2a6c009250 comp rx=0 tx=0).stop 2024-04-05T03:18:05.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 shutdown_connections 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2a6806dca0 0x7f2a68070150 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a7c0fb3b0 0x7f2a7c0fc590 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 --2- 172.21.15.153:0/122968199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a7c102f70 0x7f2a7c0fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 >> 172.21.15.153:0/122968199 conn(0x7f2a7c072310 msgr2=0x7f2a7c0f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 shutdown_connections 2024-04-05T03:18:05.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:05.993+0000 7f2a80c64700 1 -- 172.21.15.153:0/122968199 wait complete. 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- 172.21.15.153:0/2571657413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 msgr2=0x7f97480fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 --2- 172.21.15.153:0/2571657413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f97480fce30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9738004940 tx=0x7f9738009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- 172.21.15.153:0/2571657413 shutdown_connections 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 --2- 172.21.15.153:0/2571657413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f97480fce30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 --2- 172.21.15.153:0/2571657413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97481031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- 172.21.15.153:0/2571657413 >> 172.21.15.153:0/2571657413 conn(0x7f97480f89b0 msgr2=0x7f97480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- 172.21.15.153:0/2571657413 shutdown_connections 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- 172.21.15.153:0/2571657413 wait complete. 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 Processor -- start 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.095+0000 7f97500cc700 1 -- start start 2024-04-05T03:18:06.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748072210 con 0x7f97481037c0 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748070d40 con 0x7f9748102e20 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52630/0 (socket says 172.21.15.153:52630) 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 -- 172.21.15.153:0/3369525138 learned_addr learned my addr 172.21.15.153:0/3369525138 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974de68700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 -- 172.21.15.153:0/3369525138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 msgr2=0x7f97480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97480702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 -- 172.21.15.153:0/3369525138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738005040 con 0x7f97481037c0 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974de68700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97480702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f974d667700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9738009100 tx=0x7f9738009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973800e040 con 0x7f97481037c0 2024-04-05T03:18:06.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9738013420 con 0x7f97481037c0 2024-04-05T03:18:06.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748070fc0 con 0x7f97481037c0 2024-04-05T03:18:06.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973801c610 con 0x7f97481037c0 2024-04-05T03:18:06.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.096+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974811a0f0 con 0x7f97481037c0 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.097+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97481052f0 con 0x7f97481037c0 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.098+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9738013590 con 0x7f97481037c0 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.098+0000 7f973e7fc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 0x7f9734070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.099+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f973808e730 con 0x7f97481037c0 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.099+0000 7f974de68700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 0x7f9734070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.099+0000 7f974de68700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 0x7f9734070150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9744009fd0 tx=0x7f9744009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.103+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9738057dd0 con 0x7f97481037c0 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.343+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9748109f00 con 0x7f97481037c0 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.343+0000 7f973e7fc700 1 -- 172.21.15.153:0/3369525138 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f973805b490 con 0x7f97481037c0 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:18:06.342 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:18:06.343 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:18:06.344 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:06.345 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 msgr2=0x7f9734070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 0x7f9734070150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9744009fd0 tx=0x7f9744009e50 comp rx=0 tx=0).stop 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 msgr2=0x7f9748070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9738009100 tx=0x7f9738009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 shutdown_connections 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f973406dca0 0x7f9734070150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9748102e20 0x7f97480702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 --2- 172.21.15.153:0/3369525138 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97481037c0 0x7f9748070800 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 >> 172.21.15.153:0/3369525138 conn(0x7f97480f89b0 msgr2=0x7f97481021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 shutdown_connections 2024-04-05T03:18:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.346+0000 7f97500cc700 1 -- 172.21.15.153:0/3369525138 wait complete. 2024-04-05T03:18:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:18:06.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 -- 172.21.15.153:0/4285437493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 msgr2=0x7fa3f00fe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 --2- 172.21.15.153:0/4285437493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f00fe1c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0004940 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:06.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 -- 172.21.15.153:0/4285437493 shutdown_connections 2024-04-05T03:18:06.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 --2- 172.21.15.153:0/4285437493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 0x7fa3f0106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 --2- 172.21.15.153:0/4285437493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f00fe1c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 -- 172.21.15.153:0/4285437493 >> 172.21.15.153:0/4285437493 conn(0x7fa3f00f7860 msgr2=0x7fa3f00f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:06.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 -- 172.21.15.153:0/4285437493 shutdown_connections 2024-04-05T03:18:06.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.448+0000 7fa3f5964700 1 -- 172.21.15.153:0/4285437493 wait complete. 2024-04-05T03:18:06.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 Processor -- start 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 -- start start 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 0x7fa3f01140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0114700 con 0x7fa3f00fddf0 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3f5964700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010daf0 con 0x7fa3f00fe790 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3eeffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52644/0 (socket says 172.21.15.153:52644) 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3eeffd700 1 -- 172.21.15.153:0/2959499803 learned_addr learned my addr 172.21.15.153:0/2959499803 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3ee7fc700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 0x7fa3f01140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3eeffd700 1 -- 172.21.15.153:0/2959499803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 msgr2=0x7fa3f01140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.449+0000 7fa3eeffd700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 0x7fa3f01140b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3eeffd700 1 -- 172.21.15.153:0/2959499803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0005040 con 0x7fa3f00fddf0 2024-04-05T03:18:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3eeffd700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009380 tx=0x7fa3e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e000e040 con 0x7fa3f00fddf0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3e0009af0 con 0x7fa3f00fddf0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f010dd70 con 0x7fa3f00fddf0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e001c630 con 0x7fa3f00fddf0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.450+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f010e2b0 con 0x7fa3f00fddf0 2024-04-05T03:18:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.451+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0066770 con 0x7fa3f00fddf0 2024-04-05T03:18:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.451+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa3e000bbc0 con 0x7fa3f00fddf0 2024-04-05T03:18:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.452+0000 7fa3d77fe700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 0x7fa3d8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.452+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3e008e7b0 con 0x7fa3f00fddf0 2024-04-05T03:18:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.452+0000 7fa3ee7fc700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 0x7fa3d8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.452+0000 7fa3ee7fc700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 0x7fa3d8070150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3f010ecb0 tx=0x7fa3e4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.456+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3e005b980 con 0x7fa3f00fddf0 2024-04-05T03:18:06.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.669+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f0109ed0 con 0x7fa3d806dca0 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.670+0000 7fa3d77fe700 1 -- 172.21.15.153:0/2959499803 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa3f0109ed0 con 0x7fa3d806dca0 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:18:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:18:06.670 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:18:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 msgr2=0x7fa3d8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 0x7fa3d8070150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3f010ecb0 tx=0x7fa3e4009ee0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 msgr2=0x7fa3f0113b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009380 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 shutdown_connections 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3d806dca0 0x7fa3d8070150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3f00fddf0 0x7fa3f0113b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 --2- 172.21.15.153:0/2959499803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3f00fe790 0x7fa3f01140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.672+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 >> 172.21.15.153:0/2959499803 conn(0x7fa3f00f7860 msgr2=0x7fa3f00f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.673+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 shutdown_connections 2024-04-05T03:18:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.673+0000 7fa3f5964700 1 -- 172.21.15.153:0/2959499803 wait complete. 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 -- 172.21.15.153:0/4239658735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340103780 msgr2=0x7f03400fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 --2- 172.21.15.153:0/4239658735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340103780 0x7f03400fcdf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0334004940 tx=0x7f0334009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 -- 172.21.15.153:0/4239658735 shutdown_connections 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 --2- 172.21.15.153:0/4239658735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340103780 0x7f03400fcdf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 --2- 172.21.15.153:0/4239658735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340102de0 0x7f03401031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 -- 172.21.15.153:0/4239658735 >> 172.21.15.153:0/4239658735 conn(0x7f03400f8950 msgr2=0x7f03400fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:06.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 -- 172.21.15.153:0/4239658735 shutdown_connections 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.773+0000 7f0344afd700 1 -- 172.21.15.153:0/4239658735 wait complete. 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 Processor -- start 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 -- start start 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 0x7f0340113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0340114500 con 0x7f0340102de0 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f0344afd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034010db00 con 0x7f0340103780 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f033e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f033e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52660/0 (socket says 172.21.15.153:52660) 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f033e59c700 1 -- 172.21.15.153:0/2983709773 learned_addr learned my addr 172.21.15.153:0/2983709773 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.774+0000 7f033dd9b700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 0x7f0340113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f033e59c700 1 -- 172.21.15.153:0/2983709773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 msgr2=0x7f0340113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:06.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f033e59c700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 0x7f0340113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f033e59c700 1 -- 172.21.15.153:0/2983709773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334005040 con 0x7f0340102de0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f033dd9b700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 0x7f0340113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f033e59c700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0330002a50 tx=0x7f03300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0330007cc0 con 0x7f0340102de0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f033000f040 con 0x7f0340102de0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034010dde0 con 0x7f0340102de0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0330009640 con 0x7f0340102de0 2024-04-05T03:18:06.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.775+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034010e3a0 con 0x7f0340102de0 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.776+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03400fe530 con 0x7f0340102de0 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.776+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f03300097a0 con 0x7f0340102de0 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.777+0000 7f032effd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 0x7f0328070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.777+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f033008d2a0 con 0x7f0340102de0 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.777+0000 7f033dd9b700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 0x7f0328070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:06.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.777+0000 7f033dd9b700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 0x7f0328070150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f033400b3b0 tx=0x7f0334009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:06.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:06.781+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f033005a3d0 con 0x7f0340102de0 2024-04-05T03:18:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: pgmap v373: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/122968199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:18:07.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:06 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3369525138' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:18:07.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.055+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0340109ec0 con 0x7f0340102de0 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: pgmap v373: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/122968199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:18:07.055 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:06 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3369525138' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:18:07.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.056+0000 7f032effd700 1 -- 172.21.15.153:0/2983709773 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f0330056030 con 0x7f0340102de0 2024-04-05T03:18:07.055 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:18:07.056 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:18:07.056 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:18:07.056 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:18:07.056 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 msgr2=0x7f0328070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 0x7f0328070150 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f033400b3b0 tx=0x7f0334009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 msgr2=0x7f0340113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0330002a50 tx=0x7f03300046b0 comp rx=0 tx=0).stop 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 shutdown_connections 2024-04-05T03:18:07.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f032806dca0 0x7f0328070150 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:07.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0340102de0 0x7f0340113970 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:07.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 --2- 172.21.15.153:0/2983709773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0340103780 0x7f0340113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:07.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 >> 172.21.15.153:0/2983709773 conn(0x7f03400f8950 msgr2=0x7f03400f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:07.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 shutdown_connections 2024-04-05T03:18:07.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:07.058+0000 7f0344afd700 1 -- 172.21.15.153:0/2983709773 wait complete. 2024-04-05T03:18:08.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:07 smithi203 ceph-mon[95988]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:08.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:07 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2983709773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:18:08.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:07 smithi153 ceph-mon[106102]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:08.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:07 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2983709773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:18:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:08 smithi203 ceph-mon[95988]: pgmap v374: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:08 smithi153 ceph-mon[106102]: pgmap v374: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:10 smithi203 ceph-mon[95988]: pgmap v375: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:10 smithi153 ceph-mon[106102]: pgmap v375: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:12 smithi203 ceph-mon[95988]: pgmap v376: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:12 smithi153 ceph-mon[106102]: pgmap v376: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:14 smithi203 ceph-mon[95988]: pgmap v377: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:14 smithi153 ceph-mon[106102]: pgmap v377: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:16 smithi203 ceph-mon[95988]: pgmap v378: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:16 smithi153 ceph-mon[106102]: pgmap v378: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:18 smithi203 ceph-mon[95988]: pgmap v379: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:18 smithi153 ceph-mon[106102]: pgmap v379: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:20 smithi203 ceph-mon[95988]: pgmap v380: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:20 smithi153 ceph-mon[106102]: pgmap v380: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:22 smithi203 ceph-mon[95988]: pgmap v381: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:22 smithi153 ceph-mon[106102]: pgmap v381: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:24 smithi203 ceph-mon[95988]: pgmap v382: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:24 smithi153 ceph-mon[106102]: pgmap v382: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:26 smithi203 ceph-mon[95988]: pgmap v383: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:26 smithi153 ceph-mon[106102]: pgmap v383: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:28 smithi203 ceph-mon[95988]: pgmap v384: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:28 smithi153 ceph-mon[106102]: pgmap v384: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:30 smithi203 ceph-mon[95988]: pgmap v385: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:30 smithi153 ceph-mon[106102]: pgmap v385: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:32 smithi203 ceph-mon[95988]: pgmap v386: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:32 smithi153 ceph-mon[106102]: pgmap v386: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:34 smithi203 ceph-mon[95988]: pgmap v387: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:34 smithi153 ceph-mon[106102]: pgmap v387: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:36 smithi203 ceph-mon[95988]: pgmap v388: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:36 smithi153 ceph-mon[106102]: pgmap v388: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.171+0000 7f821c293700 1 -- 172.21.15.153:0/346376835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 msgr2=0x7f82141022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.171+0000 7f821c293700 1 --2- 172.21.15.153:0/346376835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f82141022d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8204004940 tx=0x7f8204009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 -- 172.21.15.153:0/346376835 shutdown_connections 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 --2- 172.21.15.153:0/346376835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 0x7f82140fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 --2- 172.21.15.153:0/346376835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f82141022d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 -- 172.21.15.153:0/346376835 >> 172.21.15.153:0/346376835 conn(0x7f82140714c0 msgr2=0x7f82140718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 -- 172.21.15.153:0/346376835 shutdown_connections 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 -- 172.21.15.153:0/346376835 wait complete. 2024-04-05T03:18:37.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 Processor -- start 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.172+0000 7f821c293700 1 -- start start 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821c293700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 0x7f8214112910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821c293700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821c293700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141134a0 con 0x7f8214101f00 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821c293700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821410caa0 con 0x7f82140fc5e0 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50438/0 (socket says 172.21.15.153:50438) 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 -- 172.21.15.153:0/1203764840 learned_addr learned my addr 172.21.15.153:0/1203764840 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821a02f700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 0x7f8214112910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 -- 172.21.15.153:0/1203764840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 msgr2=0x7f8214112910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 0x7f8214112910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 -- 172.21.15.153:0/1203764840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204005040 con 0x7f8214101f00 2024-04-05T03:18:37.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f821982e700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f821000c810 tx=0x7f8210004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.173+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8210012810 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.174+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8210012e40 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.174+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821410cd80 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.174+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821410d2c0 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.174+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82100102e0 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.175+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82140fd5f0 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.175+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8210011380 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.176+0000 7f820affd700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 0x7f8200070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.176+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f821008d970 con 0x7f8214101f00 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.176+0000 7f821a02f700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 0x7f8200070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.176+0000 7f821a02f700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 0x7f8200070150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8204009fd0 tx=0x7f8204009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.180+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f821005aa20 con 0x7f8214101f00 2024-04-05T03:18:37.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.395+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8214108fb0 con 0x7f820006dca0 2024-04-05T03:18:37.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.396+0000 7f820affd700 1 -- 172.21.15.153:0/1203764840 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8214108fb0 con 0x7f820006dca0 2024-04-05T03:18:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 msgr2=0x7f8200070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 0x7f8200070150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8204009fd0 tx=0x7f8204009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 msgr2=0x7f8214112e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f821000c810 tx=0x7f8210004470 comp rx=0 tx=0).stop 2024-04-05T03:18:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 shutdown_connections 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f820006dca0 0x7f8200070150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82140fc5e0 0x7f8214112910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 --2- 172.21.15.153:0/1203764840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8214101f00 0x7f8214112e50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 >> 172.21.15.153:0/1203764840 conn(0x7f82140714c0 msgr2=0x7f82140f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 shutdown_connections 2024-04-05T03:18:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.398+0000 7f821c293700 1 -- 172.21.15.153:0/1203764840 wait complete. 2024-04-05T03:18:37.408 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:18:37.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 -- 172.21.15.153:0/623308992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102e10 msgr2=0x7f63541031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 --2- 172.21.15.153:0/623308992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102e10 0x7f63541031e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6350004940 tx=0x7f6350009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 -- 172.21.15.153:0/623308992 shutdown_connections 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 --2- 172.21.15.153:0/623308992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63541037b0 0x7f63540fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 --2- 172.21.15.153:0/623308992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102e10 0x7f63541031e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 -- 172.21.15.153:0/623308992 >> 172.21.15.153:0/623308992 conn(0x7f63540f89a0 msgr2=0x7f63540fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 -- 172.21.15.153:0/623308992 shutdown_connections 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.501+0000 7f635c54c700 1 -- 172.21.15.153:0/623308992 wait complete. 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 Processor -- start 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 -- start start 2024-04-05T03:18:37.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 0x7f6354113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541146f0 con 0x7f63541037b0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635c54c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635410dae0 con 0x7f6354102e10 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50460/0 (socket says 172.21.15.153:50460) 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 -- 172.21.15.153:0/565352060 learned_addr learned my addr 172.21.15.153:0/565352060 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635a2e8700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 0x7f6354113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 -- 172.21.15.153:0/565352060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 msgr2=0x7f6354113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 0x7f6354113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 -- 172.21.15.153:0/565352060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350005040 con 0x7f63541037b0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f635a2e8700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 0x7f6354113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.502+0000 7f6359ae7700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f634400eb40 tx=0x7f634400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.503+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6344007cc0 con 0x7f63541037b0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.503+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f634400f040 con 0x7f63541037b0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.503+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63440095e0 con 0x7f63541037b0 2024-04-05T03:18:37.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.503+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635410dd60 con 0x7f63541037b0 2024-04-05T03:18:37.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.503+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635410e250 con 0x7f63541037b0 2024-04-05T03:18:37.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.504+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6354100120 con 0x7f63541037b0 2024-04-05T03:18:37.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.504+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6344009740 con 0x7f63541037b0 2024-04-05T03:18:37.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.504+0000 7f634affd700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 0x7f6340070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.505+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f634408d050 con 0x7f63541037b0 2024-04-05T03:18:37.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.505+0000 7f635a2e8700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 0x7f6340070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.505+0000 7f635a2e8700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 0x7f6340070150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6350000c00 tx=0x7f6350009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.509+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f634405a1b0 con 0x7f63541037b0 2024-04-05T03:18:37.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.726+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f635410f1b0 con 0x7f634006dca0 2024-04-05T03:18:37.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.728+0000 7f634affd700 1 -- 172.21.15.153:0/565352060 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f635410f1b0 con 0x7f634006dca0 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 msgr2=0x7f6340070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 0x7f6340070150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6350000c00 tx=0x7f6350009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 msgr2=0x7f63541140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f634400eb40 tx=0x7f634400b040 comp rx=0 tx=0).stop 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 shutdown_connections 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f634006dca0 0x7f6340070150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6354102e10 0x7f6354113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 --2- 172.21.15.153:0/565352060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f63541037b0 0x7f63541140a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 >> 172.21.15.153:0/565352060 conn(0x7f63540f89a0 msgr2=0x7f6354107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 shutdown_connections 2024-04-05T03:18:37.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.730+0000 7f635c54c700 1 -- 172.21.15.153:0/565352060 wait complete. 2024-04-05T03:18:37.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 -- 172.21.15.153:0/2669585978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 msgr2=0x7ff1cc1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 --2- 172.21.15.153:0/2669585978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc1020c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0004940 tx=0x7ff1c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:37.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 -- 172.21.15.153:0/2669585978 shutdown_connections 2024-04-05T03:18:37.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 --2- 172.21.15.153:0/2669585978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1cc102690 0x7ff1cc0fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 --2- 172.21.15.153:0/2669585978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc1020c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 -- 172.21.15.153:0/2669585978 >> 172.21.15.153:0/2669585978 conn(0x7ff1cc0f7860 msgr2=0x7ff1cc0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:37.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 -- 172.21.15.153:0/2669585978 shutdown_connections 2024-04-05T03:18:37.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.832+0000 7ff1d1750700 1 -- 172.21.15.153:0/2669585978 wait complete. 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 Processor -- start 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 -- start start 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1cc102690 0x7ff1cc116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1168e0 con 0x7ff1cc101cf0 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1d1750700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc10fcb0 con 0x7ff1cc102690 2024-04-05T03:18:37.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50470/0 (socket says 172.21.15.153:50470) 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 -- 172.21.15.153:0/3222086522 learned_addr learned my addr 172.21.15.153:0/3222086522 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 -- 172.21.15.153:0/3222086522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1cc102690 msgr2=0x7ff1cc116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1cc102690 0x7ff1cc116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.833+0000 7ff1caffd700 1 -- 172.21.15.153:0/3222086522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c0005040 con 0x7ff1cc101cf0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1caffd700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009150 tx=0x7ff1c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c000e040 con 0x7ff1cc101cf0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1c0013420 con 0x7ff1cc101cf0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc10ff30 con 0x7ff1cc101cf0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c001c630 con 0x7ff1cc101cf0 2024-04-05T03:18:37.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.834+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1104f0 con 0x7ff1cc101cf0 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.835+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc0fd440 con 0x7ff1cc101cf0 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.835+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff1c0013590 con 0x7ff1cc101cf0 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.836+0000 7ff1b37fe700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 0x7ff1b4070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.836+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff1c008e820 con 0x7ff1cc101cf0 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.836+0000 7ff1ca7fc700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 0x7ff1b4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:37.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.836+0000 7ff1ca7fc700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 0x7ff1b4070160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1115f0 tx=0x7ff1bc009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:37.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:37.840+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff1c005b950 con 0x7ff1cc101cf0 2024-04-05T03:18:38.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.048+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1cc108e60 con 0x7ff1b406dcb0 2024-04-05T03:18:38.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.059+0000 7ff1b37fe700 1 -- 172.21.15.153:0/3222086522 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff1cc108e60 con 0x7ff1b406dcb0 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (13m) 6m ago 24m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (10m) 6m ago 25m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (10m) 6m ago 23m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (12m) 6m ago 24m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (6m) 6m ago 18m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (7m) 6m ago 18m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (6m) 6m ago 18m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (6m) 6m ago 18m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (14m) 6m ago 26m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (14m) 6m ago 23m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (11m) 6m ago 26m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:18:38.058 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (11m) 6m ago 23m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (13m) 6m ago 24m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (13m) 6m ago 23m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (10m) 6m ago 22m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (9m) 6m ago 21m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (9m) 6m ago 21m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (8m) 6m ago 21m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (7m) 6m ago 20m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (7m) 6m ago 20m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:18:38.059 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (13m) 6m ago 22m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 msgr2=0x7ff1b4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 0x7ff1b4070160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc1115f0 tx=0x7ff1bc009ce0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 msgr2=0x7ff1cc115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009150 tx=0x7ff1c0009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 shutdown_connections 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff1b406dcb0 0x7ff1b4070160 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff1cc101cf0 0x7ff1cc115d50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 --2- 172.21.15.153:0/3222086522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1cc102690 0x7ff1cc116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 >> 172.21.15.153:0/3222086522 conn(0x7ff1cc0f7860 msgr2=0x7ff1cc0f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 shutdown_connections 2024-04-05T03:18:38.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.061+0000 7ff1d1750700 1 -- 172.21.15.153:0/3222086522 wait complete. 2024-04-05T03:18:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- 172.21.15.153:0/2106329062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 msgr2=0x7f82e40fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 --2- 172.21.15.153:0/2106329062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e40fbd30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f82cc004940 tx=0x7f82cc009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- 172.21.15.153:0/2106329062 shutdown_connections 2024-04-05T03:18:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 --2- 172.21.15.153:0/2106329062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e40fbd30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 --2- 172.21.15.153:0/2106329062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 0x7f82e41020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- 172.21.15.153:0/2106329062 >> 172.21.15.153:0/2106329062 conn(0x7f82e40f7870 msgr2=0x7f82e40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- 172.21.15.153:0/2106329062 shutdown_connections 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- 172.21.15.153:0/2106329062 wait complete. 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 Processor -- start 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.161+0000 7f82e8c13700 1 -- start start 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e8c13700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 0x7f82e4115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e8c13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e8c13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e41166d0 con 0x7f82e41026c0 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e8c13700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e410fcb0 con 0x7f82e4101d20 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50490/0 (socket says 172.21.15.153:50490) 2024-04-05T03:18:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 -- 172.21.15.153:0/16407174 learned_addr learned my addr 172.21.15.153:0/16407174 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e259c700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 0x7f82e4115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 -- 172.21.15.153:0/16407174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 msgr2=0x7f82e4115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 0x7f82e4115b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 -- 172.21.15.153:0/16407174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc005040 con 0x7f82e41026c0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e1d9b700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009180 tx=0x7f82cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82cc00e040 con 0x7f82e41026c0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82cc013420 con 0x7f82e41026c0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.162+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e410ff30 con 0x7f82e41026c0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.163+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82cc01c610 con 0x7f82e41026c0 2024-04-05T03:18:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.163+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e41104f0 con 0x7f82e41026c0 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.163+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e40fd470 con 0x7f82e41026c0 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.164+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f82cc013a00 con 0x7f82e41026c0 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.165+0000 7f82daffd700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 0x7f82d0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.165+0000 7f82e259c700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 0x7f82d0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.165+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f82cc08e600 con 0x7f82e41026c0 2024-04-05T03:18:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.165+0000 7f82e259c700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 0x7f82d0070150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009fd0 tx=0x7f82dc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.169+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f82cc05b710 con 0x7f82e41026c0 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.444+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f82e41111c0 con 0x7f82e41026c0 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.445+0000 7f82daffd700 1 -- 172.21.15.153:0/16407174 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f82cc05b2a0 con 0x7f82e41026c0 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:18:38.444 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:18:38.445 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:18:38.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.447+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 msgr2=0x7f82d0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.447+0000 7f82e8c13700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 0x7f82d0070150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009fd0 tx=0x7f82dc00b040 comp rx=0 tx=0).stop 2024-04-05T03:18:38.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.447+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 msgr2=0x7f82e4116080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.447+0000 7f82e8c13700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009180 tx=0x7f82cc009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 shutdown_connections 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f82d006dca0 0x7f82d0070150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e4101d20 0x7f82e4115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 --2- 172.21.15.153:0/16407174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f82e41026c0 0x7f82e4116080 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 >> 172.21.15.153:0/16407174 conn(0x7f82e40f7870 msgr2=0x7f82e40f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 shutdown_connections 2024-04-05T03:18:38.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.448+0000 7f82e8c13700 1 -- 172.21.15.153:0/16407174 wait complete. 2024-04-05T03:18:38.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- 172.21.15.153:0/1800644451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 msgr2=0x7f8c3c0fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 --2- 172.21.15.153:0/1800644451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c0fbd40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c004940 tx=0x7f8c2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- 172.21.15.153:0/1800644451 shutdown_connections 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 --2- 172.21.15.153:0/1800644451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c0fbd40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 --2- 172.21.15.153:0/1800644451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 0x7f8c3c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- 172.21.15.153:0/1800644451 >> 172.21.15.153:0/1800644451 conn(0x7f8c3c0f78c0 msgr2=0x7f8c3c0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- 172.21.15.153:0/1800644451 shutdown_connections 2024-04-05T03:18:38.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- 172.21.15.153:0/1800644451 wait complete. 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 Processor -- start 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- start start 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 0x7f8c3c10eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c10f520 con 0x7f8c3c1026d0 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.549+0000 7f8c439b1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c10f660 con 0x7f8c3c101d30 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50502/0 (socket says 172.21.15.153:50502) 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 -- 172.21.15.153:0/49224168 learned_addr learned my addr 172.21.15.153:0/49224168 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:38.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c4174d700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 0x7f8c3c10eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 -- 172.21.15.153:0/49224168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 msgr2=0x7f8c3c10eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 0x7f8c3c10eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 -- 172.21.15.153:0/49224168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2c005040 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c40f4c700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000c00 tx=0x7f8c2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c2c021070 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c2c013cd0 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3c10f8e0 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c2c00e070 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.550+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3c117d80 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.551+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3c0fd480 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.552+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8c2c01caf0 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.552+0000 7f8c31ffb700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 0x7f8c28070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.552+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8c2c08ea40 con 0x7f8c3c1026d0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.552+0000 7f8c4174d700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 0x7f8c28070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.553+0000 7f8c4174d700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 0x7f8c28070150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8c38004600 tx=0x7f8c3800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.557+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8c2c058040 con 0x7f8c3c1026d0 2024-04-05T03:18:38.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.794+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8c3c117f70 con 0x7f8c3c1026d0 2024-04-05T03:18:38.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.795+0000 7f8c31ffb700 1 -- 172.21.15.153:0/49224168 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f8c2c05b700 con 0x7f8c3c1026d0 2024-04-05T03:18:38.794 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:18:38.794 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:18:38.794 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:18:38.795 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:18:38.796 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:18:38.797 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:38.798 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 msgr2=0x7f8c28070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 0x7f8c28070150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8c38004600 tx=0x7f8c3800b040 comp rx=0 tx=0).stop 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 msgr2=0x7f8c3c10efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000c00 tx=0x7f8c2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 shutdown_connections 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8c2806dca0 0x7f8c28070150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c3c101d30 0x7f8c3c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 --2- 172.21.15.153:0/49224168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c3c1026d0 0x7f8c3c10efe0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 >> 172.21.15.153:0/49224168 conn(0x7f8c3c0f78c0 msgr2=0x7f8c3c0f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 shutdown_connections 2024-04-05T03:18:38.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.798+0000 7f8c439b1700 1 -- 172.21.15.153:0/49224168 wait complete. 2024-04-05T03:18:38.800 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 -- 172.21.15.153:0/346358049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 msgr2=0x7f05b80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 --2- 172.21.15.153:0/346358049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b80fbd80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f05a8004940 tx=0x7f05a8009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 -- 172.21.15.153:0/346358049 shutdown_connections 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 --2- 172.21.15.153:0/346358049 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b80fbd80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 --2- 172.21.15.153:0/346358049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 -- 172.21.15.153:0/346358049 >> 172.21.15.153:0/346358049 conn(0x7f05b8072bc0 msgr2=0x7f05b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 -- 172.21.15.153:0/346358049 shutdown_connections 2024-04-05T03:18:38.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.901+0000 7f05bf63a700 1 -- 172.21.15.153:0/346358049 wait complete. 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 Processor -- start 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 -- start start 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8115da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b8116930 con 0x7f05b8102710 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bf63a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b810fd00 con 0x7f05b8101d70 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52356/0 (socket says 172.21.15.153:52356) 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 -- 172.21.15.153:0/3074040162 learned_addr learned my addr 172.21.15.153:0/3074040162 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 -- 172.21.15.153:0/3074040162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 msgr2=0x7f05b8115da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bd3d6700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8115da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8115da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:38.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bcbd5700 1 -- 172.21.15.153:0/3074040162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8005040 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.902+0000 7f05bd3d6700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05bcbd5700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009100 tx=0x7f05a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05a800e040 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05a8013420 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b810ff80 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05a801c610 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.903+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b8110540 con 0x7f05b8102710 2024-04-05T03:18:38.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.904+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b80fd4c0 con 0x7f05b8102710 2024-04-05T03:18:38.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.904+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f05a8013590 con 0x7f05b8102710 2024-04-05T03:18:38.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.905+0000 7f05adffb700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 0x7f05a4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:38.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.905+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05a808e830 con 0x7f05b8102710 2024-04-05T03:18:38.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.905+0000 7f05bd3d6700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 0x7f05a4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:38.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.905+0000 7f05bd3d6700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 0x7f05a4070150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05b40041d0 tx=0x7f05b4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:38.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:38.909+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f05a805b960 con 0x7f05b8102710 2024-04-05T03:18:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:38 smithi203 ceph-mon[95988]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:38 smithi203 ceph-mon[95988]: pgmap v389: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:38 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/16407174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:18:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:38 smithi153 ceph-mon[106102]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:38 smithi153 ceph-mon[106102]: pgmap v389: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:38 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/16407174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:18:39.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.122+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f05b8108e50 con 0x7f05a406dca0 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.123+0000 7f05adffb700 1 -- 172.21.15.153:0/3074040162 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f05b8108e50 con 0x7f05a406dca0 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:18:39.122 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:18:39.123 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 msgr2=0x7f05a4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 0x7f05a4070150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05b40041d0 tx=0x7f05b4009040 comp rx=0 tx=0).stop 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 msgr2=0x7f05b81162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009100 tx=0x7f05a8009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 shutdown_connections 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f05a406dca0 0x7f05a4070150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05b8101d70 0x7f05b8115da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 --2- 172.21.15.153:0/3074040162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05b8102710 0x7f05b81162e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 >> 172.21.15.153:0/3074040162 conn(0x7f05b8072bc0 msgr2=0x7f05b8065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 shutdown_connections 2024-04-05T03:18:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.126+0000 7f05bf63a700 1 -- 172.21.15.153:0/3074040162 wait complete. 2024-04-05T03:18:39.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 -- 172.21.15.153:0/2378047515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc028102e00 msgr2=0x7fc0281031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2378047515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc028102e00 0x7fc0281031d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc01c004940 tx=0x7fc01c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 -- 172.21.15.153:0/2378047515 shutdown_connections 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2378047515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0281037a0 0x7fc0280fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2378047515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc028102e00 0x7fc0281031d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 -- 172.21.15.153:0/2378047515 >> 172.21.15.153:0/2378047515 conn(0x7fc0280f89b0 msgr2=0x7fc0280fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 -- 172.21.15.153:0/2378047515 shutdown_connections 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.227+0000 7fc02ce67700 1 -- 172.21.15.153:0/2378047515 wait complete. 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 Processor -- start 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 -- start start 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028102e00 0x7fc028113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc028114720 con 0x7fc0281037a0 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02ce67700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02810db10 con 0x7fc028102e00 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:39.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52386/0 (socket says 172.21.15.153:52386) 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 -- 172.21.15.153:0/2696085131 learned_addr learned my addr 172.21.15.153:0/2696085131 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc02659c700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028102e00 0x7fc028113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 -- 172.21.15.153:0/2696085131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028102e00 msgr2=0x7fc028113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028102e00 0x7fc028113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.228+0000 7fc025d9b700 1 -- 172.21.15.153:0/2696085131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c005040 con 0x7fc0281037a0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc025d9b700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc018002a50 tx=0x7fc0180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc018007cc0 con 0x7fc0281037a0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc01800f040 con 0x7fc0281037a0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02810ddf0 con 0x7fc0281037a0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc018009630 con 0x7fc0281037a0 2024-04-05T03:18:39.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.229+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02810e330 con 0x7fc0281037a0 2024-04-05T03:18:39.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.230+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc028100110 con 0x7fc0281037a0 2024-04-05T03:18:39.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.230+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc018009790 con 0x7fc0281037a0 2024-04-05T03:18:39.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.230+0000 7fc016ffd700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 0x7fc010070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:18:39.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.231+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc01808d230 con 0x7fc0281037a0 2024-04-05T03:18:39.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.231+0000 7fc02659c700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 0x7fc010070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:18:39.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.231+0000 7fc02659c700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 0x7fc010070150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc01c004f90 tx=0x7fc01c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:18:39.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.235+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc01805a390 con 0x7fc0281037a0 2024-04-05T03:18:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.513+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc02810f1e0 con 0x7fc0281037a0 2024-04-05T03:18:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.513+0000 7fc016ffd700 1 -- 172.21.15.153:0/2696085131 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc018078e60 con 0x7fc0281037a0 2024-04-05T03:18:39.513 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:18:39.513 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:18:39.513 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:18:39.513 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:18:39.513 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:18:39.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 msgr2=0x7fc010070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 0x7fc010070150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc01c004f90 tx=0x7fc01c009e30 comp rx=0 tx=0).stop 2024-04-05T03:18:39.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 msgr2=0x7fc0281140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:18:39.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc018002a50 tx=0x7fc0180046b0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 shutdown_connections 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc01006dca0 0x7fc010070150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028102e00 0x7fc028113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 --2- 172.21.15.153:0/2696085131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc0281037a0 0x7fc0281140d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 >> 172.21.15.153:0/2696085131 conn(0x7fc0280f89b0 msgr2=0x7fc0281073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 shutdown_connections 2024-04-05T03:18:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:18:39.515+0000 7fc02ce67700 1 -- 172.21.15.153:0/2696085131 wait complete. 2024-04-05T03:18:39.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:39 smithi153 ceph-mon[106102]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:39.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:39 smithi153 ceph-mon[106102]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:39.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:39 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/49224168' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:18:39.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:39 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2696085131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:18:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:39 smithi203 ceph-mon[95988]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:39 smithi203 ceph-mon[95988]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:39 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/49224168' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:18:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:39 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2696085131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:18:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:40 smithi203 ceph-mon[95988]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:40 smithi203 ceph-mon[95988]: pgmap v390: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:40 smithi153 ceph-mon[106102]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:18:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:40 smithi153 ceph-mon[106102]: pgmap v390: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:42.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:41 smithi203 ceph-mon[95988]: pgmap v391: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:42.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:41 smithi153 ceph-mon[106102]: pgmap v391: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:44.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:43 smithi203 ceph-mon[95988]: pgmap v392: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:44.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:43 smithi153 ceph-mon[106102]: pgmap v392: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:46 smithi203 ceph-mon[95988]: pgmap v393: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:46 smithi153 ceph-mon[106102]: pgmap v393: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:48 smithi203 ceph-mon[95988]: pgmap v394: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:48 smithi153 ceph-mon[106102]: pgmap v394: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:50 smithi203 ceph-mon[95988]: pgmap v395: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:50 smithi153 ceph-mon[106102]: pgmap v395: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:52 smithi203 ceph-mon[95988]: pgmap v396: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:52 smithi153 ceph-mon[106102]: pgmap v396: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:54 smithi203 ceph-mon[95988]: pgmap v397: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:54 smithi153 ceph-mon[106102]: pgmap v397: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:56 smithi203 ceph-mon[95988]: pgmap v398: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:56 smithi153 ceph-mon[106102]: pgmap v398: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:18:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:18:58 smithi203 ceph-mon[95988]: pgmap v399: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:18:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:18:58 smithi153 ceph-mon[106102]: pgmap v399: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:00 smithi203 ceph-mon[95988]: pgmap v400: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:01.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:00 smithi153 ceph-mon[106102]: pgmap v400: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:02 smithi203 ceph-mon[95988]: pgmap v401: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:02 smithi153 ceph-mon[106102]: pgmap v401: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:04 smithi203 ceph-mon[95988]: pgmap v402: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:04 smithi153 ceph-mon[106102]: pgmap v402: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:06 smithi203 ceph-mon[95988]: pgmap v403: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:06 smithi153 ceph-mon[106102]: pgmap v403: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:08 smithi203 ceph-mon[95988]: pgmap v404: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:08 smithi153 ceph-mon[106102]: pgmap v404: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:09.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 -- 172.21.15.153:0/2078868677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 msgr2=0x7f89300fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 --2- 172.21.15.153:0/2078868677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f89300fce10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8920004940 tx=0x7f8920009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:09.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 -- 172.21.15.153:0/2078868677 shutdown_connections 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 --2- 172.21.15.153:0/2078868677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f89300fce10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 --2- 172.21.15.153:0/2078868677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 0x7f89301031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 -- 172.21.15.153:0/2078868677 >> 172.21.15.153:0/2078868677 conn(0x7f89300f89b0 msgr2=0x7f89300fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 -- 172.21.15.153:0/2078868677 shutdown_connections 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 -- 172.21.15.153:0/2078868677 wait complete. 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.631+0000 7f8934923700 1 Processor -- start 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f8934923700 1 -- start start 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f8934923700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 0x7f8930113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f8934923700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f8934923700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8930114510 con 0x7f89301037a0 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f8934923700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893010db10 con 0x7f8930102e00 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38078/0 (socket says 172.21.15.153:38078) 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 -- 172.21.15.153:0/325703761 learned_addr learned my addr 172.21.15.153:0/325703761 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:09.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892e59c700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 0x7f8930113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 -- 172.21.15.153:0/325703761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 msgr2=0x7f8930113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 0x7f8930113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 -- 172.21.15.153:0/325703761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920005040 con 0x7f89301037a0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.632+0000 7f892dd9b700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8920009100 tx=0x7f8920009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.633+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892000e040 con 0x7f89301037a0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.633+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8920013420 con 0x7f89301037a0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.633+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893010dd90 con 0x7f89301037a0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.633+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892001c610 con 0x7f89301037a0 2024-04-05T03:19:09.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.633+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893010e350 con 0x7f89301037a0 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.634+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89301052d0 con 0x7f89301037a0 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.634+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8920013590 con 0x7f89301037a0 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.634+0000 7f891effd700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 0x7f8918070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.635+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f892008e830 con 0x7f89301037a0 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.635+0000 7f892e59c700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 0x7f8918070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.635+0000 7f892e59c700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 0x7f8918070150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8924009fd0 tx=0x7f8924009e90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:09.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.639+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f892005b960 con 0x7f89301037a0 2024-04-05T03:19:09.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.858+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f893010efd0 con 0x7f891806dca0 2024-04-05T03:19:09.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.859+0000 7f891effd700 1 -- 172.21.15.153:0/325703761 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f893010efd0 con 0x7f891806dca0 2024-04-05T03:19:09.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 msgr2=0x7f8918070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 0x7f8918070150 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8924009fd0 tx=0x7f8924009e90 comp rx=0 tx=0).stop 2024-04-05T03:19:09.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 msgr2=0x7f8930113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8920009100 tx=0x7f8920009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 shutdown_connections 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f891806dca0 0x7f8918070150 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8930102e00 0x7f8930113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 --2- 172.21.15.153:0/325703761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89301037a0 0x7f8930113ec0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 >> 172.21.15.153:0/325703761 conn(0x7f89300f89b0 msgr2=0x7f8930102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 shutdown_connections 2024-04-05T03:19:09.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.861+0000 7f8934923700 1 -- 172.21.15.153:0/325703761 wait complete. 2024-04-05T03:19:09.870 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:19:09.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 -- 172.21.15.153:0/2812806210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec781037a0 msgr2=0x7fec780fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 --2- 172.21.15.153:0/2812806210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec781037a0 0x7fec780fce10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec60004940 tx=0x7fec60009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:09.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 -- 172.21.15.153:0/2812806210 shutdown_connections 2024-04-05T03:19:09.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 --2- 172.21.15.153:0/2812806210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec781037a0 0x7fec780fce10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 --2- 172.21.15.153:0/2812806210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec78102e00 0x7fec781031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 -- 172.21.15.153:0/2812806210 >> 172.21.15.153:0/2812806210 conn(0x7fec780f89b0 msgr2=0x7fec780fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.963+0000 7fec7df4e700 1 -- 172.21.15.153:0/2812806210 shutdown_connections 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 -- 172.21.15.153:0/2812806210 wait complete. 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 Processor -- start 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 -- start start 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 0x7fec781140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec78114710 con 0x7fec78102e00 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec7df4e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7810db70 con 0x7fec781037a0 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38094/0 (socket says 172.21.15.153:38094) 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.964+0000 7fec777fe700 1 -- 172.21.15.153:0/1462201192 learned_addr learned my addr 172.21.15.153:0/1462201192 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec6ffff700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 0x7fec781140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec777fe700 1 -- 172.21.15.153:0/1462201192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 msgr2=0x7fec781140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec777fe700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 0x7fec781140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec777fe700 1 -- 172.21.15.153:0/1462201192 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60005040 con 0x7fec78102e00 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec6ffff700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 0x7fec781140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec777fe700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c8c0 tx=0x7fec680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec6800a980 con 0x7fec78102e00 2024-04-05T03:19:09.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec6800ce60 con 0x7fec78102e00 2024-04-05T03:19:09.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec680115e0 con 0x7fec78102e00 2024-04-05T03:19:09.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7810de50 con 0x7fec78102e00 2024-04-05T03:19:09.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.965+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7810e410 con 0x7fec78102e00 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.966+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec781052d0 con 0x7fec78102e00 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.967+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fec68012400 con 0x7fec78102e00 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.967+0000 7fec74ff9700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 0x7fec64070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.967+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec6808d590 con 0x7fec78102e00 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.967+0000 7fec6ffff700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 0x7fec64070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:09.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.968+0000 7fec6ffff700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 0x7fec64070160 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec7810f420 tx=0x7fec60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:09.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:09.972+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fec68056c30 con 0x7fec78102e00 2024-04-05T03:19:10.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.187+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fec78109ee0 con 0x7fec6406dcb0 2024-04-05T03:19:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.188+0000 7fec74ff9700 1 -- 172.21.15.153:0/1462201192 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fec78109ee0 con 0x7fec6406dcb0 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 msgr2=0x7fec64070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 0x7fec64070160 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec7810f420 tx=0x7fec60009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 msgr2=0x7fec78113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c8c0 tx=0x7fec680044c0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 shutdown_connections 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fec6406dcb0 0x7fec64070160 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fec78102e00 0x7fec78113b80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 --2- 172.21.15.153:0/1462201192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec781037a0 0x7fec781140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 >> 172.21.15.153:0/1462201192 conn(0x7fec780f89b0 msgr2=0x7fec78102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 shutdown_connections 2024-04-05T03:19:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.190+0000 7fec7df4e700 1 -- 172.21.15.153:0/1462201192 wait complete. 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 -- 172.21.15.153:0/1998366847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 msgr2=0x7fb14c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 --2- 172.21.15.153:0/1998366847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c1020f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb13c004940 tx=0x7fb13c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 -- 172.21.15.153:0/1998366847 shutdown_connections 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 --2- 172.21.15.153:0/1998366847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 --2- 172.21.15.153:0/1998366847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c1020f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 -- 172.21.15.153:0/1998366847 >> 172.21.15.153:0/1998366847 conn(0x7fb14c0f7870 msgr2=0x7fb14c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 -- 172.21.15.153:0/1998366847 shutdown_connections 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.291+0000 7fb153edd700 1 -- 172.21.15.153:0/1998366847 wait complete. 2024-04-05T03:19:10.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 Processor -- start 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 -- start start 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c1030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c104ad0 con 0x7fb14c101d20 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb153edd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c103600 con 0x7fb14c1026c0 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb151c79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb151c79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38114/0 (socket says 172.21.15.153:38114) 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.292+0000 7fb151c79700 1 -- 172.21.15.153:0/4446190 learned_addr learned my addr 172.21.15.153:0/4446190 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151478700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c1030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151c79700 1 -- 172.21.15.153:0/4446190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 msgr2=0x7fb14c1030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151c79700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c1030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151c79700 1 -- 172.21.15.153:0/4446190 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c005040 con 0x7fb14c101d20 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151478700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb151c79700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00b3b0 tx=0x7fb13c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13c00e040 con 0x7fb14c101d20 2024-04-05T03:19:10.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13c00bbc0 con 0x7fb14c101d20 2024-04-05T03:19:10.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c103880 con 0x7fb14c101d20 2024-04-05T03:19:10.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb13c01c630 con 0x7fb14c101d20 2024-04-05T03:19:10.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.293+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c11a130 con 0x7fb14c101d20 2024-04-05T03:19:10.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.294+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c0fd470 con 0x7fb14c101d20 2024-04-05T03:19:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.295+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb13c00bd30 con 0x7fb14c101d20 2024-04-05T03:19:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.295+0000 7fb1427fc700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 0x7fb138070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.295+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb13c08e820 con 0x7fb14c101d20 2024-04-05T03:19:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.295+0000 7fb151478700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 0x7fb138070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.296+0000 7fb151478700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 0x7fb138070150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb148007fd0 tx=0x7fb148009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:10.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.300+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb13c057e20 con 0x7fb14c101d20 2024-04-05T03:19:10.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.505+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb14c0f9620 con 0x7fb13806dca0 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.515+0000 7fb1427fc700 1 -- 172.21.15.153:0/4446190 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb14c0f9620 con 0x7fb13806dca0 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (13m) 7m ago 24m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (11m) 7m ago 25m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (11m) 6m ago 23m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (13m) 7m ago 24m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (7m) 7m ago 19m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (7m) 7m ago 19m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (6m) 6m ago 19m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:19:10.514 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (7m) 6m ago 19m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (15m) 7m ago 27m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (14m) 6m ago 23m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (12m) 7m ago 27m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (11m) 6m ago 23m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (14m) 7m ago 25m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (14m) 6m ago 23m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (10m) 7m ago 23m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (10m) 7m ago 22m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (9m) 7m ago 22m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (9m) 6m ago 21m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (8m) 6m ago 21m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (7m) 6m ago 20m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:19:10.515 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (14m) 6m ago 23m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:19:10.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 msgr2=0x7fb138070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 0x7fb138070150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb148007fd0 tx=0x7fb148009040 comp rx=0 tx=0).stop 2024-04-05T03:19:10.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 msgr2=0x7fb14c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00b3b0 tx=0x7fb13c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:10.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 shutdown_connections 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb13806dca0 0x7fb138070150 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb14c101d20 0x7fb14c102b80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 --2- 172.21.15.153:0/4446190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb14c1026c0 0x7fb14c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 >> 172.21.15.153:0/4446190 conn(0x7fb14c0f7870 msgr2=0x7fb14c0f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 shutdown_connections 2024-04-05T03:19:10.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.517+0000 7fb153edd700 1 -- 172.21.15.153:0/4446190 wait complete. 2024-04-05T03:19:10.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 -- 172.21.15.153:0/1227156035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 msgr2=0x7fba580fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 --2- 172.21.15.153:0/1227156035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fb820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba4c004940 tx=0x7fba4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:10.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 -- 172.21.15.153:0/1227156035 shutdown_connections 2024-04-05T03:19:10.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 --2- 172.21.15.153:0/1227156035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fb820 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 --2- 172.21.15.153:0/1227156035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 0x7fba58103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.618+0000 7fba5e356700 1 -- 172.21.15.153:0/1227156035 >> 172.21.15.153:0/1227156035 conn(0x7fba58072310 msgr2=0x7fba58072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:10.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 -- 172.21.15.153:0/1227156035 shutdown_connections 2024-04-05T03:19:10.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 -- 172.21.15.153:0/1227156035 wait complete. 2024-04-05T03:19:10.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 Processor -- start 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 -- start start 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 0x7fba580fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580fe4e0 con 0x7fba580fb3b0 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.619+0000 7fba5e356700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580fd010 con 0x7fba58102f70 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38136/0 (socket says 172.21.15.153:38136) 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 -- 172.21.15.153:0/1355663396 learned_addr learned my addr 172.21.15.153:0/1355663396 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba577fe700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 0x7fba580fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 -- 172.21.15.153:0/1355663396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 msgr2=0x7fba580fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 0x7fba580fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 -- 172.21.15.153:0/1355663396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c005040 con 0x7fba580fb3b0 2024-04-05T03:19:10.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba57fff700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba4800c830 tx=0x7fba48004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba480106f0 con 0x7fba580fb3b0 2024-04-05T03:19:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba580fd2f0 con 0x7fba580fb3b0 2024-04-05T03:19:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba48010d20 con 0x7fba580fb3b0 2024-04-05T03:19:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba480113a0 con 0x7fba580fb3b0 2024-04-05T03:19:10.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.620+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5811a110 con 0x7fba580fb3b0 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.621+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba58105350 con 0x7fba580fb3b0 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.622+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fba48010850 con 0x7fba580fb3b0 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.622+0000 7fba54ff9700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 0x7fba40070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.622+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fba4808d520 con 0x7fba580fb3b0 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.622+0000 7fba577fe700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 0x7fba40070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:10.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.623+0000 7fba577fe700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 0x7fba40070150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba580fdcc0 tx=0x7fba4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:10.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.627+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba48056b20 con 0x7fba580fb3b0 2024-04-05T03:19:10.885 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:10 smithi153 ceph-mon[106102]: pgmap v405: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:10.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.908+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fba580fda70 con 0x7fba580fb3b0 2024-04-05T03:19:10.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.909+0000 7fba54ff9700 1 -- 172.21.15.153:0/1355663396 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fba4805a1e0 con 0x7fba580fb3b0 2024-04-05T03:19:10.908 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:19:10.908 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:19:10.908 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:19:10.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:19:10.909 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 msgr2=0x7fba40070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 0x7fba40070150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba580fdcc0 tx=0x7fba4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 msgr2=0x7fba580fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba4800c830 tx=0x7fba48004470 comp rx=0 tx=0).stop 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 shutdown_connections 2024-04-05T03:19:10.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba4006dca0 0x7fba40070150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba580fb3b0 0x7fba580fc590 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 --2- 172.21.15.153:0/1355663396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba58102f70 0x7fba580fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 >> 172.21.15.153:0/1355663396 conn(0x7fba58072310 msgr2=0x7fba580f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 shutdown_connections 2024-04-05T03:19:10.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:10.911+0000 7fba5e356700 1 -- 172.21.15.153:0/1355663396 wait complete. 2024-04-05T03:19:11.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 -- 172.21.15.153:0/1171233361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 msgr2=0x7f27240fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 --2- 172.21.15.153:0/1171233361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f27240fb990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2718004940 tx=0x7f2718009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 -- 172.21.15.153:0/1171233361 shutdown_connections 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 --2- 172.21.15.153:0/1171233361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f27240fb990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 --2- 172.21.15.153:0/1171233361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f27241033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 -- 172.21.15.153:0/1171233361 >> 172.21.15.153:0/1171233361 conn(0x7f2724071530 msgr2=0x7f2724071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 -- 172.21.15.153:0/1171233361 shutdown_connections 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.013+0000 7f272a882700 1 -- 172.21.15.153:0/1171233361 wait complete. 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 Processor -- start 2024-04-05T03:19:11.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 -- start start 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f2724113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27241144c0 con 0x7f27240fb540 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f272a882700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272410db00 con 0x7f2724102fd0 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f2723fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f2723fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38144/0 (socket says 172.21.15.153:38144) 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.014+0000 7f2723fff700 1 -- 172.21.15.153:0/1272955336 learned_addr learned my addr 172.21.15.153:0/1272955336 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f27237fe700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f2724113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2723fff700 1 -- 172.21.15.153:0/1272955336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 msgr2=0x7f2724113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2723fff700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f2724113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2723fff700 1 -- 172.21.15.153:0/1272955336 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2718005040 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f27237fe700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f2724113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2723fff700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f271400c810 tx=0x7f2714004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2714012810 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272410dde0 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2714012e40 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f271400a370 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.015+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272410e3a0 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.016+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27241053b0 con 0x7f27240fb540 2024-04-05T03:19:11.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.017+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2714011810 con 0x7f27240fb540 2024-04-05T03:19:11.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.017+0000 7f2720ff9700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 0x7f270c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.017+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f271408d9f0 con 0x7f27240fb540 2024-04-05T03:19:11.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.017+0000 7f27237fe700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 0x7f270c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.017+0000 7f27237fe700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 0x7f270c070150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2718009fd0 tx=0x7f2718009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.022+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f271408fe60 con 0x7f27240fb540 2024-04-05T03:19:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:10 smithi203 ceph-mon[95988]: pgmap v405: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:11.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.261+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f272410a030 con 0x7f27240fb540 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.261+0000 7f2720ff9700 1 -- 172.21.15.153:0/1272955336 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f271405aaa0 con 0x7f27240fb540 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:11.260 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:19:11.261 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:19:11.262 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:11.263 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 msgr2=0x7f270c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 0x7f270c070150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2718009fd0 tx=0x7f2718009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 msgr2=0x7f2724113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f271400c810 tx=0x7f2714004470 comp rx=0 tx=0).stop 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 shutdown_connections 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f270c06dca0 0x7f270c070150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240fb540 0x7f2724113930 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 --2- 172.21.15.153:0/1272955336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2724102fd0 0x7f2724113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 >> 172.21.15.153:0/1272955336 conn(0x7f2724071530 msgr2=0x7f27240fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 shutdown_connections 2024-04-05T03:19:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.263+0000 7f272a882700 1 -- 172.21.15.153:0/1272955336 wait complete. 2024-04-05T03:19:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:19:11.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 -- 172.21.15.153:0/301730715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c101d70 msgr2=0x7f679c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/301730715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c101d70 0x7f679c102140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f678c004940 tx=0x7f678c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 -- 172.21.15.153:0/301730715 shutdown_connections 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/301730715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c102710 0x7f679c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/301730715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c101d70 0x7f679c102140 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 -- 172.21.15.153:0/301730715 >> 172.21.15.153:0/301730715 conn(0x7f679c072bc0 msgr2=0x7f679c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 -- 172.21.15.153:0/301730715 shutdown_connections 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 -- 172.21.15.153:0/301730715 wait complete. 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.365+0000 7f67a0a9d700 1 Processor -- start 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 -- start start 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c101d70 0x7f679c115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c116850 con 0x7f679c102710 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c10fc20 con 0x7f679c101d70 2024-04-05T03:19:11.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38160/0 (socket says 172.21.15.153:38160) 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 -- 172.21.15.153:0/646056625 learned_addr learned my addr 172.21.15.153:0/646056625 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f679a59c700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c101d70 0x7f679c115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 -- 172.21.15.153:0/646056625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c101d70 msgr2=0x7f679c115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c101d70 0x7f679c115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 -- 172.21.15.153:0/646056625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c005040 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6793fff700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6784002a50 tx=0x7f67840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6784007cc0 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f678400f040 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.366+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c10ff00 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.367+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6784009630 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.367+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c1104c0 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.368+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c0fd4c0 con 0x7f679c102710 2024-04-05T03:19:11.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.368+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6784009830 con 0x7f679c102710 2024-04-05T03:19:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.368+0000 7f6792ffd700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 0x7f6788070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.368+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f678408d180 con 0x7f679c102710 2024-04-05T03:19:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.368+0000 7f679a59c700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 0x7f6788070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.369+0000 7f679a59c700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 0x7f6788070160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f678c00b020 tx=0x7f678c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.373+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f678405a2b0 con 0x7f679c102710 2024-04-05T03:19:11.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.587+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f679c108ee0 con 0x7f678806dcb0 2024-04-05T03:19:11.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.588+0000 7f6792ffd700 1 -- 172.21.15.153:0/646056625 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f679c108ee0 con 0x7f678806dcb0 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:19:11.587 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:19:11.588 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:19:11.588 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:19:11.588 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:19:11.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 msgr2=0x7f6788070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 0x7f6788070160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f678c00b020 tx=0x7f678c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 msgr2=0x7f679c116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6784002a50 tx=0x7f67840046b0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 shutdown_connections 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f678806dcb0 0x7f6788070160 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f679c101d70 0x7f679c115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 --2- 172.21.15.153:0/646056625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f679c102710 0x7f679c116200 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.590+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 >> 172.21.15.153:0/646056625 conn(0x7f679c072bc0 msgr2=0x7f679c0658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.591+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 shutdown_connections 2024-04-05T03:19:11.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.591+0000 7f67a0a9d700 1 -- 172.21.15.153:0/646056625 wait complete. 2024-04-05T03:19:11.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 -- 172.21.15.153:0/4245964709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6504101d70 msgr2=0x7f6504102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 --2- 172.21.15.153:0/4245964709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6504101d70 0x7f6504102140 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6500004940 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 -- 172.21.15.153:0/4245964709 shutdown_connections 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 --2- 172.21.15.153:0/4245964709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65040fc5b0 0x7f65040fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 --2- 172.21.15.153:0/4245964709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6504101d70 0x7f6504102140 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 -- 172.21.15.153:0/4245964709 >> 172.21.15.153:0/4245964709 conn(0x7f65040714c0 msgr2=0x7f65040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 -- 172.21.15.153:0/4245964709 shutdown_connections 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.693+0000 7f650ab77700 1 -- 172.21.15.153:0/4245964709 wait complete. 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 Processor -- start 2024-04-05T03:19:11.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 -- start start 2024-04-05T03:19:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6504101d70 0x7f6504112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6504113640 con 0x7f65040fc5b0 2024-04-05T03:19:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f650ab77700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650410ca30 con 0x7f6504101d70 2024-04-05T03:19:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f6508913700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f6508913700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38178/0 (socket says 172.21.15.153:38178) 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f6508913700 1 -- 172.21.15.153:0/1896228806 learned_addr learned my addr 172.21.15.153:0/1896228806 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.694+0000 7f64fbfff700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6504101d70 0x7f6504112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.695+0000 7f6508913700 1 -- 172.21.15.153:0/1896228806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6504101d70 msgr2=0x7f6504112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.695+0000 7f6508913700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6504101d70 0x7f6504112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.695+0000 7f6508913700 1 -- 172.21.15.153:0/1896228806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500005040 con 0x7f65040fc5b0 2024-04-05T03:19:11.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.695+0000 7f6508913700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6500009180 tx=0x7f6500009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.696+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650000e040 con 0x7f65040fc5b0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.697+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6500013420 con 0x7f65040fc5b0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.697+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650001c610 con 0x7f65040fc5b0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.697+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650410ccb0 con 0x7f65040fc5b0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.697+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650410d270 con 0x7f65040fc5b0 2024-04-05T03:19:11.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.698+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65040fd5c0 con 0x7f65040fc5b0 2024-04-05T03:19:11.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.698+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6500013a00 con 0x7f65040fc5b0 2024-04-05T03:19:11.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.698+0000 7f64f97fa700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 0x7f64ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:11.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.699+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f650008e5d0 con 0x7f65040fc5b0 2024-04-05T03:19:11.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.699+0000 7f64fbfff700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 0x7f64ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:11.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.699+0000 7f64fbfff700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 0x7f64ec070150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f650410dbf0 tx=0x7f64f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:11.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.705+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f650005b7a0 con 0x7f65040fc5b0 2024-04-05T03:19:11.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.977+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6504108df0 con 0x7f65040fc5b0 2024-04-05T03:19:11.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:11 smithi153 ceph-mon[106102]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:11.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:11 smithi153 ceph-mon[106102]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:11.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:11 smithi153 ceph-mon[106102]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:11.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:11 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1355663396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:19:11.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:11 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1272955336' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:19:11.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.977+0000 7f64f97fa700 1 -- 172.21.15.153:0/1896228806 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f650005b330 con 0x7f65040fc5b0 2024-04-05T03:19:11.980 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:19:11.980 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:19:11.980 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:19:11.980 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:19:11.980 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 msgr2=0x7f64ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 0x7f64ec070150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f650410dbf0 tx=0x7f64f4009040 comp rx=0 tx=0).stop 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 msgr2=0x7f6504112ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6500009180 tx=0x7f6500009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 shutdown_connections 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f64ec06dca0 0x7f64ec070150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65040fc5b0 0x7f6504112ab0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 --2- 172.21.15.153:0/1896228806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6504101d70 0x7f6504112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 >> 172.21.15.153:0/1896228806 conn(0x7f65040714c0 msgr2=0x7f6504104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 shutdown_connections 2024-04-05T03:19:11.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:11.979+0000 7f650ab77700 1 -- 172.21.15.153:0/1896228806 wait complete. 2024-04-05T03:19:12.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:11 smithi203 ceph-mon[95988]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:12.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:11 smithi203 ceph-mon[95988]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:12.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:11 smithi203 ceph-mon[95988]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:12.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:11 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1355663396' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:19:12.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:11 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1272955336' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:19:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:12 smithi203 ceph-mon[95988]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:12 smithi203 ceph-mon[95988]: pgmap v406: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:12 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1896228806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:19:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:12 smithi153 ceph-mon[106102]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:12 smithi153 ceph-mon[106102]: pgmap v406: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:12 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1896228806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:19:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:14 smithi203 ceph-mon[95988]: pgmap v407: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:14 smithi153 ceph-mon[106102]: pgmap v407: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:16 smithi203 ceph-mon[95988]: pgmap v408: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:16 smithi153 ceph-mon[106102]: pgmap v408: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:18 smithi203 ceph-mon[95988]: pgmap v409: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:18 smithi153 ceph-mon[106102]: pgmap v409: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:20.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:19 smithi203 ceph-mon[95988]: pgmap v410: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:20.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:19 smithi153 ceph-mon[106102]: pgmap v410: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:22 smithi203 ceph-mon[95988]: pgmap v411: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:22 smithi153 ceph-mon[106102]: pgmap v411: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:24 smithi203 ceph-mon[95988]: pgmap v412: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:24 smithi153 ceph-mon[106102]: pgmap v412: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:27.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:26 smithi203 ceph-mon[95988]: pgmap v413: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:27.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:26 smithi153 ceph-mon[106102]: pgmap v413: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:28 smithi203 ceph-mon[95988]: pgmap v414: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:28 smithi153 ceph-mon[106102]: pgmap v414: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:30 smithi203 ceph-mon[95988]: pgmap v415: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:30 smithi153 ceph-mon[106102]: pgmap v415: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:32 smithi203 ceph-mon[95988]: pgmap v416: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:32 smithi153 ceph-mon[106102]: pgmap v416: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:34 smithi203 ceph-mon[95988]: pgmap v417: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:34 smithi153 ceph-mon[106102]: pgmap v417: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:36 smithi203 ceph-mon[95988]: pgmap v418: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:36 smithi153 ceph-mon[106102]: pgmap v418: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:38 smithi203 ceph-mon[95988]: pgmap v419: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:38 smithi153 ceph-mon[106102]: pgmap v419: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:40 smithi203 ceph-mon[95988]: pgmap v420: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:40 smithi153 ceph-mon[106102]: pgmap v420: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:42.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 -- 172.21.15.153:0/1861934774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac101d20 msgr2=0x7f51ac1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 --2- 172.21.15.153:0/1861934774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac101d20 0x7f51ac1020f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f51a0004940 tx=0x7f51a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 -- 172.21.15.153:0/1861934774 shutdown_connections 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 --2- 172.21.15.153:0/1861934774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac1026c0 0x7f51ac0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 --2- 172.21.15.153:0/1861934774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac101d20 0x7f51ac1020f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 -- 172.21.15.153:0/1861934774 >> 172.21.15.153:0/1861934774 conn(0x7f51ac0f78b0 msgr2=0x7f51ac0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 -- 172.21.15.153:0/1861934774 shutdown_connections 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 -- 172.21.15.153:0/1861934774 wait complete. 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.094+0000 7f51b30df700 1 Processor -- start 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b30df700 1 -- start start 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b30df700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac101d20 0x7f51ac10eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b30df700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b30df700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac10f5c0 con 0x7f51ac1026c0 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b30df700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac10f700 con 0x7f51ac101d20 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36348/0 (socket says 172.21.15.153:36348) 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 -- 172.21.15.153:0/4276793545 learned_addr learned my addr 172.21.15.153:0/4276793545 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51b0e7b700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac101d20 0x7f51ac10eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 -- 172.21.15.153:0/4276793545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac101d20 msgr2=0x7f51ac10eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac101d20 0x7f51ac10eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.095+0000 7f51abfff700 1 -- 172.21.15.153:0/4276793545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a0005040 con 0x7f51ac1026c0 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51abfff700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f519800c800 tx=0x7f5198004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51980106f0 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac117bb0 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5198010d20 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51980113a0 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.096+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac117f80 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.097+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac0fd470 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.097+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5198010850 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.098+0000 7f51a97fa700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 0x7f519c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.098+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f519808d4d0 con 0x7f51ac1026c0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.098+0000 7f51b0e7b700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 0x7f519c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.098+0000 7f51b0e7b700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 0x7f519c070150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51a000b3b0 tx=0x7f51a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.103+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5198056ad0 con 0x7f51ac1026c0 2024-04-05T03:19:42.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.315+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51ac1055c0 con 0x7f519c06dca0 2024-04-05T03:19:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.316+0000 7f51a97fa700 1 -- 172.21.15.153:0/4276793545 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f51ac1055c0 con 0x7f519c06dca0 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 msgr2=0x7f519c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 0x7f519c070150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f51a000b3b0 tx=0x7f51a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 msgr2=0x7f51ac10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f519800c800 tx=0x7f5198004470 comp rx=0 tx=0).stop 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 shutdown_connections 2024-04-05T03:19:42.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.318+0000 7f51b30df700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f519c06dca0 0x7f519c070150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.319+0000 7f51b30df700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51ac101d20 0x7f51ac10eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.319+0000 7f51b30df700 1 --2- 172.21.15.153:0/4276793545 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51ac1026c0 0x7f51ac10f080 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.319+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 >> 172.21.15.153:0/4276793545 conn(0x7f51ac0f78b0 msgr2=0x7f51ac104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.319+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 shutdown_connections 2024-04-05T03:19:42.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.319+0000 7f51b30df700 1 -- 172.21.15.153:0/4276793545 wait complete. 2024-04-05T03:19:42.328 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:19:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 -- 172.21.15.153:0/4171744611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac41037a0 msgr2=0x7f7ac40fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/4171744611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac41037a0 0x7f7ac40fce10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4004940 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 -- 172.21.15.153:0/4171744611 shutdown_connections 2024-04-05T03:19:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/4171744611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac41037a0 0x7f7ac40fce10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/4171744611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac4102e00 0x7f7ac41031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 -- 172.21.15.153:0/4171744611 >> 172.21.15.153:0/4171744611 conn(0x7f7ac40f89b0 msgr2=0x7f7ac40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 -- 172.21.15.153:0/4171744611 shutdown_connections 2024-04-05T03:19:42.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.421+0000 7f7ac99b3700 1 -- 172.21.15.153:0/4171744611 wait complete. 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 Processor -- start 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 -- start start 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac41037a0 0x7f7ac4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac4072210 con 0x7f7ac4102e00 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac99b3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac4070d40 con 0x7f7ac41037a0 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36376/0 (socket says 172.21.15.153:36376) 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.422+0000 7f7ac2ffd700 1 -- 172.21.15.153:0/1381331483 learned_addr learned my addr 172.21.15.153:0/1381331483 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac27fc700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac41037a0 0x7f7ac4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac2ffd700 1 -- 172.21.15.153:0/1381331483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac41037a0 msgr2=0x7f7ac4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac2ffd700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac41037a0 0x7f7ac4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac2ffd700 1 -- 172.21.15.153:0/1381331483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4005040 con 0x7f7ac4102e00 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac2ffd700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c810 tx=0x7f7ab8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab8012810 con 0x7f7ac4102e00 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ab8012e40 con 0x7f7ac4102e00 2024-04-05T03:19:42.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac4071020 con 0x7f7ac4102e00 2024-04-05T03:19:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab80102e0 con 0x7f7ac4102e00 2024-04-05T03:19:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.423+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac411a190 con 0x7f7ac4102e00 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.424+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac41052d0 con 0x7f7ac4102e00 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.425+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7ab8011380 con 0x7f7ac4102e00 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.425+0000 7f7aab7fe700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 0x7f7aac070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.425+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ab808d920 con 0x7f7ac4102e00 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.425+0000 7f7ac27fc700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 0x7f7aac070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.426+0000 7f7ac27fc700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 0x7f7aac070150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7ac40719f0 tx=0x7f7ab4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.430+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7ab8056ea0 con 0x7f7ac4102e00 2024-04-05T03:19:42.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.656+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ac40717a0 con 0x7f7aac06dca0 2024-04-05T03:19:42.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.658+0000 7f7aab7fe700 1 -- 172.21.15.153:0/1381331483 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ac40717a0 con 0x7f7aac06dca0 2024-04-05T03:19:42.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 msgr2=0x7f7aac070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 0x7f7aac070150 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7ac40719f0 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 msgr2=0x7f7ac40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800c810 tx=0x7f7ab8004470 comp rx=0 tx=0).stop 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 shutdown_connections 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7aac06dca0 0x7f7aac070150 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ac4102e00 0x7f7ac40702c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 --2- 172.21.15.153:0/1381331483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ac41037a0 0x7f7ac4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 >> 172.21.15.153:0/1381331483 conn(0x7f7ac40f89b0 msgr2=0x7f7ac41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 shutdown_connections 2024-04-05T03:19:42.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.660+0000 7f7ac99b3700 1 -- 172.21.15.153:0/1381331483 wait complete. 2024-04-05T03:19:42.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.763+0000 7f72f30f0700 1 -- 172.21.15.153:0/2410597251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 msgr2=0x7f72ec100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.763+0000 7f72f30f0700 1 --2- 172.21.15.153:0/2410597251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec100bd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f72d8004940 tx=0x7f72d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.763+0000 7f72f30f0700 1 -- 172.21.15.153:0/2410597251 shutdown_connections 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.763+0000 7f72f30f0700 1 --2- 172.21.15.153:0/2410597251 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec100bd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.763+0000 7f72f30f0700 1 --2- 172.21.15.153:0/2410597251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 0x7f72ec0fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 -- 172.21.15.153:0/2410597251 >> 172.21.15.153:0/2410597251 conn(0x7f72ec0f7870 msgr2=0x7f72ec0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 -- 172.21.15.153:0/2410597251 shutdown_connections 2024-04-05T03:19:42.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 -- 172.21.15.153:0/2410597251 wait complete. 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 Processor -- start 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 -- start start 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.764+0000 7f72f30f0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 0x7f72ec11a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f30f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f30f0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec11ac70 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f30f0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec114250 con 0x7f72ec0fbc10 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36394/0 (socket says 172.21.15.153:36394) 2024-04-05T03:19:42.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 -- 172.21.15.153:0/504828593 learned_addr learned my addr 172.21.15.153:0/504828593 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f0e8c700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 0x7f72ec11a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 -- 172.21.15.153:0/504828593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 msgr2=0x7f72ec11a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 0x7f72ec11a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 -- 172.21.15.153:0/504828593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8005040 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72ebfff700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72d800b3b0 tx=0x7f72d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72d800e040 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec1144d0 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72d8013420 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72d801c630 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.765+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec114a10 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.766+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec066b50 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.767+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f72d8013590 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.767+0000 7f72e97fa700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 0x7f72dc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:42.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.767+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f72d808e760 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.767+0000 7f72f0e8c700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 0x7f72dc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:42.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.768+0000 7f72f0e8c700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 0x7f72dc070150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72e0004f40 tx=0x7f72e000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:42.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.773+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f72d805b890 con 0x7f72ec0fc5b0 2024-04-05T03:19:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.978+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f72ec1157a0 con 0x7f72dc06dca0 2024-04-05T03:19:42.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:42 smithi153 ceph-mon[106102]: pgmap v421: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:42.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.988+0000 7f72e97fa700 1 -- 172.21.15.153:0/504828593 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f72ec1157a0 con 0x7f72dc06dca0 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (14m) 7m ago 25m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (11m) 7m ago 26m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (11m) 7m ago 24m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (14m) 7m ago 25m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:19:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (7m) 7m ago 19m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (8m) 7m ago 19m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (7m) 7m ago 19m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (7m) 7m ago 19m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (15m) 7m ago 27m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (15m) 7m ago 24m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (12m) 7m ago 27m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (12m) 7m ago 24m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (14m) 7m ago 25m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (14m) 7m ago 24m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (11m) 7m ago 23m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (10m) 7m ago 23m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (10m) 7m ago 22m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (9m) 7m ago 22m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (9m) 7m ago 21m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:19:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (8m) 7m ago 21m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (14m) 7m ago 23m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.990+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 msgr2=0x7f72dc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.990+0000 7f72f30f0700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 0x7f72dc070150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72e0004f40 tx=0x7f72e000b040 comp rx=0 tx=0).stop 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.990+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 msgr2=0x7f72ec11a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.990+0000 7f72f30f0700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72d800b3b0 tx=0x7f72d8009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:42.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 shutdown_connections 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f72dc06dca0 0x7f72dc070150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72ec0fbc10 0x7f72ec11a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 --2- 172.21.15.153:0/504828593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f72ec0fc5b0 0x7f72ec11a620 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 >> 172.21.15.153:0/504828593 conn(0x7f72ec0f7870 msgr2=0x7f72ec0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 shutdown_connections 2024-04-05T03:19:42.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:42.991+0000 7f72f30f0700 1 -- 172.21.15.153:0/504828593 wait complete. 2024-04-05T03:19:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:42 smithi203 ceph-mon[95988]: pgmap v421: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:43.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.091+0000 7f0e01285700 1 -- 172.21.15.153:0/1448151766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc103780 msgr2=0x7f0dfc0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.091+0000 7f0e01285700 1 --2- 172.21.15.153:0/1448151766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc103780 0x7f0dfc0fcdf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0df0004940 tx=0x7f0df0009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.091+0000 7f0e01285700 1 -- 172.21.15.153:0/1448151766 shutdown_connections 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.091+0000 7f0e01285700 1 --2- 172.21.15.153:0/1448151766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc103780 0x7f0dfc0fcdf0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 --2- 172.21.15.153:0/1448151766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc102de0 0x7f0dfc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- 172.21.15.153:0/1448151766 >> 172.21.15.153:0/1448151766 conn(0x7f0dfc0f8970 msgr2=0x7f0dfc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- 172.21.15.153:0/1448151766 shutdown_connections 2024-04-05T03:19:43.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- 172.21.15.153:0/1448151766 wait complete. 2024-04-05T03:19:43.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 Processor -- start 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- start start 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc103780 0x7f0dfc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc070d90 con 0x7f0dfc102de0 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.092+0000 7f0e01285700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc070ed0 con 0x7f0dfc103780 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36420/0 (socket says 172.21.15.153:36420) 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 -- 172.21.15.153:0/1135526442 learned_addr learned my addr 172.21.15.153:0/1135526442 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfa7fc700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc103780 0x7f0dfc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 -- 172.21.15.153:0/1135526442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc103780 msgr2=0x7f0dfc070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc103780 0x7f0dfc070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 -- 172.21.15.153:0/1135526442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df0005040 con 0x7f0dfc102de0 2024-04-05T03:19:43.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0dfaffd700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00eb40 tx=0x7f0dec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dec007cc0 con 0x7f0dfc102de0 2024-04-05T03:19:43.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dec00f040 con 0x7f0dfc102de0 2024-04-05T03:19:43.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc071120 con 0x7f0dfc102de0 2024-04-05T03:19:43.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dec00ca10 con 0x7f0dfc102de0 2024-04-05T03:19:43.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.093+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc11a170 con 0x7f0dfc102de0 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.094+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc0fe530 con 0x7f0dfc102de0 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.095+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f0dec00cc80 con 0x7f0dfc102de0 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.095+0000 7f0de37fe700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 0x7f0de4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.095+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0dec08d140 con 0x7f0dfc102de0 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.095+0000 7f0dfa7fc700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 0x7f0de4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.096+0000 7f0dfa7fc700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 0x7f0de4070150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0df0004890 tx=0x7f0df0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.100+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0dec056770 con 0x7f0dfc102de0 2024-04-05T03:19:43.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.375+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0dfc04bf60 con 0x7f0dfc102de0 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.376+0000 7f0de37fe700 1 -- 172.21.15.153:0/1135526442 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0dec059e30 con 0x7f0dfc102de0 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:19:43.375 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:19:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:19:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:19:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:19:43.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:19:43.376 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 msgr2=0x7f0de4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 0x7f0de4070150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0df0004890 tx=0x7f0df0009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 msgr2=0x7f0dfc0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0dec00eb40 tx=0x7f0dec00b040 comp rx=0 tx=0).stop 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 shutdown_connections 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0de406dca0 0x7f0de4070150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dfc102de0 0x7f0dfc0702c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 --2- 172.21.15.153:0/1135526442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc103780 0x7f0dfc070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 >> 172.21.15.153:0/1135526442 conn(0x7f0dfc0f8970 msgr2=0x7f0dfc0f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:43.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 shutdown_connections 2024-04-05T03:19:43.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.378+0000 7f0e01285700 1 -- 172.21.15.153:0/1135526442 wait complete. 2024-04-05T03:19:43.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 -- 172.21.15.153:0/3801056919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a601026c0 msgr2=0x7f8a600fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 --2- 172.21.15.153:0/3801056919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a601026c0 0x7f8a600fbd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c004940 tx=0x7f8a5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:43.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 -- 172.21.15.153:0/3801056919 shutdown_connections 2024-04-05T03:19:43.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 --2- 172.21.15.153:0/3801056919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a601026c0 0x7f8a600fbd30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 --2- 172.21.15.153:0/3801056919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a60101d20 0x7f8a601020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 -- 172.21.15.153:0/3801056919 >> 172.21.15.153:0/3801056919 conn(0x7f8a600f7870 msgr2=0x7f8a600f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 -- 172.21.15.153:0/3801056919 shutdown_connections 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.480+0000 7f8a67420700 1 -- 172.21.15.153:0/3801056919 wait complete. 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 Processor -- start 2024-04-05T03:19:43.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 -- start start 2024-04-05T03:19:43.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a601026c0 0x7f8a6010f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6010f550 con 0x7f8a60101d20 2024-04-05T03:19:43.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a67420700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6010f690 con 0x7f8a601026c0 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36440/0 (socket says 172.21.15.153:36440) 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 -- 172.21.15.153:0/1827449579 learned_addr learned my addr 172.21.15.153:0/1827449579 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 -- 172.21.15.153:0/1827449579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a601026c0 msgr2=0x7f8a6010f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a649bb700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a601026c0 0x7f8a6010f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a601026c0 0x7f8a6010f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.481+0000 7f8a651bc700 1 -- 172.21.15.153:0/1827449579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c005040 con 0x7f8a60101d20 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a651bc700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a50002a80 tx=0x7f8a500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a50007cc0 con 0x7f8a60101d20 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a5000f040 con 0x7f8a60101d20 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a500096f0 con 0x7f8a60101d20 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a60117b40 con 0x7f8a60101d20 2024-04-05T03:19:43.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.482+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a60117e90 con 0x7f8a60101d20 2024-04-05T03:19:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.483+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a600fd470 con 0x7f8a60101d20 2024-04-05T03:19:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.483+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8a5000d690 con 0x7f8a60101d20 2024-04-05T03:19:43.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.484+0000 7f8a55ffb700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 0x7f8a4c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.484+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8a5008ccd0 con 0x7f8a60101d20 2024-04-05T03:19:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.484+0000 7f8a649bb700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 0x7f8a4c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.484+0000 7f8a649bb700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 0x7f8a4c070150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.488+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8a50056250 con 0x7f8a60101d20 2024-04-05T03:19:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.731+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8a60066810 con 0x7f8a60101d20 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.731+0000 7f8a55ffb700 1 -- 172.21.15.153:0/1827449579 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f8a50059910 con 0x7f8a60101d20 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:19:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:19:43.731 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:19:43.732 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:43.733 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:43.733 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:43.733 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:19:43.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 msgr2=0x7f8a4c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 0x7f8a4c070150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c000c00 tx=0x7f8a5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 msgr2=0x7f8a6010ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8a50002a80 tx=0x7f8a500046b0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 shutdown_connections 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8a4c06dca0 0x7f8a4c070150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a60101d20 0x7f8a6010ead0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 --2- 172.21.15.153:0/1827449579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a601026c0 0x7f8a6010f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 >> 172.21.15.153:0/1827449579 conn(0x7f8a600f7870 msgr2=0x7f8a600f88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 shutdown_connections 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.733+0000 7f8a67420700 1 -- 172.21.15.153:0/1827449579 wait complete. 2024-04-05T03:19:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- 172.21.15.153:0/1158340316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fc5b0 msgr2=0x7f5af8100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/1158340316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fc5b0 0x7f5af8100bd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8004940 tx=0x7f5ae8009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- 172.21.15.153:0/1158340316 shutdown_connections 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/1158340316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fc5b0 0x7f5af8100bd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/1158340316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fbc10 0x7f5af80fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- 172.21.15.153:0/1158340316 >> 172.21.15.153:0/1158340316 conn(0x7f5af80f7870 msgr2=0x7f5af80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- 172.21.15.153:0/1158340316 shutdown_connections 2024-04-05T03:19:43.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- 172.21.15.153:0/1158340316 wait complete. 2024-04-05T03:19:43.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 Processor -- start 2024-04-05T03:19:43.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.836+0000 7f5aff5c6700 1 -- start start 2024-04-05T03:19:43.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aff5c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aff5c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fc5b0 0x7f5af811a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aff5c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af811aba0 con 0x7f5af80fbc10 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aff5c6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af81141d0 con 0x7f5af80fc5b0 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36454/0 (socket says 172.21.15.153:36454) 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 -- 172.21.15.153:0/2515704001 learned_addr learned my addr 172.21.15.153:0/2515704001 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afcb61700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fc5b0 0x7f5af811a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 -- 172.21.15.153:0/2515704001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fc5b0 msgr2=0x7f5af811a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fc5b0 0x7f5af811a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 -- 172.21.15.153:0/2515704001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8005040 con 0x7f5af80fbc10 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5afd362700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5af4002a50 tx=0x7f5af40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5af4007cc0 con 0x7f5af80fbc10 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5af400f040 con 0x7f5af80fbc10 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af81144b0 con 0x7f5af80fbc10 2024-04-05T03:19:43.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.837+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5af4009640 con 0x7f5af80fbc10 2024-04-05T03:19:43.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.838+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af81149f0 con 0x7f5af80fbc10 2024-04-05T03:19:43.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.838+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af8066b70 con 0x7f5af80fbc10 2024-04-05T03:19:43.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.839+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5af40097a0 con 0x7f5af80fbc10 2024-04-05T03:19:43.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.839+0000 7f5aedffb700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 0x7f5ae4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:43.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.839+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5af408d2a0 con 0x7f5af80fbc10 2024-04-05T03:19:43.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.839+0000 7f5afcb61700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 0x7f5ae4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:43.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.840+0000 7f5afcb61700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 0x7f5ae4070150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5af81158b0 tx=0x7f5ae8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:43.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:43.844+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5af405a3d0 con 0x7f5af80fbc10 2024-04-05T03:19:44.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:43 smithi203 ceph-mon[95988]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:44.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:43 smithi203 ceph-mon[95988]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:44.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:43 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1135526442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:19:44.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.055+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5af81156d0 con 0x7f5ae406dca0 2024-04-05T03:19:44.054 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:43 smithi153 ceph-mon[106102]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:44.054 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:43 smithi153 ceph-mon[106102]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:44.054 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:43 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1135526442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:19:44.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.057+0000 7f5aedffb700 1 -- 172.21.15.153:0/2515704001 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5af81156d0 con 0x7f5ae406dca0 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:19:44.056 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:19:44.057 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:19:44.057 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:19:44.057 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:19:44.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 msgr2=0x7f5ae4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 0x7f5ae4070150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5af81158b0 tx=0x7f5ae8009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 msgr2=0x7f5af811a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5af4002a50 tx=0x7f5af40046b0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 shutdown_connections 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ae406dca0 0x7f5ae4070150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af80fbc10 0x7f5af811a010 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 --2- 172.21.15.153:0/2515704001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5af80fc5b0 0x7f5af811a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 >> 172.21.15.153:0/2515704001 conn(0x7f5af80f7870 msgr2=0x7f5af80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 shutdown_connections 2024-04-05T03:19:44.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.060+0000 7f5aff5c6700 1 -- 172.21.15.153:0/2515704001 wait complete. 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- 172.21.15.153:0/2848176145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748102de0 msgr2=0x7f77481031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 --2- 172.21.15.153:0/2848176145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748102de0 0x7f77481031b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7744004940 tx=0x7f7744009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- 172.21.15.153:0/2848176145 shutdown_connections 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 --2- 172.21.15.153:0/2848176145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748103780 0x7f77480fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 --2- 172.21.15.153:0/2848176145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748102de0 0x7f77481031b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- 172.21.15.153:0/2848176145 >> 172.21.15.153:0/2848176145 conn(0x7f77480f8950 msgr2=0x7f77480fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:44.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- 172.21.15.153:0/2848176145 shutdown_connections 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- 172.21.15.153:0/2848176145 wait complete. 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 Processor -- start 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 -- start start 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748102de0 0x7f7748113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.162+0000 7f774f91f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774f91f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7748114590 con 0x7f7748103780 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774f91f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774810db00 con 0x7f7748102de0 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36476/0 (socket says 172.21.15.153:36476) 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 -- 172.21.15.153:0/1382268244 learned_addr learned my addr 172.21.15.153:0/1382268244 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774d6bb700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748102de0 0x7f7748113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 -- 172.21.15.153:0/1382268244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748102de0 msgr2=0x7f7748113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748102de0 0x7f7748113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 -- 172.21.15.153:0/1382268244 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744005040 con 0x7f7748103780 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774ceba700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f773800b790 tx=0x7f7738009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f773800f6f0 con 0x7f7748103780 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774810dde0 con 0x7f7748103780 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f773800fd20 con 0x7f7748103780 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.163+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77380103a0 con 0x7f7748103780 2024-04-05T03:19:44.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.164+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774810e3a0 con 0x7f7748103780 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.164+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77481000f0 con 0x7f7748103780 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.165+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7738011400 con 0x7f7748103780 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.165+0000 7f773dffb700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 0x7f7734070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.165+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f773808d450 con 0x7f7748103780 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.165+0000 7f774d6bb700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 0x7f7734070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:19:44.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.166+0000 7f774d6bb700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 0x7f7734070150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7744004f90 tx=0x7f7744009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:19:44.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.170+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f773805a500 con 0x7f7748103780 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.447+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f77480fe530 con 0x7f7748103780 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.447+0000 7f773dffb700 1 -- 172.21.15.153:0/1382268244 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f773805a090 con 0x7f7748103780 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:19:44.446 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:19:44.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 msgr2=0x7f7734070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 0x7f7734070150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7744004f90 tx=0x7f7744009e30 comp rx=0 tx=0).stop 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 msgr2=0x7f7748113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f773800b790 tx=0x7f7738009220 comp rx=0 tx=0).stop 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 shutdown_connections 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f773406dca0 0x7f7734070150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7748102de0 0x7f7748113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 --2- 172.21.15.153:0/1382268244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7748103780 0x7f7748113eb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 >> 172.21.15.153:0/1382268244 conn(0x7f77480f8950 msgr2=0x7f7748107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 shutdown_connections 2024-04-05T03:19:44.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:19:44.449+0000 7f774f91f700 1 -- 172.21.15.153:0/1382268244 wait complete. 2024-04-05T03:19:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:44 smithi203 ceph-mon[95988]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:44 smithi203 ceph-mon[95988]: pgmap v422: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:44 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1827449579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:19:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:44 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1382268244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:19:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:44 smithi153 ceph-mon[106102]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:44 smithi153 ceph-mon[106102]: pgmap v422: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:44 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1827449579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:19:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:44 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1382268244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:19:46.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:45 smithi203 ceph-mon[95988]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:46.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:45 smithi153 ceph-mon[106102]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:19:46.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:46 smithi153 ceph-mon[106102]: pgmap v423: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:46 smithi203 ceph-mon[95988]: pgmap v423: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:48 smithi203 ceph-mon[95988]: pgmap v424: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:48 smithi153 ceph-mon[106102]: pgmap v424: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:50.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:49 smithi203 ceph-mon[95988]: pgmap v425: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:50.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:49 smithi153 ceph-mon[106102]: pgmap v425: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:52 smithi203 ceph-mon[95988]: pgmap v426: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:53.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:52 smithi153 ceph-mon[106102]: pgmap v426: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:54 smithi203 ceph-mon[95988]: pgmap v427: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:54 smithi153 ceph-mon[106102]: pgmap v427: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:56 smithi203 ceph-mon[95988]: pgmap v428: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:56 smithi153 ceph-mon[106102]: pgmap v428: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:58.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:19:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:19:58 smithi203 ceph-mon[95988]: pgmap v429: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:19:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:19:58 smithi153 ceph-mon[106102]: pgmap v429: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:00.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:19:59.999+0000 7f9e332b5700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:20:00.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:19:59.999+0000 7f9e332b5700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:20:00.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:19:59.999+0000 7f9e332b5700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-05T03:20:00.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:19:59.999+0000 7f9e332b5700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:20:00.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:19:59.999+0000 7f9e332b5700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: pgmap v430: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: fs cephfs is degraded 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:20:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:00 smithi203 ceph-mon[95988]: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:20:01.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: pgmap v430: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:20:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:20:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: fs cephfs is degraded 2024-04-05T03:20:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:20:01.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:00 smithi153 ceph-mon[106102]: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:20:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:02 smithi203 ceph-mon[95988]: pgmap v431: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:02 smithi153 ceph-mon[106102]: pgmap v431: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:04 smithi203 ceph-mon[95988]: pgmap v432: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:04 smithi153 ceph-mon[106102]: pgmap v432: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:06 smithi203 ceph-mon[95988]: pgmap v433: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:06 smithi153 ceph-mon[106102]: pgmap v433: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:08 smithi203 ceph-mon[95988]: pgmap v434: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:08 smithi153 ceph-mon[106102]: pgmap v434: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:10 smithi203 ceph-mon[95988]: pgmap v435: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:10 smithi153 ceph-mon[106102]: pgmap v435: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:12 smithi203 ceph-mon[95988]: pgmap v436: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:12 smithi153 ceph-mon[106102]: pgmap v436: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 -- 172.21.15.153:0/3852049843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 msgr2=0x7f51d8070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 --2- 172.21.15.153:0/3852049843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d8070360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f51d4004970 tx=0x7f51d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 -- 172.21.15.153:0/3852049843 shutdown_connections 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 --2- 172.21.15.153:0/3852049843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 0x7f51d8070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 --2- 172.21.15.153:0/3852049843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d8070360 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 -- 172.21.15.153:0/3852049843 >> 172.21.15.153:0/3852049843 conn(0x7f51d80f89b0 msgr2=0x7f51d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 -- 172.21.15.153:0/3852049843 shutdown_connections 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.563+0000 7f51df3a5700 1 -- 172.21.15.153:0/3852049843 wait complete. 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 Processor -- start 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 -- start start 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 0x7f51d8119ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d811aa60 con 0x7f51d8071f90 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51df3a5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d81140f0 con 0x7f51d80708a0 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42584/0 (socket says 172.21.15.153:42584) 2024-04-05T03:20:14.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 -- 172.21.15.153:0/1426097282 learned_addr learned my addr 172.21.15.153:0/1426097282 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dd141700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 0x7f51d8119ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 -- 172.21.15.153:0/1426097282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 msgr2=0x7f51d8119ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 0x7f51d8119ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 -- 172.21.15.153:0/1426097282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d4005040 con 0x7f51d8071f90 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.564+0000 7f51dc940700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c8c0 tx=0x7f51c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.565+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c800a980 con 0x7f51d8071f90 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.565+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51c800ce60 con 0x7f51d8071f90 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.565+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d81143d0 con 0x7f51d8071f90 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.565+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51c80115e0 con 0x7f51d8071f90 2024-04-05T03:20:14.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.565+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d8114910 con 0x7f51d8071f90 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.566+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d8071aa0 con 0x7f51d8071f90 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.566+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f51c8012400 con 0x7f51d8071f90 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.567+0000 7f51cdffb700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 0x7f51c4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.567+0000 7f51dd141700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 0x7f51c4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.567+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f51c808d670 con 0x7f51d8071f90 2024-04-05T03:20:14.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.567+0000 7f51dd141700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 0x7f51c4070150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f51d4004f50 tx=0x7f51d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:14.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.571+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f51c8056c70 con 0x7f51d8071f90 2024-04-05T03:20:14.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.784+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d8110520 con 0x7f51c406dca0 2024-04-05T03:20:14.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.785+0000 7f51cdffb700 1 -- 172.21.15.153:0/1426097282 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f51d8110520 con 0x7f51c406dca0 2024-04-05T03:20:14.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 msgr2=0x7f51c4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 0x7f51c4070150 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f51d4004f50 tx=0x7f51d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:14.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 msgr2=0x7f51d811a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c8c0 tx=0x7f51c80044c0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 shutdown_connections 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f51c406dca0 0x7f51c4070150 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51d80708a0 0x7f51d8119ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 --2- 172.21.15.153:0/1426097282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51d8071f90 0x7f51d811a410 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 >> 172.21.15.153:0/1426097282 conn(0x7f51d80f89b0 msgr2=0x7f51d80f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 shutdown_connections 2024-04-05T03:20:14.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.787+0000 7f51df3a5700 1 -- 172.21.15.153:0/1426097282 wait complete. 2024-04-05T03:20:14.796 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:20:14.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:14 smithi153 ceph-mon[106102]: pgmap v437: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:14.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- 172.21.15.153:0/1131112770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fe7c0 msgr2=0x7f7b00106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 --2- 172.21.15.153:0/1131112770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fe7c0 0x7f7b00106e00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7af0004940 tx=0x7f7af0009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:14.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- 172.21.15.153:0/1131112770 shutdown_connections 2024-04-05T03:20:14.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 --2- 172.21.15.153:0/1131112770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fe7c0 0x7f7b00106e00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 --2- 172.21.15.153:0/1131112770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fde20 0x7f7b000fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- 172.21.15.153:0/1131112770 >> 172.21.15.153:0/1131112770 conn(0x7f7b000f7870 msgr2=0x7f7b000f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- 172.21.15.153:0/1131112770 shutdown_connections 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- 172.21.15.153:0/1131112770 wait complete. 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 Processor -- start 2024-04-05T03:20:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.891+0000 7f7b06e85700 1 -- start start 2024-04-05T03:20:14.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b06e85700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b06e85700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fe7c0 0x7f7b00113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b06e85700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00114530 con 0x7f7b000fde20 2024-04-05T03:20:14.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b06e85700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0010db30 con 0x7f7b000fe7c0 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42604/0 (socket says 172.21.15.153:42604) 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 -- 172.21.15.153:0/3084413601 learned_addr learned my addr 172.21.15.153:0/3084413601 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7af7fff700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fe7c0 0x7f7b00113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 -- 172.21.15.153:0/3084413601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fe7c0 msgr2=0x7f7b00113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fe7c0 0x7f7b00113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 -- 172.21.15.153:0/3084413601 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af0005040 con 0x7f7b000fde20 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b04c21700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7afc002a50 tx=0x7f7afc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc007cc0 con 0x7f7b000fde20 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0010de10 con 0x7f7b000fde20 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.892+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7afc00f040 con 0x7f7b000fde20 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.893+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7afc00da30 con 0x7f7b000fde20 2024-04-05T03:20:14.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.893+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0010e320 con 0x7f7b000fde20 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.894+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b00066b70 con 0x7f7b000fde20 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.894+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7afc00db90 con 0x7f7b000fde20 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.894+0000 7f7af57fa700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 0x7f7ae8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.894+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7afc08d2e0 con 0x7f7b000fde20 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.894+0000 7f7af7fff700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 0x7f7ae8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.895+0000 7f7af7fff700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 0x7f7ae8070150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b0010ee10 tx=0x7f7af0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:14.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:14.899+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7afc05a390 con 0x7f7b000fde20 2024-04-05T03:20:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:14 smithi203 ceph-mon[95988]: pgmap v437: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:15.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.111+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b00109eb0 con 0x7f7ae806dca0 2024-04-05T03:20:15.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.112+0000 7f7af57fa700 1 -- 172.21.15.153:0/3084413601 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7b00109eb0 con 0x7f7ae806dca0 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 msgr2=0x7f7ae8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 0x7f7ae8070150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b0010ee10 tx=0x7f7af0009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 msgr2=0x7f7b001139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7afc002a50 tx=0x7f7afc0046b0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 shutdown_connections 2024-04-05T03:20:15.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ae806dca0 0x7f7ae8070150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b000fde20 0x7f7b001139a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 --2- 172.21.15.153:0/3084413601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b000fe7c0 0x7f7b00113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 >> 172.21.15.153:0/3084413601 conn(0x7f7b000f7870 msgr2=0x7f7b000f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 shutdown_connections 2024-04-05T03:20:15.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.114+0000 7f7b06e85700 1 -- 172.21.15.153:0/3084413601 wait complete. 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1052476671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f8102e10 msgr2=0x7f93f81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1052476671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f8102e10 0x7f93f81031e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f93e8004940 tx=0x7f93e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1052476671 shutdown_connections 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1052476671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f81037b0 0x7f93f80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1052476671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f8102e10 0x7f93f81031e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1052476671 >> 172.21.15.153:0/1052476671 conn(0x7f93f80f89a0 msgr2=0x7f93f80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1052476671 shutdown_connections 2024-04-05T03:20:15.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.216+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1052476671 wait complete. 2024-04-05T03:20:15.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 Processor -- start 2024-04-05T03:20:15.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 -- start start 2024-04-05T03:20:15.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f8102e10 0x7f93f8113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81146f0 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93ff3ae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f810dae0 con 0x7f93f8102e10 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93fc949700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93fc949700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42620/0 (socket says 172.21.15.153:42620) 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93fc949700 1 -- 172.21.15.153:0/1154611265 learned_addr learned my addr 172.21.15.153:0/1154611265 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.217+0000 7f93fd14a700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f8102e10 0x7f93f8113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93fc949700 1 -- 172.21.15.153:0/1154611265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f8102e10 msgr2=0x7f93f8113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93fc949700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f8102e10 0x7f93f8113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93fc949700 1 -- 172.21.15.153:0/1154611265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e8005040 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93fc949700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f93f400eb40 tx=0x7f93f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93f4007cc0 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93f400f040 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.218+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f810dd60 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.219+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93f400c9d0 con 0x7f93f81037b0 2024-04-05T03:20:15.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.219+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f810e320 con 0x7f93f81037b0 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.220+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f93f400cb30 con 0x7f93f81037b0 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.220+0000 7f93edffb700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 0x7f93e4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.220+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f8100120 con 0x7f93f81037b0 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.220+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f93f408cea0 con 0x7f93f81037b0 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.220+0000 7f93fd14a700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 0x7f93e4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.221+0000 7f93fd14a700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 0x7f93e4070150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009fd0 tx=0x7f93e8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.225+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f93f405a080 con 0x7f93f81037b0 2024-04-05T03:20:15.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.427+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f93f810f1b0 con 0x7f93e406dca0 2024-04-05T03:20:15.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.439+0000 7f93edffb700 1 -- 172.21.15.153:0/1154611265 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f93f810f1b0 con 0x7f93e406dca0 2024-04-05T03:20:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:20:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (14m) 8m ago 25m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:20:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (12m) 8m ago 27m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:20:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (12m) 7m ago 24m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (14m) 8m ago 25m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (8m) 8m ago 20m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (8m) 8m ago 20m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (7m) 7m ago 20m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (8m) 7m ago 20m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (16m) 8m ago 28m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (15m) 7m ago 24m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (13m) 8m ago 28m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (13m) 7m ago 24m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (15m) 8m ago 26m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (15m) 7m ago 24m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (11m) 8m ago 24m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (11m) 8m ago 23m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (10m) 8m ago 23m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:20:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (10m) 7m ago 22m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:20:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (9m) 7m ago 22m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:20:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (9m) 7m ago 21m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:20:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (15m) 7m ago 24m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:20:15.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 msgr2=0x7f93e4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 0x7f93e4070150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009fd0 tx=0x7f93e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 msgr2=0x7f93f81140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f93f400eb40 tx=0x7f93f400b040 comp rx=0 tx=0).stop 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 shutdown_connections 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93e406dca0 0x7f93e4070150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93f8102e10 0x7f93f8113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 --2- 172.21.15.153:0/1154611265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93f81037b0 0x7f93f81140a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 >> 172.21.15.153:0/1154611265 conn(0x7f93f80f89a0 msgr2=0x7f93f8107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 shutdown_connections 2024-04-05T03:20:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.441+0000 7f93ff3ae700 1 -- 172.21.15.153:0/1154611265 wait complete. 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- 172.21.15.153:0/3044852595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 msgr2=0x7f6cd80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/3044852595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd80fce10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4004940 tx=0x7f6cc4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- 172.21.15.153:0/3044852595 shutdown_connections 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/3044852595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd80fce10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/3044852595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 0x7f6cd81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- 172.21.15.153:0/3044852595 >> 172.21.15.153:0/3044852595 conn(0x7f6cd80f89b0 msgr2=0x7f6cd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- 172.21.15.153:0/3044852595 shutdown_connections 2024-04-05T03:20:15.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- 172.21.15.153:0/3044852595 wait complete. 2024-04-05T03:20:15.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 Processor -- start 2024-04-05T03:20:15.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.541+0000 7f6cdebb7700 1 -- start start 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cdebb7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 0x7f6cd8113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cdebb7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cdebb7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8114510 con 0x7f6cd81037a0 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cdebb7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd810db10 con 0x7f6cd8102e00 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42636/0 (socket says 172.21.15.153:42636) 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 -- 172.21.15.153:0/1652917039 learned_addr learned my addr 172.21.15.153:0/1652917039 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cdc953700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 0x7f6cd8113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 -- 172.21.15.153:0/1652917039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 msgr2=0x7f6cd8113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 0x7f6cd8113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 -- 172.21.15.153:0/1652917039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4005040 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.542+0000 7f6cd7fff700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009100 tx=0x7f6cc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.543+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc400e040 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.543+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd810dd90 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.543+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc4013420 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.543+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc401c630 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.543+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd810e350 con 0x7f6cd81037a0 2024-04-05T03:20:15.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.544+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd81052d0 con 0x7f6cd81037a0 2024-04-05T03:20:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.544+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6cc4013590 con 0x7f6cd81037a0 2024-04-05T03:20:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.545+0000 7f6cd57fa700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 0x7f6cc8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.545+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6cc408e840 con 0x7f6cd81037a0 2024-04-05T03:20:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.545+0000 7f6cdc953700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 0x7f6cc8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.545+0000 7f6cdc953700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 0x7f6cc8070150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc007fd0 tx=0x7f6ccc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.550+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6cc405b8d0 con 0x7f6cd81037a0 2024-04-05T03:20:15.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.829+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6cd810efd0 con 0x7f6cd81037a0 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.829+0000 7f6cd57fa700 1 -- 172.21.15.153:0/1652917039 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6cc405b460 con 0x7f6cd81037a0 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:20:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:20:15.829 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 msgr2=0x7f6cc8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 0x7f6cc8070150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc007fd0 tx=0x7f6ccc009040 comp rx=0 tx=0).stop 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 msgr2=0x7f6cd8113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009100 tx=0x7f6cc4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 shutdown_connections 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6cc806dca0 0x7f6cc8070150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cd8102e00 0x7f6cd8113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 --2- 172.21.15.153:0/1652917039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cd81037a0 0x7f6cd8113ec0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 >> 172.21.15.153:0/1652917039 conn(0x7f6cd80f89b0 msgr2=0x7f6cd81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 shutdown_connections 2024-04-05T03:20:15.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.831+0000 7f6cdebb7700 1 -- 172.21.15.153:0/1652917039 wait complete. 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- 172.21.15.153:0/421784237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b30101b50 msgr2=0x7f9b30101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 --2- 172.21.15.153:0/421784237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b30101b50 0x7f9b30101f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c004940 tx=0x7f9b2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- 172.21.15.153:0/421784237 shutdown_connections 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 --2- 172.21.15.153:0/421784237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b301024f0 0x7f9b300fbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 --2- 172.21.15.153:0/421784237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b30101b50 0x7f9b30101f20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- 172.21.15.153:0/421784237 >> 172.21.15.153:0/421784237 conn(0x7f9b300f76a0 msgr2=0x7f9b300f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- 172.21.15.153:0/421784237 shutdown_connections 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- 172.21.15.153:0/421784237 wait complete. 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 Processor -- start 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.933+0000 7f9b3719e700 1 -- start start 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30101b50 0x7f9b301029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30103480 con 0x7f9b301024f0 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301035c0 con 0x7f9b30101b50 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42650/0 (socket says 172.21.15.153:42650) 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 -- 172.21.15.153:0/1062410317 learned_addr learned my addr 172.21.15.153:0/1062410317 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3619c700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30101b50 0x7f9b301029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 -- 172.21.15.153:0/1062410317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30101b50 msgr2=0x7f9b301029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30101b50 0x7f9b301029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 -- 172.21.15.153:0/1062410317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c005040 con 0x7f9b301024f0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3599b700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c890 tx=0x7f9b200044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b2000a700 con 0x7f9b301024f0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b2000ad30 con 0x7f9b301024f0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30119ce0 con 0x7f9b301024f0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b200113a0 con 0x7f9b301024f0 2024-04-05T03:20:15.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.934+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3011a160 con 0x7f9b301024f0 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.935+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b300fd2a0 con 0x7f9b301024f0 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.936+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9b20012400 con 0x7f9b301024f0 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.936+0000 7f9b26ffd700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 0x7f9b1c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.936+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9b2008d5f0 con 0x7f9b301024f0 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.936+0000 7f9b3619c700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 0x7f9b1c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:15.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.937+0000 7f9b3619c700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 0x7f9b1c070150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c004f90 tx=0x7f9b2c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:15.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:15.941+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9b20056bf0 con 0x7f9b301024f0 2024-04-05T03:20:16.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.181+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9b3004bed0 con 0x7f9b301024f0 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.181+0000 7f9b26ffd700 1 -- 172.21.15.153:0/1062410317 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f9b2005a2b0 con 0x7f9b301024f0 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:16.180 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:20:16.181 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:20:16.182 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 msgr2=0x7f9b1c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 0x7f9b1c070150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c004f90 tx=0x7f9b2c009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 msgr2=0x7f9b30102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c890 tx=0x7f9b200044c0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 shutdown_connections 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9b1c06dca0 0x7f9b1c070150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b30101b50 0x7f9b301029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 --2- 172.21.15.153:0/1062410317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b301024f0 0x7f9b30102ef0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.183+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 >> 172.21.15.153:0/1062410317 conn(0x7f9b300f76a0 msgr2=0x7f9b300f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.184+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 shutdown_connections 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.184+0000 7f9b3719e700 1 -- 172.21.15.153:0/1062410317 wait complete. 2024-04-05T03:20:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- 172.21.15.153:0/2561894412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694102de0 msgr2=0x7f76941031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 --2- 172.21.15.153:0/2561894412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694102de0 0x7f76941031b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7684004940 tx=0x7f7684009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- 172.21.15.153:0/2561894412 shutdown_connections 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 --2- 172.21.15.153:0/2561894412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694103780 0x7f76940fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 --2- 172.21.15.153:0/2561894412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694102de0 0x7f76941031b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- 172.21.15.153:0/2561894412 >> 172.21.15.153:0/2561894412 conn(0x7f76940f8950 msgr2=0x7f76940fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- 172.21.15.153:0/2561894412 shutdown_connections 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- 172.21.15.153:0/2561894412 wait complete. 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 Processor -- start 2024-04-05T03:20:16.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.286+0000 7f76994e0700 1 -- start start 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694102de0 0x7f76940702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7694072210 con 0x7f7694103780 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7694070d40 con 0x7f7694102de0 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42674/0 (socket says 172.21.15.153:42674) 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 -- 172.21.15.153:0/1115038033 learned_addr learned my addr 172.21.15.153:0/1115038033 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f7692ffd700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694102de0 0x7f76940702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 -- 172.21.15.153:0/1115038033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694102de0 msgr2=0x7f76940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694102de0 0x7f76940702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 -- 172.21.15.153:0/1115038033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7684005040 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76927fc700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7688002a50 tx=0x7f7688004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7688007cc0 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7694071020 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f768800f040 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f768800d9c0 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.287+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769411a130 con 0x7f7694103780 2024-04-05T03:20:16.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.288+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76941000f0 con 0x7f7694103780 2024-04-05T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.289+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f768800dbc0 con 0x7f7694103780 2024-04-05T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.289+0000 7f767b7fe700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 0x7f767c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.289+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f768808d2d0 con 0x7f7694103780 2024-04-05T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.289+0000 7f7692ffd700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 0x7f767c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.290+0000 7f7692ffd700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 0x7f767c070150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7684004f90 tx=0x7f7684009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:16.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.294+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7688056850 con 0x7f7694103780 2024-04-05T03:20:16.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.504+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7694109ee0 con 0x7f767c06dca0 2024-04-05T03:20:16.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.506+0000 7f767b7fe700 1 -- 172.21.15.153:0/1115038033 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7694109ee0 con 0x7f767c06dca0 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:20:16.505 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:20:16.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 msgr2=0x7f767c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 0x7f767c070150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7684004f90 tx=0x7f7684009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 msgr2=0x7f7694070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7688002a50 tx=0x7f7688004600 comp rx=0 tx=0).stop 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 shutdown_connections 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f767c06dca0 0x7f767c070150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7694102de0 0x7f76940702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 --2- 172.21.15.153:0/1115038033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7694103780 0x7f7694070800 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 >> 172.21.15.153:0/1115038033 conn(0x7f76940f8950 msgr2=0x7f7694107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 shutdown_connections 2024-04-05T03:20:16.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.508+0000 7f76994e0700 1 -- 172.21.15.153:0/1115038033 wait complete. 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 -- 172.21.15.153:0/193070465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fe7a0 msgr2=0x7f11bc106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 --2- 172.21.15.153:0/193070465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc106e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f11a4004940 tx=0x7f11a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 -- 172.21.15.153:0/193070465 shutdown_connections 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 --2- 172.21.15.153:0/193070465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc106e70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 --2- 172.21.15.153:0/193070465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fde00 0x7f11bc0fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 -- 172.21.15.153:0/193070465 >> 172.21.15.153:0/193070465 conn(0x7f11bc0f7880 msgr2=0x7f11bc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 -- 172.21.15.153:0/193070465 shutdown_connections 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 -- 172.21.15.153:0/193070465 wait complete. 2024-04-05T03:20:16.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.609+0000 7f11c19a2700 1 Processor -- start 2024-04-05T03:20:16.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11c19a2700 1 -- start start 2024-04-05T03:20:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11c19a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11c19a2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11c19a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc114a60 con 0x7f11bc0fde00 2024-04-05T03:20:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11c19a2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc10de30 con 0x7f11bc0fe7a0 2024-04-05T03:20:16.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42692/0 (socket says 172.21.15.153:42692) 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 -- 172.21.15.153:0/2388478538 learned_addr learned my addr 172.21.15.153:0/2388478538 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11b27fc700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 -- 172.21.15.153:0/2388478538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 msgr2=0x7f11bc114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc114410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 -- 172.21.15.153:0/2388478538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a4005040 con 0x7f11bc0fde00 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11b27fc700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.610+0000 7f11baffd700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c810 tx=0x7f11ac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.611+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11ac012810 con 0x7f11bc0fde00 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.611+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc10e110 con 0x7f11bc0fde00 2024-04-05T03:20:16.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.611+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11ac012e40 con 0x7f11bc0fde00 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.611+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11ac00a370 con 0x7f11bc0fde00 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.611+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc10e650 con 0x7f11bc0fde00 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.612+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc0666f0 con 0x7f11bc0fde00 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.612+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f11ac011810 con 0x7f11bc0fde00 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.613+0000 7f11b3fff700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 0x7f11a8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:16.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.613+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f11ac08d9f0 con 0x7f11bc0fde00 2024-04-05T03:20:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.613+0000 7f11b27fc700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 0x7f11a8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:16.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.613+0000 7f11b27fc700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 0x7f11a8070150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11a4000c00 tx=0x7f11a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:16.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.617+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f11ac08fe60 con 0x7f11bc0fde00 2024-04-05T03:20:16.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:16.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:16.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: pgmap v438: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1652917039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:20:16.822 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:16 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1062410317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.892+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f11bc05f5f0 con 0x7f11bc0fde00 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.892+0000 7f11b3fff700 1 -- 172.21.15.153:0/2388478538 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f11ac05aaa0 con 0x7f11bc0fde00 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:20:16.891 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 msgr2=0x7f11a8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 0x7f11a8070150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11a4000c00 tx=0x7f11a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 msgr2=0x7f11bc113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c810 tx=0x7f11ac004470 comp rx=0 tx=0).stop 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 shutdown_connections 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f11a806dca0 0x7f11a8070150 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11bc0fde00 0x7f11bc113ed0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 --2- 172.21.15.153:0/2388478538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11bc0fe7a0 0x7f11bc114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 >> 172.21.15.153:0/2388478538 conn(0x7f11bc0f7880 msgr2=0x7f11bc0f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.894+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 shutdown_connections 2024-04-05T03:20:16.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:16.895+0000 7f11c19a2700 1 -- 172.21.15.153:0/2388478538 wait complete. 2024-04-05T03:20:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: pgmap v438: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1652917039' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:20:17.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:16 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1062410317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:20:18.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:17 smithi203 ceph-mon[95988]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:18.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:17 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2388478538' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:20:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:17 smithi153 ceph-mon[106102]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:17 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2388478538' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:20:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:18 smithi203 ceph-mon[95988]: pgmap v439: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:18 smithi153 ceph-mon[106102]: pgmap v439: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:20.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:19 smithi203 ceph-mon[95988]: pgmap v440: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:20.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:19 smithi153 ceph-mon[106102]: pgmap v440: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:22 smithi203 ceph-mon[95988]: pgmap v441: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:22 smithi153 ceph-mon[106102]: pgmap v441: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:24 smithi203 ceph-mon[95988]: pgmap v442: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:24 smithi153 ceph-mon[106102]: pgmap v442: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:26 smithi203 ceph-mon[95988]: pgmap v443: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:26 smithi153 ceph-mon[106102]: pgmap v443: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:29.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:28 smithi203 ceph-mon[95988]: pgmap v444: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:28 smithi153 ceph-mon[106102]: pgmap v444: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:30 smithi203 ceph-mon[95988]: pgmap v445: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:30 smithi153 ceph-mon[106102]: pgmap v445: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:32 smithi203 ceph-mon[95988]: pgmap v446: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:32 smithi153 ceph-mon[106102]: pgmap v446: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:34 smithi203 ceph-mon[95988]: pgmap v447: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:34 smithi153 ceph-mon[106102]: pgmap v447: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:36 smithi203 ceph-mon[95988]: pgmap v448: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:36 smithi153 ceph-mon[106102]: pgmap v448: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:38 smithi203 ceph-mon[95988]: pgmap v449: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:38 smithi153 ceph-mon[106102]: pgmap v449: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:40 smithi203 ceph-mon[95988]: pgmap v450: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:40 smithi153 ceph-mon[106102]: pgmap v450: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:42 smithi203 ceph-mon[95988]: pgmap v451: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:42 smithi153 ceph-mon[106102]: pgmap v451: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:44 smithi203 ceph-mon[95988]: pgmap v452: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:44 smithi153 ceph-mon[106102]: pgmap v452: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 -- 172.21.15.153:0/1259281172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900102e20 msgr2=0x7f49001031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 --2- 172.21.15.153:0/1259281172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900102e20 0x7f49001031f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f48f4004940 tx=0x7f48f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 -- 172.21.15.153:0/1259281172 shutdown_connections 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 --2- 172.21.15.153:0/1259281172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49001037c0 0x7f49000fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 --2- 172.21.15.153:0/1259281172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4900102e20 0x7f49001031f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 -- 172.21.15.153:0/1259281172 >> 172.21.15.153:0/1259281172 conn(0x7f49000f89d0 msgr2=0x7f49000fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.009 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:46 smithi153 ceph-mon[106102]: pgmap v453: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 -- 172.21.15.153:0/1259281172 shutdown_connections 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.010+0000 7f49055a6700 1 -- 172.21.15.153:0/1259281172 wait complete. 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 Processor -- start 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 -- start start 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 0x7f49000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4900070d90 con 0x7f49001037c0 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f49055a6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4900070ed0 con 0x7f4900102e20 2024-04-05T03:20:47.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f48fe7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f48fe7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55278/0 (socket says 172.21.15.153:55278) 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.011+0000 7f48fe7fc700 1 -- 172.21.15.153:0/888976204 learned_addr learned my addr 172.21.15.153:0/888976204 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48feffd700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 0x7f49000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48fe7fc700 1 -- 172.21.15.153:0/888976204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 msgr2=0x7f49000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48fe7fc700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 0x7f49000702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48fe7fc700 1 -- 172.21.15.153:0/888976204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f4005040 con 0x7f49001037c0 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48feffd700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 0x7f49000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48fe7fc700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f48f0002a50 tx=0x7f48f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f0007cc0 con 0x7f49001037c0 2024-04-05T03:20:47.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48f000f040 con 0x7f49001037c0 2024-04-05T03:20:47.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48f000daf0 con 0x7f49001037c0 2024-04-05T03:20:47.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4900119e90 con 0x7f49001037c0 2024-04-05T03:20:47.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.012+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490011a260 con 0x7f49001037c0 2024-04-05T03:20:47.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.013+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900100130 con 0x7f49001037c0 2024-04-05T03:20:47.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.014+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f48f0009810 con 0x7f49001037c0 2024-04-05T03:20:47.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.014+0000 7f48e77fe700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 0x7f48e8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.014+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f48f008d240 con 0x7f49001037c0 2024-04-05T03:20:47.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.014+0000 7f48feffd700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 0x7f48e8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.015+0000 7f48feffd700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 0x7f48e8070150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48f4004560 tx=0x7f48f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.019+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f48f0056840 con 0x7f49001037c0 2024-04-05T03:20:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:46 smithi203 ceph-mon[95988]: pgmap v453: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.238+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4900071060 con 0x7f48e806dca0 2024-04-05T03:20:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.240+0000 7f48e77fe700 1 -- 172.21.15.153:0/888976204 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4900071060 con 0x7f48e806dca0 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 msgr2=0x7f48e8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 0x7f48e8070150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48f4004560 tx=0x7f48f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 msgr2=0x7f4900070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f48f0002a50 tx=0x7f48f0004600 comp rx=0 tx=0).stop 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 shutdown_connections 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48e806dca0 0x7f48e8070150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4900102e20 0x7f49000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 --2- 172.21.15.153:0/888976204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49001037c0 0x7f4900070800 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 >> 172.21.15.153:0/888976204 conn(0x7f49000f89d0 msgr2=0x7f4900107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 shutdown_connections 2024-04-05T03:20:47.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.242+0000 7f49055a6700 1 -- 172.21.15.153:0/888976204 wait complete. 2024-04-05T03:20:47.252 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- 172.21.15.153:0/62230657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 msgr2=0x7f21b40fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/62230657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fe1e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f21a4004940 tx=0x7f21a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- 172.21.15.153:0/62230657 shutdown_connections 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/62230657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b4106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/62230657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fe1e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- 172.21.15.153:0/62230657 >> 172.21.15.153:0/62230657 conn(0x7f21b40f78d0 msgr2=0x7f21b40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- 172.21.15.153:0/62230657 shutdown_connections 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- 172.21.15.153:0/62230657 wait complete. 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 Processor -- start 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- start start 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b40fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40fc7a0 con 0x7f21b40fde10 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.352+0000 7f21b8c2d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40fc8e0 con 0x7f21b40fe7b0 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55294/0 (socket says 172.21.15.153:55294) 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 -- 172.21.15.153:0/1835812894 learned_addr learned my addr 172.21.15.153:0/1835812894 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b1d9b700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b40fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 -- 172.21.15.153:0/1835812894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 msgr2=0x7f21b40fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b40fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 -- 172.21.15.153:0/1835812894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4005040 con 0x7f21b40fde10 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b1d9b700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b259c700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21a4004890 tx=0x7f21a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a400e040 con 0x7f21b40fde10 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b40fcb30 con 0x7f21b40fde10 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21a4009af0 con 0x7f21b40fde10 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21a401c630 con 0x7f21b40fde10 2024-04-05T03:20:47.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.353+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b411a230 con 0x7f21b40fde10 2024-04-05T03:20:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.354+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b40666f0 con 0x7f21b40fde10 2024-04-05T03:20:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.355+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f21a400bbc0 con 0x7f21b40fde10 2024-04-05T03:20:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.355+0000 7f21a2ffd700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 0x7f219c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.355+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f21a408e860 con 0x7f21b40fde10 2024-04-05T03:20:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.355+0000 7f21b1d9b700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 0x7f219c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.356+0000 7f21b1d9b700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 0x7f219c070150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f21b40fd610 tx=0x7f21a8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.360+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f21a4057e60 con 0x7f21b40fde10 2024-04-05T03:20:47.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.579+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21b40fd3c0 con 0x7f219c06dca0 2024-04-05T03:20:47.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.581+0000 7f21a2ffd700 1 -- 172.21.15.153:0/1835812894 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f21b40fd3c0 con 0x7f219c06dca0 2024-04-05T03:20:47.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 msgr2=0x7f219c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 0x7f219c070150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f21b40fd610 tx=0x7f21a8009040 comp rx=0 tx=0).stop 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 msgr2=0x7f21b40fbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21a4004890 tx=0x7f21a4009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 shutdown_connections 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f219c06dca0 0x7f219c070150 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21b40fde10 0x7f21b40fbcd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 --2- 172.21.15.153:0/1835812894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b40fe7b0 0x7f21b40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 >> 172.21.15.153:0/1835812894 conn(0x7f21b40f78d0 msgr2=0x7f21b40f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 shutdown_connections 2024-04-05T03:20:47.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.583+0000 7f21b8c2d700 1 -- 172.21.15.153:0/1835812894 wait complete. 2024-04-05T03:20:47.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 -- 172.21.15.153:0/3617546588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 msgr2=0x7f64941020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 --2- 172.21.15.153:0/3617546588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f64941020f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6484004940 tx=0x7f6484009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 -- 172.21.15.153:0/3617546588 shutdown_connections 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 --2- 172.21.15.153:0/3617546588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 0x7f64940fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 --2- 172.21.15.153:0/3617546588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f64941020f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.686+0000 7f649a551700 1 -- 172.21.15.153:0/3617546588 >> 172.21.15.153:0/3617546588 conn(0x7f64940f7870 msgr2=0x7f64940f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 -- 172.21.15.153:0/3617546588 shutdown_connections 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 -- 172.21.15.153:0/3617546588 wait complete. 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 Processor -- start 2024-04-05T03:20:47.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 -- start start 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 0x7f649410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6494067120 con 0x7f6494101d20 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.687+0000 7f649a551700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649410d3d0 con 0x7f64941026c0 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55312/0 (socket says 172.21.15.153:55312) 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 -- 172.21.15.153:0/1294521648 learned_addr learned my addr 172.21.15.153:0/1294521648 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f64937fe700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 0x7f649410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 -- 172.21.15.153:0/1294521648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 msgr2=0x7f649410ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 0x7f649410ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 -- 172.21.15.153:0/1294521648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484005040 con 0x7f6494101d20 2024-04-05T03:20:47.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6493fff700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f648400b3b0 tx=0x7f6484009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f648400e040 con 0x7f6494101d20 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f648400bbc0 con 0x7f6494101d20 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649410d650 con 0x7f6494101d20 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f648401c3f0 con 0x7f6494101d20 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.688+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649411a180 con 0x7f6494101d20 2024-04-05T03:20:47.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.689+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64940fd470 con 0x7f6494101d20 2024-04-05T03:20:47.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.690+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f648400bd30 con 0x7f6494101d20 2024-04-05T03:20:47.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.690+0000 7f6490ff9700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 0x7f647c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:47.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.690+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f648408e730 con 0x7f6494101d20 2024-04-05T03:20:47.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.691+0000 7f64937fe700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 0x7f647c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:47.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.691+0000 7f64937fe700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 0x7f647c070150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6488007fd0 tx=0x7f6488009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:47.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.695+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f648405b860 con 0x7f6494101d20 2024-04-05T03:20:47.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.908+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f64940668a0 con 0x7f647c06dca0 2024-04-05T03:20:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.919+0000 7f6490ff9700 1 -- 172.21.15.153:0/1294521648 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f64940668a0 con 0x7f647c06dca0 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (15m) 8m ago 26m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (12m) 8m ago 27m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (12m) 8m ago 25m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (15m) 8m ago 26m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (8m) 8m ago 20m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (9m) 8m ago 21m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (8m) 8m ago 20m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (8m) 8m ago 20m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (16m) 8m ago 28m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (16m) 8m ago 25m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (14m) 8m ago 28m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (13m) 8m ago 25m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:20:47.918 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (15m) 8m ago 26m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (15m) 8m ago 25m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (12m) 8m ago 24m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (11m) 8m ago 24m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (11m) 8m ago 23m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (10m) 8m ago 23m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (10m) 8m ago 22m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (9m) 8m ago 22m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:20:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (15m) 8m ago 24m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 msgr2=0x7f647c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 0x7f647c070150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6488007fd0 tx=0x7f6488009040 comp rx=0 tx=0).stop 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 msgr2=0x7f649410c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f648400b3b0 tx=0x7f6484009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 shutdown_connections 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f647c06dca0 0x7f647c070150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6494101d20 0x7f649410c950 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 --2- 172.21.15.153:0/1294521648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64941026c0 0x7f649410ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 >> 172.21.15.153:0/1294521648 conn(0x7f64940f7870 msgr2=0x7f64940f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 shutdown_connections 2024-04-05T03:20:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:47.921+0000 7f649a551700 1 -- 172.21.15.153:0/1294521648 wait complete. 2024-04-05T03:20:48.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.023+0000 7f367037a700 1 -- 172.21.15.153:0/2186296515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 msgr2=0x7f36680fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.023+0000 7f367037a700 1 --2- 172.21.15.153:0/2186296515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f36680fe260 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3664004940 tx=0x7f3664009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- 172.21.15.153:0/2186296515 shutdown_connections 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 --2- 172.21.15.153:0/2186296515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 0x7f3668106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 --2- 172.21.15.153:0/2186296515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f36680fe260 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- 172.21.15.153:0/2186296515 >> 172.21.15.153:0/2186296515 conn(0x7f3668072bc0 msgr2=0x7f3668072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- 172.21.15.153:0/2186296515 shutdown_connections 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- 172.21.15.153:0/2186296515 wait complete. 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 Processor -- start 2024-04-05T03:20:48.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- start start 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 0x7f36681140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3668114730 con 0x7f36680fde90 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f367037a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366810db40 con 0x7f36680fe830 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f366e116700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f366e116700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55338/0 (socket says 172.21.15.153:55338) 2024-04-05T03:20:48.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.024+0000 7f366e116700 1 -- 172.21.15.153:0/780905230 learned_addr learned my addr 172.21.15.153:0/780905230 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f366d915700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 0x7f36681140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f366e116700 1 -- 172.21.15.153:0/780905230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 msgr2=0x7f36681140e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f366e116700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 0x7f36681140e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f366e116700 1 -- 172.21.15.153:0/780905230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664005040 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f366e116700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f366400b3b0 tx=0x7f3664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366400e040 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3664013420 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366401c5f0 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366810ddc0 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.025+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366810e2b0 con 0x7f36680fde90 2024-04-05T03:20:48.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.026+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3668065e30 con 0x7f36680fde90 2024-04-05T03:20:48.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.027+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3664013590 con 0x7f36680fde90 2024-04-05T03:20:48.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.027+0000 7f365effd700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 0x7f3654070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.027+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f366408e720 con 0x7f36680fde90 2024-04-05T03:20:48.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.027+0000 7f366d915700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 0x7f3654070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.028+0000 7f366d915700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 0x7f3654070150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f366810f440 tx=0x7f36580043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.032+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3664057d20 con 0x7f36680fde90 2024-04-05T03:20:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.310+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3668109fc0 con 0x7f36680fde90 2024-04-05T03:20:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.311+0000 7f365effd700 1 -- 172.21.15.153:0/780905230 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f366405b3e0 con 0x7f36680fde90 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:48.310 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:20:48.311 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 msgr2=0x7f3654070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 0x7f3654070150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f366810f440 tx=0x7f36580043c0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 msgr2=0x7f3668113ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f366400b3b0 tx=0x7f3664009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 shutdown_connections 2024-04-05T03:20:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f365406dca0 0x7f3654070150 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36680fde90 0x7f3668113ba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 --2- 172.21.15.153:0/780905230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36680fe830 0x7f36681140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 >> 172.21.15.153:0/780905230 conn(0x7f3668072bc0 msgr2=0x7f3668103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 shutdown_connections 2024-04-05T03:20:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.313+0000 7f367037a700 1 -- 172.21.15.153:0/780905230 wait complete. 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.414+0000 7f1a5c924700 1 -- 172.21.15.153:0/1854389429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 msgr2=0x7f1a580fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.414+0000 7f1a5c924700 1 --2- 172.21.15.153:0/1854389429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a580fbd20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004940 tx=0x7f1a4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- 172.21.15.153:0/1854389429 shutdown_connections 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 --2- 172.21.15.153:0/1854389429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a580fbd20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 --2- 172.21.15.153:0/1854389429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 0x7f1a581020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- 172.21.15.153:0/1854389429 >> 172.21.15.153:0/1854389429 conn(0x7f1a580f7880 msgr2=0x7f1a580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- 172.21.15.153:0/1854389429 shutdown_connections 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- 172.21.15.153:0/1854389429 wait complete. 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 Processor -- start 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- start start 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 0x7f1a58113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a58114a00 con 0x7f1a581026b0 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a5c924700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5810ddd0 con 0x7f1a58101d10 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.415+0000 7f1a55d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55348/0 (socket says 172.21.15.153:55348) 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 -- 172.21.15.153:0/3385203477 learned_addr learned my addr 172.21.15.153:0/3385203477 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a5659c700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 0x7f1a58113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 -- 172.21.15.153:0/3385203477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 msgr2=0x7f1a58113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 0x7f1a58113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 -- 172.21.15.153:0/3385203477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c005040 con 0x7f1a581026b0 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a55d9b700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009180 tx=0x7f1a4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4c00e040 con 0x7f1a581026b0 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a4c00bbc0 con 0x7f1a581026b0 2024-04-05T03:20:48.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5810e050 con 0x7f1a581026b0 2024-04-05T03:20:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4c01c3f0 con 0x7f1a581026b0 2024-04-05T03:20:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.416+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5810e610 con 0x7f1a581026b0 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.417+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a580fd460 con 0x7f1a581026b0 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.418+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1a4c00bd30 con 0x7f1a581026b0 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.418+0000 7f1a46ffd700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 0x7f1a40070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.418+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1a4c08e7f0 con 0x7f1a581026b0 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.418+0000 7f1a5659c700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 0x7f1a40070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.419+0000 7f1a5659c700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 0x7f1a40070150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a48009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.423+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1a4c05b9c0 con 0x7f1a581026b0 2024-04-05T03:20:48.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.663+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1a5804bed0 con 0x7f1a581026b0 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.663+0000 7f1a46ffd700 1 -- 172.21.15.153:0/3385203477 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f1a4c05b550 con 0x7f1a581026b0 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:20:48.662 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:20:48.663 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:20:48.664 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:20:48.665 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:20:48.665 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:20:48.665 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:20:48.666 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:20:48.666 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:20:48.666 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:20:48.666 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:20:48.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:48.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:48.668 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:20:48.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 msgr2=0x7f1a40070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 0x7f1a40070150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a48009f60 comp rx=0 tx=0).stop 2024-04-05T03:20:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 msgr2=0x7f1a581143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009180 tx=0x7f1a4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 shutdown_connections 2024-04-05T03:20:48.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4006dca0 0x7f1a40070150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a58101d10 0x7f1a58113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 --2- 172.21.15.153:0/3385203477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a581026b0 0x7f1a581143b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 >> 172.21.15.153:0/3385203477 conn(0x7f1a580f7880 msgr2=0x7f1a580f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 shutdown_connections 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.665+0000 7f1a5c924700 1 -- 172.21.15.153:0/3385203477 wait complete. 2024-04-05T03:20:48.672 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:20:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 -- 172.21.15.153:0/1920857826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24102710 msgr2=0x7fba240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 --2- 172.21.15.153:0/1920857826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24102710 0x7fba240fbd80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fba14004940 tx=0x7fba14009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 -- 172.21.15.153:0/1920857826 shutdown_connections 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 --2- 172.21.15.153:0/1920857826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24102710 0x7fba240fbd80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 --2- 172.21.15.153:0/1920857826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24101d70 0x7fba24102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 -- 172.21.15.153:0/1920857826 >> 172.21.15.153:0/1920857826 conn(0x7fba24072bc0 msgr2=0x7fba24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 -- 172.21.15.153:0/1920857826 shutdown_connections 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 -- 172.21.15.153:0/1920857826 wait complete. 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.768+0000 7fba2b493700 1 Processor -- start 2024-04-05T03:20:48.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2b493700 1 -- start start 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2b493700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2b493700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24102710 0x7fba24112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2b493700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba241136b0 con 0x7fba24101d70 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2b493700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2410ca10 con 0x7fba24102710 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36664/0 (socket says 172.21.15.153:36664) 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 -- 172.21.15.153:0/4282877893 learned_addr learned my addr 172.21.15.153:0/4282877893 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba28a2e700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24102710 0x7fba24112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 -- 172.21.15.153:0/4282877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24102710 msgr2=0x7fba24112fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24102710 0x7fba24112fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 -- 172.21.15.153:0/4282877893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14005040 con 0x7fba24101d70 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.769+0000 7fba2922f700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fba2000b790 tx=0x7fba20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.770+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba2000f6f0 con 0x7fba24101d70 2024-04-05T03:20:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.770+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2410ccf0 con 0x7fba24101d70 2024-04-05T03:20:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.770+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba2000fd20 con 0x7fba24101d70 2024-04-05T03:20:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.770+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba200103a0 con 0x7fba24101d70 2024-04-05T03:20:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.770+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2410d280 con 0x7fba24101d70 2024-04-05T03:20:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.771+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba240ff080 con 0x7fba24101d70 2024-04-05T03:20:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.771+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fba20011400 con 0x7fba24101d70 2024-04-05T03:20:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.771+0000 7fba19ffb700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 0x7fba10070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.772+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fba2008d380 con 0x7fba24101d70 2024-04-05T03:20:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.772+0000 7fba28a2e700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 0x7fba10070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.772+0000 7fba28a2e700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 0x7fba10070150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fba2410dcf0 tx=0x7fba14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.776+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba2005a4b0 con 0x7fba24101d70 2024-04-05T03:20:48.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.990+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba2410e150 con 0x7fba1006dca0 2024-04-05T03:20:48.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:48 smithi153 ceph-mon[106102]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:48.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:48 smithi153 ceph-mon[106102]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:48.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:48 smithi153 ceph-mon[106102]: pgmap v454: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:48.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:48 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/780905230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:20:48.989 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:48 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3385203477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.991+0000 7fba19ffb700 1 -- 172.21.15.153:0/4282877893 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fba2410e150 con 0x7fba1006dca0 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:20:48.990 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:20:48.991 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:20:48.991 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:20:48.991 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:20:48.991 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:20:48.991 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:20:48.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 msgr2=0x7fba10070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 0x7fba10070150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fba2410dcf0 tx=0x7fba14009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:48.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 msgr2=0x7fba24112a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:48.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fba2000b790 tx=0x7fba20009250 comp rx=0 tx=0).stop 2024-04-05T03:20:48.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 shutdown_connections 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fba1006dca0 0x7fba10070150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba24101d70 0x7fba24112a90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 --2- 172.21.15.153:0/4282877893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba24102710 0x7fba24112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.993+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 >> 172.21.15.153:0/4282877893 conn(0x7fba24072bc0 msgr2=0x7fba240faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.994+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 shutdown_connections 2024-04-05T03:20:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:48.994+0000 7fba2b493700 1 -- 172.21.15.153:0/4282877893 wait complete. 2024-04-05T03:20:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:48 smithi203 ceph-mon[95988]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:48 smithi203 ceph-mon[95988]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:48 smithi203 ceph-mon[95988]: pgmap v454: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:48 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/780905230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:20:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:48 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3385203477' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:20:49.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 -- 172.21.15.153:0/445975464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 msgr2=0x7f28681020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 --2- 172.21.15.153:0/445975464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f28681020f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2858004940 tx=0x7f2858009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 -- 172.21.15.153:0/445975464 shutdown_connections 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 --2- 172.21.15.153:0/445975464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f28680fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 --2- 172.21.15.153:0/445975464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f28681020f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 -- 172.21.15.153:0/445975464 >> 172.21.15.153:0/445975464 conn(0x7f28680f7870 msgr2=0x7f28680f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 -- 172.21.15.153:0/445975464 shutdown_connections 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.094+0000 7f2870414700 1 -- 172.21.15.153:0/445975464 wait complete. 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 Processor -- start 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 -- start start 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f2868113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28681145f0 con 0x7f2868101d20 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f2870414700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286810dbd0 con 0x7f28681026c0 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:49.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36680/0 (socket says 172.21.15.153:36680) 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 -- 172.21.15.153:0/1811092904 learned_addr learned my addr 172.21.15.153:0/1811092904 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 -- 172.21.15.153:0/1811092904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 msgr2=0x7f2868113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286d9af700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f2868113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f2868113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 -- 172.21.15.153:0/1811092904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858005040 con 0x7f2868101d20 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286d9af700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f2868113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.095+0000 7f286e1b0700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f285800b3b0 tx=0x7f2858009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.096+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285800e040 con 0x7f2868101d20 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.096+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f285800bbc0 con 0x7f2868101d20 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.096+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286810de50 con 0x7f2868101d20 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.096+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f285801c3f0 con 0x7f2868101d20 2024-04-05T03:20:49.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.096+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286810e390 con 0x7f2868101d20 2024-04-05T03:20:49.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.097+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28680fd470 con 0x7f2868101d20 2024-04-05T03:20:49.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.097+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f285800bd30 con 0x7f2868101d20 2024-04-05T03:20:49.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.097+0000 7f285effd700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 0x7f2854070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:20:49.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.098+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f285808e730 con 0x7f2868101d20 2024-04-05T03:20:49.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.098+0000 7f286d9af700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 0x7f2854070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:20:49.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.098+0000 7f286d9af700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 0x7f2854070150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2864007fd0 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:20:49.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.102+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f285805b860 con 0x7f2868101d20 2024-04-05T03:20:49.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.377+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f286810f0b0 con 0x7f2868101d20 2024-04-05T03:20:49.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.378+0000 7f285effd700 1 -- 172.21.15.153:0/1811092904 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f285805b3f0 con 0x7f2868101d20 2024-04-05T03:20:49.376 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:20:49.377 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:20:49.377 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:20:49.377 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:20:49.377 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:20:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 msgr2=0x7f2854070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 0x7f2854070150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2864007fd0 tx=0x7f2864009040 comp rx=0 tx=0).stop 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 msgr2=0x7f2868113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f285800b3b0 tx=0x7f2858009e30 comp rx=0 tx=0).stop 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 shutdown_connections 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f285406dca0 0x7f2854070150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2868101d20 0x7f2868113a60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 --2- 172.21.15.153:0/1811092904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28681026c0 0x7f2868113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 >> 172.21.15.153:0/1811092904 conn(0x7f28680f7870 msgr2=0x7f28680f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 shutdown_connections 2024-04-05T03:20:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:20:49.380+0000 7f2870414700 1 -- 172.21.15.153:0/1811092904 wait complete. 2024-04-05T03:20:50.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:49 smithi203 ceph-mon[95988]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:50.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:49 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1811092904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:20:50.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:49 smithi153 ceph-mon[106102]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:50.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:49 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1811092904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:20:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:50 smithi203 ceph-mon[95988]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:50 smithi203 ceph-mon[95988]: pgmap v455: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:50 smithi153 ceph-mon[106102]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:20:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:50 smithi153 ceph-mon[106102]: pgmap v455: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:52.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:51 smithi203 ceph-mon[95988]: pgmap v456: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:52.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:51 smithi153 ceph-mon[106102]: pgmap v456: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:54 smithi203 ceph-mon[95988]: pgmap v457: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:54 smithi153 ceph-mon[106102]: pgmap v457: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:56 smithi203 ceph-mon[95988]: pgmap v458: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:56 smithi153 ceph-mon[106102]: pgmap v458: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:20:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:20:58 smithi203 ceph-mon[95988]: pgmap v459: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:20:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:20:58 smithi153 ceph-mon[106102]: pgmap v459: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:00 smithi203 ceph-mon[95988]: pgmap v460: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:01.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:00 smithi153 ceph-mon[106102]: pgmap v460: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:02 smithi203 ceph-mon[95988]: pgmap v461: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:02 smithi153 ceph-mon[106102]: pgmap v461: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:04 smithi203 ceph-mon[95988]: pgmap v462: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:04 smithi153 ceph-mon[106102]: pgmap v462: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:06 smithi203 ceph-mon[95988]: pgmap v463: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:06 smithi153 ceph-mon[106102]: pgmap v463: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:08 smithi203 ceph-mon[95988]: pgmap v464: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:08 smithi153 ceph-mon[106102]: pgmap v464: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:10 smithi203 ceph-mon[95988]: pgmap v465: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:10 smithi153 ceph-mon[106102]: pgmap v465: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:12 smithi203 ceph-mon[95988]: pgmap v466: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:12 smithi153 ceph-mon[106102]: pgmap v466: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:14 smithi203 ceph-mon[95988]: pgmap v467: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:14 smithi153 ceph-mon[106102]: pgmap v467: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:16 smithi203 ceph-mon[95988]: pgmap v468: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:16 smithi153 ceph-mon[106102]: pgmap v468: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:18 smithi203 ceph-mon[95988]: pgmap v469: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:18 smithi153 ceph-mon[106102]: pgmap v469: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 -- 172.21.15.153:0/1268996936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f81026c0 msgr2=0x7f61f80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 --2- 172.21.15.153:0/1268996936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f81026c0 0x7f61f80fbd30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004940 tx=0x7f61ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 -- 172.21.15.153:0/1268996936 shutdown_connections 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 --2- 172.21.15.153:0/1268996936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f81026c0 0x7f61f80fbd30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 --2- 172.21.15.153:0/1268996936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8101d20 0x7f61f81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 -- 172.21.15.153:0/1268996936 >> 172.21.15.153:0/1268996936 conn(0x7f61f80f78f0 msgr2=0x7f61f80f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:19.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 -- 172.21.15.153:0/1268996936 shutdown_connections 2024-04-05T03:21:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.496+0000 7f61fda1c700 1 -- 172.21.15.153:0/1268996936 wait complete. 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 Processor -- start 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 -- start start 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f81026c0 0x7f61f8113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8114590 con 0x7f61f8101d20 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61fda1c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f810db00 con 0x7f61f81026c0 2024-04-05T03:21:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61f6ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60472/0 (socket says 172.21.15.153:60472) 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.497+0000 7f61f6ffd700 1 -- 172.21.15.153:0/2533127292 learned_addr learned my addr 172.21.15.153:0/2533127292 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61f67fc700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f81026c0 0x7f61f8113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61f6ffd700 1 -- 172.21.15.153:0/2533127292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f81026c0 msgr2=0x7f61f8113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61f6ffd700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f81026c0 0x7f61f8113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61f6ffd700 1 -- 172.21.15.153:0/2533127292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec005040 con 0x7f61f8101d20 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61f6ffd700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61f000bbb0 tx=0x7f61f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f00092e0 con 0x7f61f8101d20 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61f000cb20 con 0x7f61f8101d20 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f810dde0 con 0x7f61f8101d20 2024-04-05T03:21:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f0015d70 con 0x7f61f8101d20 2024-04-05T03:21:19.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.498+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f810e370 con 0x7f61f8101d20 2024-04-05T03:21:19.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.499+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f80fd470 con 0x7f61f8101d20 2024-04-05T03:21:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.500+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f61f0012070 con 0x7f61f8101d20 2024-04-05T03:21:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.500+0000 7f61df7fe700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 0x7f61e0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.500+0000 7f61f67fc700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 0x7f61e0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.500+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f61f008d2c0 con 0x7f61f8101d20 2024-04-05T03:21:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.501+0000 7f61f67fc700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 0x7f61e0070150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004890 tx=0x7f61ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.505+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f61f005a370 con 0x7f61f8101d20 2024-04-05T03:21:19.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.723+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f61f8108ee0 con 0x7f61e006dca0 2024-04-05T03:21:19.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.724+0000 7f61df7fe700 1 -- 172.21.15.153:0/2533127292 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f61f8108ee0 con 0x7f61e006dca0 2024-04-05T03:21:19.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 msgr2=0x7f61e0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 0x7f61e0070150 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004890 tx=0x7f61ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 msgr2=0x7f61f8113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f61f000bbb0 tx=0x7f61f0009840 comp rx=0 tx=0).stop 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 shutdown_connections 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e006dca0 0x7f61e0070150 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61f8101d20 0x7f61f8113970 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 --2- 172.21.15.153:0/2533127292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f81026c0 0x7f61f8113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 >> 172.21.15.153:0/2533127292 conn(0x7f61f80f78f0 msgr2=0x7f61f8066290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 shutdown_connections 2024-04-05T03:21:19.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.727+0000 7f61fda1c700 1 -- 172.21.15.153:0/2533127292 wait complete. 2024-04-05T03:21:19.736 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 -- 172.21.15.153:0/3394388840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 msgr2=0x7f95400fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 --2- 172.21.15.153:0/3394388840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f95400fbd80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9528004940 tx=0x7f9528009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 -- 172.21.15.153:0/3394388840 shutdown_connections 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 --2- 172.21.15.153:0/3394388840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f95400fbd80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 --2- 172.21.15.153:0/3394388840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 0x7f9540102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 -- 172.21.15.153:0/3394388840 >> 172.21.15.153:0/3394388840 conn(0x7f9540072bc0 msgr2=0x7f9540072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 -- 172.21.15.153:0/3394388840 shutdown_connections 2024-04-05T03:21:19.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 -- 172.21.15.153:0/3394388840 wait complete. 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.830+0000 7f954604f700 1 Processor -- start 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f954604f700 1 -- start start 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f954604f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 0x7f9540112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f954604f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f954604f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9540113630 con 0x7f9540102710 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f954604f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954010ca20 con 0x7f9540101d70 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60490/0 (socket says 172.21.15.153:60490) 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 -- 172.21.15.153:0/173954351 learned_addr learned my addr 172.21.15.153:0/173954351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f953f7fe700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 0x7f9540112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 -- 172.21.15.153:0/173954351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 msgr2=0x7f9540112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 0x7f9540112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 -- 172.21.15.153:0/173954351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528005040 con 0x7f9540102710 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f9536ffd700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9528009100 tx=0x7f9528009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:19.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f952800e040 con 0x7f9540102710 2024-04-05T03:21:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.831+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9528009af0 con 0x7f9540102710 2024-04-05T03:21:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.832+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954010cca0 con 0x7f9540102710 2024-04-05T03:21:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.832+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f952801c3f0 con 0x7f9540102710 2024-04-05T03:21:19.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.832+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954010d260 con 0x7f9540102710 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.832+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540104240 con 0x7f9540102710 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.833+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f952800baf0 con 0x7f9540102710 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.833+0000 7f953cff9700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 0x7f952c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.833+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f952808e7c0 con 0x7f9540102710 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.833+0000 7f953f7fe700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 0x7f952c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:19.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.834+0000 7f953f7fe700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 0x7f952c070150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9530004eb0 tx=0x7f953000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:19.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:19.838+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f952805b8f0 con 0x7f9540102710 2024-04-05T03:21:20.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.051+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f954010e0f0 con 0x7f952c06dca0 2024-04-05T03:21:20.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.052+0000 7f953cff9700 1 -- 172.21.15.153:0/173954351 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f954010e0f0 con 0x7f952c06dca0 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 msgr2=0x7f952c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 0x7f952c070150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9530004eb0 tx=0x7f953000b040 comp rx=0 tx=0).stop 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 msgr2=0x7f9540112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9528009100 tx=0x7f9528009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 shutdown_connections 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f952c06dca0 0x7f952c070150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9540101d70 0x7f9540112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 --2- 172.21.15.153:0/173954351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9540102710 0x7f9540112fe0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 >> 172.21.15.153:0/173954351 conn(0x7f9540072bc0 msgr2=0x7f95400fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 shutdown_connections 2024-04-05T03:21:20.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.055+0000 7f954604f700 1 -- 172.21.15.153:0/173954351 wait complete. 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 -- 172.21.15.153:0/2459367779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 msgr2=0x7fbdc80fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 --2- 172.21.15.153:0/2459367779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc80fbd40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8004940 tx=0x7fbdb8009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 -- 172.21.15.153:0/2459367779 shutdown_connections 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 --2- 172.21.15.153:0/2459367779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc80fbd40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 --2- 172.21.15.153:0/2459367779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 0x7fbdc8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 -- 172.21.15.153:0/2459367779 >> 172.21.15.153:0/2459367779 conn(0x7fbdc80f78c0 msgr2=0x7fbdc80f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 -- 172.21.15.153:0/2459367779 shutdown_connections 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.156+0000 7fbdd055f700 1 -- 172.21.15.153:0/2459367779 wait complete. 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 Processor -- start 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 -- start start 2024-04-05T03:21:20.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 0x7fbdc810eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc810f580 con 0x7fbdc81026d0 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdd055f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc810f6c0 con 0x7fbdc8101d30 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdcdafa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdcdafa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60508/0 (socket says 172.21.15.153:60508) 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.157+0000 7fbdcdafa700 1 -- 172.21.15.153:0/1155115810 learned_addr learned my addr 172.21.15.153:0/1155115810 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdce2fb700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 0x7fbdc810eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdcdafa700 1 -- 172.21.15.153:0/1155115810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 msgr2=0x7fbdc810eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdcdafa700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 0x7fbdc810eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdcdafa700 1 -- 172.21.15.153:0/1155115810 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8005040 con 0x7fbdc81026d0 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdcdafa700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009100 tx=0x7fbdb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb800e040 con 0x7fbdc81026d0 2024-04-05T03:21:20.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc810f940 con 0x7fbdc81026d0 2024-04-05T03:21:20.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.158+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdb8009af0 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.159+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdb801c3f0 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.159+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8117f10 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.160+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbdb800baf0 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.160+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc80fd480 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.160+0000 7fbdbeffd700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 0x7fbdb4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.160+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbdb808e4d0 con 0x7fbdc81026d0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.160+0000 7fbdce2fb700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 0x7fbdb4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.161+0000 7fbdce2fb700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 0x7fbdb4070150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4004eb0 tx=0x7fbdc400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.166+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbdb8057ad0 con 0x7fbdc81026d0 2024-04-05T03:21:20.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.370+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdc8108e60 con 0x7fbdb406dca0 2024-04-05T03:21:20.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.380+0000 7fbdbeffd700 1 -- 172.21.15.153:0/1155115810 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fbdc8108e60 con 0x7fbdb406dca0 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (15m) 9m ago 27m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (13m) 9m ago 28m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (13m) 8m ago 25m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (15m) 9m ago 26m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (9m) 9m ago 21m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (9m) 9m ago 21m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (8m) 8m ago 21m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (9m) 8m ago 21m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (17m) 9m ago 29m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (16m) 8m ago 25m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (14m) 9m ago 29m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (14m) 8m ago 25m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:21:20.379 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (16m) 9m ago 27m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (16m) 8m ago 25m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (13m) 9m ago 25m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (12m) 9m ago 24m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (11m) 9m ago 24m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (11m) 8m ago 23m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (10m) 8m ago 23m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (10m) 8m ago 22m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:21:20.380 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (16m) 8m ago 25m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 msgr2=0x7fbdb4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 0x7fbdb4070150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4004eb0 tx=0x7fbdc400b040 comp rx=0 tx=0).stop 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 msgr2=0x7fbdc810f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009100 tx=0x7fbdb8009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 shutdown_connections 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbdb406dca0 0x7fbdb4070150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdc8101d30 0x7fbdc810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 --2- 172.21.15.153:0/1155115810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdc81026d0 0x7fbdc810f040 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 >> 172.21.15.153:0/1155115810 conn(0x7fbdc80f78c0 msgr2=0x7fbdc80f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 shutdown_connections 2024-04-05T03:21:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.382+0000 7fbdd055f700 1 -- 172.21.15.153:0/1155115810 wait complete. 2024-04-05T03:21:20.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 -- 172.21.15.153:0/2054296351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 msgr2=0x7f9a900fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/2054296351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a900fbd80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9a84004940 tx=0x7f9a84009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 -- 172.21.15.153:0/2054296351 shutdown_connections 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/2054296351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a900fbd80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/2054296351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 0x7f9a90102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 -- 172.21.15.153:0/2054296351 >> 172.21.15.153:0/2054296351 conn(0x7f9a90072bc0 msgr2=0x7f9a90072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 -- 172.21.15.153:0/2054296351 shutdown_connections 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.483+0000 7f9a97ef1700 1 -- 172.21.15.153:0/2054296351 wait complete. 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 Processor -- start 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 -- start start 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 0x7f9a90113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a901146f0 con 0x7f9a90102710 2024-04-05T03:21:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a97ef1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9010dae0 con 0x7f9a90101d70 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60522/0 (socket says 172.21.15.153:60522) 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 -- 172.21.15.153:0/1391148793 learned_addr learned my addr 172.21.15.153:0/1391148793 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a95c8d700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 0x7f9a90113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 -- 172.21.15.153:0/1391148793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 msgr2=0x7f9a90113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 0x7f9a90113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 -- 172.21.15.153:0/1391148793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84005040 con 0x7f9a90102710 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.484+0000 7f9a9548c700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009180 tx=0x7f9a84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.485+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a8400e040 con 0x7f9a90102710 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.485+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a84013420 con 0x7f9a90102710 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.485+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a8401c610 con 0x7f9a90102710 2024-04-05T03:21:20.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.485+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9010dd60 con 0x7f9a90102710 2024-04-05T03:21:20.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.485+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9010e320 con 0x7f9a90102710 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.486+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a900fd4c0 con 0x7f9a90102710 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.486+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9a84013a00 con 0x7f9a90102710 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.487+0000 7f9a8e7fc700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 0x7f9a78070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.487+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9a8408e5d0 con 0x7f9a90102710 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.487+0000 7f9a95c8d700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 0x7f9a78070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.487+0000 7f9a95c8d700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 0x7f9a78070150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a80004fd0 tx=0x7f9a80009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.491+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9a8405b7a0 con 0x7f9a90102710 2024-04-05T03:21:20.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.768+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9a90108e50 con 0x7f9a90102710 2024-04-05T03:21:20.767 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:20 smithi153 ceph-mon[106102]: pgmap v470: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:20.767 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:20 smithi153 ceph-mon[106102]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:20.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.768+0000 7f9a8e7fc700 1 -- 172.21.15.153:0/1391148793 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9a8405b330 con 0x7f9a90102710 2024-04-05T03:21:20.767 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:21:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:21:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:21:20.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:21:20.768 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:21:20.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 msgr2=0x7f9a78070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 0x7f9a78070150 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a80004fd0 tx=0x7f9a80009040 comp rx=0 tx=0).stop 2024-04-05T03:21:20.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 msgr2=0x7f9a901140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009180 tx=0x7f9a84009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 shutdown_connections 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9a7806dca0 0x7f9a78070150 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a90101d70 0x7f9a90113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 --2- 172.21.15.153:0/1391148793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a90102710 0x7f9a901140a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 >> 172.21.15.153:0/1391148793 conn(0x7f9a90072bc0 msgr2=0x7f9a900659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 shutdown_connections 2024-04-05T03:21:20.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.770+0000 7f9a97ef1700 1 -- 172.21.15.153:0/1391148793 wait complete. 2024-04-05T03:21:20.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 -- 172.21.15.153:0/1241352025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 msgr2=0x7f9acc1090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 --2- 172.21.15.153:0/1241352025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc1090f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9abc004970 tx=0x7f9abc009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 -- 172.21.15.153:0/1241352025 shutdown_connections 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 --2- 172.21.15.153:0/1241352025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc1090f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 --2- 172.21.15.153:0/1241352025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 0x7f9acc070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.871+0000 7f9ad1495700 1 -- 172.21.15.153:0/1241352025 >> 172.21.15.153:0/1241352025 conn(0x7f9acc0f8970 msgr2=0x7f9acc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 -- 172.21.15.153:0/1241352025 shutdown_connections 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 -- 172.21.15.153:0/1241352025 wait complete. 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 Processor -- start 2024-04-05T03:21:20.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 -- start start 2024-04-05T03:21:20.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 0x7f9acc115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc1168e0 con 0x7f9acc070c70 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.872+0000 7f9ad1495700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc10fcb0 con 0x7f9acc070360 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60536/0 (socket says 172.21.15.153:60536) 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 -- 172.21.15.153:0/2687637469 learned_addr learned my addr 172.21.15.153:0/2687637469 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9acaffd700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 0x7f9acc115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 -- 172.21.15.153:0/2687637469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 msgr2=0x7f9acc115d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 0x7f9acc115d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 -- 172.21.15.153:0/2687637469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc005040 con 0x7f9acc070c70 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9aca7fc700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9abc000c00 tx=0x7f9abc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9abc00e040 con 0x7f9acc070c70 2024-04-05T03:21:20.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9abc00bc10 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc10ff30 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9abc01c3f0 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.873+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc110470 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.874+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc071aa0 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.875+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9abc013420 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.875+0000 7f9ac37fe700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 0x7f9ab8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.875+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9abc08e760 con 0x7f9acc070c70 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.875+0000 7f9acaffd700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 0x7f9ab8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.876+0000 7f9acaffd700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 0x7f9ab8070150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004f40 tx=0x7f9ab400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:20.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:20.880+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9abc05b890 con 0x7f9acc070c70 2024-04-05T03:21:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:20 smithi203 ceph-mon[95988]: pgmap v470: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:21.070 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:20 smithi203 ceph-mon[95988]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:21.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.117+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9acc04bed0 con 0x7f9acc070c70 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.118+0000 7f9ac37fe700 1 -- 172.21.15.153:0/2687637469 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f9abc004de0 con 0x7f9acc070c70 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:21:21.117 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:21:21.118 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:21.119 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:21:21.120 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:21.120 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:21.120 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:21.120 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:21.120 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 msgr2=0x7f9ab8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 0x7f9ab8070150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4004f40 tx=0x7f9ab400b040 comp rx=0 tx=0).stop 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 msgr2=0x7f9acc116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9abc000c00 tx=0x7f9abc009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 shutdown_connections 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9ab806dca0 0x7f9ab8070150 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9acc070360 0x7f9acc115d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 --2- 172.21.15.153:0/2687637469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9acc070c70 0x7f9acc116290 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 >> 172.21.15.153:0/2687637469 conn(0x7f9acc0f8970 msgr2=0x7f9acc0f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 shutdown_connections 2024-04-05T03:21:21.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.121+0000 7f9ad1495700 1 -- 172.21.15.153:0/2687637469 wait complete. 2024-04-05T03:21:21.122 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- 172.21.15.153:0/3332567801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 msgr2=0x7f38041092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 --2- 172.21.15.153:0/3332567801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f38041092f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f37f4004940 tx=0x7f37f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- 172.21.15.153:0/3332567801 shutdown_connections 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 --2- 172.21.15.153:0/3332567801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f38041092f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 --2- 172.21.15.153:0/3332567801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f38040fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- 172.21.15.153:0/3332567801 >> 172.21.15.153:0/3332567801 conn(0x7f3804072bc0 msgr2=0x7f3804072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- 172.21.15.153:0/3332567801 shutdown_connections 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- 172.21.15.153:0/3332567801 wait complete. 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 Processor -- start 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.223+0000 7f380c3ca700 1 -- start start 2024-04-05T03:21:21.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f3804115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38041168c0 con 0x7f38040fc590 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380410fc90 con 0x7f38040fbc80 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60550/0 (socket says 172.21.15.153:60550) 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 -- 172.21.15.153:0/1454530496 learned_addr learned my addr 172.21.15.153:0/1454530496 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380a166700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f3804115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 -- 172.21.15.153:0/1454530496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 msgr2=0x7f3804115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f3804115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 -- 172.21.15.153:0/1454530496 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4005040 con 0x7f38040fc590 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380a166700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f3804115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f3809965700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b3b0 tx=0x7f37f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f400e040 con 0x7f38040fc590 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380410ff10 con 0x7f38040fc590 2024-04-05T03:21:21.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37f400bbc0 con 0x7f38040fc590 2024-04-05T03:21:21.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37f401c3f0 con 0x7f38040fc590 2024-04-05T03:21:21.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.224+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38041104d0 con 0x7f38040fc590 2024-04-05T03:21:21.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.225+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3804065e30 con 0x7f38040fc590 2024-04-05T03:21:21.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.226+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f37f400bd30 con 0x7f38040fc590 2024-04-05T03:21:21.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.226+0000 7f37faffd700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 0x7f37f0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.226+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f37f408e830 con 0x7f38040fc590 2024-04-05T03:21:21.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.226+0000 7f380a166700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 0x7f37f0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.227+0000 7f380a166700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 0x7f37f0070150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004ee0 tx=0x7f37fc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:21.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.231+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f37f405b8e0 con 0x7f38040fc590 2024-04-05T03:21:21.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.444+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f380410c160 con 0x7f37f006dca0 2024-04-05T03:21:21.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.446+0000 7f37faffd700 1 -- 172.21.15.153:0/1454530496 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f380410c160 con 0x7f37f006dca0 2024-04-05T03:21:21.445 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:21:21.445 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:21:21.445 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:21:21.445 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:21:21.446 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:21:21.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 msgr2=0x7f37f0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 0x7f37f0070150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f37fc004ee0 tx=0x7f37fc009040 comp rx=0 tx=0).stop 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 msgr2=0x7f3804116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f37f400b3b0 tx=0x7f37f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 shutdown_connections 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f37f006dca0 0x7f37f0070150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38040fbc80 0x7f3804115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 --2- 172.21.15.153:0/1454530496 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38040fc590 0x7f3804116270 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 >> 172.21.15.153:0/1454530496 conn(0x7f3804072bc0 msgr2=0x7f3804105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 shutdown_connections 2024-04-05T03:21:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.448+0000 7f380c3ca700 1 -- 172.21.15.153:0/1454530496 wait complete. 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.549+0000 7eff4aa16700 1 -- 172.21.15.153:0/1505140839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44102e00 msgr2=0x7eff441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.549+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1505140839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44102e00 0x7eff441031d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7eff34004940 tx=0x7eff34009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- 172.21.15.153:0/1505140839 shutdown_connections 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1505140839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff441037a0 0x7eff440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1505140839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44102e00 0x7eff441031d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- 172.21.15.153:0/1505140839 >> 172.21.15.153:0/1505140839 conn(0x7eff440f89b0 msgr2=0x7eff440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- 172.21.15.153:0/1505140839 shutdown_connections 2024-04-05T03:21:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- 172.21.15.153:0/1505140839 wait complete. 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 Processor -- start 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- start start 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 0x7eff44115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff44116780 con 0x7eff441037a0 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.550+0000 7eff4aa16700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4410fcd0 con 0x7eff44102e00 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60556/0 (socket says 172.21.15.153:60556) 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 -- 172.21.15.153:0/1103078912 learned_addr learned my addr 172.21.15.153:0/1103078912 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff43fff700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 0x7eff44115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 -- 172.21.15.153:0/1103078912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 msgr2=0x7eff44115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 0x7eff44115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 -- 172.21.15.153:0/1103078912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34005040 con 0x7eff441037a0 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff43fff700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 0x7eff44115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff437fe700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b790 tx=0x7eff38009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff3800f6f0 con 0x7eff441037a0 2024-04-05T03:21:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4410ff50 con 0x7eff441037a0 2024-04-05T03:21:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff3800fd20 con 0x7eff441037a0 2024-04-05T03:21:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff380103a0 con 0x7eff441037a0 2024-04-05T03:21:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.551+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff44110510 con 0x7eff441037a0 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.552+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff440fe550 con 0x7eff441037a0 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.553+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7eff38011400 con 0x7eff441037a0 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.553+0000 7eff40ff9700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 0x7eff2c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.553+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7eff3808d450 con 0x7eff441037a0 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.553+0000 7eff43fff700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 0x7eff2c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.554+0000 7eff43fff700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 0x7eff2c070150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7eff34004560 tx=0x7eff34009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.558+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7eff3805a500 con 0x7eff441037a0 2024-04-05T03:21:21.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.836+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7eff44109f70 con 0x7eff441037a0 2024-04-05T03:21:21.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.836+0000 7eff40ff9700 1 -- 172.21.15.153:0/1103078912 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7eff3805a090 con 0x7eff441037a0 2024-04-05T03:21:21.836 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:21:21.836 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:21:21.836 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:21:21.836 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:21:21.836 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 msgr2=0x7eff2c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 0x7eff2c070150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7eff34004560 tx=0x7eff34009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 msgr2=0x7eff441160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b790 tx=0x7eff38009220 comp rx=0 tx=0).stop 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 shutdown_connections 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7eff2c06dca0 0x7eff2c070150 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff44102e00 0x7eff44115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 --2- 172.21.15.153:0/1103078912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441037a0 0x7eff441160a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 >> 172.21.15.153:0/1103078912 conn(0x7eff440f89b0 msgr2=0x7eff44071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 shutdown_connections 2024-04-05T03:21:21.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:21.838+0000 7eff4aa16700 1 -- 172.21.15.153:0/1103078912 wait complete. 2024-04-05T03:21:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:21 smithi203 ceph-mon[95988]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:21 smithi203 ceph-mon[95988]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:21 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1391148793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:21:22.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:21 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2687637469' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:21:22.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:21 smithi153 ceph-mon[106102]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:22.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:21 smithi153 ceph-mon[106102]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:22.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:21 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1391148793' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:21:22.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:21 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2687637469' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:21:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:22 smithi203 ceph-mon[95988]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:22 smithi203 ceph-mon[95988]: pgmap v471: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1103078912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:21:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:22 smithi153 ceph-mon[106102]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:22 smithi153 ceph-mon[106102]: pgmap v471: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1103078912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:21:24.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:23 smithi203 ceph-mon[95988]: pgmap v472: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:24.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:23 smithi153 ceph-mon[106102]: pgmap v472: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:26 smithi203 ceph-mon[95988]: pgmap v473: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:26 smithi153 ceph-mon[106102]: pgmap v473: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:28 smithi203 ceph-mon[95988]: pgmap v474: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:28 smithi153 ceph-mon[106102]: pgmap v474: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:30 smithi203 ceph-mon[95988]: pgmap v475: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:30 smithi153 ceph-mon[106102]: pgmap v475: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:32 smithi203 ceph-mon[95988]: pgmap v476: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:32 smithi153 ceph-mon[106102]: pgmap v476: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:34 smithi203 ceph-mon[95988]: pgmap v477: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:34 smithi153 ceph-mon[106102]: pgmap v477: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:37.047 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:36 smithi203 ceph-mon[95988]: pgmap v478: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:37.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:36 smithi153 ceph-mon[106102]: pgmap v478: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:38 smithi203 ceph-mon[95988]: pgmap v479: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:38 smithi153 ceph-mon[106102]: pgmap v479: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:40 smithi203 ceph-mon[95988]: pgmap v480: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:40 smithi153 ceph-mon[106102]: pgmap v480: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:42 smithi203 ceph-mon[95988]: pgmap v481: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:42 smithi153 ceph-mon[106102]: pgmap v481: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:44 smithi203 ceph-mon[95988]: pgmap v482: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:44 smithi153 ceph-mon[106102]: pgmap v482: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:46 smithi203 ceph-mon[95988]: pgmap v483: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:46 smithi153 ceph-mon[106102]: pgmap v483: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:48.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:47 smithi203 ceph-mon[95988]: pgmap v484: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:48.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:47 smithi153 ceph-mon[106102]: pgmap v484: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:50 smithi203 ceph-mon[95988]: pgmap v485: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:50 smithi153 ceph-mon[106102]: pgmap v485: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:51.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/1112767150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fe7a0 msgr2=0x7fe3e4106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/1112767150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e4106de0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4004940 tx=0x7fe3d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/1112767150 shutdown_connections 2024-04-05T03:21:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/1112767150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e4106de0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/1112767150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fde00 0x7fe3e40fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:51.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/1112767150 >> 172.21.15.153:0/1112767150 conn(0x7fe3e40f78d0 msgr2=0x7fe3e40f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/1112767150 shutdown_connections 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.953+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/1112767150 wait complete. 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 Processor -- start 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 -- start start 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e41140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e4114700 con 0x7fe3e40fde00 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ec6c6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e410daf0 con 0x7fe3e40fe7a0 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ea462700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ea462700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44922/0 (socket says 172.21.15.153:44922) 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3ea462700 1 -- 172.21.15.153:0/221944529 learned_addr learned my addr 172.21.15.153:0/221944529 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.954+0000 7fe3e9c61700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e41140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ea462700 1 -- 172.21.15.153:0/221944529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fe7a0 msgr2=0x7fe3e41140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ea462700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e41140b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:51.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ea462700 1 -- 172.21.15.153:0/221944529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d4005040 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ea462700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c8c0 tx=0x7fe3e00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e000a980 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3e000ce60 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e410ddd0 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3e00115e0 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.955+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e410e310 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.956+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e4066770 con 0x7fe3e40fde00 2024-04-05T03:21:51.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.956+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe3e0012400 con 0x7fe3e40fde00 2024-04-05T03:21:51.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.957+0000 7fe3daffd700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 0x7fe3d0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:51.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.957+0000 7fe3e9c61700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 0x7fe3d0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:51.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.957+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe3e008d590 con 0x7fe3e40fde00 2024-04-05T03:21:51.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.958+0000 7fe3e9c61700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 0x7fe3d0070150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3e410ecb0 tx=0x7fe3d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:51.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:51.962+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe3e005a760 con 0x7fe3e40fde00 2024-04-05T03:21:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.175+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3e4109e90 con 0x7fe3d006dca0 2024-04-05T03:21:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.176+0000 7fe3daffd700 1 -- 172.21.15.153:0/221944529 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe3e4109e90 con 0x7fe3d006dca0 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 msgr2=0x7fe3d0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 0x7fe3d0070150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe3e410ecb0 tx=0x7fe3d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 msgr2=0x7fe3e4113b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c8c0 tx=0x7fe3e00044c0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 shutdown_connections 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe3d006dca0 0x7fe3d0070150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe3e40fde00 0x7fe3e4113b70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 --2- 172.21.15.153:0/221944529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3e40fe7a0 0x7fe3e41140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.178+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 >> 172.21.15.153:0/221944529 conn(0x7fe3e40f78d0 msgr2=0x7fe3e40f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.179+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 shutdown_connections 2024-04-05T03:21:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.179+0000 7fe3ec6c6700 1 -- 172.21.15.153:0/221944529 wait complete. 2024-04-05T03:21:52.187 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:21:52.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 -- 172.21.15.153:0/1918012539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 msgr2=0x7f8164100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 --2- 172.21.15.153:0/1918012539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f8164100b90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8154004940 tx=0x7f8154009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 -- 172.21.15.153:0/1918012539 shutdown_connections 2024-04-05T03:21:52.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 --2- 172.21.15.153:0/1918012539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f8164100b90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 --2- 172.21.15.153:0/1918012539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 0x7f81640fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.280+0000 7f816b33d700 1 -- 172.21.15.153:0/1918012539 >> 172.21.15.153:0/1918012539 conn(0x7f81640f7890 msgr2=0x7f81640f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 -- 172.21.15.153:0/1918012539 shutdown_connections 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 -- 172.21.15.153:0/1918012539 wait complete. 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 Processor -- start 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 -- start start 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 0x7f8164119fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816411ab50 con 0x7f81640fc570 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f816b33d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8164114180 con 0x7f81640fbbd0 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.281+0000 7f81688d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81690d9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 0x7f8164119fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44952/0 (socket says 172.21.15.153:44952) 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 -- 172.21.15.153:0/1416042137 learned_addr learned my addr 172.21.15.153:0/1416042137 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 -- 172.21.15.153:0/1416042137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 msgr2=0x7f8164119fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 0x7f8164119fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 -- 172.21.15.153:0/1416042137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8154005040 con 0x7f81640fc570 2024-04-05T03:21:52.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f81688d8700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f815400b8b0 tx=0x7f8154009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f815400e040 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8154004290 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f815401c3f0 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8164114400 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.282+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8164114940 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.283+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8164066bd0 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.284+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8154013420 con 0x7f81640fc570 2024-04-05T03:21:52.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.285+0000 7f8159ffb700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 0x7f8150070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.285+0000 7f81690d9700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 0x7f8150070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.285+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f815408e670 con 0x7f81640fc570 2024-04-05T03:21:52.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.285+0000 7f81690d9700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 0x7f8150070150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f815c007a40 tx=0x7f815c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.289+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f815405b840 con 0x7f81640fc570 2024-04-05T03:21:52.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.503+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8164110550 con 0x7f815006dca0 2024-04-05T03:21:52.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.504+0000 7f8159ffb700 1 -- 172.21.15.153:0/1416042137 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8164110550 con 0x7f815006dca0 2024-04-05T03:21:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 msgr2=0x7f8150070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 0x7f8150070150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f815c007a40 tx=0x7f815c009da0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 msgr2=0x7f816411a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f815400b8b0 tx=0x7f8154009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 shutdown_connections 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f815006dca0 0x7f8150070150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81640fbbd0 0x7f8164119fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 --2- 172.21.15.153:0/1416042137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81640fc570 0x7f816411a500 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 >> 172.21.15.153:0/1416042137 conn(0x7f81640f7890 msgr2=0x7f81640f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 shutdown_connections 2024-04-05T03:21:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.507+0000 7f816b33d700 1 -- 172.21.15.153:0/1416042137 wait complete. 2024-04-05T03:21:52.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 -- 172.21.15.153:0/507132353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f01037c0 msgr2=0x7f54f00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 --2- 172.21.15.153:0/507132353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f01037c0 0x7f54f00fce30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f54e0004940 tx=0x7f54e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 -- 172.21.15.153:0/507132353 shutdown_connections 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 --2- 172.21.15.153:0/507132353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f01037c0 0x7f54f00fce30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 --2- 172.21.15.153:0/507132353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f0102e20 0x7f54f01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 -- 172.21.15.153:0/507132353 >> 172.21.15.153:0/507132353 conn(0x7f54f00f89b0 msgr2=0x7f54f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 -- 172.21.15.153:0/507132353 shutdown_connections 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.611+0000 7f54f8013700 1 -- 172.21.15.153:0/507132353 wait complete. 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 Processor -- start 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 -- start start 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 0x7f54f0113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0114530 con 0x7f54f0102e20 2024-04-05T03:21:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f8013700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f010db30 con 0x7f54f01037c0 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44976/0 (socket says 172.21.15.153:44976) 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 -- 172.21.15.153:0/871053343 learned_addr learned my addr 172.21.15.153:0/871053343 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 -- 172.21.15.153:0/871053343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 msgr2=0x7f54f0113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f55ae700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 0x7f54f0113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 0x7f54f0113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f5daf700 1 -- 172.21.15.153:0/871053343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e0005040 con 0x7f54f0102e20 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.612+0000 7f54f55ae700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 0x7f54f0113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54f5daf700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f54e8001d00 tx=0x7f54e8009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e8004020 con 0x7f54f0102e20 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54e800f040 con 0x7f54f0102e20 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f010de10 con 0x7f54f0102e20 2024-04-05T03:21:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e8009310 con 0x7f54f0102e20 2024-04-05T03:21:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.613+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f010e3d0 con 0x7f54f0102e20 2024-04-05T03:21:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.614+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f01052f0 con 0x7f54f0102e20 2024-04-05T03:21:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.614+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f54e8009580 con 0x7f54f0102e20 2024-04-05T03:21:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.615+0000 7f54e67fc700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 0x7f54dc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.615+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f54e808cfa0 con 0x7f54f0102e20 2024-04-05T03:21:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.615+0000 7f54f55ae700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 0x7f54dc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.615+0000 7f54f55ae700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 0x7f54dc070150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b3b0 tx=0x7f54e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.619+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f54e805a050 con 0x7f54f0102e20 2024-04-05T03:21:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.826+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f54f010eff0 con 0x7f54dc06dca0 2024-04-05T03:21:52.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.836+0000 7f54e67fc700 1 -- 172.21.15.153:0/871053343 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f54f010eff0 con 0x7f54dc06dca0 2024-04-05T03:21:52.834 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:21:52.834 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (16m) 10m ago 27m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (13m) 10m ago 28m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (13m) 9m ago 26m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (16m) 10m ago 27m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (10m) 10m ago 22m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (10m) 10m ago 22m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (9m) 9m ago 21m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (9m) 9m ago 22m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (17m) 10m ago 29m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (17m) 9m ago 26m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (15m) 10m ago 30m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (14m) 9m ago 26m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (17m) 10m ago 27m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (16m) 9m ago 26m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (13m) 10m ago 25m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (12m) 10m ago 25m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:21:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (12m) 10m ago 24m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (11m) 9m ago 24m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (11m) 9m ago 23m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (10m) 9m ago 23m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (16m) 9m ago 26m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 msgr2=0x7f54dc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 0x7f54dc070150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54e000b3b0 tx=0x7f54e0009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 msgr2=0x7f54f01139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f54e8001d00 tx=0x7f54e8009f30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 shutdown_connections 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f54dc06dca0 0x7f54dc070150 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54f0102e20 0x7f54f01139a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 --2- 172.21.15.153:0/871053343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f01037c0 0x7f54f0113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 >> 172.21.15.153:0/871053343 conn(0x7f54f00f89b0 msgr2=0x7f54f01021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 shutdown_connections 2024-04-05T03:21:52.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.838+0000 7f54f8013700 1 -- 172.21.15.153:0/871053343 wait complete. 2024-04-05T03:21:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 -- 172.21.15.153:0/3877215055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 msgr2=0x7f737c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 --2- 172.21.15.153:0/3877215055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c0fbd30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7370004940 tx=0x7f7370009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 -- 172.21.15.153:0/3877215055 shutdown_connections 2024-04-05T03:21:52.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 --2- 172.21.15.153:0/3877215055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c0fbd30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 --2- 172.21.15.153:0/3877215055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 0x7f737c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 -- 172.21.15.153:0/3877215055 >> 172.21.15.153:0/3877215055 conn(0x7f737c0f7870 msgr2=0x7f737c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:52.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 -- 172.21.15.153:0/3877215055 shutdown_connections 2024-04-05T03:21:52.939 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:52 smithi153 ceph-mon[106102]: pgmap v486: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.938+0000 7f73811d1700 1 -- 172.21.15.153:0/3877215055 wait complete. 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 Processor -- start 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 -- start start 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 0x7f737c1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c114580 con 0x7f737c1026c0 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.939+0000 7f73811d1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c10db80 con 0x7f737c101d20 2024-04-05T03:21:52.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45006/0 (socket says 172.21.15.153:45006) 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 -- 172.21.15.153:0/1161895889 learned_addr learned my addr 172.21.15.153:0/1161895889 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737ad9d700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 0x7f737c1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 -- 172.21.15.153:0/1161895889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 msgr2=0x7f737c1139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 0x7f737c1139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 -- 172.21.15.153:0/1161895889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370005040 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f737a59c700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7370009100 tx=0x7f7370009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737000e040 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c10de00 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7370009af0 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737001c3f0 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.940+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c10e3c0 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.942+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c0fd470 con 0x7f737c1026c0 2024-04-05T03:21:52.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.942+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f737000baf0 con 0x7f737c1026c0 2024-04-05T03:21:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.943+0000 7f73637fe700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 0x7f7364070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.943+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f737008e820 con 0x7f737c1026c0 2024-04-05T03:21:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.943+0000 7f737ad9d700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 0x7f7364070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:52.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.943+0000 7f737ad9d700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 0x7f7364070150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f736c004eb0 tx=0x7f736c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:52.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:52.947+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f737005b8f0 con 0x7f737c1026c0 2024-04-05T03:21:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:52 smithi203 ceph-mon[95988]: pgmap v486: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:53.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.220+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f737c04c710 con 0x7f737c1026c0 2024-04-05T03:21:53.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.220+0000 7f73637fe700 1 -- 172.21.15.153:0/1161895889 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f737005b480 con 0x7f737c1026c0 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:21:53.220 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:21:53.221 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 msgr2=0x7f7364070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 0x7f7364070150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f736c004eb0 tx=0x7f736c00b040 comp rx=0 tx=0).stop 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 msgr2=0x7f737c113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7370009100 tx=0x7f7370009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 shutdown_connections 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f736406dca0 0x7f7364070150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f737c101d20 0x7f737c1139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 --2- 172.21.15.153:0/1161895889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f737c1026c0 0x7f737c113f30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 >> 172.21.15.153:0/1161895889 conn(0x7f737c0f7870 msgr2=0x7f737c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 shutdown_connections 2024-04-05T03:21:53.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.223+0000 7f73811d1700 1 -- 172.21.15.153:0/1161895889 wait complete. 2024-04-05T03:21:53.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 -- 172.21.15.153:0/1831240404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 msgr2=0x7f2da4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/1831240404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da4103240 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2d94004940 tx=0x7f2d94009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:53.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 -- 172.21.15.153:0/1831240404 shutdown_connections 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/1831240404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/1831240404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da4103240 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 -- 172.21.15.153:0/1831240404 >> 172.21.15.153:0/1831240404 conn(0x7f2da40714c0 msgr2=0x7f2da40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 -- 172.21.15.153:0/1831240404 shutdown_connections 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.324+0000 7f2dab4e4700 1 -- 172.21.15.153:0/1831240404 wait complete. 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 Processor -- start 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 -- start start 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da40fd150 con 0x7f2da4102e70 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2dab4e4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da40fd290 con 0x7f2da40fb4a0 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45022/0 (socket says 172.21.15.153:45022) 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 -- 172.21.15.153:0/3657070611 learned_addr learned my addr 172.21.15.153:0/3657070611 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da9280700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 -- 172.21.15.153:0/3657070611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 msgr2=0x7f2da40fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 -- 172.21.15.153:0/3657070611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94005040 con 0x7f2da4102e70 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da9280700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2da8a7f700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2da0002a50 tx=0x7f2da00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.325+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da0007cc0 con 0x7f2da4102e70 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.326+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2da000f040 con 0x7f2da4102e70 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.326+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da4119f10 con 0x7f2da4102e70 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.326+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2da0009640 con 0x7f2da4102e70 2024-04-05T03:21:53.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.326+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da411a310 con 0x7f2da4102e70 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.327+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da40653a0 con 0x7f2da4102e70 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.327+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2da00097a0 con 0x7f2da4102e70 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.327+0000 7f2d99ffb700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 0x7f2d90070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.328+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2da008d340 con 0x7f2da4102e70 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.328+0000 7f2da9280700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 0x7f2d90070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.328+0000 7f2da9280700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 0x7f2d90070150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d94004560 tx=0x7f2d94009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:53.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.332+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2da00568c0 con 0x7f2da4102e70 2024-04-05T03:21:53.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.573+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2da404bed0 con 0x7f2da4102e70 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.573+0000 7f2d99ffb700 1 -- 172.21.15.153:0/3657070611 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f2da0059f80 con 0x7f2da4102e70 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:21:53.572 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:21:53.573 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:21:53.574 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 msgr2=0x7f2d90070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 0x7f2d90070150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d94004560 tx=0x7f2d94009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:53.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 msgr2=0x7f2da40fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2da0002a50 tx=0x7f2da00046b0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 shutdown_connections 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2d9006dca0 0x7f2d90070150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2da40fb4a0 0x7f2da40fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 --2- 172.21.15.153:0/3657070611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da4102e70 0x7f2da40fcbc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 >> 172.21.15.153:0/3657070611 conn(0x7f2da40714c0 msgr2=0x7f2da40fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 shutdown_connections 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.575+0000 7f2dab4e4700 1 -- 172.21.15.153:0/3657070611 wait complete. 2024-04-05T03:21:53.576 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:21:53.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- 172.21.15.153:0/4086412936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 msgr2=0x7f0704070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 --2- 172.21.15.153:0/4086412936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704070d10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f06f4004970 tx=0x7f06f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:53.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- 172.21.15.153:0/4086412936 shutdown_connections 2024-04-05T03:21:53.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 --2- 172.21.15.153:0/4086412936 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704070d10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 --2- 172.21.15.153:0/4086412936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 0x7f0704070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- 172.21.15.153:0/4086412936 >> 172.21.15.153:0/4086412936 conn(0x7f07040f89b0 msgr2=0x7f07040fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- 172.21.15.153:0/4086412936 shutdown_connections 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- 172.21.15.153:0/4086412936 wait complete. 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 Processor -- start 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- start start 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 0x7f070411a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070411ab50 con 0x7f07040708a0 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.680+0000 7f070b41f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0704114150 con 0x7f0704071f90 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45040/0 (socket says 172.21.15.153:45040) 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 -- 172.21.15.153:0/2326560414 learned_addr learned my addr 172.21.15.153:0/2326560414 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07089ba700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 0x7f070411a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 -- 172.21.15.153:0/2326560414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 msgr2=0x7f070411a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 0x7f070411a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 -- 172.21.15.153:0/2326560414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4005040 con 0x7f07040708a0 2024-04-05T03:21:53.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f07091bb700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f070000b790 tx=0x7f0700009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:53.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070000f6f0 con 0x7f07040708a0 2024-04-05T03:21:53.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0704114430 con 0x7f07040708a0 2024-04-05T03:21:53.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f070000fd20 con 0x7f07040708a0 2024-04-05T03:21:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.681+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07000103a0 con 0x7f07040708a0 2024-04-05T03:21:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.682+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0704114940 con 0x7f07040708a0 2024-04-05T03:21:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.683+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0704071aa0 con 0x7f07040708a0 2024-04-05T03:21:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.683+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f0700011400 con 0x7f07040708a0 2024-04-05T03:21:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.683+0000 7f06f9ffb700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 0x7f06f0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:53.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.684+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f070008d380 con 0x7f07040708a0 2024-04-05T03:21:53.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.684+0000 7f07089ba700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 0x7f06f0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:53.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.684+0000 7f07089ba700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 0x7f06f0070150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009fd0 tx=0x7f06f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:53.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.688+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0700056980 con 0x7f07040708a0 2024-04-05T03:21:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.901+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07041105a0 con 0x7f06f006dca0 2024-04-05T03:21:53.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:53 smithi153 ceph-mon[106102]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:53.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:53 smithi153 ceph-mon[106102]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:53.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:53 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1161895889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:21:53.900 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:53 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3657070611' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:21:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.902+0000 7f06f9ffb700 1 -- 172.21.15.153:0/2326560414 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07041105a0 con 0x7f06f006dca0 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:21:53.901 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:21:53.902 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 msgr2=0x7f06f0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 0x7f06f0070150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009fd0 tx=0x7f06f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 msgr2=0x7f0704119f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f070000b790 tx=0x7f0700009250 comp rx=0 tx=0).stop 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 shutdown_connections 2024-04-05T03:21:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f06f006dca0 0x7f06f0070150 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07040708a0 0x7f0704119f30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 --2- 172.21.15.153:0/2326560414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0704071f90 0x7f070411a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.904+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 >> 172.21.15.153:0/2326560414 conn(0x7f07040f89b0 msgr2=0x7f07040f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.905+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 shutdown_connections 2024-04-05T03:21:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:53.905+0000 7f070b41f700 1 -- 172.21.15.153:0/2326560414 wait complete. 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 -- 172.21.15.153:0/1916254914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fd1d0 msgr2=0x7f2dd80fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/1916254914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd80fd5a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8004940 tx=0x7f2dc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 -- 172.21.15.153:0/1916254914 shutdown_connections 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/1916254914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd80fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/1916254914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd80fd5a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 -- 172.21.15.153:0/1916254914 >> 172.21.15.153:0/1916254914 conn(0x7f2dd80f89b0 msgr2=0x7f2dd80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 -- 172.21.15.153:0/1916254914 shutdown_connections 2024-04-05T03:21:54.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.005+0000 7f2ddd5de700 1 -- 172.21.15.153:0/1916254914 wait complete. 2024-04-05T03:21:54.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 Processor -- start 2024-04-05T03:21:54.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 -- start start 2024-04-05T03:21:54.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd8113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:54.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:54.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd81144d0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2ddd5de700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd810dad0 con 0x7f2dd80fd1d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2dd67fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2dd67fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45064/0 (socket says 172.21.15.153:45064) 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2dd67fc700 1 -- 172.21.15.153:0/931942890 learned_addr learned my addr 172.21.15.153:0/931942890 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.006+0000 7f2dd6ffd700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd8113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dd67fc700 1 -- 172.21.15.153:0/931942890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fd1d0 msgr2=0x7f2dd8113940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dd67fc700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd8113940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dd67fc700 1 -- 172.21.15.153:0/931942890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8005040 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dd67fc700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc002a50 tx=0x7f2dcc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dcc007cc0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dcc00f040 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd810ddb0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dcc009640 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.007+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd810e370 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.008+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd80fe570 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.008+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2dcc0097a0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.009+0000 7f2dbf7fe700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 0x7f2dc0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:21:54.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.009+0000 7f2dd6ffd700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 0x7f2dc0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:21:54.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.009+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2dcc08d2a0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.009+0000 7f2dd6ffd700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 0x7f2dc0070150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8004f90 tx=0x7f2dc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:21:54.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.014+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2dcc05a3d0 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:53 smithi203 ceph-mon[95988]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:54.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:53 smithi203 ceph-mon[95988]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:54.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:53 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1161895889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:21:54.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:53 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3657070611' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.290+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2dd810ef90 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.290+0000 7f2dbf7fe700 1 -- 172.21.15.153:0/931942890 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2dcc056030 con 0x7f2dd80fe9d0 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:21:54.289 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:21:54.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 msgr2=0x7f2dc0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 0x7f2dc0070150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8004f90 tx=0x7f2dc8009e30 comp rx=0 tx=0).stop 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 msgr2=0x7f2dd8113e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc002a50 tx=0x7f2dcc0046b0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 shutdown_connections 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2dc006dca0 0x7f2dc0070150 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd80fd1d0 0x7f2dd8113940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 --2- 172.21.15.153:0/931942890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dd80fe9d0 0x7f2dd8113e80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 >> 172.21.15.153:0/931942890 conn(0x7f2dd80f89b0 msgr2=0x7f2dd80f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 shutdown_connections 2024-04-05T03:21:54.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:21:54.293+0000 7f2ddd5de700 1 -- 172.21.15.153:0/931942890 wait complete. 2024-04-05T03:21:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:54 smithi203 ceph-mon[95988]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:54 smithi203 ceph-mon[95988]: pgmap v487: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:54 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/931942890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:21:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:54 smithi153 ceph-mon[106102]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:54 smithi153 ceph-mon[106102]: pgmap v487: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:54 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/931942890' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:21:56.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:55 smithi203 ceph-mon[95988]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:56.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:55 smithi153 ceph-mon[106102]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:21:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:56 smithi203 ceph-mon[95988]: pgmap v488: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:56 smithi153 ceph-mon[106102]: pgmap v488: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:21:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:21:58 smithi203 ceph-mon[95988]: pgmap v489: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:21:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:21:58 smithi153 ceph-mon[106102]: pgmap v489: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:00 smithi203 ceph-mon[95988]: pgmap v490: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:01.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:00 smithi153 ceph-mon[106102]: pgmap v490: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:02 smithi203 ceph-mon[95988]: pgmap v491: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:02 smithi153 ceph-mon[106102]: pgmap v491: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:04 smithi203 ceph-mon[95988]: pgmap v492: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:04 smithi153 ceph-mon[106102]: pgmap v492: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:06 smithi203 ceph-mon[95988]: pgmap v493: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:06 smithi153 ceph-mon[106102]: pgmap v493: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:08 smithi203 ceph-mon[95988]: pgmap v494: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:08 smithi153 ceph-mon[106102]: pgmap v494: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:10.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:09 smithi203 ceph-mon[95988]: pgmap v495: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:10.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:09 smithi153 ceph-mon[106102]: pgmap v495: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:12 smithi203 ceph-mon[95988]: pgmap v496: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:12 smithi153 ceph-mon[106102]: pgmap v496: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:14 smithi203 ceph-mon[95988]: pgmap v497: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:14 smithi153 ceph-mon[106102]: pgmap v497: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:16 smithi203 ceph-mon[95988]: pgmap v498: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:17.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:16 smithi153 ceph-mon[106102]: pgmap v498: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:18 smithi203 ceph-mon[95988]: pgmap v499: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:18 smithi153 ceph-mon[106102]: pgmap v499: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:20 smithi203 ceph-mon[95988]: pgmap v500: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:20 smithi153 ceph-mon[106102]: pgmap v500: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:22 smithi203 ceph-mon[95988]: pgmap v501: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:23.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:22 smithi153 ceph-mon[106102]: pgmap v501: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/3123138164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fde20 msgr2=0x7f7fa00fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/3123138164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fe1f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004940 tx=0x7f7f90009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/3123138164 shutdown_connections 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/3123138164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa0106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/3123138164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fe1f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/3123138164 >> 172.21.15.153:0/3123138164 conn(0x7f7fa00f7870 msgr2=0x7f7fa00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/3123138164 shutdown_connections 2024-04-05T03:22:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.406+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/3123138164 wait complete. 2024-04-05T03:22:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 Processor -- start 2024-04-05T03:22:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 -- start start 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00fdba0 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa6bb0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00fc6d0 con 0x7f7fa00fde20 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7f9ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7f9ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48986/0 (socket says 172.21.15.153:48986) 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7f9ffff700 1 -- 172.21.15.153:0/863306806 learned_addr learned my addr 172.21.15.153:0/863306806 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.407+0000 7f7fa494c700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9ffff700 1 -- 172.21.15.153:0/863306806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fde20 msgr2=0x7f7fa00fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9ffff700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9ffff700 1 -- 172.21.15.153:0/863306806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90005040 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9ffff700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c8c0 tx=0x7f7f940044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9400a980 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa00fc9b0 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f9400ce60 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f940115e0 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.408+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa011a160 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.409+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0066b70 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.409+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7f94012400 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.410+0000 7f7f9d7fa700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 0x7f7f88070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.410+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7f9408d670 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.410+0000 7f7fa494c700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 0x7f7f88070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.410+0000 7f7fa494c700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 0x7f7f88070150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004f90 tx=0x7f7f90009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.415+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7f94056c70 con 0x7f7fa00fe7c0 2024-04-05T03:22:24.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.627+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa0109f20 con 0x7f7f8806dca0 2024-04-05T03:22:24.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.628+0000 7f7f9d7fa700 1 -- 172.21.15.153:0/863306806 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7fa0109f20 con 0x7f7f8806dca0 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 msgr2=0x7f7f88070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 0x7f7f88070150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004f90 tx=0x7f7f90009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 msgr2=0x7f7fa00fc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c8c0 tx=0x7f7f940044c0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 shutdown_connections 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7f8806dca0 0x7f7f88070150 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00fde20 0x7f7fa00fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 --2- 172.21.15.153:0/863306806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fa00fe7c0 0x7f7fa00fc190 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 >> 172.21.15.153:0/863306806 conn(0x7f7fa00f7870 msgr2=0x7f7fa0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 shutdown_connections 2024-04-05T03:22:24.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.630+0000 7f7fa6bb0700 1 -- 172.21.15.153:0/863306806 wait complete. 2024-04-05T03:22:24.640 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 -- 172.21.15.153:0/1537895380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 msgr2=0x7fc5f00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 --2- 172.21.15.153:0/1537895380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f00fce10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4004940 tx=0x7fc5e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 -- 172.21.15.153:0/1537895380 shutdown_connections 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 --2- 172.21.15.153:0/1537895380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f00fce10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 --2- 172.21.15.153:0/1537895380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.733+0000 7fc5f570e700 1 -- 172.21.15.153:0/1537895380 >> 172.21.15.153:0/1537895380 conn(0x7fc5f00f89b0 msgr2=0x7fc5f00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.734+0000 7fc5f570e700 1 -- 172.21.15.153:0/1537895380 shutdown_connections 2024-04-05T03:22:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.734+0000 7fc5f570e700 1 -- 172.21.15.153:0/1537895380 wait complete. 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.734+0000 7fc5f570e700 1 Processor -- start 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.734+0000 7fc5f570e700 1 -- start start 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5f570e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5f570e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5f570e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0114510 con 0x7fc5f01037a0 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5f570e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f010db10 con 0x7fc5f0102e00 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49014/0 (socket says 172.21.15.153:49014) 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 -- 172.21.15.153:0/3282812198 learned_addr learned my addr 172.21.15.153:0/3282812198 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5eeffd700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 -- 172.21.15.153:0/3282812198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 msgr2=0x7fc5f0113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f0113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 -- 172.21.15.153:0/3282812198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4005040 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5eeffd700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f0113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5ee7fc700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009100 tx=0x7fc5e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e400e040 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5e4013420 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f010dd90 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.735+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5e401c610 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.736+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f010e350 con 0x7fc5f01037a0 2024-04-05T03:22:24.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.736+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f01052d0 con 0x7fc5f01037a0 2024-04-05T03:22:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.737+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc5e4013590 con 0x7fc5f01037a0 2024-04-05T03:22:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.737+0000 7fc5d77fe700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 0x7fc5d8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.737+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc5e408e840 con 0x7fc5f01037a0 2024-04-05T03:22:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.737+0000 7fc5eeffd700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 0x7fc5d8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.738+0000 7fc5eeffd700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 0x7fc5d8070150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009fd0 tx=0x7fc5e0009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.742+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc5e405b970 con 0x7fc5f01037a0 2024-04-05T03:22:24.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.956+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5f010efd0 con 0x7fc5d806dca0 2024-04-05T03:22:24.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.957+0000 7fc5d77fe700 1 -- 172.21.15.153:0/3282812198 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc5f010efd0 con 0x7fc5d806dca0 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 msgr2=0x7fc5d8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 0x7fc5d8070150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009fd0 tx=0x7fc5e0009e50 comp rx=0 tx=0).stop 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 msgr2=0x7fc5f0113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009100 tx=0x7fc5e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 shutdown_connections 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc5d806dca0 0x7fc5d8070150 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f0102e00 0x7fc5f0113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 --2- 172.21.15.153:0/3282812198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5f01037a0 0x7fc5f0113ec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 >> 172.21.15.153:0/3282812198 conn(0x7fc5f00f89b0 msgr2=0x7fc5f01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 shutdown_connections 2024-04-05T03:22:24.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:24.959+0000 7fc5f570e700 1 -- 172.21.15.153:0/3282812198 wait complete. 2024-04-05T03:22:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:24 smithi203 ceph-mon[95988]: pgmap v502: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:25.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 -- 172.21.15.153:0/1829277165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0102710 msgr2=0x7fb8b00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/1829277165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0102710 0x7fb8b00fbd80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0004940 tx=0x7fb8a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:25.060 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:24 smithi153 ceph-mon[106102]: pgmap v502: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 -- 172.21.15.153:0/1829277165 shutdown_connections 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/1829277165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0102710 0x7fb8b00fbd80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/1829277165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0101d70 0x7fb8b0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 -- 172.21.15.153:0/1829277165 >> 172.21.15.153:0/1829277165 conn(0x7fb8b0072bc0 msgr2=0x7fb8b0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 -- 172.21.15.153:0/1829277165 shutdown_connections 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.060+0000 7fb8b82a0700 1 -- 172.21.15.153:0/1829277165 wait complete. 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 Processor -- start 2024-04-05T03:22:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 -- start start 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0102710 0x7fb8b0112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b0113630 con 0x7fb8b0101d70 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.061+0000 7fb8b82a0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b010ca20 con 0x7fb8b0102710 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49034/0 (socket says 172.21.15.153:49034) 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 -- 172.21.15.153:0/2957140806 learned_addr learned my addr 172.21.15.153:0/2957140806 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b583b700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0102710 0x7fb8b0112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 -- 172.21.15.153:0/2957140806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0102710 msgr2=0x7fb8b0112fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0102710 0x7fb8b0112fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 -- 172.21.15.153:0/2957140806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0005040 con 0x7fb8b0101d70 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b603c700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c830 tx=0x7fb8ac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac0106f0 con 0x7fb8b0101d70 2024-04-05T03:22:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8ac010d20 con 0x7fb8b0101d70 2024-04-05T03:22:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b010cd00 con 0x7fb8b0101d70 2024-04-05T03:22:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8ac0113a0 con 0x7fb8b0101d70 2024-04-05T03:22:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.062+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b010d290 con 0x7fb8b0101d70 2024-04-05T03:22:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.063+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b0104240 con 0x7fb8b0101d70 2024-04-05T03:22:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.064+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb8ac012400 con 0x7fb8b0101d70 2024-04-05T03:22:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.064+0000 7fb8a6ffd700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 0x7fb89c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.064+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb8ac08d420 con 0x7fb8b0101d70 2024-04-05T03:22:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.064+0000 7fb8b583b700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 0x7fb89c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.065+0000 7fb8b583b700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 0x7fb89c070150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0004890 tx=0x7fb8a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.069+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb8ac05a5f0 con 0x7fb8b0101d70 2024-04-05T03:22:25.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.272+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8b010e160 con 0x7fb89c06dca0 2024-04-05T03:22:25.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.283+0000 7fb8a6ffd700 1 -- 172.21.15.153:0/2957140806 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb8b010e160 con 0x7fb89c06dca0 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (16m) 10m ago 28m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (14m) 10m ago 29m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (14m) 10m ago 27m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (16m) 10m ago 27m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (10m) 10m ago 22m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (10m) 10m ago 22m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (10m) 10m ago 22m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (10m) 10m ago 22m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (18m) 10m ago 30m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (17m) 10m ago 26m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (15m) 10m ago 30m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:22:25.282 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (15m) 10m ago 26m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (17m) 10m ago 28m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (17m) 10m ago 26m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (14m) 10m ago 26m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (13m) 10m ago 25m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (12m) 10m ago 25m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (12m) 10m ago 24m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (11m) 10m ago 24m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (11m) 10m ago 24m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:22:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (17m) 10m ago 26m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 msgr2=0x7fb89c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 0x7fb89c070150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0004890 tx=0x7fb8a0009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 msgr2=0x7fb8b0112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c830 tx=0x7fb8ac004470 comp rx=0 tx=0).stop 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 shutdown_connections 2024-04-05T03:22:25.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb89c06dca0 0x7fb89c070150 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8b0101d70 0x7fb8b0112aa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 --2- 172.21.15.153:0/2957140806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8b0102710 0x7fb8b0112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 >> 172.21.15.153:0/2957140806 conn(0x7fb8b0072bc0 msgr2=0x7fb8b00fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 shutdown_connections 2024-04-05T03:22:25.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.285+0000 7fb8b82a0700 1 -- 172.21.15.153:0/2957140806 wait complete. 2024-04-05T03:22:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- 172.21.15.153:0/1513005322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a60102e00 msgr2=0x7f4a601031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 --2- 172.21.15.153:0/1513005322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a60102e00 0x7f4a601031d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c004940 tx=0x7f4a5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- 172.21.15.153:0/1513005322 shutdown_connections 2024-04-05T03:22:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 --2- 172.21.15.153:0/1513005322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a601037a0 0x7f4a600fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 --2- 172.21.15.153:0/1513005322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a60102e00 0x7f4a601031d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- 172.21.15.153:0/1513005322 >> 172.21.15.153:0/1513005322 conn(0x7f4a600f89b0 msgr2=0x7f4a600fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- 172.21.15.153:0/1513005322 shutdown_connections 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- 172.21.15.153:0/1513005322 wait complete. 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 Processor -- start 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.386+0000 7f4a679c9700 1 -- start start 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a679c9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60102e00 0x7f4a60113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a679c9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a679c9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60114520 con 0x7f4a601037a0 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a679c9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6010db70 con 0x7f4a60102e00 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49044/0 (socket says 172.21.15.153:49044) 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 -- 172.21.15.153:0/3277736645 learned_addr learned my addr 172.21.15.153:0/3277736645 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a65765700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60102e00 0x7f4a60113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 -- 172.21.15.153:0/3277736645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60102e00 msgr2=0x7f4a60113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60102e00 0x7f4a60113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 -- 172.21.15.153:0/3277736645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c005040 con 0x7f4a601037a0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a64f64700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c890 tx=0x7f4a500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5000a700 con 0x7f4a601037a0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a5000ad30 con 0x7f4a601037a0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.387+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6010de50 con 0x7f4a601037a0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.388+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a500113a0 con 0x7f4a601037a0 2024-04-05T03:22:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.388+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6010e410 con 0x7f4a601037a0 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.389+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a60100110 con 0x7f4a601037a0 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.389+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4a50012400 con 0x7f4a601037a0 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.390+0000 7f4a55ffb700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 0x7f4a4c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.390+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a5008d540 con 0x7f4a601037a0 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.390+0000 7f4a65765700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 0x7f4a4c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.390+0000 7f4a65765700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 0x7f4a4c070150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009fd0 tx=0x7f4a5c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.394+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4a5005a710 con 0x7f4a601037a0 2024-04-05T03:22:25.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.672+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4a6010efe0 con 0x7f4a601037a0 2024-04-05T03:22:25.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.673+0000 7f4a55ffb700 1 -- 172.21.15.153:0/3277736645 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f4a5005a2a0 con 0x7f4a601037a0 2024-04-05T03:22:25.671 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:22:25.672 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:22:25.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 msgr2=0x7f4a4c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 0x7f4a4c070150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009fd0 tx=0x7f4a5c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 msgr2=0x7f4a60113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000c890 tx=0x7f4a500044c0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 shutdown_connections 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4a4c06dca0 0x7f4a4c070150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60102e00 0x7f4a60113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 --2- 172.21.15.153:0/3277736645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a601037a0 0x7f4a60113ed0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 >> 172.21.15.153:0/3277736645 conn(0x7f4a600f89b0 msgr2=0x7f4a601073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 shutdown_connections 2024-04-05T03:22:25.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.675+0000 7f4a679c9700 1 -- 172.21.15.153:0/3277736645 wait complete. 2024-04-05T03:22:25.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 -- 172.21.15.153:0/250357115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 msgr2=0x7f4dac103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 --2- 172.21.15.153:0/250357115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac103260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009ce0 tx=0x7f4d9c004d10 comp rx=0 tx=0).stop 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 -- 172.21.15.153:0/250357115 shutdown_connections 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 --2- 172.21.15.153:0/250357115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 0x7f4dac0fb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 --2- 172.21.15.153:0/250357115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac103260 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 -- 172.21.15.153:0/250357115 >> 172.21.15.153:0/250357115 conn(0x7f4dac0f6bd0 msgr2=0x7f4dac0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 -- 172.21.15.153:0/250357115 shutdown_connections 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 -- 172.21.15.153:0/250357115 wait complete. 2024-04-05T03:22:25.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.778+0000 7f4db21c3700 1 Processor -- start 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 -- start start 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 0x7f4dac113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac1146e0 con 0x7f4dac102e90 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac10dc30 con 0x7f4dac0faec0 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49062/0 (socket says 172.21.15.153:49062) 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 -- 172.21.15.153:0/2072148809 learned_addr learned my addr 172.21.15.153:0/2072148809 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4dab7fe700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 0x7f4dac113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 -- 172.21.15.153:0/2072148809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 msgr2=0x7f4dac113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 0x7f4dac113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 -- 172.21.15.153:0/2072148809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c009940 con 0x7f4dac102e90 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4daaffd700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4da000b790 tx=0x7f4da0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da000f770 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac10deb0 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4da000fda0 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.779+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da000d310 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.780+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac10e470 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.780+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac1001d0 con 0x7f4dac102e90 2024-04-05T03:22:25.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.781+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4da0010420 con 0x7f4dac102e90 2024-04-05T03:22:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.781+0000 7f4d93fff700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 0x7f4d94070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.782+0000 7f4dab7fe700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 0x7f4d94070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.782+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4da008d640 con 0x7f4dac102e90 2024-04-05T03:22:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.782+0000 7f4dab7fe700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 0x7f4d94070150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c015040 tx=0x7f4d9c004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:25.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:25.786+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4da005a710 con 0x7f4dac102e90 2024-04-05T03:22:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.026+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4dac10f110 con 0x7f4dac102e90 2024-04-05T03:22:26.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:25 smithi153 ceph-mon[106102]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:26.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:25 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3277736645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.027+0000 7f4d93fff700 1 -- 172.21.15.153:0/2072148809 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f4da005a2a0 con 0x7f4dac102e90 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:22:26.026 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:22:26.027 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:22:26.027 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:22:26.027 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:22:26.027 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:22:26.029 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:22:26.029 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:22:26.029 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:22:26.029 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:22:26.030 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:26.031 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:26.032 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:26.032 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 msgr2=0x7f4d94070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 0x7f4d94070150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c015040 tx=0x7f4d9c004d10 comp rx=0 tx=0).stop 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 msgr2=0x7f4dac114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4da000b790 tx=0x7f4da0009250 comp rx=0 tx=0).stop 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 shutdown_connections 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4d9406dca0 0x7f4d94070150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4dac0faec0 0x7f4dac113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 --2- 172.21.15.153:0/2072148809 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4dac102e90 0x7f4dac114000 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 >> 172.21.15.153:0/2072148809 conn(0x7f4dac0f6bd0 msgr2=0x7f4dac0f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 shutdown_connections 2024-04-05T03:22:26.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.029+0000 7f4db21c3700 1 -- 172.21.15.153:0/2072148809 wait complete. 2024-04-05T03:22:26.034 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:22:26.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:25 smithi203 ceph-mon[95988]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:26.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:25 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3277736645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- 172.21.15.153:0/3226610187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 msgr2=0x7fb9501020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 --2- 172.21.15.153:0/3226610187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb9501020b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb94c004940 tx=0x7fb94c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- 172.21.15.153:0/3226610187 shutdown_connections 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 --2- 172.21.15.153:0/3226610187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb9500fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 --2- 172.21.15.153:0/3226610187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb9501020b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- 172.21.15.153:0/3226610187 >> 172.21.15.153:0/3226610187 conn(0x7fb9500f7890 msgr2=0x7fb9500f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- 172.21.15.153:0/3226610187 shutdown_connections 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- 172.21.15.153:0/3226610187 wait complete. 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 Processor -- start 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.132+0000 7fb957554700 1 -- start start 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb957554700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb957554700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb950113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb957554700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb950114490 con 0x7fb950101ce0 2024-04-05T03:22:26.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb957554700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95010dae0 con 0x7fb950102680 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49078/0 (socket says 172.21.15.153:49078) 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 -- 172.21.15.153:0/1983179940 learned_addr learned my addr 172.21.15.153:0/1983179940 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb954aef700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb950113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 -- 172.21.15.153:0/1983179940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 msgr2=0x7fb950113e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb950113e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 -- 172.21.15.153:0/1983179940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c005040 con 0x7fb950101ce0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb954aef700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb950113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb9552f0700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009b30 tx=0x7fb94c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb94c00e070 con 0x7fb950101ce0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb94c013cd0 con 0x7fb950101ce0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb94c00b720 con 0x7fb950101ce0 2024-04-05T03:22:26.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.133+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95010dd60 con 0x7fb950101ce0 2024-04-05T03:22:26.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.134+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95010e240 con 0x7fb950101ce0 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.134+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500fd430 con 0x7fb950101ce0 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.135+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb94c01cab0 con 0x7fb950101ce0 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.135+0000 7fb945ffb700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 0x7fb93c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.135+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb94c08edf0 con 0x7fb950101ce0 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.136+0000 7fb954aef700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 0x7fb93c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.136+0000 7fb954aef700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 0x7fb93c070150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb940004600 tx=0x7fb94000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:26.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.140+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb94c05bfc0 con 0x7fb950101ce0 2024-04-05T03:22:26.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.356+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb950108de0 con 0x7fb93c06dca0 2024-04-05T03:22:26.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.358+0000 7fb945ffb700 1 -- 172.21.15.153:0/1983179940 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb950108de0 con 0x7fb93c06dca0 2024-04-05T03:22:26.356 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:22:26.356 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:22:26.356 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:22:26.356 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:22:26.357 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 msgr2=0x7fb93c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 0x7fb93c070150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb940004600 tx=0x7fb94000b040 comp rx=0 tx=0).stop 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 msgr2=0x7fb950113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009b30 tx=0x7fb94c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 shutdown_connections 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb93c06dca0 0x7fb93c070150 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb950101ce0 0x7fb950113900 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 --2- 172.21.15.153:0/1983179940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb950102680 0x7fb950113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 >> 172.21.15.153:0/1983179940 conn(0x7fb9500f7890 msgr2=0x7fb9500f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 shutdown_connections 2024-04-05T03:22:26.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.360+0000 7fb957554700 1 -- 172.21.15.153:0/1983179940 wait complete. 2024-04-05T03:22:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 -- 172.21.15.153:0/2024040096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74102710 msgr2=0x7f0e740fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 --2- 172.21.15.153:0/2024040096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74102710 0x7f0e740fbd80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0e64004940 tx=0x7f0e64009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 -- 172.21.15.153:0/2024040096 shutdown_connections 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 --2- 172.21.15.153:0/2024040096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74102710 0x7f0e740fbd80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 --2- 172.21.15.153:0/2024040096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74101d70 0x7f0e74102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 -- 172.21.15.153:0/2024040096 >> 172.21.15.153:0/2024040096 conn(0x7f0e74072bc0 msgr2=0x7f0e74072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 -- 172.21.15.153:0/2024040096 shutdown_connections 2024-04-05T03:22:26.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.462+0000 7f0e7c024700 1 -- 172.21.15.153:0/2024040096 wait complete. 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 Processor -- start 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 -- start start 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74102710 0x7f0e7410f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7410f7a0 con 0x7f0e74101d70 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e7c024700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7410f8e0 con 0x7f0e74102710 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e79dc0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e79dc0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49100/0 (socket says 172.21.15.153:49100) 2024-04-05T03:22:26.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.463+0000 7f0e79dc0700 1 -- 172.21.15.153:0/1158924236 learned_addr learned my addr 172.21.15.153:0/1158924236 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e795bf700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74102710 0x7f0e7410f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e79dc0700 1 -- 172.21.15.153:0/1158924236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74102710 msgr2=0x7f0e7410f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e79dc0700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74102710 0x7f0e7410f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e79dc0700 1 -- 172.21.15.153:0/1158924236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64005040 con 0x7f0e74101d70 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e79dc0700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b790 tx=0x7f0e70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7000f6f0 con 0x7f0e74101d70 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e7000fd20 con 0x7f0e74101d70 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e700103a0 con 0x7f0e74101d70 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7410c920 con 0x7f0e74101d70 2024-04-05T03:22:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.464+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7410ceb0 con 0x7f0e74101d70 2024-04-05T03:22:26.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.465+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e740ff080 con 0x7f0e74101d70 2024-04-05T03:22:26.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.466+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f0e70011400 con 0x7f0e74101d70 2024-04-05T03:22:26.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.466+0000 7f0e6a7fc700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 0x7f0e60070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:26.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.466+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e7008d230 con 0x7f0e74101d70 2024-04-05T03:22:26.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.466+0000 7f0e795bf700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 0x7f0e60070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:26.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.467+0000 7f0e795bf700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 0x7f0e60070150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0e64004890 tx=0x7f0e64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:26.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.471+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0e700568d0 con 0x7f0e74101d70 2024-04-05T03:22:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.746+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0e740fd4c0 con 0x7f0e74101d70 2024-04-05T03:22:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.747+0000 7f0e6a7fc700 1 -- 172.21.15.153:0/1158924236 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f0e70059f90 con 0x7f0e74101d70 2024-04-05T03:22:26.745 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:22:26.746 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:22:26.746 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:22:26.746 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:22:26.746 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:22:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 msgr2=0x7f0e60070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 0x7f0e60070150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0e64004890 tx=0x7f0e64009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 msgr2=0x7f0e7410ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b790 tx=0x7f0e70009250 comp rx=0 tx=0).stop 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 shutdown_connections 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0e6006dca0 0x7f0e60070150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e74101d70 0x7f0e7410ec90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 --2- 172.21.15.153:0/1158924236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e74102710 0x7f0e7410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 >> 172.21.15.153:0/1158924236 conn(0x7f0e74072bc0 msgr2=0x7f0e740faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 shutdown_connections 2024-04-05T03:22:26.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:26.749+0000 7f0e7c024700 1 -- 172.21.15.153:0/1158924236 wait complete. 2024-04-05T03:22:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:26 smithi203 ceph-mon[95988]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:26 smithi203 ceph-mon[95988]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:26 smithi203 ceph-mon[95988]: pgmap v503: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:26 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2072148809' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:22:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:26 smithi153 ceph-mon[106102]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:26 smithi153 ceph-mon[106102]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:26 smithi153 ceph-mon[106102]: pgmap v503: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:26 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2072148809' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:22:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:27 smithi203 ceph-mon[95988]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:27 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1158924236' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:22:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:27 smithi153 ceph-mon[106102]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:27 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1158924236' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:22:28.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:28 smithi203 ceph-mon[95988]: pgmap v504: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:29.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:28 smithi153 ceph-mon[106102]: pgmap v504: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:30 smithi203 ceph-mon[95988]: pgmap v505: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:30 smithi153 ceph-mon[106102]: pgmap v505: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:31 smithi203 ceph-mon[95988]: pgmap v506: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:32.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:31 smithi153 ceph-mon[106102]: pgmap v506: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:34 smithi203 ceph-mon[95988]: pgmap v507: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:34 smithi153 ceph-mon[106102]: pgmap v507: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:36 smithi203 ceph-mon[95988]: pgmap v508: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:36 smithi153 ceph-mon[106102]: pgmap v508: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:38 smithi203 ceph-mon[95988]: pgmap v509: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:39.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:38 smithi153 ceph-mon[106102]: pgmap v509: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:41.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:40 smithi203 ceph-mon[95988]: pgmap v510: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:41.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:40 smithi153 ceph-mon[106102]: pgmap v510: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:42 smithi203 ceph-mon[95988]: pgmap v511: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:42 smithi153 ceph-mon[106102]: pgmap v511: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:44 smithi203 ceph-mon[95988]: pgmap v512: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:44 smithi153 ceph-mon[106102]: pgmap v512: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:46 smithi203 ceph-mon[95988]: pgmap v513: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:47.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:46 smithi153 ceph-mon[106102]: pgmap v513: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:48 smithi203 ceph-mon[95988]: pgmap v514: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:48 smithi153 ceph-mon[106102]: pgmap v514: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:50 smithi203 ceph-mon[95988]: pgmap v515: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:51.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:50 smithi153 ceph-mon[106102]: pgmap v515: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:52 smithi203 ceph-mon[95988]: pgmap v516: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:52 smithi153 ceph-mon[106102]: pgmap v516: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:54 smithi203 ceph-mon[95988]: pgmap v517: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:55.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:54 smithi153 ceph-mon[106102]: pgmap v517: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 -- 172.21.15.153:0/3871998353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6040fb540 msgr2=0x7fa6040fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 --2- 172.21.15.153:0/3871998353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6040fb540 0x7fa6040fb990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4004940 tx=0x7fa5f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 -- 172.21.15.153:0/3871998353 shutdown_connections 2024-04-05T03:22:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 --2- 172.21.15.153:0/3871998353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6040fb540 0x7fa6040fb990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:56.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 --2- 172.21.15.153:0/3871998353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa604102fd0 0x7fa6041033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 -- 172.21.15.153:0/3871998353 >> 172.21.15.153:0/3871998353 conn(0x7fa604071530 msgr2=0x7fa604071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 -- 172.21.15.153:0/3871998353 shutdown_connections 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 -- 172.21.15.153:0/3871998353 wait complete. 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.865+0000 7fa60c7db700 1 Processor -- start 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60c7db700 1 -- start start 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60c7db700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 0x7fa6040fc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60c7db700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60c7db700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6040fe650 con 0x7fa604102fd0 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60c7db700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6040fd1a0 con 0x7fa6040fb540 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60230/0 (socket says 172.21.15.153:60230) 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 -- 172.21.15.153:0/4273361756 learned_addr learned my addr 172.21.15.153:0/4273361756 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60a577700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 0x7fa6040fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:56.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 -- 172.21.15.153:0/4273361756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 msgr2=0x7fa6040fc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 0x7fa6040fc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 -- 172.21.15.153:0/4273361756 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f4005040 con 0x7fa604102fd0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa60a577700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 0x7fa6040fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.866+0000 7fa609d76700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400b3b0 tx=0x7fa5f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.867+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5f400e040 con 0x7fa604102fd0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.867+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5f400bbc0 con 0x7fa604102fd0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.867+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5f401c3f0 con 0x7fa604102fd0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.867+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6040fd420 con 0x7fa604102fd0 2024-04-05T03:22:56.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.867+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60411a020 con 0x7fa604102fd0 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.868+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6041053b0 con 0x7fa604102fd0 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.868+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa5f400bd30 con 0x7fa604102fd0 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.869+0000 7fa5faffd700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 0x7fa5f0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.869+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa5f408e830 con 0x7fa604102fd0 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.869+0000 7fa60a577700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 0x7fa5f0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:56.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.869+0000 7fa60a577700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 0x7fa5f0070150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa600009fd0 tx=0x7fa600009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:56.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:56.874+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa5f4057db0 con 0x7fa604102fd0 2024-04-05T03:22:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:56 smithi203 ceph-mon[95988]: pgmap v518: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:57.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:56 smithi153 ceph-mon[106102]: pgmap v518: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:57.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.087+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa60410a010 con 0x7fa5f006dca0 2024-04-05T03:22:57.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.088+0000 7fa5faffd700 1 -- 172.21.15.153:0/4273361756 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa60410a010 con 0x7fa5f006dca0 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 msgr2=0x7fa5f0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 0x7fa5f0070150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa600009fd0 tx=0x7fa600009ee0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 msgr2=0x7fa6040fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400b3b0 tx=0x7fa5f4009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 shutdown_connections 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5f006dca0 0x7fa5f0070150 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6040fb540 0x7fa6040fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 --2- 172.21.15.153:0/4273361756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa604102fd0 0x7fa6040fcc60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 >> 172.21.15.153:0/4273361756 conn(0x7fa604071530 msgr2=0x7fa6040fa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 shutdown_connections 2024-04-05T03:22:57.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.090+0000 7fa60c7db700 1 -- 172.21.15.153:0/4273361756 wait complete. 2024-04-05T03:22:57.099 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 -- 172.21.15.153:0/2792207712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408101d70 msgr2=0x7fd408102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/2792207712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408101d70 0x7fd408102140 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc004940 tx=0x7fd3fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 -- 172.21.15.153:0/2792207712 shutdown_connections 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/2792207712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd408102710 0x7fd4080fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/2792207712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408101d70 0x7fd408102140 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 -- 172.21.15.153:0/2792207712 >> 172.21.15.153:0/2792207712 conn(0x7fd408072bc0 msgr2=0x7fd408072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 -- 172.21.15.153:0/2792207712 shutdown_connections 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 -- 172.21.15.153:0/2792207712 wait complete. 2024-04-05T03:22:57.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.191+0000 7fd40e4d4700 1 Processor -- start 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd40e4d4700 1 -- start start 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd40e4d4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd408101d70 0x7fd408113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd40e4d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd40e4d4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081146b0 con 0x7fd408102710 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd40e4d4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40810daa0 con 0x7fd408101d70 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60246/0 (socket says 172.21.15.153:60246) 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 -- 172.21.15.153:0/1180166826 learned_addr learned my addr 172.21.15.153:0/1180166826 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 -- 172.21.15.153:0/1180166826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd408101d70 msgr2=0x7fd408113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd408101d70 0x7fd408113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 -- 172.21.15.153:0/1180166826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc005040 con 0x7fd408102710 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.192+0000 7fd4077fe700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8002a50 tx=0x7fd3f8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.193+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3f8007cc0 con 0x7fd408102710 2024-04-05T03:22:57.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.193+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3f800f040 con 0x7fd408102710 2024-04-05T03:22:57.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.193+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40810dd80 con 0x7fd408102710 2024-04-05T03:22:57.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.193+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3f800daf0 con 0x7fd408102710 2024-04-05T03:22:57.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.193+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40810e340 con 0x7fd408102710 2024-04-05T03:22:57.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.194+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4080fd4c0 con 0x7fd408102710 2024-04-05T03:22:57.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.194+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd3f8009810 con 0x7fd408102710 2024-04-05T03:22:57.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.195+0000 7fd404ff9700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 0x7fd3f0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.195+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd3f808d230 con 0x7fd408102710 2024-04-05T03:22:57.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.195+0000 7fd407fff700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 0x7fd3f0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.195+0000 7fd407fff700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 0x7fd3f0070160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc004f90 tx=0x7fd3fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.199+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd3f805a360 con 0x7fd408102710 2024-04-05T03:22:57.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.411+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd408108e70 con 0x7fd3f006dcb0 2024-04-05T03:22:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.413+0000 7fd404ff9700 1 -- 172.21.15.153:0/1180166826 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd408108e70 con 0x7fd3f006dcb0 2024-04-05T03:22:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 msgr2=0x7fd3f0070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 0x7fd3f0070160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc004f90 tx=0x7fd3fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 msgr2=0x7fd408114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8002a50 tx=0x7fd3f8004600 comp rx=0 tx=0).stop 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 shutdown_connections 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd3f006dcb0 0x7fd3f0070160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd408101d70 0x7fd408113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 --2- 172.21.15.153:0/1180166826 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd408102710 0x7fd408114060 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 >> 172.21.15.153:0/1180166826 conn(0x7fd408072bc0 msgr2=0x7fd408105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 shutdown_connections 2024-04-05T03:22:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.415+0000 7fd40e4d4700 1 -- 172.21.15.153:0/1180166826 wait complete. 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 -- 172.21.15.153:0/558130041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234102710 msgr2=0x7f52340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 --2- 172.21.15.153:0/558130041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234102710 0x7f52340fbd80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5228004940 tx=0x7f5228009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 -- 172.21.15.153:0/558130041 shutdown_connections 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 --2- 172.21.15.153:0/558130041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234102710 0x7f52340fbd80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 --2- 172.21.15.153:0/558130041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234101d70 0x7f5234102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 -- 172.21.15.153:0/558130041 >> 172.21.15.153:0/558130041 conn(0x7f5234072bc0 msgr2=0x7f5234072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 -- 172.21.15.153:0/558130041 shutdown_connections 2024-04-05T03:22:57.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.517+0000 7f5239035700 1 -- 172.21.15.153:0/558130041 wait complete. 2024-04-05T03:22:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 Processor -- start 2024-04-05T03:22:57.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 -- start start 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 0x7f5234116280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52341168d0 con 0x7f5234101d70 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.518+0000 7f5239035700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523410fca0 con 0x7f5234102710 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60276/0 (socket says 172.21.15.153:60276) 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 -- 172.21.15.153:0/4177546040 learned_addr learned my addr 172.21.15.153:0/4177546040 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f523259c700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 0x7f5234116280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 -- 172.21.15.153:0/4177546040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 msgr2=0x7f5234116280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 0x7f5234116280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 -- 172.21.15.153:0/4177546040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228005040 con 0x7f5234101d70 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f523259c700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 0x7f5234116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5232d9d700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f522400c8c0 tx=0x7f52240044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522400a980 con 0x7f5234101d70 2024-04-05T03:22:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f522400ce60 con 0x7f5234101d70 2024-04-05T03:22:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523410ff80 con 0x7f5234101d70 2024-04-05T03:22:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52240115e0 con 0x7f5234101d70 2024-04-05T03:22:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.519+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5234110540 con 0x7f5234101d70 2024-04-05T03:22:57.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.520+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52340fd4c0 con 0x7f5234101d70 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.521+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5224012400 con 0x7f5234101d70 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.521+0000 7f521b7fe700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 0x7f521c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.521+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f522408d590 con 0x7f5234101d70 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.521+0000 7f523259c700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 0x7f521c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.522+0000 7f523259c700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 0x7f521c070150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f522800b3b0 tx=0x7f5228009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.526+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f522405a760 con 0x7f5234101d70 2024-04-05T03:22:57.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.733+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f523404bed0 con 0x7f521c06dca0 2024-04-05T03:22:57.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.743+0000 7f521b7fe700 1 -- 172.21.15.153:0/4177546040 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f523404bed0 con 0x7f521c06dca0 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (17m) 11m ago 28m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (15m) 11m ago 29m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (14m) 10m ago 27m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (17m) 11m ago 28m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (11m) 11m ago 23m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (11m) 11m ago 23m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:22:57.742 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (10m) 10m ago 23m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (10m) 10m ago 23m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (19m) 11m ago 30m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (18m) 10m ago 27m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (16m) 11m ago 31m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (15m) 10m ago 27m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (18m) 11m ago 28m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (18m) 10m ago 27m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (14m) 11m ago 26m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (14m) 11m ago 26m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (13m) 11m ago 25m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (12m) 10m ago 25m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (12m) 10m ago 25m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:22:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (11m) 10m ago 24m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:22:57.744 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (17m) 10m ago 27m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:22:57.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.745+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 msgr2=0x7f521c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.745+0000 7f5239035700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 0x7f521c070150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f522800b3b0 tx=0x7f5228009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.745+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 msgr2=0x7f5234115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.745+0000 7f5239035700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f522400c8c0 tx=0x7f52240044c0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.745+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 shutdown_connections 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f521c06dca0 0x7f521c070150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5234101d70 0x7f5234115d40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 --2- 172.21.15.153:0/4177546040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5234102710 0x7f5234116280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 >> 172.21.15.153:0/4177546040 conn(0x7f5234072bc0 msgr2=0x7f52340659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 shutdown_connections 2024-04-05T03:22:57.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.746+0000 7f5239035700 1 -- 172.21.15.153:0/4177546040 wait complete. 2024-04-05T03:22:57.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 -- 172.21.15.153:0/1702635767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c101d70 msgr2=0x7f231c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1702635767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c101d70 0x7f231c102140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f230c004940 tx=0x7f230c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 -- 172.21.15.153:0/1702635767 shutdown_connections 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1702635767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c102710 0x7f231c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1702635767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c101d70 0x7f231c102140 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 -- 172.21.15.153:0/1702635767 >> 172.21.15.153:0/1702635767 conn(0x7f231c072bc0 msgr2=0x7f231c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 -- 172.21.15.153:0/1702635767 shutdown_connections 2024-04-05T03:22:57.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 -- 172.21.15.153:0/1702635767 wait complete. 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.846+0000 7f2321ac9700 1 Processor -- start 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2321ac9700 1 -- start start 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2321ac9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c101d70 0x7f231c10c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2321ac9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2321ac9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c10d3b0 con 0x7f231c102710 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2321ac9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c10d520 con 0x7f231c101d70 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60300/0 (socket says 172.21.15.153:60300) 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 -- 172.21.15.153:0/1034986910 learned_addr learned my addr 172.21.15.153:0/1034986910 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:57.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f231b7fe700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c101d70 0x7f231c10c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 -- 172.21.15.153:0/1034986910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c101d70 msgr2=0x7f231c10c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c101d70 0x7f231c10c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.847+0000 7f2312ffd700 1 -- 172.21.15.153:0/1034986910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c005040 con 0x7f231c102710 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2312ffd700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f230400b790 tx=0x7f2304009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f230400f6f0 con 0x7f231c102710 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f230400fd20 con 0x7f231c102710 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c119f10 con 0x7f231c102710 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23040103a0 con 0x7f231c102710 2024-04-05T03:22:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.848+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c11a340 con 0x7f231c102710 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.849+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c0ff080 con 0x7f231c102710 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.849+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f230400f850 con 0x7f231c102710 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.850+0000 7f2318ff9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 0x7f2308070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.850+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f230408d370 con 0x7f231c102710 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.850+0000 7f231b7fe700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 0x7f2308070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.850+0000 7f231b7fe700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 0x7f2308070150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f230c004540 tx=0x7f230c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:57.854+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f230405a420 con 0x7f231c102710 2024-04-05T03:22:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:58.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:22:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.133+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f231c0fd4c0 con 0x7f231c102710 2024-04-05T03:22:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.133+0000 7f2318ff9700 1 -- 172.21.15.153:0/1034986910 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f2304059fb0 con 0x7f231c102710 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:22:58.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:22:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:22:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:22:58.133 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:22:58.133 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:22:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 msgr2=0x7f2308070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 0x7f2308070150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f230c004540 tx=0x7f230c009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 msgr2=0x7f231c10cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f230400b790 tx=0x7f2304009220 comp rx=0 tx=0).stop 2024-04-05T03:22:58.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 shutdown_connections 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f230806dca0 0x7f2308070150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f231c101d70 0x7f231c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 --2- 172.21.15.153:0/1034986910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f231c102710 0x7f231c10cde0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 >> 172.21.15.153:0/1034986910 conn(0x7f231c072bc0 msgr2=0x7f231c0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 shutdown_connections 2024-04-05T03:22:58.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.136+0000 7f2321ac9700 1 -- 172.21.15.153:0/1034986910 wait complete. 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 -- 172.21.15.153:0/834541783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 msgr2=0x7f23f40fee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 --2- 172.21.15.153:0/834541783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f40fee40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f23e8004940 tx=0x7f23e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 -- 172.21.15.153:0/834541783 shutdown_connections 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 --2- 172.21.15.153:0/834541783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f40fee40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 --2- 172.21.15.153:0/834541783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f40fd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 -- 172.21.15.153:0/834541783 >> 172.21.15.153:0/834541783 conn(0x7f23f40f89b0 msgr2=0x7f23f40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 -- 172.21.15.153:0/834541783 shutdown_connections 2024-04-05T03:22:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.237+0000 7f23f999c700 1 -- 172.21.15.153:0/834541783 wait complete. 2024-04-05T03:22:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 Processor -- start 2024-04-05T03:22:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 -- start start 2024-04-05T03:22:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f4115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4116770 con 0x7f23f40fe9d0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f999c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f410fd00 con 0x7f23f40fd1d0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.238+0000 7f23f27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60316/0 (socket says 172.21.15.153:60316) 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 -- 172.21.15.153:0/682302510 learned_addr learned my addr 172.21.15.153:0/682302510 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f2ffd700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f4115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 -- 172.21.15.153:0/682302510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 msgr2=0x7f23f4115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f4115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 -- 172.21.15.153:0/682302510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8005040 con 0x7f23f40fe9d0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f2ffd700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f4115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f27fc700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009180 tx=0x7f23e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e800e040 con 0x7f23f40fe9d0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f410ff80 con 0x7f23f40fe9d0 2024-04-05T03:22:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23e800bbc0 con 0x7f23f40fe9d0 2024-04-05T03:22:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23e801c630 con 0x7f23f40fe9d0 2024-04-05T03:22:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.239+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f4110540 con 0x7f23f40fe9d0 2024-04-05T03:22:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.240+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f40fe570 con 0x7f23f40fe9d0 2024-04-05T03:22:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.241+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f23e800bd30 con 0x7f23f40fe9d0 2024-04-05T03:22:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.241+0000 7f23db7fe700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 0x7f23dc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.241+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f23e808ea20 con 0x7f23f40fe9d0 2024-04-05T03:22:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.241+0000 7f23f2ffd700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 0x7f23dc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.242+0000 7f23f2ffd700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 0x7f23dc070150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006fd0 tx=0x7f23e4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.246+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f23e8057f80 con 0x7f23f40fe9d0 2024-04-05T03:22:58.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.484+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f23f4111210 con 0x7f23f40fe9d0 2024-04-05T03:22:58.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.485+0000 7f23db7fe700 1 -- 172.21.15.153:0/682302510 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f23e805b640 con 0x7f23f40fe9d0 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:22:58.484 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:22:58.485 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:58.486 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:58.487 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:22:58.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 msgr2=0x7f23dc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 0x7f23dc070150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006fd0 tx=0x7f23e4008040 comp rx=0 tx=0).stop 2024-04-05T03:22:58.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 msgr2=0x7f23f4116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009180 tx=0x7f23e8009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 shutdown_connections 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f23dc06dca0 0x7f23dc070150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f40fd1d0 0x7f23f4115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 --2- 172.21.15.153:0/682302510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23f40fe9d0 0x7f23f4116090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 >> 172.21.15.153:0/682302510 conn(0x7f23f40f89b0 msgr2=0x7f23f4071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 shutdown_connections 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.487+0000 7f23f999c700 1 -- 172.21.15.153:0/682302510 wait complete. 2024-04-05T03:22:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:22:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 -- 172.21.15.153:0/3414981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 msgr2=0x7fabe80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 --2- 172.21.15.153:0/3414981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe80fbd30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fabd8004940 tx=0x7fabd8009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 -- 172.21.15.153:0/3414981799 shutdown_connections 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 --2- 172.21.15.153:0/3414981799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe80fbd30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 --2- 172.21.15.153:0/3414981799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabe8101d20 0x7fabe81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 -- 172.21.15.153:0/3414981799 >> 172.21.15.153:0/3414981799 conn(0x7fabe80f7870 msgr2=0x7fabe80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 -- 172.21.15.153:0/3414981799 shutdown_connections 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.597+0000 7fabeccb7700 1 -- 172.21.15.153:0/3414981799 wait complete. 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 Processor -- start 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 -- start start 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabe8101d20 0x7fabe810eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe810f5c0 con 0x7fabe81026c0 2024-04-05T03:22:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabeccb7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe810f700 con 0x7fabe8101d20 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60340/0 (socket says 172.21.15.153:60340) 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 -- 172.21.15.153:0/455074680 learned_addr learned my addr 172.21.15.153:0/455074680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 -- 172.21.15.153:0/455074680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabe8101d20 msgr2=0x7fabe810eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabe8101d20 0x7fabe810eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.598+0000 7fabe5d9b700 1 -- 172.21.15.153:0/455074680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd8005040 con 0x7fabe81026c0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabe5d9b700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009100 tx=0x7fabd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd800e040 con 0x7fabe81026c0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe810f980 con 0x7fabe81026c0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabd8004290 con 0x7fabe81026c0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd801c3f0 con 0x7fabe81026c0 2024-04-05T03:22:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.599+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe8117f00 con 0x7fabe81026c0 2024-04-05T03:22:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.600+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe80fd470 con 0x7fabe81026c0 2024-04-05T03:22:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.600+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fabd8013790 con 0x7fabe81026c0 2024-04-05T03:22:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.601+0000 7fabd6ffd700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 0x7fabd0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.601+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fabd808e750 con 0x7fabe81026c0 2024-04-05T03:22:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.601+0000 7fabe659c700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 0x7fabd0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.601+0000 7fabe659c700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 0x7fabd0070150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabdc004d00 tx=0x7fabdc009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.605+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fabd8057df0 con 0x7fabe81026c0 2024-04-05T03:22:58.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.820+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabe80f9030 con 0x7fabd006dca0 2024-04-05T03:22:58.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.821+0000 7fabd6ffd700 1 -- 172.21.15.153:0/455074680 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fabe80f9030 con 0x7fabd006dca0 2024-04-05T03:22:58.820 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:22:58.821 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:22:58.821 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:22:58.821 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:22:58.821 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:22:58.821 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:22:58.822 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:22:58.823 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:22:58.823 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:22:58.823 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:22:58.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.823+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 msgr2=0x7fabd0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.823+0000 7fabeccb7700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 0x7fabd0070150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabdc004d00 tx=0x7fabdc009ce0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.823+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 msgr2=0x7fabe810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.823+0000 7fabeccb7700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009100 tx=0x7fabd8009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 shutdown_connections 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fabd006dca0 0x7fabd0070150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabe8101d20 0x7fabe810eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 --2- 172.21.15.153:0/455074680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fabe81026c0 0x7fabe810f080 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 >> 172.21.15.153:0/455074680 conn(0x7fabe80f7870 msgr2=0x7fabe80f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 shutdown_connections 2024-04-05T03:22:58.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.824+0000 7fabeccb7700 1 -- 172.21.15.153:0/455074680 wait complete. 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 -- 172.21.15.153:0/2751949710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fe810 msgr2=0x7f7700106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 --2- 172.21.15.153:0/2751949710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fe810 0x7f7700106e50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f76fc004940 tx=0x7f76fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 -- 172.21.15.153:0/2751949710 shutdown_connections 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 --2- 172.21.15.153:0/2751949710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fe810 0x7f7700106e50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 --2- 172.21.15.153:0/2751949710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fde70 0x7f77000fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 -- 172.21.15.153:0/2751949710 >> 172.21.15.153:0/2751949710 conn(0x7f77000f7880 msgr2=0x7f77000f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 -- 172.21.15.153:0/2751949710 shutdown_connections 2024-04-05T03:22:58.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.925+0000 7f7707dba700 1 -- 172.21.15.153:0/2751949710 wait complete. 2024-04-05T03:22:58.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 Processor -- start 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 -- start start 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fe810 0x7f77001142c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700114910 con 0x7f77000fde70 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7707dba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770010dce0 con 0x7f77000fe810 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38338/0 (socket says 172.21.15.153:38338) 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 -- 172.21.15.153:0/4198327140 learned_addr learned my addr 172.21.15.153:0/4198327140 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705355700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fe810 0x7f77001142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 -- 172.21.15.153:0/4198327140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fe810 msgr2=0x7f77001142c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fe810 0x7f77001142c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.926+0000 7f7705b56700 1 -- 172.21.15.153:0/4198327140 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc005040 con 0x7f77000fde70 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f7705b56700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f76f0002a80 tx=0x7f76f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f0007cc0 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76f000f040 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f00096f0 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770010dfc0 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.927+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770010e500 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.928+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700066710 con 0x7f77000fde70 2024-04-05T03:22:58.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.928+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f76f000d690 con 0x7f77000fde70 2024-04-05T03:22:58.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.929+0000 7f76f67fc700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 0x7f76ec070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:22:58.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.929+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f76f008ccf0 con 0x7f77000fde70 2024-04-05T03:22:58.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.929+0000 7f7705355700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 0x7f76ec070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:22:58.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.929+0000 7f7705355700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 0x7f76ec070160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f770010f620 tx=0x7f76fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:22:58.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:58.933+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f76f0059da0 con 0x7f77000fde70 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: pgmap v519: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1034986910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:22:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:58 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/682302510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: pgmap v519: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1034986910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:22:59.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:58 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/682302510' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:22:59.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.209+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f770004bed0 con 0x7f77000fde70 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.209+0000 7f76f67fc700 1 -- 172.21.15.153:0/4198327140 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f76f0059930 con 0x7f77000fde70 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:22:59.208 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 msgr2=0x7f76ec070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 0x7f76ec070160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f770010f620 tx=0x7f76fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 msgr2=0x7f7700113d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f76f0002a80 tx=0x7f76f00046b0 comp rx=0 tx=0).stop 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 shutdown_connections 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f76ec06dcb0 0x7f76ec070160 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77000fde70 0x7f7700113d80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:59.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 --2- 172.21.15.153:0/4198327140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77000fe810 0x7f77001142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:22:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 >> 172.21.15.153:0/4198327140 conn(0x7f77000f7880 msgr2=0x7f77000f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:22:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 shutdown_connections 2024-04-05T03:22:59.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:22:59.211+0000 7f7707dba700 1 -- 172.21.15.153:0/4198327140 wait complete. 2024-04-05T03:23:00.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:59 smithi203 ceph-mon[95988]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:00.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:59 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4198327140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:23:00.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:22:59 smithi203 ceph-mon[95988]: pgmap v520: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:00.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:59 smithi153 ceph-mon[106102]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:00.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:59 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4198327140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:23:00.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:22:59 smithi153 ceph-mon[106102]: pgmap v520: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:02 smithi203 ceph-mon[95988]: pgmap v521: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:03.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:02 smithi153 ceph-mon[106102]: pgmap v521: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:04 smithi203 ceph-mon[95988]: pgmap v522: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:04 smithi153 ceph-mon[106102]: pgmap v522: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:06 smithi203 ceph-mon[95988]: pgmap v523: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:06 smithi153 ceph-mon[106102]: pgmap v523: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:08 smithi203 ceph-mon[95988]: pgmap v524: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:09.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:08 smithi153 ceph-mon[106102]: pgmap v524: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:10 smithi203 ceph-mon[95988]: pgmap v525: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:10 smithi153 ceph-mon[106102]: pgmap v525: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:12 smithi203 ceph-mon[95988]: pgmap v526: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:12 smithi153 ceph-mon[106102]: pgmap v526: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:14 smithi203 ceph-mon[95988]: pgmap v527: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:14 smithi153 ceph-mon[106102]: pgmap v527: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:16 smithi203 ceph-mon[95988]: pgmap v528: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:16 smithi153 ceph-mon[106102]: pgmap v528: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:18 smithi203 ceph-mon[95988]: pgmap v529: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:19.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:18 smithi153 ceph-mon[106102]: pgmap v529: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:20 smithi203 ceph-mon[95988]: pgmap v530: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:20 smithi153 ceph-mon[106102]: pgmap v530: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:22 smithi203 ceph-mon[95988]: pgmap v531: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:23.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:22 smithi153 ceph-mon[106102]: pgmap v531: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:24.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:23 smithi203 ceph-mon[95988]: pgmap v532: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:24.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:23 smithi153 ceph-mon[106102]: pgmap v532: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:26 smithi203 ceph-mon[95988]: pgmap v533: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:27.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:26 smithi153 ceph-mon[106102]: pgmap v533: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:28 smithi203 ceph-mon[95988]: pgmap v534: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:28 smithi153 ceph-mon[106102]: pgmap v534: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:29.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 -- 172.21.15.153:0/982124723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 msgr2=0x7ff7900fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 --2- 172.21.15.153:0/982124723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff7900fe250 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff78c004940 tx=0x7ff78c009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:29.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 -- 172.21.15.153:0/982124723 shutdown_connections 2024-04-05T03:23:29.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 --2- 172.21.15.153:0/982124723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7900fe820 0x7ff790106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 --2- 172.21.15.153:0/982124723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff7900fe250 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 -- 172.21.15.153:0/982124723 >> 172.21.15.153:0/982124723 conn(0x7ff7900f7870 msgr2=0x7ff7900f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 -- 172.21.15.153:0/982124723 shutdown_connections 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.325+0000 7ff7973d1700 1 -- 172.21.15.153:0/982124723 wait complete. 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 Processor -- start 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 -- start start 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7900fe820 0x7ff790113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff790114530 con 0x7ff7900fde80 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff7973d1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79010daa0 con 0x7ff7900fe820 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47640/0 (socket says 172.21.15.153:47640) 2024-04-05T03:23:29.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 -- 172.21.15.153:0/433941043 learned_addr learned my addr 172.21.15.153:0/433941043 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 -- 172.21.15.153:0/433941043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7900fe820 msgr2=0x7ff790113e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7900fe820 0x7ff790113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 -- 172.21.15.153:0/433941043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c005040 con 0x7ff7900fde80 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.326+0000 7ff79516d700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009380 tx=0x7ff78c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.327+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78c00e040 con 0x7ff7900fde80 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.327+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff78c009af0 con 0x7ff7900fde80 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.327+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79010dd80 con 0x7ff7900fde80 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.327+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78c01c630 con 0x7ff7900fde80 2024-04-05T03:23:29.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.327+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79010e2c0 con 0x7ff7900fde80 2024-04-05T03:23:29.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.328+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790066af0 con 0x7ff7900fde80 2024-04-05T03:23:29.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.328+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff78c00bbc0 con 0x7ff7900fde80 2024-04-05T03:23:29.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.329+0000 7ff785ffb700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 0x7ff77c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.329+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff78c08e760 con 0x7ff7900fde80 2024-04-05T03:23:29.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.329+0000 7ff79496c700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 0x7ff77c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.329+0000 7ff79496c700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 0x7ff77c070150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff780006fd0 tx=0x7ff780006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.334+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff78c05b930 con 0x7ff7900fde80 2024-04-05T03:23:29.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.546+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff790109ec0 con 0x7ff77c06dca0 2024-04-05T03:23:29.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.547+0000 7ff785ffb700 1 -- 172.21.15.153:0/433941043 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff790109ec0 con 0x7ff77c06dca0 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 msgr2=0x7ff77c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 0x7ff77c070150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff780006fd0 tx=0x7ff780006db0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 msgr2=0x7ff790113910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009380 tx=0x7ff78c009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 shutdown_connections 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff77c06dca0 0x7ff77c070150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7900fde80 0x7ff790113910 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 --2- 172.21.15.153:0/433941043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7900fe820 0x7ff790113e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 >> 172.21.15.153:0/433941043 conn(0x7ff7900f7870 msgr2=0x7ff7900f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:29.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 shutdown_connections 2024-04-05T03:23:29.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.549+0000 7ff7973d1700 1 -- 172.21.15.153:0/433941043 wait complete. 2024-04-05T03:23:29.558 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 -- 172.21.15.153:0/700758057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 msgr2=0x7f5c900fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/700758057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c900fbd80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5c78004940 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 -- 172.21.15.153:0/700758057 shutdown_connections 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/700758057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c900fbd80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/700758057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c90102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.652+0000 7f5c95ac0700 1 -- 172.21.15.153:0/700758057 >> 172.21.15.153:0/700758057 conn(0x7f5c90072bc0 msgr2=0x7f5c90072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 -- 172.21.15.153:0/700758057 shutdown_connections 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 -- 172.21.15.153:0/700758057 wait complete. 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 Processor -- start 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 -- start start 2024-04-05T03:23:29.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c9010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9010f610 con 0x7f5c90102710 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c95ac0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9010f750 con 0x7f5c90101d70 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.653+0000 7f5c87fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47664/0 (socket says 172.21.15.153:47664) 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 -- 172.21.15.153:0/870524107 learned_addr learned my addr 172.21.15.153:0/870524107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c8f7fe700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c9010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 -- 172.21.15.153:0/870524107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 msgr2=0x7f5c9010ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c9010ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 -- 172.21.15.153:0/870524107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78005040 con 0x7f5c90102710 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c8f7fe700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c9010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c87fff700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5c78000c00 tx=0x7f5c78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7800e070 con 0x7f5c90102710 2024-04-05T03:23:29.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c90117af0 con 0x7f5c90102710 2024-04-05T03:23:29.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c78013c00 con 0x7f5c90102710 2024-04-05T03:23:29.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7800b700 con 0x7f5c90102710 2024-04-05T03:23:29.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.654+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c90117f80 con 0x7f5c90102710 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.655+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c90104240 con 0x7f5c90102710 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.656+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5c7801d460 con 0x7f5c90102710 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.656+0000 7f5c8cff9700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 0x7f5c7c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.656+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5c7808e780 con 0x7f5c90102710 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.656+0000 7f5c8f7fe700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 0x7f5c7c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.657+0000 7f5c8f7fe700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 0x7f5c7c070160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c80004600 tx=0x7f5c8000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.661+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5c78057ed0 con 0x7f5c90102710 2024-04-05T03:23:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.875+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c90065e20 con 0x7f5c7c06dcb0 2024-04-05T03:23:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.877+0000 7f5c8cff9700 1 -- 172.21.15.153:0/870524107 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5c90065e20 con 0x7f5c7c06dcb0 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 msgr2=0x7f5c7c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 0x7f5c7c070160 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c80004600 tx=0x7f5c8000b040 comp rx=0 tx=0).stop 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 msgr2=0x7f5c9010efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5c78000c00 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 shutdown_connections 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c7c06dcb0 0x7f5c7c070160 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90101d70 0x7f5c9010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 --2- 172.21.15.153:0/870524107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c90102710 0x7f5c9010efc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 >> 172.21.15.153:0/870524107 conn(0x7f5c90072bc0 msgr2=0x7f5c900faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 shutdown_connections 2024-04-05T03:23:29.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.879+0000 7f5c95ac0700 1 -- 172.21.15.153:0/870524107 wait complete. 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- 172.21.15.153:0/1549334752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238102e20 msgr2=0x7fd2381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 --2- 172.21.15.153:0/1549334752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238102e20 0x7fd2381031f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd234004940 tx=0x7fd234009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- 172.21.15.153:0/1549334752 shutdown_connections 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 --2- 172.21.15.153:0/1549334752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2381037c0 0x7fd2380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 --2- 172.21.15.153:0/1549334752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd238102e20 0x7fd2381031f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- 172.21.15.153:0/1549334752 >> 172.21.15.153:0/1549334752 conn(0x7fd2380f89d0 msgr2=0x7fd2380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:29.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- 172.21.15.153:0/1549334752 shutdown_connections 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- 172.21.15.153:0/1549334752 wait complete. 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 Processor -- start 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.981+0000 7fd23fc79700 1 -- start start 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23fc79700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 0x7fd238113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23fc79700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 0x7fd2381140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23fc79700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238114720 con 0x7fd2381037c0 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23fc79700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23810db10 con 0x7fd238102e20 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23da15700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 0x7fd238113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23d214700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 0x7fd2381140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23da15700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 0x7fd238113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.153:42842/0 (socket says 172.21.15.153:42842) 2024-04-05T03:23:29.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23da15700 1 -- 172.21.15.153:0/346310427 learned_addr learned my addr 172.21.15.153:0/346310427 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:29.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23d214700 1 -- 172.21.15.153:0/346310427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 msgr2=0x7fd238113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:29.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23d214700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 0x7fd238113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23d214700 1 -- 172.21.15.153:0/346310427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd234005040 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.982+0000 7fd23d214700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 0x7fd2381140d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd228004fd0 tx=0x7fd228004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.983+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd228007cc0 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.983+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23810ddf0 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.983+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd22800f040 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.983+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd228013860 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.983+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23810e3b0 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.984+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238100130 con 0x7fd2381037c0 2024-04-05T03:23:29.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.984+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd22801c400 con 0x7fd2381037c0 2024-04-05T03:23:29.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.985+0000 7fd22e7fc700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 0x7fd224070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:29.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.985+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd22808d1e0 con 0x7fd2381037c0 2024-04-05T03:23:29.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.985+0000 7fd23da15700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 0x7fd224070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:29.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.985+0000 7fd23da15700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 0x7fd224070150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd234004560 tx=0x7fd234009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:29.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:29.989+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd22805a290 con 0x7fd2381037c0 2024-04-05T03:23:30.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.194+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd23810f1e0 con 0x7fd22406dca0 2024-04-05T03:23:30.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.204+0000 7fd22e7fc700 1 -- 172.21.15.153:0/346310427 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd23810f1e0 con 0x7fd22406dca0 2024-04-05T03:23:30.202 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:23:30.202 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (18m) 11m ago 29m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:23:30.202 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (15m) 11m ago 30m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (15m) 11m ago 28m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (17m) 11m ago 29m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (11m) 11m ago 23m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (11m) 11m ago 23m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (11m) 11m ago 23m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (11m) 11m ago 23m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (19m) 11m ago 31m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (18m) 11m ago 27m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (16m) 11m ago 31m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (16m) 11m ago 27m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (18m) 11m ago 29m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (18m) 11m ago 28m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (15m) 11m ago 27m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (14m) 11m ago 26m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:23:30.203 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (14m) 11m ago 26m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:23:30.204 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (13m) 11m ago 26m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:23:30.204 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (12m) 11m ago 25m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:23:30.204 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (12m) 11m ago 25m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:23:30.204 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (18m) 11m ago 27m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:23:30.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 msgr2=0x7fd224070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 0x7fd224070150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd234004560 tx=0x7fd234009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 msgr2=0x7fd2381140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 0x7fd2381140d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd228004fd0 tx=0x7fd228004570 comp rx=0 tx=0).stop 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 shutdown_connections 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd22406dca0 0x7fd224070150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd238102e20 0x7fd238113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 --2- 172.21.15.153:0/346310427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2381037c0 0x7fd2381140d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 >> 172.21.15.153:0/346310427 conn(0x7fd2380f89d0 msgr2=0x7fd238107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 shutdown_connections 2024-04-05T03:23:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.206+0000 7fd23fc79700 1 -- 172.21.15.153:0/346310427 wait complete. 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/2493109726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f81026c0 msgr2=0x7fa5f80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/2493109726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f81026c0 0x7fa5f80fbd30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec004940 tx=0x7fa5ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/2493109726 shutdown_connections 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/2493109726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f81026c0 0x7fa5f80fbd30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/2493109726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f8101d20 0x7fa5f81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/2493109726 >> 172.21.15.153:0/2493109726 conn(0x7fa5f80f7870 msgr2=0x7fa5f80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/2493109726 shutdown_connections 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/2493109726 wait complete. 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 Processor -- start 2024-04-05T03:23:30.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 -- start start 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.306+0000 7fa5fcfd6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f81026c0 0x7fa5f81160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5fcfd6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8116710 con 0x7fa5f8101d20 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5fcfd6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f810fd30 con 0x7fa5f81026c0 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47692/0 (socket says 172.21.15.153:47692) 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 -- 172.21.15.153:0/3253405604 learned_addr learned my addr 172.21.15.153:0/3253405604 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:30.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f5d9b700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f81026c0 0x7fa5f81160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 -- 172.21.15.153:0/3253405604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f81026c0 msgr2=0x7fa5f81160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f81026c0 0x7fa5f81160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 -- 172.21.15.153:0/3253405604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec005040 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5f659c700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800c830 tx=0x7fa5e8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e80106f0 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8110010 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5e8010d20 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e80113a0 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.307+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f81105a0 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.309+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f80fd470 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.309+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa5e8012400 con 0x7fa5f8101d20 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.309+0000 7fa5e6ffd700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 0x7fa5e0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.309+0000 7fa5f5d9b700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 0x7fa5e0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.309+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa5e808d420 con 0x7fa5f8101d20 2024-04-05T03:23:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.310+0000 7fa5f5d9b700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 0x7fa5e0070150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec004890 tx=0x7fa5ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.314+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa5e8056ac0 con 0x7fa5f8101d20 2024-04-05T03:23:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.594+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa5f8111210 con 0x7fa5f8101d20 2024-04-05T03:23:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.595+0000 7fa5e6ffd700 1 -- 172.21.15.153:0/3253405604 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fa5e805a180 con 0x7fa5f8101d20 2024-04-05T03:23:30.593 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:23:30.593 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:23:30.593 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:23:30.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:23:30.594 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:23:30.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 msgr2=0x7fa5e0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 0x7fa5e0070150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec004890 tx=0x7fa5ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 msgr2=0x7fa5f8115b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800c830 tx=0x7fa5e8004470 comp rx=0 tx=0).stop 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 shutdown_connections 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa5e006dca0 0x7fa5e0070150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101d20 0x7fa5f8115b80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 --2- 172.21.15.153:0/3253405604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5f81026c0 0x7fa5f81160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 >> 172.21.15.153:0/3253405604 conn(0x7fa5f80f7870 msgr2=0x7fa5f80662d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 shutdown_connections 2024-04-05T03:23:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.597+0000 7fa5fcfd6700 1 -- 172.21.15.153:0/3253405604 wait complete. 2024-04-05T03:23:30.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 -- 172.21.15.153:0/4108380068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 msgr2=0x7f57001018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 --2- 172.21.15.153:0/4108380068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f57001018c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009ce0 tx=0x7f56f8004d10 comp rx=0 tx=0).stop 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 -- 172.21.15.153:0/4108380068 shutdown_connections 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 --2- 172.21.15.153:0/4108380068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f57000fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 --2- 172.21.15.153:0/4108380068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f57001018c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 -- 172.21.15.153:0/4108380068 >> 172.21.15.153:0/4108380068 conn(0x7f57000714c0 msgr2=0x7f57000718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 -- 172.21.15.153:0/4108380068 shutdown_connections 2024-04-05T03:23:30.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.698+0000 7f5708254700 1 -- 172.21.15.153:0/4108380068 wait complete. 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 Processor -- start 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 -- start start 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f570010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570010f930 con 0x7f57001014f0 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5708254700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570010fa70 con 0x7f57000fb4a0 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47712/0 (socket says 172.21.15.153:47712) 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 -- 172.21.15.153:0/1388804920 learned_addr learned my addr 172.21.15.153:0/1388804920 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5705ff0700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f570010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 -- 172.21.15.153:0/1388804920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 msgr2=0x7f570010eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f570010eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 -- 172.21.15.153:0/1388804920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009940 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f5705ff0700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f570010eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.699+0000 7f57057ef700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f56f4002a50 tx=0x7f56f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.700+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f4007cc0 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.700+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56f400f040 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.700+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700117d60 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.700+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56f4009640 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.700+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5700118290 con 0x7f57001014f0 2024-04-05T03:23:30.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.701+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5700103950 con 0x7f57001014f0 2024-04-05T03:23:30.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.701+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f56f4009820 con 0x7f57001014f0 2024-04-05T03:23:30.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.702+0000 7f56fe7fc700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 0x7f56e8070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:30.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.702+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f56f408d350 con 0x7f57001014f0 2024-04-05T03:23:30.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.702+0000 7f5705ff0700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 0x7f56e8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:30.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.702+0000 7f5705ff0700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 0x7f56e8070160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b740 tx=0x7f56f8004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:30.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.706+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f56f40568d0 con 0x7f57001014f0 2024-04-05T03:23:30.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:30 smithi153 ceph-mon[106102]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:30.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:30 smithi153 ceph-mon[106102]: pgmap v535: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:30.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:30 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3253405604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:23:30.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.947+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5700108560 con 0x7f57001014f0 2024-04-05T03:23:30.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.948+0000 7f56fe7fc700 1 -- 172.21.15.153:0/1388804920 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f56f4059f90 con 0x7f57001014f0 2024-04-05T03:23:30.946 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:23:30.946 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:23:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:23:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:23:30.949 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 msgr2=0x7f56e8070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 0x7f56e8070160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b740 tx=0x7f56f8004d10 comp rx=0 tx=0).stop 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 msgr2=0x7f570010f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f56f4002a50 tx=0x7f56f40046b0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 shutdown_connections 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f56e806dcb0 0x7f56e8070160 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57000fb4a0 0x7f570010eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 --2- 172.21.15.153:0/1388804920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57001014f0 0x7f570010f3f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 >> 172.21.15.153:0/1388804920 conn(0x7f57000714c0 msgr2=0x7f57000fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 shutdown_connections 2024-04-05T03:23:30.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:30.950+0000 7f5708254700 1 -- 172.21.15.153:0/1388804920 wait complete. 2024-04-05T03:23:30.951 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:23:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:30 smithi203 ceph-mon[95988]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:30 smithi203 ceph-mon[95988]: pgmap v535: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:30 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3253405604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 -- 172.21.15.153:0/184395990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c102e00 msgr2=0x7fd26c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 --2- 172.21.15.153:0/184395990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c102e00 0x7fd26c1031d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd25c004940 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 -- 172.21.15.153:0/184395990 shutdown_connections 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 --2- 172.21.15.153:0/184395990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c1037a0 0x7fd26c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 --2- 172.21.15.153:0/184395990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c102e00 0x7fd26c1031d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.051+0000 7fd271095700 1 -- 172.21.15.153:0/184395990 >> 172.21.15.153:0/184395990 conn(0x7fd26c0f89b0 msgr2=0x7fd26c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:31.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 -- 172.21.15.153:0/184395990 shutdown_connections 2024-04-05T03:23:31.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 -- 172.21.15.153:0/184395990 wait complete. 2024-04-05T03:23:31.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 Processor -- start 2024-04-05T03:23:31.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 -- start start 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c102e00 0x7fd26c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c114520 con 0x7fd26c1037a0 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.052+0000 7fd271095700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c10db70 con 0x7fd26c102e00 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47738/0 (socket says 172.21.15.153:47738) 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 -- 172.21.15.153:0/1411746130 learned_addr learned my addr 172.21.15.153:0/1411746130 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26ad9d700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c102e00 0x7fd26c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 -- 172.21.15.153:0/1411746130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c102e00 msgr2=0x7fd26c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c102e00 0x7fd26c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 -- 172.21.15.153:0/1411746130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c005040 con 0x7fd26c1037a0 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd26a59c700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd26000c800 tx=0x7fd260004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2600106f0 con 0x7fd26c1037a0 2024-04-05T03:23:31.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c10de50 con 0x7fd26c1037a0 2024-04-05T03:23:31.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd260010d20 con 0x7fd26c1037a0 2024-04-05T03:23:31.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2600113a0 con 0x7fd26c1037a0 2024-04-05T03:23:31.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.053+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c10e410 con 0x7fd26c1037a0 2024-04-05T03:23:31.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.054+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c100110 con 0x7fd26c1037a0 2024-04-05T03:23:31.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.055+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd260012400 con 0x7fd26c1037a0 2024-04-05T03:23:31.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.055+0000 7fd2537fe700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 0x7fd254070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.055+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd26008d540 con 0x7fd26c1037a0 2024-04-05T03:23:31.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.055+0000 7fd26ad9d700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 0x7fd254070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.056+0000 7fd26ad9d700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 0x7fd254070150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd25c004f90 tx=0x7fd25c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:31.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.060+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd26005a5d0 con 0x7fd26c1037a0 2024-04-05T03:23:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.275+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd26c10f050 con 0x7fd25406dca0 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.277+0000 7fd2537fe700 1 -- 172.21.15.153:0/1411746130 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd26c10f050 con 0x7fd25406dca0 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:23:31.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:23:31.276 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:23:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 msgr2=0x7fd254070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 0x7fd254070150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd25c004f90 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2024-04-05T03:23:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 msgr2=0x7fd26c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd26000c800 tx=0x7fd260004470 comp rx=0 tx=0).stop 2024-04-05T03:23:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 shutdown_connections 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd25406dca0 0x7fd254070150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd26c102e00 0x7fd26c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 --2- 172.21.15.153:0/1411746130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd26c1037a0 0x7fd26c113ed0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 >> 172.21.15.153:0/1411746130 conn(0x7fd26c0f89b0 msgr2=0x7fd26c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 shutdown_connections 2024-04-05T03:23:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.279+0000 7fd271095700 1 -- 172.21.15.153:0/1411746130 wait complete. 2024-04-05T03:23:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 -- 172.21.15.153:0/447638645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4102e10 msgr2=0x7f96b41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 --2- 172.21.15.153:0/447638645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4102e10 0x7f96b41031e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009ce0 tx=0x7f96ac004d10 comp rx=0 tx=0).stop 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 -- 172.21.15.153:0/447638645 shutdown_connections 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 --2- 172.21.15.153:0/447638645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b41037b0 0x7f96b40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 --2- 172.21.15.153:0/447638645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b4102e10 0x7f96b41031e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 -- 172.21.15.153:0/447638645 >> 172.21.15.153:0/447638645 conn(0x7f96b40f8980 msgr2=0x7f96b40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 -- 172.21.15.153:0/447638645 shutdown_connections 2024-04-05T03:23:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.381+0000 7f96bb358700 1 -- 172.21.15.153:0/447638645 wait complete. 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 Processor -- start 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 -- start start 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b4102e10 0x7f96b4113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4114850 con 0x7f96b41037b0 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96bb358700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410dc20 con 0x7f96b4102e10 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47746/0 (socket says 172.21.15.153:47746) 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 -- 172.21.15.153:0/989284737 learned_addr learned my addr 172.21.15.153:0/989284737 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b90f4700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b4102e10 0x7f96b4113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 -- 172.21.15.153:0/989284737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b4102e10 msgr2=0x7f96b4113cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b4102e10 0x7f96b4113cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.382+0000 7f96b88f3700 1 -- 172.21.15.153:0/989284737 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009940 con 0x7f96b41037b0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96b88f3700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c890 tx=0x7f96a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a800a700 con 0x7f96b41037b0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96a800ad30 con 0x7f96b41037b0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b410df00 con 0x7f96b41037b0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96a80113a0 con 0x7f96b41037b0 2024-04-05T03:23:31.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.383+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b410e4c0 con 0x7f96b41037b0 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.384+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b4100120 con 0x7f96b41037b0 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.384+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f96a8012400 con 0x7f96b41037b0 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.385+0000 7f96b1ffb700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 0x7f969c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.385+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f96a808d610 con 0x7f96b41037b0 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.385+0000 7f96b90f4700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 0x7f969c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:23:31.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.385+0000 7f96b90f4700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 0x7f969c070150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00b740 tx=0x7f96ac004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:23:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.389+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f96a805a740 con 0x7f96b41037b0 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.668+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f96b40fe560 con 0x7f96b41037b0 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.668+0000 7f96b1ffb700 1 -- 172.21.15.153:0/989284737 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f96a805a2d0 con 0x7f96b41037b0 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:23:31.667 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:23:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 msgr2=0x7f969c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 0x7f969c070150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96ac00b740 tx=0x7f96ac004d10 comp rx=0 tx=0).stop 2024-04-05T03:23:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 msgr2=0x7f96b4114200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:23:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c890 tx=0x7f96a80044c0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 shutdown_connections 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f969c06dca0 0x7f969c070150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96b4102e10 0x7f96b4113cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 --2- 172.21.15.153:0/989284737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f96b41037b0 0x7f96b4114200 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 >> 172.21.15.153:0/989284737 conn(0x7f96b40f8980 msgr2=0x7f96b41073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 shutdown_connections 2024-04-05T03:23:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:23:31.671+0000 7f96bb358700 1 -- 172.21.15.153:0/989284737 wait complete. 2024-04-05T03:23:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:31 smithi203 ceph-mon[95988]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:31 smithi203 ceph-mon[95988]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1388804920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:23:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/989284737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:23:32.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:31 smithi153 ceph-mon[106102]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:32.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:31 smithi153 ceph-mon[106102]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:32.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1388804920' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:23:32.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/989284737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:23:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:32 smithi203 ceph-mon[95988]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:32 smithi203 ceph-mon[95988]: pgmap v536: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:32 smithi153 ceph-mon[106102]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:23:33.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:32 smithi153 ceph-mon[106102]: pgmap v536: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:34 smithi203 ceph-mon[95988]: pgmap v537: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:34 smithi153 ceph-mon[106102]: pgmap v537: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:36 smithi203 ceph-mon[95988]: pgmap v538: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:36 smithi153 ceph-mon[106102]: pgmap v538: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:38 smithi203 ceph-mon[95988]: pgmap v539: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:38 smithi153 ceph-mon[106102]: pgmap v539: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:39 smithi203 ceph-mon[95988]: pgmap v540: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:40.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:39 smithi153 ceph-mon[106102]: pgmap v540: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:42 smithi203 ceph-mon[95988]: pgmap v541: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:42 smithi153 ceph-mon[106102]: pgmap v541: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:44 smithi203 ceph-mon[95988]: pgmap v542: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:45.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:44 smithi153 ceph-mon[106102]: pgmap v542: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:46 smithi203 ceph-mon[95988]: pgmap v543: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:46 smithi153 ceph-mon[106102]: pgmap v543: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:48 smithi203 ceph-mon[95988]: pgmap v544: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:49.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:48 smithi153 ceph-mon[106102]: pgmap v544: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:50 smithi203 ceph-mon[95988]: pgmap v545: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:50 smithi153 ceph-mon[106102]: pgmap v545: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:52 smithi203 ceph-mon[95988]: pgmap v546: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:52 smithi153 ceph-mon[106102]: pgmap v546: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:54 smithi203 ceph-mon[95988]: pgmap v547: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:54 smithi153 ceph-mon[106102]: pgmap v547: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:56 smithi203 ceph-mon[95988]: pgmap v548: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:57.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:56 smithi153 ceph-mon[106102]: pgmap v548: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:23:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:58 smithi203 ceph-mon[95988]: pgmap v549: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:23:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:58 smithi153 ceph-mon[106102]: pgmap v549: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:00.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:23:59 smithi203 ceph-mon[95988]: pgmap v550: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:00.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:23:59 smithi153 ceph-mon[106102]: pgmap v550: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- 172.21.15.153:0/3446215346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 msgr2=0x7f94081013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 --2- 172.21.15.153:0/3446215346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f94081013f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9404004940 tx=0x7f9404009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- 172.21.15.153:0/3446215346 shutdown_connections 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 --2- 172.21.15.153:0/3446215346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 0x7f94080fb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 --2- 172.21.15.153:0/3446215346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f94081013f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- 172.21.15.153:0/3446215346 >> 172.21.15.153:0/3446215346 conn(0x7f94080f6bd0 msgr2=0x7f94080f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:01.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- 172.21.15.153:0/3446215346 shutdown_connections 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- 172.21.15.153:0/3446215346 wait complete. 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 Processor -- start 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.787+0000 7f940fe5a700 1 -- start start 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940fe5a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940fe5a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 0x7f94081034d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940fe5a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408104ee0 con 0x7f9408101020 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940fe5a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408103a10 con 0x7f94081019c0 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50598/0 (socket says 172.21.15.153:50598) 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 -- 172.21.15.153:0/1474187901 learned_addr learned my addr 172.21.15.153:0/1474187901 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940d3f5700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 0x7f94081034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 -- 172.21.15.153:0/1474187901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 msgr2=0x7f94081034d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 0x7f94081034d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 -- 172.21.15.153:0/1474187901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404005040 con 0x7f9408101020 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940dbf6700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f940400b3b0 tx=0x7f9404009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f940400e040 con 0x7f9408101020 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f940400bbc0 con 0x7f9408101020 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9408103c90 con 0x7f9408101020 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.788+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f940401c3f0 con 0x7f9408101020 2024-04-05T03:24:01.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.789+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940811a0a0 con 0x7f9408101020 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.789+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94080fc770 con 0x7f9408101020 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.790+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f940400bd30 con 0x7f9408101020 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.790+0000 7f93fe7fc700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 0x7f93f4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.790+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f940408e7b0 con 0x7f9408101020 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.790+0000 7f940d3f5700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 0x7f93f4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:01.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.791+0000 7f940d3f5700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 0x7f93f4070150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f94081046c0 tx=0x7f93f8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:01.795+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9404057d30 con 0x7f9408101020 2024-04-05T03:24:02.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.009+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9408104510 con 0x7f93f406dca0 2024-04-05T03:24:02.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.011+0000 7f93fe7fc700 1 -- 172.21.15.153:0/1474187901 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9408104510 con 0x7f93f406dca0 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 msgr2=0x7f93f4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 0x7f93f4070150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f94081046c0 tx=0x7f93f8009ee0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 msgr2=0x7f9408102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f940400b3b0 tx=0x7f9404009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 shutdown_connections 2024-04-05T03:24:02.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93f406dca0 0x7f93f4070150 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9408101020 0x7f9408102f90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 --2- 172.21.15.153:0/1474187901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94081019c0 0x7f94081034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 >> 172.21.15.153:0/1474187901 conn(0x7f94080f6bd0 msgr2=0x7f9408105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 shutdown_connections 2024-04-05T03:24:02.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.013+0000 7f940fe5a700 1 -- 172.21.15.153:0/1474187901 wait complete. 2024-04-05T03:24:02.021 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.116+0000 7f78c17c1700 1 -- 172.21.15.153:0/3592767513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc0fb540 msgr2=0x7f78bc0fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.116+0000 7f78c17c1700 1 --2- 172.21.15.153:0/3592767513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc0fb540 0x7f78bc0fb990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f78b0004940 tx=0x7f78b0009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 -- 172.21.15.153:0/3592767513 shutdown_connections 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 --2- 172.21.15.153:0/3592767513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc0fb540 0x7f78bc0fb990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 --2- 172.21.15.153:0/3592767513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc102fd0 0x7f78bc1033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 -- 172.21.15.153:0/3592767513 >> 172.21.15.153:0/3592767513 conn(0x7f78bc071530 msgr2=0x7f78bc071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 -- 172.21.15.153:0/3592767513 shutdown_connections 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 -- 172.21.15.153:0/3592767513 wait complete. 2024-04-05T03:24:02.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 Processor -- start 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 -- start start 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.117+0000 7f78c17c1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc0fb540 0x7f78bc113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78c17c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78c17c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc114310 con 0x7f78bc102fd0 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78c17c1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc10d960 con 0x7f78bc0fb540 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50618/0 (socket says 172.21.15.153:50618) 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 -- 172.21.15.153:0/4220658492 learned_addr learned my addr 172.21.15.153:0/4220658492 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78baffd700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc0fb540 0x7f78bc113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 -- 172.21.15.153:0/4220658492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc0fb540 msgr2=0x7f78bc113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc0fb540 0x7f78bc113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 -- 172.21.15.153:0/4220658492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b0005040 con 0x7f78bc102fd0 2024-04-05T03:24:02.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78ba7fc700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b3b0 tx=0x7f78b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b000e040 con 0x7f78bc102fd0 2024-04-05T03:24:02.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78b000bd40 con 0x7f78bc102fd0 2024-04-05T03:24:02.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc10dbe0 con 0x7f78bc102fd0 2024-04-05T03:24:02.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b001c6c0 con 0x7f78bc102fd0 2024-04-05T03:24:02.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.118+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc10e120 con 0x7f78bc102fd0 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.119+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc1053b0 con 0x7f78bc102fd0 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.120+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f78b0013420 con 0x7f78bc102fd0 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.120+0000 7f78a37fe700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 0x7f78a4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.120+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f78b008eb30 con 0x7f78bc102fd0 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.120+0000 7f78baffd700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 0x7f78a4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.120+0000 7f78baffd700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 0x7f78a4070150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f78ac007fd0 tx=0x7f78ac009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.125+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f78b005bd10 con 0x7f78bc102fd0 2024-04-05T03:24:02.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.337+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78bc10edd0 con 0x7f78a406dca0 2024-04-05T03:24:02.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.338+0000 7f78a37fe700 1 -- 172.21.15.153:0/4220658492 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f78bc10edd0 con 0x7f78a406dca0 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 msgr2=0x7f78a4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 0x7f78a4070150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f78ac007fd0 tx=0x7f78ac009040 comp rx=0 tx=0).stop 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 msgr2=0x7f78bc113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b3b0 tx=0x7f78b0009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 shutdown_connections 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f78a406dca0 0x7f78a4070150 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78bc0fb540 0x7f78bc113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 --2- 172.21.15.153:0/4220658492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f78bc102fd0 0x7f78bc113cc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 >> 172.21.15.153:0/4220658492 conn(0x7f78bc071530 msgr2=0x7f78bc0fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 shutdown_connections 2024-04-05T03:24:02.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.340+0000 7f78c17c1700 1 -- 172.21.15.153:0/4220658492 wait complete. 2024-04-05T03:24:02.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 -- 172.21.15.153:0/3762406059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 msgr2=0x7fa1240fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 --2- 172.21.15.153:0/3762406059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa1240fbd30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa118004940 tx=0x7fa118009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 -- 172.21.15.153:0/3762406059 shutdown_connections 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 --2- 172.21.15.153:0/3762406059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa1240fbd30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 --2- 172.21.15.153:0/3762406059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa1241020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 -- 172.21.15.153:0/3762406059 >> 172.21.15.153:0/3762406059 conn(0x7fa1240f7870 msgr2=0x7fa1240f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 -- 172.21.15.153:0/3762406059 shutdown_connections 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.442+0000 7fa12a454700 1 -- 172.21.15.153:0/3762406059 wait complete. 2024-04-05T03:24:02.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 Processor -- start 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 -- start start 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa124113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1241144f0 con 0x7fa1241026c0 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa12a454700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12410db40 con 0x7fa124101d20 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50640/0 (socket says 172.21.15.153:50640) 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 -- 172.21.15.153:0/2329436504 learned_addr learned my addr 172.21.15.153:0/2329436504 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:02.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa123fff700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa124113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 -- 172.21.15.153:0/2329436504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 msgr2=0x7fa124113a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa124113a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa1237fe700 1 -- 172.21.15.153:0/2329436504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118005040 con 0x7fa1241026c0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.443+0000 7fa123fff700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa124113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa1237fe700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa118009100 tx=0x7fa118009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa11800e040 con 0x7fa1241026c0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12410ddc0 con 0x7fa1241026c0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa118013420 con 0x7fa1241026c0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12410e380 con 0x7fa1241026c0 2024-04-05T03:24:02.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.444+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa11801c610 con 0x7fa1241026c0 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.445+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1240fd470 con 0x7fa1241026c0 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.445+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa118013590 con 0x7fa1241026c0 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.446+0000 7fa120ff9700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 0x7fa10c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.446+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa11808e6f0 con 0x7fa1241026c0 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.446+0000 7fa123fff700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 0x7fa10c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.447+0000 7fa123fff700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 0x7fa10c070150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa1140041d0 tx=0x7fa114009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.451+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa118057d90 con 0x7fa1241026c0 2024-04-05T03:24:02.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.658+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa124108e00 con 0x7fa10c06dca0 2024-04-05T03:24:02.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.668+0000 7fa120ff9700 1 -- 172.21.15.153:0/2329436504 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fa124108e00 con 0x7fa10c06dca0 2024-04-05T03:24:02.666 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (18m) 12m ago 29m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (16m) 12m ago 30m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (16m) 11m ago 28m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (18m) 12m ago 29m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (12m) 12m ago 24m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (12m) 12m ago 24m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (11m) 11m ago 24m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (11m) 11m ago 24m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (20m) 12m ago 32m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (19m) 11m ago 28m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (17m) 12m ago 32m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (16m) 11m ago 28m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (19m) 12m ago 29m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:24:02.667 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (19m) 11m ago 28m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (15m) 12m ago 27m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (15m) 12m ago 27m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (14m) 12m ago 26m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (13m) 11m ago 26m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (13m) 11m ago 26m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (12m) 11m ago 25m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:24:02.668 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (18m) 11m ago 28m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.670+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 msgr2=0x7fa10c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.670+0000 7fa12a454700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 0x7fa10c070150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa1140041d0 tx=0x7fa114009040 comp rx=0 tx=0).stop 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.670+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 msgr2=0x7fa124113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa118009100 tx=0x7fa118009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 shutdown_connections 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa10c06dca0 0x7fa10c070150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa124101d20 0x7fa124113a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 --2- 172.21.15.153:0/2329436504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1241026c0 0x7fa124113f40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 >> 172.21.15.153:0/2329436504 conn(0x7fa1240f7870 msgr2=0x7fa124066290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 shutdown_connections 2024-04-05T03:24:02.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.671+0000 7fa12a454700 1 -- 172.21.15.153:0/2329436504 wait complete. 2024-04-05T03:24:02.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.772+0000 7f918ea2c700 1 -- 172.21.15.153:0/2749949025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 msgr2=0x7f91880fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.772+0000 7f918ea2c700 1 --2- 172.21.15.153:0/2749949025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f91880fbfe0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9178004940 tx=0x7f9178009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 -- 172.21.15.153:0/2749949025 shutdown_connections 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 --2- 172.21.15.153:0/2749949025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 0x7f9188100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 --2- 172.21.15.153:0/2749949025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f91880fbfe0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 -- 172.21.15.153:0/2749949025 >> 172.21.15.153:0/2749949025 conn(0x7f91880f7850 msgr2=0x7f91880f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 -- 172.21.15.153:0/2749949025 shutdown_connections 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 -- 172.21.15.153:0/2749949025 wait complete. 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 Processor -- start 2024-04-05T03:24:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 -- start start 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.773+0000 7f918ea2c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50658/0 (socket says 172.21.15.153:50658) 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f918ea2c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 0x7f918811a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 -- 172.21.15.153:0/1644554210 learned_addr learned my addr 172.21.15.153:0/1644554210 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918811ab80 con 0x7f91880fbc10 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91881141b0 con 0x7f91880fc5b0 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 -- 172.21.15.153:0/1644554210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 msgr2=0x7f918811a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f91877fe700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 0x7f918811a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 0x7f918811a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:02.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9178005040 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9187fff700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f917800b3b0 tx=0x7f9178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917800e040 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f917800bbc0 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9188114430 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917801c3f0 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.774+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9188114970 con 0x7f91880fbc10 2024-04-05T03:24:02.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.775+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9188066b50 con 0x7f91880fbc10 2024-04-05T03:24:02.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.776+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f917800bd30 con 0x7f91880fbc10 2024-04-05T03:24:02.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.776+0000 7f9184ff9700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 0x7f9170070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:02.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.776+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f917808e830 con 0x7f91880fbc10 2024-04-05T03:24:02.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.776+0000 7f91877fe700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 0x7f9170070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:02.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.777+0000 7f91877fe700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 0x7f9170070150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9188115890 tx=0x7f917c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:02.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:02.781+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f917805b8e0 con 0x7f91880fbc10 2024-04-05T03:24:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:02 smithi203 ceph-mon[95988]: pgmap v551: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.058+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9188115640 con 0x7f91880fbc10 2024-04-05T03:24:03.056 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:02 smithi153 ceph-mon[106102]: pgmap v551: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.058+0000 7f9184ff9700 1 -- 172.21.15.153:0/1644554210 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f917805b470 con 0x7f91880fbc10 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:24:03.057 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:24:03.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:03.058 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:24:03.058 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:24:03.058 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:24:03.058 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:24:03.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 msgr2=0x7f9170070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 0x7f9170070150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9188115890 tx=0x7f917c00b040 comp rx=0 tx=0).stop 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 msgr2=0x7f9188119ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f917800b3b0 tx=0x7f9178009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 shutdown_connections 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f917006dca0 0x7f9170070150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91880fbc10 0x7f9188119ff0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 --2- 172.21.15.153:0/1644554210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91880fc5b0 0x7f918811a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 >> 172.21.15.153:0/1644554210 conn(0x7f91880f7850 msgr2=0x7f91880f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 shutdown_connections 2024-04-05T03:24:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.061+0000 7f918ea2c700 1 -- 172.21.15.153:0/1644554210 wait complete. 2024-04-05T03:24:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 -- 172.21.15.153:0/3573740357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 msgr2=0x7f7e940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3573740357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e940fbd80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7e84004940 tx=0x7f7e84009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 -- 172.21.15.153:0/3573740357 shutdown_connections 2024-04-05T03:24:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3573740357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e940fbd80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3573740357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 -- 172.21.15.153:0/3573740357 >> 172.21.15.153:0/3573740357 conn(0x7f7e94072bc0 msgr2=0x7f7e94072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 -- 172.21.15.153:0/3573740357 shutdown_connections 2024-04-05T03:24:03.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.162+0000 7f7e9b783700 1 -- 172.21.15.153:0/3573740357 wait complete. 2024-04-05T03:24:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 Processor -- start 2024-04-05T03:24:03.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 -- start start 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94116900 con 0x7f7e94102710 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9b783700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9410fd20 con 0x7f7e94101d70 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50666/0 (socket says 172.21.15.153:50666) 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 -- 172.21.15.153:0/3394569341 learned_addr learned my addr 172.21.15.153:0/3394569341 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9951f700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 -- 172.21.15.153:0/3394569341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 msgr2=0x7f7e94115d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94115d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 -- 172.21.15.153:0/3394569341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84005040 con 0x7f7e94102710 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e9951f700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e98d1e700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009100 tx=0x7f7e84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.163+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e8400e040 con 0x7f7e94102710 2024-04-05T03:24:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.164+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9410ffa0 con 0x7f7e94102710 2024-04-05T03:24:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.164+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e84013420 con 0x7f7e94102710 2024-04-05T03:24:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.164+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e8401c610 con 0x7f7e94102710 2024-04-05T03:24:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.164+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94110560 con 0x7f7e94102710 2024-04-05T03:24:03.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.165+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940fd4c0 con 0x7f7e94102710 2024-04-05T03:24:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.165+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7e84013590 con 0x7f7e94102710 2024-04-05T03:24:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.166+0000 7f7e89ffb700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 0x7f7e80070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.166+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7e8408e840 con 0x7f7e94102710 2024-04-05T03:24:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.166+0000 7f7e9951f700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 0x7f7e80070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.166+0000 7f7e9951f700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 0x7f7e80070150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e90004a10 tx=0x7f7e90009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.170+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7e8405b970 con 0x7f7e94102710 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.409+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7e9404bed0 con 0x7f7e94102710 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.409+0000 7f7e89ffb700 1 -- 172.21.15.153:0/3394569341 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f7e8408ce60 con 0x7f7e94102710 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:03.408 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:24:03.409 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:24:03.410 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:03.411 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:24:03.412 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:03.412 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:03.412 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:03.412 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:03.412 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 msgr2=0x7f7e80070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 0x7f7e80070150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e90004a10 tx=0x7f7e90009ce0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 msgr2=0x7f7e941162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009100 tx=0x7f7e84009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 shutdown_connections 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7e8006dca0 0x7f7e80070150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e94101d70 0x7f7e94115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 --2- 172.21.15.153:0/3394569341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e94102710 0x7f7e941162b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.412+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 >> 172.21.15.153:0/3394569341 conn(0x7f7e94072bc0 msgr2=0x7f7e940659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.413+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 shutdown_connections 2024-04-05T03:24:03.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.413+0000 7f7e9b783700 1 -- 172.21.15.153:0/3394569341 wait complete. 2024-04-05T03:24:03.414 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 -- 172.21.15.153:0/3573538191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc101d20 msgr2=0x7f45fc1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 --2- 172.21.15.153:0/3573538191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc101d20 0x7f45fc1020f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f45ec004940 tx=0x7f45ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 -- 172.21.15.153:0/3573538191 shutdown_connections 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 --2- 172.21.15.153:0/3573538191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc1026c0 0x7f45fc0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 --2- 172.21.15.153:0/3573538191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc101d20 0x7f45fc1020f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 -- 172.21.15.153:0/3573538191 >> 172.21.15.153:0/3573538191 conn(0x7f45fc0f7870 msgr2=0x7f45fc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 -- 172.21.15.153:0/3573538191 shutdown_connections 2024-04-05T03:24:03.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.514+0000 7f460121f700 1 -- 172.21.15.153:0/3573538191 wait complete. 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 Processor -- start 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 -- start start 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc101d20 0x7f45fc10eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc10f560 con 0x7f45fc1026c0 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f460121f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc10f6a0 con 0x7f45fc101d20 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.515+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50686/0 (socket says 172.21.15.153:50686) 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 -- 172.21.15.153:0/3550175430 learned_addr learned my addr 172.21.15.153:0/3550175430 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:03.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fad9d700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc101d20 0x7f45fc10eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 -- 172.21.15.153:0/3550175430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc101d20 msgr2=0x7f45fc10eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc101d20 0x7f45fc10eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 -- 172.21.15.153:0/3550175430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec005040 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45fa59c700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f45f0002a50 tx=0x7f45f00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45f0007cc0 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45f000f040 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc117b50 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45f0009630 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.516+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc117f20 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.517+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc0fd470 con 0x7f45fc1026c0 2024-04-05T03:24:03.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.518+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f45f0009790 con 0x7f45fc1026c0 2024-04-05T03:24:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.518+0000 7f45e37fe700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 0x7f45e4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.518+0000 7f45fad9d700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 0x7f45e4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.518+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f45f008d150 con 0x7f45fc1026c0 2024-04-05T03:24:03.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.519+0000 7f45fad9d700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 0x7f45e4070150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f45ec004f90 tx=0x7f45ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.523+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f45f0056800 con 0x7f45fc1026c0 2024-04-05T03:24:03.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.734+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45fc108ec0 con 0x7f45e406dca0 2024-04-05T03:24:03.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.735+0000 7f45e37fe700 1 -- 172.21.15.153:0/3550175430 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f45ec00b5c0 con 0x7f45e406dca0 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:24:03.734 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:24:03.735 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:24:03.735 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:24:03.735 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 msgr2=0x7f45e4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 0x7f45e4070150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f45ec004f90 tx=0x7f45ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 msgr2=0x7f45fc10f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f45f0002a50 tx=0x7f45f00046b0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 shutdown_connections 2024-04-05T03:24:03.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f45e406dca0 0x7f45e4070150 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45fc101d20 0x7f45fc10eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 --2- 172.21.15.153:0/3550175430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45fc1026c0 0x7f45fc10f020 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 >> 172.21.15.153:0/3550175430 conn(0x7f45fc0f7870 msgr2=0x7f45fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 shutdown_connections 2024-04-05T03:24:03.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.738+0000 7f460121f700 1 -- 172.21.15.153:0/3550175430 wait complete. 2024-04-05T03:24:03.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 -- 172.21.15.153:0/771948747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c201026c0 msgr2=0x7f5c200fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 --2- 172.21.15.153:0/771948747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c201026c0 0x7f5c200fbd30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c004940 tx=0x7f5c0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 -- 172.21.15.153:0/771948747 shutdown_connections 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 --2- 172.21.15.153:0/771948747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c201026c0 0x7f5c200fbd30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 --2- 172.21.15.153:0/771948747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c20101d20 0x7f5c201020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 -- 172.21.15.153:0/771948747 >> 172.21.15.153:0/771948747 conn(0x7f5c200f7870 msgr2=0x7f5c200f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 -- 172.21.15.153:0/771948747 shutdown_connections 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.839+0000 7f5c26e70700 1 -- 172.21.15.153:0/771948747 wait complete. 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 Processor -- start 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 -- start start 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c201026c0 0x7f5c2010ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20067120 con 0x7f5c20101d20 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c26e70700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2010d3d0 con 0x7f5c201026c0 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c24c0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c24c0c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50704/0 (socket says 172.21.15.153:50704) 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.840+0000 7f5c24c0c700 1 -- 172.21.15.153:0/346507788 learned_addr learned my addr 172.21.15.153:0/346507788 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c24c0c700 1 -- 172.21.15.153:0/346507788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c201026c0 msgr2=0x7f5c2010ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c1ffff700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c201026c0 0x7f5c2010ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c24c0c700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c201026c0 0x7f5c2010ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c24c0c700 1 -- 172.21.15.153:0/346507788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c005040 con 0x7f5c20101d20 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c24c0c700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5c14002a50 tx=0x7f5c140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c14007cc0 con 0x7f5c20101d20 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2010d6b0 con 0x7f5c20101d20 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c1400f040 con 0x7f5c20101d20 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c14009640 con 0x7f5c20101d20 2024-04-05T03:24:03.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.841+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2011a200 con 0x7f5c20101d20 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.842+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c200fd470 con 0x7f5c20101d20 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.842+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5c140097a0 con 0x7f5c20101d20 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.843+0000 7f5c1d7fa700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 0x7f5c10070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.843+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5c1408d3a0 con 0x7f5c20101d20 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.843+0000 7f5c1ffff700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 0x7f5c10070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:03.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.843+0000 7f5c1ffff700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 0x7f5c10070150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009fd0 tx=0x7f5c0c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:03.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:03.847+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5c1405a3f0 con 0x7f5c20101d20 2024-04-05T03:24:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:03 smithi203 ceph-mon[95988]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:03 smithi203 ceph-mon[95988]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:03 smithi203 ceph-mon[95988]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:03 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1644554210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:24:04.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:03 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3394569341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:24:04.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:03 smithi153 ceph-mon[106102]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:03 smithi153 ceph-mon[106102]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:03 smithi153 ceph-mon[106102]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:04.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:03 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1644554210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:24:04.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:03 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3394569341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:24:04.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.124+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5c2011a390 con 0x7f5c20101d20 2024-04-05T03:24:04.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.124+0000 7f5c1d7fa700 1 -- 172.21.15.153:0/346507788 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5c14059f80 con 0x7f5c20101d20 2024-04-05T03:24:04.123 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:24:04.123 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:24:04.123 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:24:04.123 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:24:04.123 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 msgr2=0x7f5c10070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 0x7f5c10070150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009fd0 tx=0x7f5c0c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 msgr2=0x7f5c2010c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5c14002a50 tx=0x7f5c140046b0 comp rx=0 tx=0).stop 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 shutdown_connections 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5c1006dca0 0x7f5c10070150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c20101d20 0x7f5c2010c950 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 --2- 172.21.15.153:0/346507788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c201026c0 0x7f5c2010ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:04.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.126+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 >> 172.21.15.153:0/346507788 conn(0x7f5c200f7870 msgr2=0x7f5c200f8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:04.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.127+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 shutdown_connections 2024-04-05T03:24:04.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:04.127+0000 7f5c26e70700 1 -- 172.21.15.153:0/346507788 wait complete. 2024-04-05T03:24:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:04 smithi203 ceph-mon[95988]: pgmap v552: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:04 smithi203 ceph-mon[95988]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:04 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/346507788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:24:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:04 smithi153 ceph-mon[106102]: pgmap v552: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:04 smithi153 ceph-mon[106102]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:05.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:04 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/346507788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:24:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:06 smithi203 ceph-mon[95988]: pgmap v553: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:07.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:06 smithi153 ceph-mon[106102]: pgmap v553: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:08 smithi203 ceph-mon[95988]: pgmap v554: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:08 smithi153 ceph-mon[106102]: pgmap v554: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:10 smithi203 ceph-mon[95988]: pgmap v555: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:11.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:10 smithi153 ceph-mon[106102]: pgmap v555: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:12 smithi203 ceph-mon[95988]: pgmap v556: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:12 smithi153 ceph-mon[106102]: pgmap v556: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:13.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:14 smithi203 ceph-mon[95988]: pgmap v557: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:14 smithi153 ceph-mon[106102]: pgmap v557: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:16.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:15 smithi153 ceph-mon[106102]: pgmap v558: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:16.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:15 smithi203 ceph-mon[95988]: pgmap v558: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:18 smithi203 ceph-mon[95988]: pgmap v559: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:19.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:18 smithi153 ceph-mon[106102]: pgmap v559: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:20 smithi203 ceph-mon[95988]: pgmap v560: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:21.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:20 smithi153 ceph-mon[106102]: pgmap v560: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:23.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:22 smithi203 ceph-mon[95988]: pgmap v561: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:22 smithi153 ceph-mon[106102]: pgmap v561: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:24 smithi203 ceph-mon[95988]: pgmap v562: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:24 smithi153 ceph-mon[106102]: pgmap v562: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:26 smithi203 ceph-mon[95988]: pgmap v563: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:26 smithi153 ceph-mon[106102]: pgmap v563: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:28.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:28 smithi203 ceph-mon[95988]: pgmap v564: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:28 smithi153 ceph-mon[106102]: pgmap v564: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:29 smithi203 ceph-mon[95988]: pgmap v565: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:29 smithi153 ceph-mon[106102]: pgmap v565: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:32 smithi203 ceph-mon[95988]: pgmap v566: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:32 smithi153 ceph-mon[106102]: pgmap v566: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 -- 172.21.15.153:0/4265011391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3080fc5b0 msgr2=0x7fb3080fca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 --2- 172.21.15.153:0/4265011391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3080fc5b0 0x7fb3080fca20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8004940 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 -- 172.21.15.153:0/4265011391 shutdown_connections 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 --2- 172.21.15.153:0/4265011391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb3080fc5b0 0x7fb3080fca20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 --2- 172.21.15.153:0/4265011391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb308101d70 0x7fb308102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 -- 172.21.15.153:0/4265011391 >> 172.21.15.153:0/4265011391 conn(0x7fb3080714c0 msgr2=0x7fb3080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 -- 172.21.15.153:0/4265011391 shutdown_connections 2024-04-05T03:24:34.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.241+0000 7fb310199700 1 -- 172.21.15.153:0/4265011391 wait complete. 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 Processor -- start 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 -- start start 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080fc5b0 0x7fb308113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb308114720 con 0x7fb308101d70 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb310199700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30810db30 con 0x7fb3080fc5b0 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb30d734700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb30d734700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33334/0 (socket says 172.21.15.153:33334) 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb30d734700 1 -- 172.21.15.153:0/4284714938 learned_addr learned my addr 172.21.15.153:0/4284714938 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.242+0000 7fb30df35700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080fc5b0 0x7fb308113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb30d734700 1 -- 172.21.15.153:0/4284714938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080fc5b0 msgr2=0x7fb308113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb30d734700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080fc5b0 0x7fb308113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb30d734700 1 -- 172.21.15.153:0/4284714938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8005040 con 0x7fb308101d70 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb30d734700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009150 tx=0x7fb2f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2f800e040 con 0x7fb308101d70 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2f8013420 con 0x7fb308101d70 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30810ddb0 con 0x7fb308101d70 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2f801c610 con 0x7fb308101d70 2024-04-05T03:24:34.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.243+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30810e370 con 0x7fb308101d70 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.244+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3080fd5c0 con 0x7fb308101d70 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.244+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb2f800baf0 con 0x7fb308101d70 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.245+0000 7fb2fe7fc700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 0x7fb2f4070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.245+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb2f808e760 con 0x7fb308101d70 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.245+0000 7fb30df35700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 0x7fb2f4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.245+0000 7fb30df35700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 0x7fb2f4070160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb308064f70 tx=0x7fb30400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.250+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb2f8057d60 con 0x7fb308101d70 2024-04-05T03:24:34.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.463+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb30810f280 con 0x7fb2f406dcb0 2024-04-05T03:24:34.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.465+0000 7fb2fe7fc700 1 -- 172.21.15.153:0/4284714938 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb30810f280 con 0x7fb2f406dcb0 2024-04-05T03:24:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 msgr2=0x7fb2f4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 0x7fb2f4070160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb308064f70 tx=0x7fb30400b040 comp rx=0 tx=0).stop 2024-04-05T03:24:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 msgr2=0x7fb3081140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009150 tx=0x7fb2f8009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:34.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 shutdown_connections 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb2f406dcb0 0x7fb2f4070160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3080fc5b0 0x7fb308113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 --2- 172.21.15.153:0/4284714938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308101d70 0x7fb3081140d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 >> 172.21.15.153:0/4284714938 conn(0x7fb3080714c0 msgr2=0x7fb308104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 shutdown_connections 2024-04-05T03:24:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.467+0000 7fb310199700 1 -- 172.21.15.153:0/4284714938 wait complete. 2024-04-05T03:24:34.475 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.570+0000 7f36a401e700 1 -- 172.21.15.153:0/1475841864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fe770 msgr2=0x7f369c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.570+0000 7f36a401e700 1 --2- 172.21.15.153:0/1475841864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fe770 0x7f369c106e40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f368c004940 tx=0x7f368c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 -- 172.21.15.153:0/1475841864 shutdown_connections 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 --2- 172.21.15.153:0/1475841864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fe770 0x7f369c106e40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 --2- 172.21.15.153:0/1475841864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fddd0 0x7f369c0fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 -- 172.21.15.153:0/1475841864 >> 172.21.15.153:0/1475841864 conn(0x7f369c0f7870 msgr2=0x7f369c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 -- 172.21.15.153:0/1475841864 shutdown_connections 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 -- 172.21.15.153:0/1475841864 wait complete. 2024-04-05T03:24:34.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 Processor -- start 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.571+0000 7f36a401e700 1 -- start start 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 0x7f369c0fc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0fdbe0 con 0x7f369c0fddd0 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0fc710 con 0x7f369c0fe770 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33348/0 (socket says 172.21.15.153:33348) 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 -- 172.21.15.153:0/1910879159 learned_addr learned my addr 172.21.15.153:0/1910879159 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a15b9700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 0x7f369c0fc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 -- 172.21.15.153:0/1910879159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 msgr2=0x7f369c0fc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 0x7f369c0fc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 -- 172.21.15.153:0/1910879159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c005040 con 0x7f369c0fddd0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a15b9700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 0x7f369c0fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a1dba700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f369800c8c0 tx=0x7f36980044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f369800a980 con 0x7f369c0fddd0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c0fc9f0 con 0x7f369c0fddd0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f369800ce60 con 0x7f369c0fddd0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36980115e0 con 0x7f369c0fddd0 2024-04-05T03:24:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.572+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c11a150 con 0x7f369c0fddd0 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.573+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c066b50 con 0x7f369c0fddd0 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.574+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3698012400 con 0x7f369c0fddd0 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.574+0000 7f36927fc700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 0x7f3688070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.575+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f369808d670 con 0x7f369c0fddd0 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.575+0000 7f36a15b9700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 0x7f3688070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.575+0000 7f36a15b9700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 0x7f3688070150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f369c0fd3c0 tx=0x7f368c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.579+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3698056c70 con 0x7f369c0fddd0 2024-04-05T03:24:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.793+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f369c0fd210 con 0x7f368806dca0 2024-04-05T03:24:34.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.795+0000 7f36927fc700 1 -- 172.21.15.153:0/1910879159 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f369c0fd210 con 0x7f368806dca0 2024-04-05T03:24:34.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 msgr2=0x7f3688070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 0x7f3688070150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f369c0fd3c0 tx=0x7f368c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 msgr2=0x7f369c0fbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f369800c8c0 tx=0x7f36980044c0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 shutdown_connections 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f368806dca0 0x7f3688070150 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f369c0fddd0 0x7f369c0fbc90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 --2- 172.21.15.153:0/1910879159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f369c0fe770 0x7f369c0fc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 >> 172.21.15.153:0/1910879159 conn(0x7f369c0f7870 msgr2=0x7f369c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 shutdown_connections 2024-04-05T03:24:34.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.797+0000 7f36a401e700 1 -- 172.21.15.153:0/1910879159 wait complete. 2024-04-05T03:24:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:34 smithi153 ceph-mon[106102]: pgmap v567: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- 172.21.15.153:0/4187391873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c0fb4d0 msgr2=0x7f527c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 --2- 172.21.15.153:0/4187391873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c0fb4d0 0x7f527c0fb940 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5264004940 tx=0x7f5264009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- 172.21.15.153:0/4187391873 shutdown_connections 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 --2- 172.21.15.153:0/4187391873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c0fb4d0 0x7f527c0fb940 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 --2- 172.21.15.153:0/4187391873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c102f50 0x7f527c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- 172.21.15.153:0/4187391873 >> 172.21.15.153:0/4187391873 conn(0x7f527c0714c0 msgr2=0x7f527c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- 172.21.15.153:0/4187391873 shutdown_connections 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- 172.21.15.153:0/4187391873 wait complete. 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 Processor -- start 2024-04-05T03:24:34.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.899+0000 7f52821a0700 1 -- start start 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c0fb4d0 0x7f527c1139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c114530 con 0x7f527c102f50 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c10db50 con 0x7f527c0fb4d0 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33376/0 (socket says 172.21.15.153:33376) 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 -- 172.21.15.153:0/617970196 learned_addr learned my addr 172.21.15.153:0/617970196 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f527b7fe700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c0fb4d0 0x7f527c1139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 -- 172.21.15.153:0/617970196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c0fb4d0 msgr2=0x7f527c1139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c0fb4d0 0x7f527c1139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 -- 172.21.15.153:0/617970196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5264005040 con 0x7f527c102f50 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5272ffd700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f526400b3b0 tx=0x7f5264009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526400e040 con 0x7f527c102f50 2024-04-05T03:24:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c10ddd0 con 0x7f527c102f50 2024-04-05T03:24:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5264013420 con 0x7f527c102f50 2024-04-05T03:24:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f526401c630 con 0x7f527c102f50 2024-04-05T03:24:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.900+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c10e310 con 0x7f527c102f50 2024-04-05T03:24:34.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.901+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c105330 con 0x7f527c102f50 2024-04-05T03:24:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.902+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5264013590 con 0x7f527c102f50 2024-04-05T03:24:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.902+0000 7f5278ff9700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 0x7f5268070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.902+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f526408e6b0 con 0x7f527c102f50 2024-04-05T03:24:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.902+0000 7f527b7fe700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 0x7f5268070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:34.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.903+0000 7f527b7fe700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 0x7f5268070150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f527c065c10 tx=0x7f526c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:34.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:34.907+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5264057d50 con 0x7f527c102f50 2024-04-05T03:24:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:34 smithi203 ceph-mon[95988]: pgmap v567: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:35.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.109+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f527c04c7b0 con 0x7f526806dca0 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.118+0000 7f5278ff9700 1 -- 172.21.15.153:0/617970196 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f527c04c7b0 con 0x7f526806dca0 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (19m) 12m ago 30m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (16m) 12m ago 31m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (16m) 12m ago 29m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (18m) 12m ago 30m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (12m) 12m ago 24m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (13m) 12m ago 24m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (12m) 12m ago 24m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (12m) 12m ago 24m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:24:35.117 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (20m) 12m ago 32m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (20m) 12m ago 29m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (17m) 12m ago 32m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (17m) 12m ago 29m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (19m) 12m ago 30m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (19m) 12m ago 29m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (16m) 12m ago 28m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (15m) 12m ago 27m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (15m) 12m ago 27m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (14m) 12m ago 27m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (13m) 12m ago 26m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (13m) 12m ago 26m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:24:35.118 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (19m) 12m ago 28m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.120+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 msgr2=0x7f5268070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.120+0000 7f52821a0700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 0x7f5268070150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f527c065c10 tx=0x7f526c00b040 comp rx=0 tx=0).stop 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.120+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 msgr2=0x7f527c113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.120+0000 7f52821a0700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f526400b3b0 tx=0x7f5264009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 shutdown_connections 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f526806dca0 0x7f5268070150 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f527c0fb4d0 0x7f527c1139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 --2- 172.21.15.153:0/617970196 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f527c102f50 0x7f527c113ee0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 >> 172.21.15.153:0/617970196 conn(0x7f527c0714c0 msgr2=0x7f527c0fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 shutdown_connections 2024-04-05T03:24:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.121+0000 7f52821a0700 1 -- 172.21.15.153:0/617970196 wait complete. 2024-04-05T03:24:35.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/2076418052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed4101d20 msgr2=0x7f1ed41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/2076418052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed4101d20 0x7f1ed41020f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4004940 tx=0x7f1ec4009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/2076418052 shutdown_connections 2024-04-05T03:24:35.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/2076418052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed41026c0 0x7f1ed40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/2076418052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed4101d20 0x7f1ed41020f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.220+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/2076418052 >> 172.21.15.153:0/2076418052 conn(0x7f1ed40f7870 msgr2=0x7f1ed40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.221+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/2076418052 shutdown_connections 2024-04-05T03:24:35.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.221+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/2076418052 wait complete. 2024-04-05T03:24:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.221+0000 7f1ed9b0b700 1 Processor -- start 2024-04-05T03:24:35.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.221+0000 7f1ed9b0b700 1 -- start start 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed9b0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed4101d20 0x7f1ed41128d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed9b0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed9b0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed4113460 con 0x7f1ed41026c0 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed9b0b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed410ca60 con 0x7f1ed4101d20 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33394/0 (socket says 172.21.15.153:33394) 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 -- 172.21.15.153:0/1989057976 learned_addr learned my addr 172.21.15.153:0/1989057976 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed37fe700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed4101d20 0x7f1ed41128d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 -- 172.21.15.153:0/1989057976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed4101d20 msgr2=0x7f1ed41128d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed4101d20 0x7f1ed41128d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 -- 172.21.15.153:0/1989057976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4005040 con 0x7f1ed41026c0 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.222+0000 7f1ed2ffd700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800c7e0 tx=0x7f1ec8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.223+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec8012810 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.223+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed410cd40 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.223+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ec8012e40 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.223+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec80102e0 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.223+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed410d300 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.224+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed40fd470 con 0x7f1ed41026c0 2024-04-05T03:24:35.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.225+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1ec8011380 con 0x7f1ed41026c0 2024-04-05T03:24:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.225+0000 7f1ebbfff700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 0x7f1ebc070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.225+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1ec808d8f0 con 0x7f1ed41026c0 2024-04-05T03:24:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.225+0000 7f1ed37fe700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 0x7f1ebc070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.226+0000 7f1ed37fe700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 0x7f1ebc070150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3b0 tx=0x7f1ec4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.232+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1ec805a9a0 con 0x7f1ed41026c0 2024-04-05T03:24:35.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.503+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1ed4108e40 con 0x7f1ed41026c0 2024-04-05T03:24:35.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.504+0000 7f1ebbfff700 1 -- 172.21.15.153:0/1989057976 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1ec805a530 con 0x7f1ed41026c0 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:24:35.503 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:24:35.504 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:24:35.504 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:24:35.504 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:24:35.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 msgr2=0x7f1ebc070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 0x7f1ebc070150 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3b0 tx=0x7f1ec4009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 msgr2=0x7f1ed4112e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1ec800c7e0 tx=0x7f1ec8007f90 comp rx=0 tx=0).stop 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 shutdown_connections 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1ebc06dca0 0x7f1ebc070150 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed4101d20 0x7f1ed41128d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 --2- 172.21.15.153:0/1989057976 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ed41026c0 0x7f1ed4112e10 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 >> 172.21.15.153:0/1989057976 conn(0x7f1ed40f7870 msgr2=0x7f1ed40f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 shutdown_connections 2024-04-05T03:24:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.506+0000 7f1ed9b0b700 1 -- 172.21.15.153:0/1989057976 wait complete. 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- 172.21.15.153:0/2046873180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd550102e20 msgr2=0x7fd5501031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 --2- 172.21.15.153:0/2046873180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd550102e20 0x7fd5501031f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd540004940 tx=0x7fd540009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- 172.21.15.153:0/2046873180 shutdown_connections 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 --2- 172.21.15.153:0/2046873180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5501037c0 0x7fd5500fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 --2- 172.21.15.153:0/2046873180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd550102e20 0x7fd5501031f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- 172.21.15.153:0/2046873180 >> 172.21.15.153:0/2046873180 conn(0x7fd5500f89b0 msgr2=0x7fd5500fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- 172.21.15.153:0/2046873180 shutdown_connections 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- 172.21.15.153:0/2046873180 wait complete. 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 Processor -- start 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 -- start start 2024-04-05T03:24:35.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.608+0000 7fd555e55700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 0x7fd550113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd555e55700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd555e55700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550114500 con 0x7fd5501037c0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd555e55700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55010db50 con 0x7fd550102e20 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33422/0 (socket says 172.21.15.153:33422) 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 -- 172.21.15.153:0/2336027850 learned_addr learned my addr 172.21.15.153:0/2336027850 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd54f7fe700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 0x7fd550113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 -- 172.21.15.153:0/2336027850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 msgr2=0x7fd550113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 0x7fd550113970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 -- 172.21.15.153:0/2336027850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540005040 con 0x7fd5501037c0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd54f7fe700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 0x7fd550113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd547fff700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd538002a50 tx=0x7fd538004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd538007cc0 con 0x7fd5501037c0 2024-04-05T03:24:35.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd53800f040 con 0x7fd5501037c0 2024-04-05T03:24:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd53800daf0 con 0x7fd5501037c0 2024-04-05T03:24:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55010de30 con 0x7fd5501037c0 2024-04-05T03:24:35.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.609+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55010e3f0 con 0x7fd5501037c0 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.610+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550100130 con 0x7fd5501037c0 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.611+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd538009810 con 0x7fd5501037c0 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.611+0000 7fd54cff9700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 0x7fd53c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.611+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd53808d230 con 0x7fd5501037c0 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.611+0000 7fd54f7fe700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 0x7fd53c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.612+0000 7fd54f7fe700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 0x7fd53c070150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd540004560 tx=0x7fd540009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.616+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd53805a360 con 0x7fd5501037c0 2024-04-05T03:24:35.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:35 smithi153 ceph-mon[106102]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:35.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:35 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1989057976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:24:35.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.864+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd5500fe570 con 0x7fd5501037c0 2024-04-05T03:24:35.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.864+0000 7fd54cff9700 1 -- 172.21.15.153:0/2336027850 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fd538059ef0 con 0x7fd5501037c0 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:24:35.863 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:24:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:35.865 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 msgr2=0x7fd53c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 0x7fd53c070150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd540004560 tx=0x7fd540009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 msgr2=0x7fd550113eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd538002a50 tx=0x7fd538004600 comp rx=0 tx=0).stop 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 shutdown_connections 2024-04-05T03:24:35.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd53c06dca0 0x7fd53c070150 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd550102e20 0x7fd550113970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 --2- 172.21.15.153:0/2336027850 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd5501037c0 0x7fd550113eb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 >> 172.21.15.153:0/2336027850 conn(0x7fd5500f89b0 msgr2=0x7fd550107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 shutdown_connections 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.866+0000 7fd555e55700 1 -- 172.21.15.153:0/2336027850 wait complete. 2024-04-05T03:24:35.867 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/1426975499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b781026a0 msgr2=0x7f6b780fbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/1426975499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b781026a0 0x7f6b780fbd10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c004940 tx=0x7f6b6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/1426975499 shutdown_connections 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/1426975499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b781026a0 0x7f6b780fbd10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/1426975499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b78101d00 0x7f6b781020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/1426975499 >> 172.21.15.153:0/1426975499 conn(0x7f6b780f78d0 msgr2=0x7f6b780f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/1426975499 shutdown_connections 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/1426975499 wait complete. 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 Processor -- start 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.968+0000 7f6b7d1b5700 1 -- start start 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b781026a0 0x7f6b781140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b78114710 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7810db20 con 0x7f6b781026a0 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33430/0 (socket says 172.21.15.153:33430) 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 -- 172.21.15.153:0/2952048643 learned_addr learned my addr 172.21.15.153:0/2952048643 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7659c700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b781026a0 0x7f6b781140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 -- 172.21.15.153:0/2952048643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b781026a0 msgr2=0x7f6b781140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b781026a0 0x7f6b781140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 -- 172.21.15.153:0/2952048643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c005040 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b76d9d700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c8c0 tx=0x7f6b680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b6800a980 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7810de00 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b6800ce60 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b680115e0 con 0x7f6b78101d00 2024-04-05T03:24:35.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.969+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7810e3c0 con 0x7f6b78101d00 2024-04-05T03:24:35.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.970+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b780fd450 con 0x7f6b78101d00 2024-04-05T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.971+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6b68012400 con 0x7f6b78101d00 2024-04-05T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.971+0000 7f6b5f7fe700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 0x7f6b60070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.971+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6b6808d670 con 0x7f6b78101d00 2024-04-05T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.971+0000 7f6b7659c700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 0x7f6b60070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:35.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.972+0000 7f6b7659c700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 0x7f6b60070150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000c00 tx=0x7f6b6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:35.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:35.976+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6b68056c70 con 0x7f6b78101d00 2024-04-05T03:24:36.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:35 smithi203 ceph-mon[95988]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:36.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:35 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1989057976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:24:36.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.186+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b7804bed0 con 0x7f6b6006dca0 2024-04-05T03:24:36.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.187+0000 7f6b5f7fe700 1 -- 172.21.15.153:0/2952048643 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6b7804bed0 con 0x7f6b6006dca0 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:24:36.186 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:24:36.187 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:24:36.187 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:24:36.187 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:24:36.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.189+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 msgr2=0x7f6b60070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.189+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 0x7f6b60070150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c000c00 tx=0x7f6b6c009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:36.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 msgr2=0x7f6b78113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c8c0 tx=0x7f6b680044c0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 shutdown_connections 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b6006dca0 0x7f6b60070150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b78101d00 0x7f6b78113b80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 --2- 172.21.15.153:0/2952048643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b781026a0 0x7f6b781140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 >> 172.21.15.153:0/2952048643 conn(0x7f6b780f78d0 msgr2=0x7f6b780f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 shutdown_connections 2024-04-05T03:24:36.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.190+0000 7f6b7d1b5700 1 -- 172.21.15.153:0/2952048643 wait complete. 2024-04-05T03:24:36.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 -- 172.21.15.153:0/3055032453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 msgr2=0x7fe5940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 --2- 172.21.15.153:0/3055032453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe5940fbd80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe588004940 tx=0x7fe588009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 -- 172.21.15.153:0/3055032453 shutdown_connections 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 --2- 172.21.15.153:0/3055032453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe5940fbd80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 --2- 172.21.15.153:0/3055032453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 0x7fe594102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 -- 172.21.15.153:0/3055032453 >> 172.21.15.153:0/3055032453 conn(0x7fe594072bc0 msgr2=0x7fe594072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 -- 172.21.15.153:0/3055032453 shutdown_connections 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 -- 172.21.15.153:0/3055032453 wait complete. 2024-04-05T03:24:36.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.292+0000 7fe59a063700 1 Processor -- start 2024-04-05T03:24:36.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe59a063700 1 -- start start 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe59a063700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 0x7fe59410eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe59a063700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe59a063700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59410f6b0 con 0x7fe594102710 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe59a063700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59410f7f0 con 0x7fe594101d70 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33442/0 (socket says 172.21.15.153:33442) 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 -- 172.21.15.153:0/3122735568 learned_addr learned my addr 172.21.15.153:0/3122735568 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe5937fe700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 0x7fe59410eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 -- 172.21.15.153:0/3122735568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 msgr2=0x7fe59410eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 0x7fe59410eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 -- 172.21.15.153:0/3122735568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe588005040 con 0x7fe594102710 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.293+0000 7fe592ffd700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe588009100 tx=0x7fe588009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:36.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.294+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58800e040 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.294+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe588009af0 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.294+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594117b90 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.294+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58801c3f0 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.294+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5941180a0 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.295+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe594104240 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.295+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe58800baf0 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.296+0000 7fe57bfff700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 0x7fe57c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.296+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe58808e790 con 0x7fe594102710 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.296+0000 7fe5937fe700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 0x7fe57c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:24:36.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.296+0000 7fe5937fe700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 0x7fe57c070150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe584004ed0 tx=0x7fe58400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:24:36.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.300+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe588057d90 con 0x7fe594102710 2024-04-05T03:24:36.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.585+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe594108e70 con 0x7fe594102710 2024-04-05T03:24:36.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.586+0000 7fe57bfff700 1 -- 172.21.15.153:0/3122735568 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fe58805b450 con 0x7fe594102710 2024-04-05T03:24:36.584 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:24:36.584 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:24:36.584 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:24:36.585 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:24:36.585 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:24:36.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 msgr2=0x7fe57c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 0x7fe57c070150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe584004ed0 tx=0x7fe58400b040 comp rx=0 tx=0).stop 2024-04-05T03:24:36.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 msgr2=0x7fe59410f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:24:36.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe588009100 tx=0x7fe588009e30 comp rx=0 tx=0).stop 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 shutdown_connections 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe57c06dca0 0x7fe57c070150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe594101d70 0x7fe59410eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 --2- 172.21.15.153:0/3122735568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe594102710 0x7fe59410f060 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 >> 172.21.15.153:0/3122735568 conn(0x7fe594072bc0 msgr2=0x7fe5940fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 shutdown_connections 2024-04-05T03:24:36.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:24:36.588+0000 7fe59a063700 1 -- 172.21.15.153:0/3122735568 wait complete. 2024-04-05T03:24:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:36 smithi203 ceph-mon[95988]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:36 smithi203 ceph-mon[95988]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:36 smithi203 ceph-mon[95988]: pgmap v568: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:36 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2336027850' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:24:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:36 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3122735568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:24:37.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:36 smithi153 ceph-mon[106102]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:36 smithi153 ceph-mon[106102]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:36 smithi153 ceph-mon[106102]: pgmap v568: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:36 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2336027850' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:24:37.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:36 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3122735568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:24:38.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:37 smithi203 ceph-mon[95988]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:38.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:37 smithi153 ceph-mon[106102]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:24:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:38 smithi203 ceph-mon[95988]: pgmap v569: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:38 smithi153 ceph-mon[106102]: pgmap v569: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:40.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:39 smithi203 ceph-mon[95988]: pgmap v570: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:40.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:39 smithi153 ceph-mon[106102]: pgmap v570: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:42 smithi203 ceph-mon[95988]: pgmap v571: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:42 smithi153 ceph-mon[106102]: pgmap v571: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:45.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:44 smithi203 ceph-mon[95988]: pgmap v572: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:45.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:44 smithi153 ceph-mon[106102]: pgmap v572: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:46 smithi203 ceph-mon[95988]: pgmap v573: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:46 smithi153 ceph-mon[106102]: pgmap v573: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:48 smithi203 ceph-mon[95988]: pgmap v574: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:49.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:48 smithi153 ceph-mon[106102]: pgmap v574: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:50 smithi203 ceph-mon[95988]: pgmap v575: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:50 smithi153 ceph-mon[106102]: pgmap v575: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:53.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:52 smithi203 ceph-mon[95988]: pgmap v576: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:52 smithi153 ceph-mon[106102]: pgmap v576: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:54 smithi203 ceph-mon[95988]: pgmap v577: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:54 smithi153 ceph-mon[106102]: pgmap v577: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:56.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:55 smithi153 ceph-mon[106102]: pgmap v578: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:56.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:55 smithi203 ceph-mon[95988]: pgmap v578: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:57.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:24:58.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:24:58 smithi153 ceph-mon[106102]: pgmap v579: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:24:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:24:58 smithi203 ceph-mon[95988]: pgmap v579: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:00 smithi203 ceph-mon[95988]: pgmap v580: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:01.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:00 smithi153 ceph-mon[106102]: pgmap v580: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:02 smithi203 ceph-mon[95988]: pgmap v581: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:02 smithi153 ceph-mon[106102]: pgmap v581: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:04 smithi203 ceph-mon[95988]: pgmap v582: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:04 smithi153 ceph-mon[106102]: pgmap v582: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:06.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- 172.21.15.153:0/1323800970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd080fb4d0 msgr2=0x7efd080fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:06.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 --2- 172.21.15.153:0/1323800970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd080fb4d0 0x7efd080fb940 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efcf8004940 tx=0x7efcf8009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- 172.21.15.153:0/1323800970 shutdown_connections 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 --2- 172.21.15.153:0/1323800970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd080fb4d0 0x7efd080fb940 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 --2- 172.21.15.153:0/1323800970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd08102f50 0x7efd08103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- 172.21.15.153:0/1323800970 >> 172.21.15.153:0/1323800970 conn(0x7efd080714c0 msgr2=0x7efd080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- 172.21.15.153:0/1323800970 shutdown_connections 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- 172.21.15.153:0/1323800970 wait complete. 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 Processor -- start 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.703+0000 7efd101f2700 1 -- start start 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd101f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd080fb4d0 0x7efd08113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd101f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd101f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd081145b0 con 0x7efd08102f50 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd101f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0810db90 con 0x7efd080fb4d0 2024-04-05T03:25:06.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53616/0 (socket says 172.21.15.153:53616) 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 -- 172.21.15.153:0/2511933658 learned_addr learned my addr 172.21.15.153:0/2511933658 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0df8e700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd080fb4d0 0x7efd08113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 -- 172.21.15.153:0/2511933658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd080fb4d0 msgr2=0x7efd08113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd080fb4d0 0x7efd08113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 -- 172.21.15.153:0/2511933658 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf8005040 con 0x7efd08102f50 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efd0d78d700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcf800b8b0 tx=0x7efcf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.704+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcf800e040 con 0x7efd08102f50 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.705+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0810de10 con 0x7efd08102f50 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.705+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcf8004290 con 0x7efd08102f50 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.705+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcf801c3f0 con 0x7efd08102f50 2024-04-05T03:25:06.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.705+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0810e350 con 0x7efd08102f50 2024-04-05T03:25:06.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.705+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08100200 con 0x7efd08102f50 2024-04-05T03:25:06.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.706+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efcf80137b0 con 0x7efd08102f50 2024-04-05T03:25:06.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.707+0000 7efcfe7fc700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 0x7efcf4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:06.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.707+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efcf808f320 con 0x7efd08102f50 2024-04-05T03:25:06.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.707+0000 7efd0df8e700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 0x7efcf4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:06.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.707+0000 7efd0df8e700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 0x7efcf4070150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd08065c10 tx=0x7efd04009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:06.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.711+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efcf805c4f0 con 0x7efd08102f50 2024-04-05T03:25:06.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.925+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efd0810f0e0 con 0x7efcf406dca0 2024-04-05T03:25:06.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.927+0000 7efcfe7fc700 1 -- 172.21.15.153:0/2511933658 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efd0810f0e0 con 0x7efcf406dca0 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 msgr2=0x7efcf4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 0x7efcf4070150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd08065c10 tx=0x7efd04009ee0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 msgr2=0x7efd08113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcf800b8b0 tx=0x7efcf8009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 shutdown_connections 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efcf406dca0 0x7efcf4070150 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd080fb4d0 0x7efd08113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 --2- 172.21.15.153:0/2511933658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd08102f50 0x7efd08113f60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:06.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 >> 172.21.15.153:0/2511933658 conn(0x7efd080714c0 msgr2=0x7efd080fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:06.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 shutdown_connections 2024-04-05T03:25:06.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:06.929+0000 7efd101f2700 1 -- 172.21.15.153:0/2511933658 wait complete. 2024-04-05T03:25:06.937 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:25:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 -- 172.21.15.153:0/3102412559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c102e00 msgr2=0x7f608c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 --2- 172.21.15.153:0/3102412559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c102e00 0x7f608c1031d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6080004940 tx=0x7f6080009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 -- 172.21.15.153:0/3102412559 shutdown_connections 2024-04-05T03:25:07.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 --2- 172.21.15.153:0/3102412559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c1037a0 0x7f608c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:06 smithi153 ceph-mon[106102]: pgmap v583: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:07.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 --2- 172.21.15.153:0/3102412559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c102e00 0x7f608c1031d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.032+0000 7f6092a7b700 1 -- 172.21.15.153:0/3102412559 >> 172.21.15.153:0/3102412559 conn(0x7f608c0f89b0 msgr2=0x7f608c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.033+0000 7f6092a7b700 1 -- 172.21.15.153:0/3102412559 shutdown_connections 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.033+0000 7f6092a7b700 1 -- 172.21.15.153:0/3102412559 wait complete. 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.033+0000 7f6092a7b700 1 Processor -- start 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.033+0000 7f6092a7b700 1 -- start start 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6092a7b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c102e00 0x7f608c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6092a7b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6092a7b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c114520 con 0x7f608c1037a0 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6092a7b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c10db70 con 0x7f608c102e00 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53630/0 (socket says 172.21.15.153:53630) 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 -- 172.21.15.153:0/2057926205 learned_addr learned my addr 172.21.15.153:0/2057926205 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6090817700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c102e00 0x7f608c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 -- 172.21.15.153:0/2057926205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c102e00 msgr2=0x7f608c113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c102e00 0x7f608c113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 -- 172.21.15.153:0/2057926205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080005040 con 0x7f608c1037a0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f608bfff700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f607800c800 tx=0x7f6078004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f607800a760 con 0x7f608c1037a0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.034+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c10de50 con 0x7f608c1037a0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.035+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f607800ad90 con 0x7f608c1037a0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.035+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60780113a0 con 0x7f608c1037a0 2024-04-05T03:25:07.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.035+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c10e410 con 0x7f608c1037a0 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.035+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c100110 con 0x7f608c1037a0 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.036+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6078012400 con 0x7f608c1037a0 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.037+0000 7f60897fa700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 0x7f607c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.037+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f607808d4b0 con 0x7f608c1037a0 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.037+0000 7f6090817700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 0x7f607c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.037+0000 7f6090817700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 0x7f607c070150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f608000b3b0 tx=0x7f6080009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.041+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f607805a680 con 0x7f608c1037a0 2024-04-05T03:25:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:06 smithi203 ceph-mon[95988]: pgmap v583: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:07.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.253+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f608c109f00 con 0x7f607c06dca0 2024-04-05T03:25:07.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.254+0000 7f60897fa700 1 -- 172.21.15.153:0/2057926205 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f608c109f00 con 0x7f607c06dca0 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 msgr2=0x7f607c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 0x7f607c070150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f608000b3b0 tx=0x7f6080009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 msgr2=0x7f608c113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f607800c800 tx=0x7f6078004470 comp rx=0 tx=0).stop 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 shutdown_connections 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f607c06dca0 0x7f607c070150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f608c102e00 0x7f608c113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 --2- 172.21.15.153:0/2057926205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f608c1037a0 0x7f608c113ed0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.256+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 >> 172.21.15.153:0/2057926205 conn(0x7f608c0f89b0 msgr2=0x7f608c1073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.257+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 shutdown_connections 2024-04-05T03:25:07.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.257+0000 7f6092a7b700 1 -- 172.21.15.153:0/2057926205 wait complete. 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.359+0000 7f2838907700 1 -- 172.21.15.153:0/3343653681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834102e00 msgr2=0x7f28341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.359+0000 7f2838907700 1 --2- 172.21.15.153:0/3343653681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834102e00 0x7f28341031d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2824004940 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- 172.21.15.153:0/3343653681 shutdown_connections 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 --2- 172.21.15.153:0/3343653681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28341037a0 0x7f28340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 --2- 172.21.15.153:0/3343653681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2834102e00 0x7f28341031d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- 172.21.15.153:0/3343653681 >> 172.21.15.153:0/3343653681 conn(0x7f28340f89b0 msgr2=0x7f28340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- 172.21.15.153:0/3343653681 shutdown_connections 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- 172.21.15.153:0/3343653681 wait complete. 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 Processor -- start 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- start start 2024-04-05T03:25:07.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2834102e00 0x7f28341138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834114460 con 0x7f28341037a0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2838907700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283410dab0 con 0x7f2834102e00 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2831d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2831d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53634/0 (socket says 172.21.15.153:53634) 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.360+0000 7f2831d9b700 1 -- 172.21.15.153:0/2354058016 learned_addr learned my addr 172.21.15.153:0/2354058016 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2831d9b700 1 -- 172.21.15.153:0/2354058016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2834102e00 msgr2=0x7f28341138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2831d9b700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2834102e00 0x7f28341138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2831d9b700 1 -- 172.21.15.153:0/2354058016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824005040 con 0x7f28341037a0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2831d9b700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2828002a50 tx=0x7f28280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2828007cc0 con 0x7f28341037a0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283410dd90 con 0x7f28341037a0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f282800f040 con 0x7f28341037a0 2024-04-05T03:25:07.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2828009630 con 0x7f28341037a0 2024-04-05T03:25:07.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.361+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283410e350 con 0x7f28341037a0 2024-04-05T03:25:07.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.362+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834100110 con 0x7f28341037a0 2024-04-05T03:25:07.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.362+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2828009790 con 0x7f28341037a0 2024-04-05T03:25:07.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.363+0000 7f2822ffd700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 0x7f281c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.363+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f282808d230 con 0x7f28341037a0 2024-04-05T03:25:07.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.363+0000 7f283259c700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 0x7f281c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.363+0000 7f283259c700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 0x7f281c070150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2824004f90 tx=0x7f2824009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.368+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f282805a390 con 0x7f28341037a0 2024-04-05T03:25:07.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.574+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f283410ef90 con 0x7f281c06dca0 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.584+0000 7f2822ffd700 1 -- 172.21.15.153:0/2354058016 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f283410ef90 con 0x7f281c06dca0 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (19m) 13m ago 30m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (17m) 13m ago 31m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (17m) 12m ago 29m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:25:07.582 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (19m) 13m ago 30m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (13m) 13m ago 25m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (13m) 13m ago 25m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (12m) 12m ago 25m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (13m) 12m ago 25m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (21m) 13m ago 33m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (20m) 12m ago 29m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (18m) 13m ago 33m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (17m) 12m ago 29m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (20m) 13m ago 31m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (20m) 12m ago 29m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (16m) 13m ago 29m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (16m) 13m ago 28m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (15m) 13m ago 28m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (15m) 12m ago 27m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (14m) 12m ago 27m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:25:07.583 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (13m) 12m ago 26m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:25:07.584 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (19m) 12m ago 29m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:25:07.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 msgr2=0x7f281c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 0x7f281c070150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2824004f90 tx=0x7f2824009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 msgr2=0x7f2834113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2828002a50 tx=0x7f28280046b0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 shutdown_connections 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f281c06dca0 0x7f281c070150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2834102e00 0x7f28341138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 --2- 172.21.15.153:0/2354058016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28341037a0 0x7f2834113e10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 >> 172.21.15.153:0/2354058016 conn(0x7f28340f89b0 msgr2=0x7f28340fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 shutdown_connections 2024-04-05T03:25:07.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.586+0000 7f2838907700 1 -- 172.21.15.153:0/2354058016 wait complete. 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 -- 172.21.15.153:0/3994350091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102e10 msgr2=0x7fd18c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 --2- 172.21.15.153:0/3994350091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102e10 0x7fd18c1031e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd17c004940 tx=0x7fd17c009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 -- 172.21.15.153:0/3994350091 shutdown_connections 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 --2- 172.21.15.153:0/3994350091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c1037b0 0x7fd18c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 --2- 172.21.15.153:0/3994350091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c102e10 0x7fd18c1031e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 -- 172.21.15.153:0/3994350091 >> 172.21.15.153:0/3994350091 conn(0x7fd18c0f8980 msgr2=0x7fd18c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 -- 172.21.15.153:0/3994350091 shutdown_connections 2024-04-05T03:25:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.686+0000 7fd1929b7700 1 -- 172.21.15.153:0/3994350091 wait complete. 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 Processor -- start 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 -- start start 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102e10 0x7fd18c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c070d90 con 0x7fd18c1037b0 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd1929b7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c070ed0 con 0x7fd18c102e10 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd18b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd18b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53652/0 (socket says 172.21.15.153:53652) 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd18b7fe700 1 -- 172.21.15.153:0/1535109226 learned_addr learned my addr 172.21.15.153:0/1535109226 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.687+0000 7fd18bfff700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102e10 0x7fd18c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd18b7fe700 1 -- 172.21.15.153:0/1535109226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102e10 msgr2=0x7fd18c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd18b7fe700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102e10 0x7fd18c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd18b7fe700 1 -- 172.21.15.153:0/1535109226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c005040 con 0x7fd18c1037b0 2024-04-05T03:25:07.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd18b7fe700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd180002a50 tx=0x7fd180004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd180007cc0 con 0x7fd18c1037b0 2024-04-05T03:25:07.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c119ee0 con 0x7fd18c1037b0 2024-04-05T03:25:07.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd18000f040 con 0x7fd18c1037b0 2024-04-05T03:25:07.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd18000d9c0 con 0x7fd18c1037b0 2024-04-05T03:25:07.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.688+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c11a360 con 0x7fd18c1037b0 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.689+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c100120 con 0x7fd18c1037b0 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.690+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd180009730 con 0x7fd18c1037b0 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.690+0000 7fd188ff9700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 0x7fd174070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.691+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd18008d2e0 con 0x7fd18c1037b0 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.691+0000 7fd18bfff700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 0x7fd174070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:07.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.691+0000 7fd18bfff700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 0x7fd174070150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd17c004f90 tx=0x7fd17c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:07.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.695+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd180056860 con 0x7fd18c1037b0 2024-04-05T03:25:07.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.974+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd18c0710e0 con 0x7fd18c1037b0 2024-04-05T03:25:07.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.975+0000 7fd188ff9700 1 -- 172.21.15.153:0/1535109226 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd180059f20 con 0x7fd18c1037b0 2024-04-05T03:25:07.973 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:25:07.973 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:25:07.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:07.975 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:25:07.975 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:25:07.975 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:25:07.975 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 msgr2=0x7fd174070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 0x7fd174070150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd17c004f90 tx=0x7fd17c009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 msgr2=0x7fd18c070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd180002a50 tx=0x7fd180004600 comp rx=0 tx=0).stop 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 shutdown_connections 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd17406dca0 0x7fd174070150 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102e10 0x7fd18c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 --2- 172.21.15.153:0/1535109226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd18c1037b0 0x7fd18c070800 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 >> 172.21.15.153:0/1535109226 conn(0x7fd18c0f8980 msgr2=0x7fd18c107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 shutdown_connections 2024-04-05T03:25:07.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:07.977+0000 7fd1929b7700 1 -- 172.21.15.153:0/1535109226 wait complete. 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 -- 172.21.15.153:0/3658766777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc1026c0 msgr2=0x7fc6fc0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3658766777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc0fbd30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0004940 tx=0x7fc6f0009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 -- 172.21.15.153:0/3658766777 shutdown_connections 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3658766777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc0fbd30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3658766777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc101d20 0x7fc6fc1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 -- 172.21.15.153:0/3658766777 >> 172.21.15.153:0/3658766777 conn(0x7fc6fc0f7870 msgr2=0x7fc6fc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 -- 172.21.15.153:0/3658766777 shutdown_connections 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.077+0000 7fc701cf8700 1 -- 172.21.15.153:0/3658766777 wait complete. 2024-04-05T03:25:08.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 Processor -- start 2024-04-05T03:25:08.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 -- start start 2024-04-05T03:25:08.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc10f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc10f5e0 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc701cf8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc10f720 con 0x7fc6fc1026c0 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc6fb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc6fb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53664/0 (socket says 172.21.15.153:53664) 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.078+0000 7fc6fb7fe700 1 -- 172.21.15.153:0/3850639575 learned_addr learned my addr 172.21.15.153:0/3850639575 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6faffd700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc10f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6fb7fe700 1 -- 172.21.15.153:0/3850639575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc1026c0 msgr2=0x7fc6fc10f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6fb7fe700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc10f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6fb7fe700 1 -- 172.21.15.153:0/3850639575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f0005040 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6fb7fe700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec002a50 tx=0x7fc6ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6ec007cc0 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6ec00f040 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6ec009640 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc117bd0 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.079+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc117f20 con 0x7fc6fc101d20 2024-04-05T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.080+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc0fd470 con 0x7fc6fc101d20 2024-04-05T03:25:08.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.081+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc6ec0097a0 con 0x7fc6fc101d20 2024-04-05T03:25:08.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.081+0000 7fc6e3fff700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 0x7fc6e4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.081+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc6ec08d350 con 0x7fc6fc101d20 2024-04-05T03:25:08.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.081+0000 7fc6faffd700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 0x7fc6e4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.081+0000 7fc6faffd700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 0x7fc6e4070150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc066a60 tx=0x7fc6f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.086+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc6ec0568d0 con 0x7fc6fc101d20 2024-04-05T03:25:08.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.323+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc6fc066870 con 0x7fc6fc101d20 2024-04-05T03:25:08.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.324+0000 7fc6e3fff700 1 -- 172.21.15.153:0/3850639575 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fc6ec059f90 con 0x7fc6fc101d20 2024-04-05T03:25:08.322 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:25:08.322 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:25:08.323 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:25:08.324 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:08.325 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:08.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 msgr2=0x7fc6e4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 0x7fc6e4070150 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc066a60 tx=0x7fc6f0009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:08.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 msgr2=0x7fc6fc10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec002a50 tx=0x7fc6ec0046b0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 shutdown_connections 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc6e406dca0 0x7fc6e4070150 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6fc101d20 0x7fc6fc10eb60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 --2- 172.21.15.153:0/3850639575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6fc1026c0 0x7fc6fc10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 >> 172.21.15.153:0/3850639575 conn(0x7fc6fc0f7870 msgr2=0x7fc6fc0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 shutdown_connections 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.326+0000 7fc701cf8700 1 -- 172.21.15.153:0/3850639575 wait complete. 2024-04-05T03:25:08.327 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:25:08.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 -- 172.21.15.153:0/2162720589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 msgr2=0x7f97941031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 --2- 172.21.15.153:0/2162720589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f97941031d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9788004940 tx=0x7f9788009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 -- 172.21.15.153:0/2162720589 shutdown_connections 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 --2- 172.21.15.153:0/2162720589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 0x7f97940fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 --2- 172.21.15.153:0/2162720589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f97941031d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 -- 172.21.15.153:0/2162720589 >> 172.21.15.153:0/2162720589 conn(0x7f97940f89b0 msgr2=0x7f97940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:08.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.428+0000 7f979ad88700 1 -- 172.21.15.153:0/2162720589 shutdown_connections 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 -- 172.21.15.153:0/2162720589 wait complete. 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 Processor -- start 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 -- start start 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 0x7f9794113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9794114520 con 0x7f9794102e00 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f979ad88700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979410db70 con 0x7f97941037a0 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.429+0000 7f9798b24700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53688/0 (socket says 172.21.15.153:53688) 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 -- 172.21.15.153:0/4108402112 learned_addr learned my addr 172.21.15.153:0/4108402112 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9793fff700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 0x7f9794113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 -- 172.21.15.153:0/4108402112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 msgr2=0x7f9794113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 0x7f9794113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 -- 172.21.15.153:0/4108402112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9788005040 con 0x7f9794102e00 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f9798b24700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f978800b3b0 tx=0x7f9788009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f978800e040 con 0x7f9794102e00 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9788013420 con 0x7f9794102e00 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979410ddf0 con 0x7f9794102e00 2024-04-05T03:25:08.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f978801c630 con 0x7f9794102e00 2024-04-05T03:25:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.430+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979410e3b0 con 0x7f9794102e00 2024-04-05T03:25:08.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.431+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9794100110 con 0x7f9794102e00 2024-04-05T03:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.432+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9788013590 con 0x7f9794102e00 2024-04-05T03:25:08.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.432+0000 7f97917fa700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 0x7f9784070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.432+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f978808e760 con 0x7f9794102e00 2024-04-05T03:25:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.432+0000 7f9793fff700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 0x7f9784070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.433+0000 7f9793fff700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 0x7f9784070150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f979410ece0 tx=0x7f978000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.437+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f978805b890 con 0x7f9794102e00 2024-04-05T03:25:08.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.649+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f979410f050 con 0x7f978406dca0 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.650+0000 7f97917fa700 1 -- 172.21.15.153:0/4108402112 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f979410f050 con 0x7f978406dca0 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:25:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:25:08.649 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:25:08.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 msgr2=0x7f9784070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 0x7f9784070150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f979410ece0 tx=0x7f978000b040 comp rx=0 tx=0).stop 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 msgr2=0x7f9794113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f978800b3b0 tx=0x7f9788009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 shutdown_connections 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f978406dca0 0x7f9784070150 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9794102e00 0x7f9794113990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 --2- 172.21.15.153:0/4108402112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97941037a0 0x7f9794113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 >> 172.21.15.153:0/4108402112 conn(0x7f97940f89b0 msgr2=0x7f97941073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 shutdown_connections 2024-04-05T03:25:08.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.652+0000 7f979ad88700 1 -- 172.21.15.153:0/4108402112 wait complete. 2024-04-05T03:25:08.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 -- 172.21.15.153:0/3273165422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27401037b0 msgr2=0x7f27400fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 --2- 172.21.15.153:0/3273165422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27401037b0 0x7f27400fce20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2730004940 tx=0x7f2730009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 -- 172.21.15.153:0/3273165422 shutdown_connections 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 --2- 172.21.15.153:0/3273165422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27401037b0 0x7f27400fce20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 --2- 172.21.15.153:0/3273165422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2740102e10 0x7f27401031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 -- 172.21.15.153:0/3273165422 >> 172.21.15.153:0/3273165422 conn(0x7f27400f8980 msgr2=0x7f27400fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 -- 172.21.15.153:0/3273165422 shutdown_connections 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.753+0000 7f2746515700 1 -- 172.21.15.153:0/3273165422 wait complete. 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 Processor -- start 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 -- start start 2024-04-05T03:25:08.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27401037b0 0x7f2740114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2740114670 con 0x7f2740102e10 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f2746515700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274010da60 con 0x7f27401037b0 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f273ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f273ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45800/0 (socket says 172.21.15.153:45800) 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.754+0000 7f273ffff700 1 -- 172.21.15.153:0/1523591096 learned_addr learned my addr 172.21.15.153:0/1523591096 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273ffff700 1 -- 172.21.15.153:0/1523591096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27401037b0 msgr2=0x7f2740114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273ffff700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27401037b0 0x7f2740114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273ffff700 1 -- 172.21.15.153:0/1523591096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730005040 con 0x7f2740102e10 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273ffff700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f273400eb40 tx=0x7f273400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2734007cc0 con 0x7f2740102e10 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f273400f040 con 0x7f2740102e10 2024-04-05T03:25:08.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274010dce0 con 0x7f2740102e10 2024-04-05T03:25:08.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2734009620 con 0x7f2740102e10 2024-04-05T03:25:08.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.755+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274010e220 con 0x7f2740102e10 2024-04-05T03:25:08.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.756+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27401052e0 con 0x7f2740102e10 2024-04-05T03:25:08.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.756+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2734009780 con 0x7f2740102e10 2024-04-05T03:25:08.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.757+0000 7f273cff9700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 0x7f2728070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:08.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.757+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f273408d200 con 0x7f2740102e10 2024-04-05T03:25:08.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.757+0000 7f273f7fe700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 0x7f2728070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:08.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.757+0000 7f273f7fe700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 0x7f2728070150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f2730009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:08.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:08.761+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f273405a360 con 0x7f2740102e10 2024-04-05T03:25:09.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.036+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f274010f130 con 0x7f2740102e10 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: pgmap v584: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1535109226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:25:09.035 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:08 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3850639575' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:25:09.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.037+0000 7f273cff9700 1 -- 172.21.15.153:0/1523591096 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2734059ef0 con 0x7f2740102e10 2024-04-05T03:25:09.036 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:25:09.036 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:25:09.036 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:25:09.036 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:25:09.036 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 msgr2=0x7f2728070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 0x7f2728070150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2730000c00 tx=0x7f2730009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 msgr2=0x7f2740113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f273400eb40 tx=0x7f273400b040 comp rx=0 tx=0).stop 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 shutdown_connections 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f272806dca0 0x7f2728070150 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:09.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2740102e10 0x7f2740113ae0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:09.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 --2- 172.21.15.153:0/1523591096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27401037b0 0x7f2740114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:09.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 >> 172.21.15.153:0/1523591096 conn(0x7f27400f8980 msgr2=0x7f27401021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:09.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 shutdown_connections 2024-04-05T03:25:09.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:09.040+0000 7f2746515700 1 -- 172.21.15.153:0/1523591096 wait complete. 2024-04-05T03:25:09.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: pgmap v584: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1535109226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:25:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:08 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3850639575' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:25:10.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:09 smithi203 ceph-mon[95988]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:10.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:09 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1523591096' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:25:10.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:09 smithi203 ceph-mon[95988]: pgmap v585: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:10.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:09 smithi153 ceph-mon[106102]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:10.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:09 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1523591096' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:25:10.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:09 smithi153 ceph-mon[106102]: pgmap v585: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:12 smithi203 ceph-mon[95988]: pgmap v586: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:12 smithi153 ceph-mon[106102]: pgmap v586: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:14 smithi203 ceph-mon[95988]: pgmap v587: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:14 smithi153 ceph-mon[106102]: pgmap v587: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:17.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:16 smithi203 ceph-mon[95988]: pgmap v588: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:17.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:16 smithi153 ceph-mon[106102]: pgmap v588: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:18 smithi203 ceph-mon[95988]: pgmap v589: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:18 smithi153 ceph-mon[106102]: pgmap v589: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:20 smithi203 ceph-mon[95988]: pgmap v590: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:21.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:20 smithi153 ceph-mon[106102]: pgmap v590: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:22.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:21 smithi203 ceph-mon[95988]: pgmap v591: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:22.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:21 smithi153 ceph-mon[106102]: pgmap v591: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:25.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:24 smithi203 ceph-mon[95988]: pgmap v592: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:25.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:24 smithi153 ceph-mon[106102]: pgmap v592: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:27.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:26 smithi203 ceph-mon[95988]: pgmap v593: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:26 smithi153 ceph-mon[106102]: pgmap v593: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:29.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:28 smithi203 ceph-mon[95988]: pgmap v594: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:28 smithi153 ceph-mon[106102]: pgmap v594: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:30.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:29 smithi203 ceph-mon[95988]: pgmap v595: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:29 smithi153 ceph-mon[106102]: pgmap v595: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:32 smithi203 ceph-mon[95988]: pgmap v596: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:32 smithi153 ceph-mon[106102]: pgmap v596: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:34 smithi203 ceph-mon[95988]: pgmap v597: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:35.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:34 smithi153 ceph-mon[106102]: pgmap v597: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:37.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:36 smithi203 ceph-mon[95988]: pgmap v598: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:37.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:36 smithi153 ceph-mon[106102]: pgmap v598: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:38 smithi203 ceph-mon[95988]: pgmap v599: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:39.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:38 smithi153 ceph-mon[106102]: pgmap v599: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:39.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 -- 172.21.15.153:0/2612203433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 msgr2=0x7f24b81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 --2- 172.21.15.153:0/2612203433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b81031d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f24ac004940 tx=0x7f24ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:39.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 -- 172.21.15.153:0/2612203433 shutdown_connections 2024-04-05T03:25:39.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 --2- 172.21.15.153:0/2612203433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 --2- 172.21.15.153:0/2612203433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b81031d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 -- 172.21.15.153:0/2612203433 >> 172.21.15.153:0/2612203433 conn(0x7f24b80f89b0 msgr2=0x7f24b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:39.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 -- 172.21.15.153:0/2612203433 shutdown_connections 2024-04-05T03:25:39.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.155+0000 7f24bee40700 1 -- 172.21.15.153:0/2612203433 wait complete. 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 Processor -- start 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 -- start start 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b81140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b81147d0 con 0x7f24b8102e00 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bee40700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b810db30 con 0x7f24b81037a0 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bcbdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bcbdc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38396/0 (socket says 172.21.15.153:38396) 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bcbdc700 1 -- 172.21.15.153:0/2881278042 learned_addr learned my addr 172.21.15.153:0/2881278042 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24bcbdc700 1 -- 172.21.15.153:0/2881278042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 msgr2=0x7f24b81140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.156+0000 7f24b7fff700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b81140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24bcbdc700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b81140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24bcbdc700 1 -- 172.21.15.153:0/2881278042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac005040 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24b7fff700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b81140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24bcbdc700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00b8b0 tx=0x7f24ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24ac00e040 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24ac004290 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b810ddb0 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24ac01c3f0 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.157+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b810e370 con 0x7f24b8102e00 2024-04-05T03:25:39.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.158+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b80fe550 con 0x7f24b8102e00 2024-04-05T03:25:39.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.159+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f24ac0137b0 con 0x7f24b8102e00 2024-04-05T03:25:39.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.159+0000 7f24b57fa700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 0x7f24a8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.159+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f24ac08e8e0 con 0x7f24b8102e00 2024-04-05T03:25:39.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.159+0000 7f24b7fff700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 0x7f24a8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.159+0000 7f24b7fff700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 0x7f24a8070150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f24a4006fd0 tx=0x7f24a4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.164+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f24ac05ba10 con 0x7f24b8102e00 2024-04-05T03:25:39.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.376+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24b804bed0 con 0x7f24a806dca0 2024-04-05T03:25:39.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.377+0000 7f24b57fa700 1 -- 172.21.15.153:0/2881278042 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f24b804bed0 con 0x7f24a806dca0 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.379+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 msgr2=0x7f24a8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.379+0000 7f24bee40700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 0x7f24a8070150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f24a4006fd0 tx=0x7f24a4008040 comp rx=0 tx=0).stop 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.379+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 msgr2=0x7f24b8113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.379+0000 7f24bee40700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00b8b0 tx=0x7f24ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 shutdown_connections 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f24a806dca0 0x7f24a8070150 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24b8102e00 0x7f24b8113bb0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 --2- 172.21.15.153:0/2881278042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24b81037a0 0x7f24b81140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 >> 172.21.15.153:0/2881278042 conn(0x7f24b80f89b0 msgr2=0x7f24b80f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:39.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 shutdown_connections 2024-04-05T03:25:39.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.380+0000 7f24bee40700 1 -- 172.21.15.153:0/2881278042 wait complete. 2024-04-05T03:25:39.388 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1100662558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe504102f70 msgr2=0x7fe504103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1100662558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe504102f70 0x7fe504103340 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009ce0 tx=0x7fe4fc004d10 comp rx=0 tx=0).stop 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1100662558 shutdown_connections 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1100662558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1100662558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe504102f70 0x7fe504103340 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1100662558 >> 172.21.15.153:0/1100662558 conn(0x7fe504072310 msgr2=0x7fe504072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:39.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1100662558 shutdown_connections 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.482+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1100662558 wait complete. 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 Processor -- start 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 -- start start 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe504102f70 0x7fe5040fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040fe530 con 0x7fe5040fb3b0 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe50b2ed700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040fd010 con 0x7fe504102f70 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38418/0 (socket says 172.21.15.153:38418) 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 -- 172.21.15.153:0/1022980018 learned_addr learned my addr 172.21.15.153:0/1022980018 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe508888700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe504102f70 0x7fe5040fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 -- 172.21.15.153:0/1022980018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe504102f70 msgr2=0x7fe5040fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe504102f70 0x7fe5040fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 -- 172.21.15.153:0/1022980018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc009940 con 0x7fe5040fb3b0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.483+0000 7fe509089700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00be10 tx=0x7fe4fc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.484+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc00e040 con 0x7fe5040fb3b0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.484+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4fc004610 con 0x7fe5040fb3b0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.484+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5040fd260 con 0x7fe5040fb3b0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.484+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc01c640 con 0x7fe5040fb3b0 2024-04-05T03:25:39.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.484+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50411a1c0 con 0x7fe5040fb3b0 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.485+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5041053b0 con 0x7fe5040fb3b0 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.486+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe4fc0129e0 con 0x7fe5040fb3b0 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.486+0000 7fe501ffb700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 0x7fe4ec070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.486+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe4fc08eef0 con 0x7fe5040fb3b0 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.486+0000 7fe508888700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 0x7fe4ec070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.487+0000 7fe508888700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 0x7fe4ec070150 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5040fdcc0 tx=0x7fe4f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.491+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe4fc0584f0 con 0x7fe5040fb3b0 2024-04-05T03:25:39.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.701+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5040fda70 con 0x7fe4ec06dca0 2024-04-05T03:25:39.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.702+0000 7fe501ffb700 1 -- 172.21.15.153:0/1022980018 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe5040fda70 con 0x7fe4ec06dca0 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 msgr2=0x7fe4ec070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 0x7fe4ec070150 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5040fdcc0 tx=0x7fe4f800b040 comp rx=0 tx=0).stop 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 msgr2=0x7fe5040fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00be10 tx=0x7fe4fc004d10 comp rx=0 tx=0).stop 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 shutdown_connections 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe4ec06dca0 0x7fe4ec070150 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5040fb3b0 0x7fe5040fc590 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 --2- 172.21.15.153:0/1022980018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe504102f70 0x7fe5040fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 >> 172.21.15.153:0/1022980018 conn(0x7fe504072310 msgr2=0x7fe5040f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 shutdown_connections 2024-04-05T03:25:39.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.705+0000 7fe50b2ed700 1 -- 172.21.15.153:0/1022980018 wait complete. 2024-04-05T03:25:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 -- 172.21.15.153:0/379876805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f44102e00 msgr2=0x7f0f441031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/379876805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f44102e00 0x7f0f441031d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f38004940 tx=0x7f0f38009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:39.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 -- 172.21.15.153:0/379876805 shutdown_connections 2024-04-05T03:25:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/379876805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f441037a0 0x7f0f440fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/379876805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f44102e00 0x7f0f441031d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 -- 172.21.15.153:0/379876805 >> 172.21.15.153:0/379876805 conn(0x7f0f440f89b0 msgr2=0x7f0f440fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 -- 172.21.15.153:0/379876805 shutdown_connections 2024-04-05T03:25:39.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.807+0000 7f0f4aa67700 1 -- 172.21.15.153:0/379876805 wait complete. 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 Processor -- start 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 -- start start 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f44102e00 0x7f0f440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440721c0 con 0x7f0f441037a0 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f4aa67700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f44070d40 con 0x7f0f44102e00 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38446/0 (socket says 172.21.15.153:38446) 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.808+0000 7f0f43fff700 1 -- 172.21.15.153:0/1891588405 learned_addr learned my addr 172.21.15.153:0/1891588405 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f48803700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f44102e00 0x7f0f440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f43fff700 1 -- 172.21.15.153:0/1891588405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f44102e00 msgr2=0x7f0f440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f43fff700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f44102e00 0x7f0f440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f43fff700 1 -- 172.21.15.153:0/1891588405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f38005040 con 0x7f0f441037a0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f43fff700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0f30002a50 tx=0x7f0f30004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f30007cc0 con 0x7f0f441037a0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f3000f040 con 0x7f0f441037a0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f44071020 con 0x7f0f441037a0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4411a180 con 0x7f0f441037a0 2024-04-05T03:25:39.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.809+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f3000da40 con 0x7f0f441037a0 2024-04-05T03:25:39.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.810+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f280052b0 con 0x7f0f441037a0 2024-04-05T03:25:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.811+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f0f300097c0 con 0x7f0f441037a0 2024-04-05T03:25:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.811+0000 7f0f417fa700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 0x7f0f34070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.811+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0f3008d390 con 0x7f0f441037a0 2024-04-05T03:25:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.811+0000 7f0f48803700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 0x7f0f34070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:39.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.812+0000 7f0f48803700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 0x7f0f34070150 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800b020 tx=0x7f0f38009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:39.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:39.816+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0f3005a440 con 0x7f0f441037a0 2024-04-05T03:25:40.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.021+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f28000bc0 con 0x7f0f3406dca0 2024-04-05T03:25:40.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.031+0000 7f0f417fa700 1 -- 172.21.15.153:0/1891588405 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0f28000bc0 con 0x7f0f3406dca0 2024-04-05T03:25:40.030 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:25:40.030 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (20m) 13m ago 31m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:25:40.030 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (17m) 13m ago 32m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:25:40.030 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (17m) 13m ago 30m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (19m) 13m ago 31m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (13m) 13m ago 25m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (14m) 13m ago 25m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (13m) 13m ago 25m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (13m) 13m ago 25m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:25:40.031 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (21m) 13m ago 33m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (21m) 13m ago 30m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (18m) 13m ago 33m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (18m) 13m ago 30m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (20m) 13m ago 31m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (20m) 13m ago 30m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:25:40.032 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (17m) 13m ago 29m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (16m) 13m ago 29m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (16m) 13m ago 28m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (15m) 13m ago 28m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (15m) 13m ago 27m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (14m) 13m ago 27m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:25:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (20m) 13m ago 29m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:25:40.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 msgr2=0x7f0f34070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 0x7f0f34070150 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800b020 tx=0x7f0f38009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 msgr2=0x7f0f44070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0f30002a50 tx=0x7f0f30004600 comp rx=0 tx=0).stop 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 shutdown_connections 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f0f3406dca0 0x7f0f34070150 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f44102e00 0x7f0f440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 --2- 172.21.15.153:0/1891588405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f441037a0 0x7f0f44070800 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.033+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 >> 172.21.15.153:0/1891588405 conn(0x7f0f440f89b0 msgr2=0x7f0f441073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.034+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 shutdown_connections 2024-04-05T03:25:40.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.034+0000 7f0f4aa67700 1 -- 172.21.15.153:0/1891588405 wait complete. 2024-04-05T03:25:40.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:39 smithi153 ceph-mon[106102]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:40.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:39 smithi153 ceph-mon[106102]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:40.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:39 smithi153 ceph-mon[106102]: pgmap v600: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 -- 172.21.15.153:0/1996601247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c0fb3b0 msgr2=0x7fcf2c0fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 --2- 172.21.15.153:0/1996601247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c0fb820 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcf18004940 tx=0x7fcf18009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 -- 172.21.15.153:0/1996601247 shutdown_connections 2024-04-05T03:25:40.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 --2- 172.21.15.153:0/1996601247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c0fb820 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 --2- 172.21.15.153:0/1996601247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c102f70 0x7fcf2c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 -- 172.21.15.153:0/1996601247 >> 172.21.15.153:0/1996601247 conn(0x7fcf2c072310 msgr2=0x7fcf2c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 -- 172.21.15.153:0/1996601247 shutdown_connections 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.133+0000 7fcf331bc700 1 -- 172.21.15.153:0/1996601247 wait complete. 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 Processor -- start 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 -- start start 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1144f0 con 0x7fcf2c102f70 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf331bc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c10daf0 con 0x7fcf2c0fb3b0 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38462/0 (socket says 172.21.15.153:38462) 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 -- 172.21.15.153:0/2197368578 learned_addr learned my addr 172.21.15.153:0/2197368578 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf30f58700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 -- 172.21.15.153:0/2197368578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 msgr2=0x7fcf2c113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf2bfff700 1 -- 172.21.15.153:0/2197368578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18005040 con 0x7fcf2c102f70 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.134+0000 7fcf30f58700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf2bfff700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009b30 tx=0x7fcf18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1800e070 con 0x7fcf2c102f70 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf18013cd0 con 0x7fcf2c102f70 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c10dd70 con 0x7fcf2c102f70 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1800b6e0 con 0x7fcf2c102f70 2024-04-05T03:25:40.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.135+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c10e250 con 0x7fcf2c102f70 2024-04-05T03:25:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.136+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c105350 con 0x7fcf2c102f70 2024-04-05T03:25:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.136+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fcf1801cab0 con 0x7fcf2c102f70 2024-04-05T03:25:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.137+0000 7fcf297fa700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 0x7fcf1c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.137+0000 7fcf30f58700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 0x7fcf1c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.137+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcf1808ef30 con 0x7fcf2c102f70 2024-04-05T03:25:40.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.137+0000 7fcf30f58700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 0x7fcf1c070150 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcf20004600 tx=0x7fcf2000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.141+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcf180c3280 con 0x7fcf2c102f70 2024-04-05T03:25:40.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:39 smithi203 ceph-mon[95988]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:40.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:39 smithi203 ceph-mon[95988]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:40.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:39 smithi203 ceph-mon[95988]: pgmap v600: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:40.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.417+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcf2c10efb0 con 0x7fcf2c102f70 2024-04-05T03:25:40.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.418+0000 7fcf297fa700 1 -- 172.21.15.153:0/2197368578 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fcf1805c060 con 0x7fcf2c102f70 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:25:40.417 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:25:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:25:40.418 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:25:40.418 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:25:40.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 msgr2=0x7fcf1c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 0x7fcf1c070150 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcf20004600 tx=0x7fcf2000b040 comp rx=0 tx=0).stop 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 msgr2=0x7fcf2c113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009b30 tx=0x7fcf18009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 shutdown_connections 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fcf1c06dca0 0x7fcf1c070150 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf2c0fb3b0 0x7fcf2c113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 --2- 172.21.15.153:0/2197368578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c102f70 0x7fcf2c113ea0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 >> 172.21.15.153:0/2197368578 conn(0x7fcf2c072310 msgr2=0x7fcf2c0f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 shutdown_connections 2024-04-05T03:25:40.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.420+0000 7fcf331bc700 1 -- 172.21.15.153:0/2197368578 wait complete. 2024-04-05T03:25:40.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 -- 172.21.15.153:0/3369360293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc574070bf0 msgr2=0x7fc5741091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 --2- 172.21.15.153:0/3369360293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc574070bf0 0x7fc5741091a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc55c004940 tx=0x7fc55c009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 -- 172.21.15.153:0/3369360293 shutdown_connections 2024-04-05T03:25:40.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 --2- 172.21.15.153:0/3369360293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc574070bf0 0x7fc5741091a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 --2- 172.21.15.153:0/3369360293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5740702e0 0x7fc5740706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 -- 172.21.15.153:0/3369360293 >> 172.21.15.153:0/3369360293 conn(0x7fc5740f8980 msgr2=0x7fc5740fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 -- 172.21.15.153:0/3369360293 shutdown_connections 2024-04-05T03:25:40.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 -- 172.21.15.153:0/3369360293 wait complete. 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.522+0000 7fc579367700 1 Processor -- start 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 -- start start 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc574070bf0 0x7fc574116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574116a40 con 0x7fc5740702e0 2024-04-05T03:25:40.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57410fd80 con 0x7fc574070bf0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38480/0 (socket says 172.21.15.153:38480) 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 -- 172.21.15.153:0/2909889204 learned_addr learned my addr 172.21.15.153:0/2909889204 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc5727fc700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc574070bf0 0x7fc574116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 -- 172.21.15.153:0/2909889204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc574070bf0 msgr2=0x7fc574116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc574070bf0 0x7fc574116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 -- 172.21.15.153:0/2909889204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c005040 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc572ffd700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc56400bbb0 tx=0x7fc564009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5640092e0 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc56400cb20 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.523+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc574110000 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.524+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc564015d70 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.524+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc574110540 con 0x7fc5740702e0 2024-04-05T03:25:40.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.525+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc574071a20 con 0x7fc5740702e0 2024-04-05T03:25:40.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.525+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc564012070 con 0x7fc5740702e0 2024-04-05T03:25:40.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.525+0000 7fc56b7fe700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 0x7fc560070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.525+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc56408d2c0 con 0x7fc5740702e0 2024-04-05T03:25:40.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.525+0000 7fc5727fc700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 0x7fc560070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.526+0000 7fc5727fc700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 0x7fc560070160 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc5741116c0 tx=0x7fc55c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.530+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc56405a370 con 0x7fc5740702e0 2024-04-05T03:25:40.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.768+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc574111470 con 0x7fc5740702e0 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.768+0000 7fc56b7fe700 1 -- 172.21.15.153:0/2909889204 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7fc564059f00 con 0x7fc5740702e0 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:25:40.767 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:25:40.768 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:40.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:40.770 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 msgr2=0x7fc560070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 0x7fc560070160 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc5741116c0 tx=0x7fc55c009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 msgr2=0x7fc574115e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc56400bbb0 tx=0x7fc564009840 comp rx=0 tx=0).stop 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 shutdown_connections 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc56006dcb0 0x7fc560070160 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc5740702e0 0x7fc574115e20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 --2- 172.21.15.153:0/2909889204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc574070bf0 0x7fc574116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 >> 172.21.15.153:0/2909889204 conn(0x7fc5740f8980 msgr2=0x7fc5740f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 shutdown_connections 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.771+0000 7fc579367700 1 -- 172.21.15.153:0/2909889204 wait complete. 2024-04-05T03:25:40.771 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 -- 172.21.15.153:0/1764157091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 msgr2=0x7ff6400fbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 --2- 172.21.15.153:0/1764157091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff6400fbb60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff630004940 tx=0x7ff630009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 -- 172.21.15.153:0/1764157091 shutdown_connections 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 --2- 172.21.15.153:0/1764157091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6400fc0a0 0x7ff64010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 --2- 172.21.15.153:0/1764157091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff6400fbb60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 -- 172.21.15.153:0/1764157091 >> 172.21.15.153:0/1764157091 conn(0x7ff640071530 msgr2=0x7ff640071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.873+0000 7ff645710700 1 -- 172.21.15.153:0/1764157091 shutdown_connections 2024-04-05T03:25:40.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 -- 172.21.15.153:0/1764157091 wait complete. 2024-04-05T03:25:40.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 Processor -- start 2024-04-05T03:25:40.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 -- start start 2024-04-05T03:25:40.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6400fc0a0 0x7ff64011a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64011abc0 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff645710700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6401141f0 con 0x7ff6400fc0a0 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.874+0000 7ff63effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38496/0 (socket says 172.21.15.153:38496) 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 -- 172.21.15.153:0/1747252226 learned_addr learned my addr 172.21.15.153:0/1747252226 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 -- 172.21.15.153:0/1747252226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6400fc0a0 msgr2=0x7ff64011a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6400fc0a0 0x7ff64011a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 -- 172.21.15.153:0/1747252226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630005040 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff63effd700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff630009150 tx=0x7ff630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63000e040 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff630013420 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff640114470 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63001c630 con 0x7ff6400fb790 2024-04-05T03:25:40.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.875+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6401149b0 con 0x7ff6400fb790 2024-04-05T03:25:40.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.876+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6400fe110 con 0x7ff6400fb790 2024-04-05T03:25:40.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.877+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff630013590 con 0x7ff6400fb790 2024-04-05T03:25:40.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.877+0000 7ff6277fe700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 0x7ff628070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:40.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.877+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff63008e820 con 0x7ff6400fb790 2024-04-05T03:25:40.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.877+0000 7ff63e7fc700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 0x7ff628070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:40.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.877+0000 7ff63e7fc700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 0x7ff628070150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff6401158d0 tx=0x7ff634009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:40.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:40.882+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff63005b950 con 0x7ff6400fb790 2024-04-05T03:25:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:40 smithi153 ceph-mon[106102]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:40 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2197368578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:25:41.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:40 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2909889204' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:25:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.094+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff64004c780 con 0x7ff62806dca0 2024-04-05T03:25:41.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.095+0000 7ff6277fe700 1 -- 172.21.15.153:0/1747252226 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff64004c780 con 0x7ff62806dca0 2024-04-05T03:25:41.094 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:25:41.094 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:25:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:25:41.096 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:25:41.096 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:25:41.096 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:25:41.096 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:25:41.096 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:25:41.097 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:25:41.097 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:25:41.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 msgr2=0x7ff628070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 0x7ff628070150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff6401158d0 tx=0x7ff634009ce0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 msgr2=0x7ff64011a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff630009150 tx=0x7ff630009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 shutdown_connections 2024-04-05T03:25:41.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff62806dca0 0x7ff628070150 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6400fb790 0x7ff64011a030 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 --2- 172.21.15.153:0/1747252226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6400fc0a0 0x7ff64011a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 >> 172.21.15.153:0/1747252226 conn(0x7ff640071530 msgr2=0x7ff640102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 shutdown_connections 2024-04-05T03:25:41.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.097+0000 7ff645710700 1 -- 172.21.15.153:0/1747252226 wait complete. 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.198+0000 7f446e2d6700 1 -- 172.21.15.153:0/1352536141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 msgr2=0x7f44680fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1352536141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f44680fe250 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4458004940 tx=0x7f4458009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 -- 172.21.15.153:0/1352536141 shutdown_connections 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1352536141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 0x7f4468106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1352536141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f44680fe250 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 -- 172.21.15.153:0/1352536141 >> 172.21.15.153:0/1352536141 conn(0x7f44680f7870 msgr2=0x7f44680f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 -- 172.21.15.153:0/1352536141 shutdown_connections 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 -- 172.21.15.153:0/1352536141 wait complete. 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 Processor -- start 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.199+0000 7f446e2d6700 1 -- start start 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 0x7f4468116050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:41.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681166a0 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446810fc80 con 0x7f44680fe820 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38512/0 (socket says 172.21.15.153:38512) 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 -- 172.21.15.153:0/1718596790 learned_addr learned my addr 172.21.15.153:0/1718596790 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f44677fe700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 0x7f4468116050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 -- 172.21.15.153:0/1718596790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 msgr2=0x7f4468116050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 0x7f4468116050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 -- 172.21.15.153:0/1718596790 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458005040 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4467fff700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4458009380 tx=0x7f4458009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445800e040 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4458009af0 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446810ff60 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445801c3f0 con 0x7f44680fde80 2024-04-05T03:25:41.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.200+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44681104a0 con 0x7f44680fde80 2024-04-05T03:25:41.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.201+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468066b10 con 0x7f44680fde80 2024-04-05T03:25:41.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.202+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4458013420 con 0x7f44680fde80 2024-04-05T03:25:41.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.202+0000 7f4464ff9700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 0x7f4450070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:25:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.202+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f445808e820 con 0x7f44680fde80 2024-04-05T03:25:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.202+0000 7f44677fe700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 0x7f4450070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:25:41.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.203+0000 7f44677fe700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 0x7f4450070150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f44681113b0 tx=0x7f445c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:25:41.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.207+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f445805b950 con 0x7f44680fde80 2024-04-05T03:25:41.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:40 smithi203 ceph-mon[95988]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:41.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:40 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2197368578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:25:41.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:40 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2909889204' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:25:41.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.483+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4468111160 con 0x7f44680fde80 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.483+0000 7f4464ff9700 1 -- 172.21.15.153:0/1718596790 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4458060e60 con 0x7f44680fde80 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:25:41.482 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.485+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 msgr2=0x7f4450070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.485+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 0x7f4450070150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f44681113b0 tx=0x7f445c00b040 comp rx=0 tx=0).stop 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.485+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 msgr2=0x7f4468115b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.485+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4458009380 tx=0x7f4458009e30 comp rx=0 tx=0).stop 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 shutdown_connections 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f445006dca0 0x7f4450070150 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44680fde80 0x7f4468115b10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 --2- 172.21.15.153:0/1718596790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44680fe820 0x7f4468116050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:25:41.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 >> 172.21.15.153:0/1718596790 conn(0x7f44680f7870 msgr2=0x7f44680f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:25:41.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 shutdown_connections 2024-04-05T03:25:41.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:25:41.486+0000 7f446e2d6700 1 -- 172.21.15.153:0/1718596790 wait complete. 2024-04-05T03:25:42.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:41 smithi203 ceph-mon[95988]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:42.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:41 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1718596790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:25:42.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:41 smithi203 ceph-mon[95988]: pgmap v601: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:42.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:41 smithi153 ceph-mon[106102]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:25:42.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:41 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1718596790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:25:42.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:41 smithi153 ceph-mon[106102]: pgmap v601: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:43.196 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:43.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:44.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:43 smithi203 ceph-mon[95988]: pgmap v602: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:44.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:43 smithi153 ceph-mon[106102]: pgmap v602: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:47.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:46 smithi203 ceph-mon[95988]: pgmap v603: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:46 smithi153 ceph-mon[106102]: pgmap v603: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:48 smithi203 ceph-mon[95988]: pgmap v604: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:49.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:48 smithi153 ceph-mon[106102]: pgmap v604: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:51.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:50 smithi203 ceph-mon[95988]: pgmap v605: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:50 smithi153 ceph-mon[106102]: pgmap v605: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:52.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:51 smithi153 ceph-mon[106102]: pgmap v606: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:52.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:51 smithi203 ceph-mon[95988]: pgmap v606: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:54 smithi203 ceph-mon[95988]: pgmap v607: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:54 smithi153 ceph-mon[106102]: pgmap v607: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:56 smithi203 ceph-mon[95988]: pgmap v608: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:57.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:56 smithi153 ceph-mon[106102]: pgmap v608: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:58.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:25:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:58 smithi203 ceph-mon[95988]: pgmap v609: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:25:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:58 smithi153 ceph-mon[106102]: pgmap v609: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:00.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:25:59 smithi203 ceph-mon[95988]: pgmap v610: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:00.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:25:59 smithi153 ceph-mon[106102]: pgmap v610: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:02 smithi203 ceph-mon[95988]: pgmap v611: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:02 smithi153 ceph-mon[106102]: pgmap v611: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:05.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:04 smithi203 ceph-mon[95988]: pgmap v612: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:04 smithi153 ceph-mon[106102]: pgmap v612: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:06 smithi203 ceph-mon[95988]: pgmap v613: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:06 smithi153 ceph-mon[106102]: pgmap v613: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:08.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:07 smithi153 ceph-mon[106102]: pgmap v614: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:08.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:07 smithi203 ceph-mon[95988]: pgmap v614: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:11.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:10 smithi203 ceph-mon[95988]: pgmap v615: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:11.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:10 smithi153 ceph-mon[106102]: pgmap v615: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:11.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- 172.21.15.153:0/4199435155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 msgr2=0x7fb7e80fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/4199435155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fe1e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc004940 tx=0x7fb7dc009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- 172.21.15.153:0/4199435155 shutdown_connections 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/4199435155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 0x7fb7e8106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/4199435155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fe1e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- 172.21.15.153:0/4199435155 >> 172.21.15.153:0/4199435155 conn(0x7fb7e80f7870 msgr2=0x7fb7e80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- 172.21.15.153:0/4199435155 shutdown_connections 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- 172.21.15.153:0/4199435155 wait complete. 2024-04-05T03:26:11.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 Processor -- start 2024-04-05T03:26:11.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.601+0000 7fb7ef15e700 1 -- start start 2024-04-05T03:26:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ef15e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ef15e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 0x7fb7e80fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ef15e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80fdc20 con 0x7fb7e80fde10 2024-04-05T03:26:11.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ef15e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80fc750 con 0x7fb7e80fe7b0 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38774/0 (socket says 172.21.15.153:38774) 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 -- 172.21.15.153:0/614355832 learned_addr learned my addr 172.21.15.153:0/614355832 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7e7fff700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 0x7fb7e80fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 -- 172.21.15.153:0/614355832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 msgr2=0x7fb7e80fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 0x7fb7e80fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 -- 172.21.15.153:0/614355832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc005040 con 0x7fb7e80fde10 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.602+0000 7fb7ecefa700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009380 tx=0x7fb7dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.603+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7dc00e040 con 0x7fb7e80fde10 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.603+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7dc009af0 con 0x7fb7e80fde10 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.603+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e80fc9d0 con 0x7fb7e80fde10 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.603+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7dc01c3f0 con 0x7fb7e80fde10 2024-04-05T03:26:11.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.603+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e811a0a0 con 0x7fb7e80fde10 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.604+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e8066af0 con 0x7fb7e80fde10 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.604+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb7dc013420 con 0x7fb7e80fde10 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.604+0000 7fb7e57fa700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 0x7fb7d8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.605+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb7dc08e8a0 con 0x7fb7e80fde10 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.605+0000 7fb7e7fff700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 0x7fb7d8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.605+0000 7fb7e7fff700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 0x7fb7d8070150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80fd400 tx=0x7fb7d40043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:11.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.609+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb7dc057e20 con 0x7fb7e80fde10 2024-04-05T03:26:11.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.822+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7e80f9490 con 0x7fb7d806dca0 2024-04-05T03:26:11.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.823+0000 7fb7e57fa700 1 -- 172.21.15.153:0/614355832 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb7e80f9490 con 0x7fb7d806dca0 2024-04-05T03:26:11.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 msgr2=0x7fb7d8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 0x7fb7d8070150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80fd400 tx=0x7fb7d40043c0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 msgr2=0x7fb7e80fbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009380 tx=0x7fb7dc009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 shutdown_connections 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb7d806dca0 0x7fb7d8070150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7e80fde10 0x7fb7e80fbcd0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 --2- 172.21.15.153:0/614355832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7e80fe7b0 0x7fb7e80fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 >> 172.21.15.153:0/614355832 conn(0x7fb7e80f7870 msgr2=0x7fb7e80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 shutdown_connections 2024-04-05T03:26:11.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.825+0000 7fb7ef15e700 1 -- 172.21.15.153:0/614355832 wait complete. 2024-04-05T03:26:11.834 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 -- 172.21.15.153:0/420381693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 msgr2=0x7f6b341031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/420381693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b341031d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6b30004940 tx=0x7f6b30009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 -- 172.21.15.153:0/420381693 shutdown_connections 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/420381693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b340fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/420381693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b341031d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.927+0000 7f6b3bb30700 1 -- 172.21.15.153:0/420381693 >> 172.21.15.153:0/420381693 conn(0x7f6b340f89b0 msgr2=0x7f6b340fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 -- 172.21.15.153:0/420381693 shutdown_connections 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 -- 172.21.15.153:0/420381693 wait complete. 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 Processor -- start 2024-04-05T03:26:11.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 -- start start 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b34070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340721c0 con 0x7f6b34102e00 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b3bb30700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34070d40 con 0x7f6b341037a0 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.928+0000 7f6b398cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38802/0 (socket says 172.21.15.153:38802) 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 -- 172.21.15.153:0/2304138073 learned_addr learned my addr 172.21.15.153:0/2304138073 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b390cb700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b34070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 -- 172.21.15.153:0/2304138073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 msgr2=0x7f6b34070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b34070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 -- 172.21.15.153:0/2304138073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30005040 con 0x7f6b34102e00 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b390cb700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b34070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b398cc700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000b3b0 tx=0x7f6b30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3000e040 con 0x7f6b34102e00 2024-04-05T03:26:11.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b3000bbc0 con 0x7f6b34102e00 2024-04-05T03:26:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b34070fc0 con 0x7f6b34102e00 2024-04-05T03:26:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3001c3f0 con 0x7f6b34102e00 2024-04-05T03:26:11.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.929+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3411a130 con 0x7f6b34102e00 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.930+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34100110 con 0x7f6b34102e00 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.931+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f6b3000bd30 con 0x7f6b34102e00 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.931+0000 7f6b2a7fc700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 0x7f6b20070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.931+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6b3008e7f0 con 0x7f6b34102e00 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.931+0000 7f6b390cb700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 0x7f6b20070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:11.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.932+0000 7f6b390cb700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 0x7f6b20070150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b24007fd0 tx=0x7f6b24009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:11.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:11.936+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6b30057d70 con 0x7f6b34102e00 2024-04-05T03:26:12.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.150+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b34071750 con 0x7f6b2006dca0 2024-04-05T03:26:12.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.152+0000 7f6b2a7fc700 1 -- 172.21.15.153:0/2304138073 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6b34071750 con 0x7f6b2006dca0 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 msgr2=0x7f6b20070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 0x7f6b20070150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6b24007fd0 tx=0x7f6b24009040 comp rx=0 tx=0).stop 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 msgr2=0x7f6b340702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000b3b0 tx=0x7f6b30009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 shutdown_connections 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f6b2006dca0 0x7f6b20070150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b34102e00 0x7f6b340702c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 --2- 172.21.15.153:0/2304138073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b341037a0 0x7f6b34070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 >> 172.21.15.153:0/2304138073 conn(0x7f6b340f89b0 msgr2=0x7f6b341073a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 shutdown_connections 2024-04-05T03:26:12.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.154+0000 7f6b3bb30700 1 -- 172.21.15.153:0/2304138073 wait complete. 2024-04-05T03:26:12.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- 172.21.15.153:0/459389873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c1026c0 msgr2=0x7f1f7c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 --2- 172.21.15.153:0/459389873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c0fbd30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1f78004940 tx=0x7f1f78009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- 172.21.15.153:0/459389873 shutdown_connections 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 --2- 172.21.15.153:0/459389873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c0fbd30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 --2- 172.21.15.153:0/459389873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c101d20 0x7f1f7c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- 172.21.15.153:0/459389873 >> 172.21.15.153:0/459389873 conn(0x7f1f7c0f7870 msgr2=0x7f1f7c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- 172.21.15.153:0/459389873 shutdown_connections 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- 172.21.15.153:0/459389873 wait complete. 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 Processor -- start 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.255+0000 7f1f847c8700 1 -- start start 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f847c8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f847c8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f847c8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c114520 con 0x7f1f7c101d20 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f847c8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c10db40 con 0x7f1f7c1026c0 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38824/0 (socket says 172.21.15.153:38824) 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 -- 172.21.15.153:0/431327455 learned_addr learned my addr 172.21.15.153:0/431327455 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f81d63700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 -- 172.21.15.153:0/431327455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c1026c0 msgr2=0x7f1f7c113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 -- 172.21.15.153:0/431327455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78005040 con 0x7f1f7c101d20 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f82564700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c002a50 tx=0x7f1f6c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f6c007cc0 con 0x7f1f7c101d20 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f6c00f040 con 0x7f1f7c101d20 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.256+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c10de20 con 0x7f1f7c101d20 2024-04-05T03:26:12.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.257+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f6c009640 con 0x7f1f7c101d20 2024-04-05T03:26:12.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.257+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c10e3e0 con 0x7f1f7c101d20 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.257+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c0fd470 con 0x7f1f7c101d20 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.259+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1f6c0097c0 con 0x7f1f7c101d20 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.259+0000 7f1f72ffd700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 0x7f1f68070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.259+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1f6c08d260 con 0x7f1f7c101d20 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.259+0000 7f1f81d63700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 0x7f1f68070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.259+0000 7f1f81d63700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 0x7f1f68070150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c10f230 tx=0x7f1f78009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.264+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1f6c056860 con 0x7f1f7c101d20 2024-04-05T03:26:12.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.468+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7c10f080 con 0x7f1f6806dca0 2024-04-05T03:26:12.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.478+0000 7f1f72ffd700 1 -- 172.21.15.153:0/431327455 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1f7c10f080 con 0x7f1f6806dca0 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (20m) 14m ago 31m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (18m) 14m ago 33m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (18m) 13m ago 30m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (20m) 14m ago 31m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (14m) 14m ago 26m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (14m) 14m ago 26m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (13m) 13m ago 26m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (14m) 13m ago 26m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (22m) 14m ago 34m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (21m) 13m ago 30m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (19m) 14m ago 34m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:26:12.477 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (18m) 13m ago 30m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (21m) 14m ago 32m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (21m) 13m ago 30m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (17m) 14m ago 30m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (17m) 14m ago 29m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (16m) 14m ago 29m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (16m) 13m ago 28m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (15m) 13m ago 28m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (14m) 13m ago 27m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:26:12.478 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (21m) 13m ago 30m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:26:12.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 msgr2=0x7f1f68070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 0x7f1f68070150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c10f230 tx=0x7f1f78009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 msgr2=0x7f1f7c113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c002a50 tx=0x7f1f6c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 shutdown_connections 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1f6806dca0 0x7f1f68070150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f7c101d20 0x7f1f7c113990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 --2- 172.21.15.153:0/431327455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f7c1026c0 0x7f1f7c113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 >> 172.21.15.153:0/431327455 conn(0x7f1f7c0f7870 msgr2=0x7f1f7c0662c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 shutdown_connections 2024-04-05T03:26:12.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.481+0000 7f1f847c8700 1 -- 172.21.15.153:0/431327455 wait complete. 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- 172.21.15.153:0/811813694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fe7c0 msgr2=0x7f9f20106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 --2- 172.21.15.153:0/811813694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fe7c0 0x7f9f20106e90 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9f10004940 tx=0x7f9f10009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- 172.21.15.153:0/811813694 shutdown_connections 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 --2- 172.21.15.153:0/811813694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fe7c0 0x7f9f20106e90 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 --2- 172.21.15.153:0/811813694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fde20 0x7f9f200fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- 172.21.15.153:0/811813694 >> 172.21.15.153:0/811813694 conn(0x7f9f200f78e0 msgr2=0x7f9f200f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- 172.21.15.153:0/811813694 shutdown_connections 2024-04-05T03:26:12.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- 172.21.15.153:0/811813694 wait complete. 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 Processor -- start 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.592+0000 7f9f27272700 1 -- start start 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f27272700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f27272700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 0x7f9f20113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f27272700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20114510 con 0x7f9f200fde20 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f27272700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2010db10 con 0x7f9f200fe7c0 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38840/0 (socket says 172.21.15.153:38840) 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 -- 172.21.15.153:0/84145935 learned_addr learned my addr 172.21.15.153:0/84145935 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2480d700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 0x7f9f20113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 -- 172.21.15.153:0/84145935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 msgr2=0x7f9f20113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 0x7f9f20113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 -- 172.21.15.153:0/84145935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10005040 con 0x7f9f200fde20 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2480d700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 0x7f9f20113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f2500e700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c810 tx=0x7f9f1c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1c012810 con 0x7f9f200fde20 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f1c012e40 con 0x7f9f200fde20 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2010ddf0 con 0x7f9f200fde20 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.593+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f1c00a370 con 0x7f9f200fde20 2024-04-05T03:26:12.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.594+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2010e330 con 0x7f9f200fde20 2024-04-05T03:26:12.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.595+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f20066ad0 con 0x7f9f200fde20 2024-04-05T03:26:12.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.595+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9f1c011810 con 0x7f9f200fde20 2024-04-05T03:26:12.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.595+0000 7f9f15ffb700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 0x7f9f0c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.595+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9f1c08d980 con 0x7f9f200fde20 2024-04-05T03:26:12.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.596+0000 7f9f2480d700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 0x7f9f0c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.596+0000 7f9f2480d700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 0x7f9f0c070150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000b3b0 tx=0x7f9f10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.600+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9f1c05aa50 con 0x7f9f200fde20 2024-04-05T03:26:12.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.881+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9f20109ed0 con 0x7f9f200fde20 2024-04-05T03:26:12.880 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:12 smithi153 ceph-mon[106102]: pgmap v616: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:12.880 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:12.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.882+0000 7f9f15ffb700 1 -- 172.21.15.153:0/84145935 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9f1c05a5e0 con 0x7f9f200fde20 2024-04-05T03:26:12.880 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:26:12.880 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:26:12.881 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:26:12.882 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:26:12.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 msgr2=0x7f9f0c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 0x7f9f0c070150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000b3b0 tx=0x7f9f10009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 msgr2=0x7f9f20113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c810 tx=0x7f9f1c004470 comp rx=0 tx=0).stop 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 shutdown_connections 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9f0c06dca0 0x7f9f0c070150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f200fde20 0x7f9f20113980 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 --2- 172.21.15.153:0/84145935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f200fe7c0 0x7f9f20113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 >> 172.21.15.153:0/84145935 conn(0x7f9f200f78e0 msgr2=0x7f9f200f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 shutdown_connections 2024-04-05T03:26:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.884+0000 7f9f27272700 1 -- 172.21.15.153:0/84145935 wait complete. 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 -- 172.21.15.153:0/3675763842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 msgr2=0x7ff8141031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/3675763842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff8141031f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff810004940 tx=0x7ff810009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 -- 172.21.15.153:0/3675763842 shutdown_connections 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/3675763842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 0x7ff8140fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/3675763842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff8141031f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.985+0000 7ff81b2c1700 1 -- 172.21.15.153:0/3675763842 >> 172.21.15.153:0/3675763842 conn(0x7ff8140f89b0 msgr2=0x7ff8140fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 -- 172.21.15.153:0/3675763842 shutdown_connections 2024-04-05T03:26:12.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 -- 172.21.15.153:0/3675763842 wait complete. 2024-04-05T03:26:12.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 Processor -- start 2024-04-05T03:26:12.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 -- start start 2024-04-05T03:26:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 0x7ff814113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff814114500 con 0x7ff814102e20 2024-04-05T03:26:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81b2c1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81410db50 con 0x7ff8141037c0 2024-04-05T03:26:12.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81905d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81905d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38860/0 (socket says 172.21.15.153:38860) 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.986+0000 7ff81905d700 1 -- 172.21.15.153:0/171797285 learned_addr learned my addr 172.21.15.153:0/171797285 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81885c700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 0x7ff814113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81905d700 1 -- 172.21.15.153:0/171797285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 msgr2=0x7ff814113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81905d700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 0x7ff814113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81905d700 1 -- 172.21.15.153:0/171797285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810005040 con 0x7ff814102e20 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81905d700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b3b0 tx=0x7ff810009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81000e040 con 0x7ff814102e20 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff81000bbc0 con 0x7ff814102e20 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81410ddd0 con 0x7ff814102e20 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81001c3f0 con 0x7ff814102e20 2024-04-05T03:26:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.987+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81410e390 con 0x7ff814102e20 2024-04-05T03:26:12.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.988+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff814100130 con 0x7ff814102e20 2024-04-05T03:26:12.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.989+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff81000bd30 con 0x7ff814102e20 2024-04-05T03:26:12.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.989+0000 7ff809ffb700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 0x7ff800070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:12.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.989+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff81008e830 con 0x7ff814102e20 2024-04-05T03:26:12.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.989+0000 7ff81885c700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 0x7ff800070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:12.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.990+0000 7ff81885c700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 0x7ff800070150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff81410ecc0 tx=0x7ff80400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:12.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:12.994+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff81005b8e0 con 0x7ff814102e20 2024-04-05T03:26:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:12 smithi203 ceph-mon[95988]: pgmap v616: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.229+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff81410efc0 con 0x7ff814102e20 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.230+0000 7ff809ffb700 1 -- 172.21.15.153:0/171797285 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7ff81005b470 con 0x7ff814102e20 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:26:13.228 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:26:13.229 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:13.230 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:26:13.231 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:13.231 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:13.231 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:13.231 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:13.231 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 msgr2=0x7ff800070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 0x7ff800070150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff81410ecc0 tx=0x7ff80400b040 comp rx=0 tx=0).stop 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 msgr2=0x7ff814113970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b3b0 tx=0x7ff810009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 shutdown_connections 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff80006dca0 0x7ff800070150 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814102e20 0x7ff814113970 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 --2- 172.21.15.153:0/171797285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8141037c0 0x7ff814113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 >> 172.21.15.153:0/171797285 conn(0x7ff8140f89b0 msgr2=0x7ff814107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 shutdown_connections 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.232+0000 7ff81b2c1700 1 -- 172.21.15.153:0/171797285 wait complete. 2024-04-05T03:26:13.232 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- 172.21.15.153:0/3301882868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb8101d20 msgr2=0x7fbcb81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 --2- 172.21.15.153:0/3301882868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb8101d20 0x7fbcb81020f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4004940 tx=0x7fbcb4009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- 172.21.15.153:0/3301882868 shutdown_connections 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 --2- 172.21.15.153:0/3301882868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb81026c0 0x7fbcb80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 --2- 172.21.15.153:0/3301882868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb8101d20 0x7fbcb81020f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- 172.21.15.153:0/3301882868 >> 172.21.15.153:0/3301882868 conn(0x7fbcb80f7870 msgr2=0x7fbcb80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- 172.21.15.153:0/3301882868 shutdown_connections 2024-04-05T03:26:13.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- 172.21.15.153:0/3301882868 wait complete. 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 Processor -- start 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.335+0000 7fbcbf973700 1 -- start start 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbf973700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb8101d20 0x7fbcb8112860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbf973700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbf973700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb81133f0 con 0x7fbcb81026c0 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbf973700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb810c9f0 con 0x7fbcb8101d20 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38876/0 (socket says 172.21.15.153:38876) 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 -- 172.21.15.153:0/2273292693 learned_addr learned my addr 172.21.15.153:0/2273292693 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbd70f700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb8101d20 0x7fbcb8112860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 -- 172.21.15.153:0/2273292693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb8101d20 msgr2=0x7fbcb8112860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb8101d20 0x7fbcb8112860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 -- 172.21.15.153:0/2273292693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb4005040 con 0x7fbcb81026c0 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbcf0e700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbca800eb40 tx=0x7fbca800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca8007cc0 con 0x7fbcb81026c0 2024-04-05T03:26:13.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbca800f040 con 0x7fbcb81026c0 2024-04-05T03:26:13.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb810cc70 con 0x7fbcb81026c0 2024-04-05T03:26:13.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.336+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca800cad0 con 0x7fbcb81026c0 2024-04-05T03:26:13.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.337+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb810d1b0 con 0x7fbcb81026c0 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.338+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb80fd470 con 0x7fbcb81026c0 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.338+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbca800cc30 con 0x7fbcb81026c0 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.339+0000 7fbcadffb700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 0x7fbca4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.339+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbca808d030 con 0x7fbcb81026c0 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.339+0000 7fbcbd70f700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 0x7fbca4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.339+0000 7fbcbd70f700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 0x7fbca4070150 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009b30 tx=0x7fbcb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:13.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.343+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbca805a160 con 0x7fbcb81026c0 2024-04-05T03:26:13.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.559+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcb8112530 con 0x7fbca406dca0 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.561+0000 7fbcadffb700 1 -- 172.21.15.153:0/2273292693 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbcb8112530 con 0x7fbca406dca0 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:26:13.559 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:26:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:26:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:26:13.560 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:26:13.560 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 msgr2=0x7fbca4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 0x7fbca4070150 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009b30 tx=0x7fbcb4009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 msgr2=0x7fbcb8112da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbca800eb40 tx=0x7fbca800b040 comp rx=0 tx=0).stop 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 shutdown_connections 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbca406dca0 0x7fbca4070150 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcb8101d20 0x7fbcb8112860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 --2- 172.21.15.153:0/2273292693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcb81026c0 0x7fbcb8112da0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.563+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 >> 172.21.15.153:0/2273292693 conn(0x7fbcb80f7870 msgr2=0x7fbcb80f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.564+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 shutdown_connections 2024-04-05T03:26:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.564+0000 7fbcbf973700 1 -- 172.21.15.153:0/2273292693 wait complete. 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- 172.21.15.153:0/4246175655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc1037b0 msgr2=0x7f61fc0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 --2- 172.21.15.153:0/4246175655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc1037b0 0x7f61fc0fce20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004940 tx=0x7f61ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- 172.21.15.153:0/4246175655 shutdown_connections 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 --2- 172.21.15.153:0/4246175655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc1037b0 0x7f61fc0fce20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 --2- 172.21.15.153:0/4246175655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc102e10 0x7f61fc1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- 172.21.15.153:0/4246175655 >> 172.21.15.153:0/4246175655 conn(0x7f61fc0f8980 msgr2=0x7f61fc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- 172.21.15.153:0/4246175655 shutdown_connections 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- 172.21.15.153:0/4246175655 wait complete. 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 Processor -- start 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.664+0000 7f62043d5700 1 -- start start 2024-04-05T03:26:13.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f62043d5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f62043d5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc1037b0 0x7f61fc1140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f62043d5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc114710 con 0x7f61fc102e10 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f62043d5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc10db00 con 0x7f61fc1037b0 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38894/0 (socket says 172.21.15.153:38894) 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 -- 172.21.15.153:0/1344679723 learned_addr learned my addr 172.21.15.153:0/1344679723 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6201970700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc1037b0 0x7f61fc1140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 -- 172.21.15.153:0/1344679723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc1037b0 msgr2=0x7f61fc1140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc1037b0 0x7f61fc1140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 -- 172.21.15.153:0/1344679723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec005040 con 0x7f61fc102e10 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f6202171700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f61f8002a50 tx=0x7f61f80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f8007cc0 con 0x7f61fc102e10 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61f800f040 con 0x7f61fc102e10 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc10dde0 con 0x7f61fc102e10 2024-04-05T03:26:13.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.665+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61f800da30 con 0x7f61fc102e10 2024-04-05T03:26:13.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.666+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc10e370 con 0x7f61fc102e10 2024-04-05T03:26:13.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.666+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc1052e0 con 0x7f61fc102e10 2024-04-05T03:26:13.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.667+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f61f800db90 con 0x7f61fc102e10 2024-04-05T03:26:13.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.667+0000 7f61f2ffd700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 0x7f61e8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:13.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.667+0000 7f6201970700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 0x7f61e8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:13.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.667+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f61f808d310 con 0x7f61fc102e10 2024-04-05T03:26:13.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.668+0000 7f6201970700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 0x7f61e8070150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004890 tx=0x7f61ec009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:13.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.672+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f61f805a3c0 con 0x7f61fc102e10 2024-04-05T03:26:13.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:13 smithi153 ceph-mon[106102]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:13.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:13 smithi153 ceph-mon[106102]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:13.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:13 smithi153 ceph-mon[106102]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:13.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:13 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/84145935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:26:13.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:13 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/171797285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:26:13.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.950+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f61fc10f1d0 con 0x7f61fc102e10 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.950+0000 7f61f2ffd700 1 -- 172.21.15.153:0/1344679723 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f61f8059f50 con 0x7f61fc102e10 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:26:13.949 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 msgr2=0x7f61e8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 0x7f61e8070150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f61ec004890 tx=0x7f61ec009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 msgr2=0x7f61fc113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f61f8002a50 tx=0x7f61f80046b0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 shutdown_connections 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f61e806dca0 0x7f61e8070150 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61fc102e10 0x7f61fc113b80 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 --2- 172.21.15.153:0/1344679723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61fc1037b0 0x7f61fc1140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.952+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 >> 172.21.15.153:0/1344679723 conn(0x7f61fc0f8980 msgr2=0x7f61fc1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:13.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.953+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 shutdown_connections 2024-04-05T03:26:13.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:13.953+0000 7f62043d5700 1 -- 172.21.15.153:0/1344679723 wait complete. 2024-04-05T03:26:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:13 smithi203 ceph-mon[95988]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:13 smithi203 ceph-mon[95988]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:13 smithi203 ceph-mon[95988]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:13 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/84145935' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:26:14.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:13 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/171797285' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:26:15.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:14 smithi203 ceph-mon[95988]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:14 smithi203 ceph-mon[95988]: pgmap v617: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:14 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1344679723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:26:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:14 smithi153 ceph-mon[106102]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:14 smithi153 ceph-mon[106102]: pgmap v617: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:15.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:14 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1344679723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:26:16.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:15 smithi203 ceph-mon[95988]: pgmap v618: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:16.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:15 smithi153 ceph-mon[106102]: pgmap v618: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:18 smithi203 ceph-mon[95988]: pgmap v619: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:18 smithi153 ceph-mon[106102]: pgmap v619: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:21.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:20 smithi203 ceph-mon[95988]: pgmap v620: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:21.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:20 smithi153 ceph-mon[106102]: pgmap v620: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:22.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:21 smithi203 ceph-mon[95988]: pgmap v621: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:22.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:21 smithi153 ceph-mon[106102]: pgmap v621: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:24 smithi203 ceph-mon[95988]: pgmap v622: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:24 smithi153 ceph-mon[106102]: pgmap v622: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:26 smithi203 ceph-mon[95988]: pgmap v623: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:26 smithi153 ceph-mon[106102]: pgmap v623: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:28 smithi203 ceph-mon[95988]: pgmap v624: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:28 smithi153 ceph-mon[106102]: pgmap v624: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:30.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:29 smithi203 ceph-mon[95988]: pgmap v625: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:29 smithi153 ceph-mon[106102]: pgmap v625: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:32 smithi203 ceph-mon[95988]: pgmap v626: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:32 smithi153 ceph-mon[106102]: pgmap v626: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:34 smithi203 ceph-mon[95988]: pgmap v627: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:34 smithi153 ceph-mon[106102]: pgmap v627: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:36.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:35 smithi203 ceph-mon[95988]: pgmap v628: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:36.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:35 smithi153 ceph-mon[106102]: pgmap v628: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:39.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:38 smithi203 ceph-mon[95988]: pgmap v629: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:39.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:38 smithi153 ceph-mon[106102]: pgmap v629: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:40.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:39 smithi203 ceph-mon[95988]: pgmap v630: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:40.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:39 smithi153 ceph-mon[106102]: pgmap v630: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:42 smithi203 ceph-mon[95988]: pgmap v631: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:42 smithi153 ceph-mon[106102]: pgmap v631: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:43.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:44.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 -- 172.21.15.153:0/2767141504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc102e10 msgr2=0x7fa8fc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 --2- 172.21.15.153:0/2767141504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc102e10 0x7fa8fc1031e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0004940 tx=0x7fa8f0009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 -- 172.21.15.153:0/2767141504 shutdown_connections 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 --2- 172.21.15.153:0/2767141504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 --2- 172.21.15.153:0/2767141504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc102e10 0x7fa8fc1031e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 -- 172.21.15.153:0/2767141504 >> 172.21.15.153:0/2767141504 conn(0x7fa8fc0f89a0 msgr2=0x7fa8fc0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 -- 172.21.15.153:0/2767141504 shutdown_connections 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 -- 172.21.15.153:0/2767141504 wait complete. 2024-04-05T03:26:44.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.068+0000 7fa901e53700 1 Processor -- start 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa901e53700 1 -- start start 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa901e53700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc102e10 0x7fa8fc113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa901e53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa901e53700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc114740 con 0x7fa8fc1037b0 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa901e53700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8fc10dba0 con 0x7fa8fc102e10 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60156/0 (socket says 172.21.15.153:60156) 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 -- 172.21.15.153:0/382162285 learned_addr learned my addr 172.21.15.153:0/382162285 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8fb7fe700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc102e10 0x7fa8fc113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 -- 172.21.15.153:0/382162285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc102e10 msgr2=0x7fa8fc113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc102e10 0x7fa8fc113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.069+0000 7fa8faffd700 1 -- 172.21.15.153:0/382162285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f0005040 con 0x7fa8fc1037b0 2024-04-05T03:26:44.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa8faffd700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec002a80 tx=0x7fa8ec004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8ec007cc0 con 0x7fa8fc1037b0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc10de80 con 0x7fa8fc1037b0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8ec00f040 con 0x7fa8fc1037b0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8ec00daf0 con 0x7fa8fc1037b0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.070+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8fc10e440 con 0x7fa8fc1037b0 2024-04-05T03:26:44.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.071+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc100120 con 0x7fa8fc1037b0 2024-04-05T03:26:44.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.071+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa8ec00dc90 con 0x7fa8fc1037b0 2024-04-05T03:26:44.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.072+0000 7fa8e3fff700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 0x7fa8e4070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.072+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa8ec08cc70 con 0x7fa8fc1037b0 2024-04-05T03:26:44.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.072+0000 7fa8fb7fe700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 0x7fa8e4070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.072+0000 7fa8fb7fe700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 0x7fa8e4070160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b3b0 tx=0x7fa8f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.077+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa8ec0561f0 con 0x7fa8fc1037b0 2024-04-05T03:26:44.075 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:43 smithi153 ceph-mon[106102]: pgmap v632: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:44.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.290+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8fc10f200 con 0x7fa8e406dcb0 2024-04-05T03:26:44.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.291+0000 7fa8e3fff700 1 -- 172.21.15.153:0/382162285 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa8fc10f200 con 0x7fa8e406dcb0 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.293+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 msgr2=0x7fa8e4070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.293+0000 7fa901e53700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 0x7fa8e4070160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000b3b0 tx=0x7fa8f0009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.293+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 msgr2=0x7fa8fc1140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.293+0000 7fa901e53700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec002a80 tx=0x7fa8ec004630 comp rx=0 tx=0).stop 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 shutdown_connections 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa8e406dcb0 0x7fa8e4070160 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8fc102e10 0x7fa8fc113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 --2- 172.21.15.153:0/382162285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8fc1037b0 0x7fa8fc1140f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 >> 172.21.15.153:0/382162285 conn(0x7fa8fc0f89a0 msgr2=0x7fa8fc0facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 shutdown_connections 2024-04-05T03:26:44.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.294+0000 7fa901e53700 1 -- 172.21.15.153:0/382162285 wait complete. 2024-04-05T03:26:44.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:43 smithi203 ceph-mon[95988]: pgmap v632: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:44.302 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:26:44.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 -- 172.21.15.153:0/1018116020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 msgr2=0x7f08e80fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 --2- 172.21.15.153:0/1018116020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fb820 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f08e4004940 tx=0x7f08e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 -- 172.21.15.153:0/1018116020 shutdown_connections 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 --2- 172.21.15.153:0/1018116020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fb820 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 --2- 172.21.15.153:0/1018116020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 0x7f08e8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 -- 172.21.15.153:0/1018116020 >> 172.21.15.153:0/1018116020 conn(0x7f08e8072310 msgr2=0x7f08e8072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 -- 172.21.15.153:0/1018116020 shutdown_connections 2024-04-05T03:26:44.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.396+0000 7f08ef828700 1 -- 172.21.15.153:0/1018116020 wait complete. 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 Processor -- start 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 -- start start 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 0x7f08e80fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e80fe4e0 con 0x7f08e80fb3b0 2024-04-05T03:26:44.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ef828700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e80fd010 con 0x7f08e8102f70 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60174/0 (socket says 172.21.15.153:60174) 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 -- 172.21.15.153:0/3320006035 learned_addr learned my addr 172.21.15.153:0/3320006035 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ecdc3700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 0x7f08e80fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 -- 172.21.15.153:0/3320006035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 msgr2=0x7f08e80fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 0x7f08e80fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.397+0000 7f08ed5c4700 1 -- 172.21.15.153:0/3320006035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4005040 con 0x7f08e80fb3b0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ed5c4700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f08d800eb40 tx=0x7f08d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d8007cc0 con 0x7f08e80fb3b0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08d800f040 con 0x7f08e80fb3b0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e80fd290 con 0x7f08e80fb3b0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08d8009620 con 0x7f08e80fb3b0 2024-04-05T03:26:44.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.398+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e811a030 con 0x7f08e80fb3b0 2024-04-05T03:26:44.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.399+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8105350 con 0x7f08e80fb3b0 2024-04-05T03:26:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.399+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f08d8009780 con 0x7f08e80fb3b0 2024-04-05T03:26:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.400+0000 7f08ddffb700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 0x7f08d4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.400+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f08d808d220 con 0x7f08e80fb3b0 2024-04-05T03:26:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.400+0000 7f08ecdc3700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 0x7f08d4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.400+0000 7f08ecdc3700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 0x7f08d4070150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f08e80fdcc0 tx=0x7f08e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.405+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f08d8056850 con 0x7f08e80fb3b0 2024-04-05T03:26:44.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.618+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f08e80fda70 con 0x7f08d406dca0 2024-04-05T03:26:44.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.620+0000 7f08ddffb700 1 -- 172.21.15.153:0/3320006035 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f08e80fda70 con 0x7f08d406dca0 2024-04-05T03:26:44.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 msgr2=0x7f08d4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 0x7f08d4070150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f08e80fdcc0 tx=0x7f08e4009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 msgr2=0x7f08e80fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f08d800eb40 tx=0x7f08d800b040 comp rx=0 tx=0).stop 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 shutdown_connections 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08d406dca0 0x7f08d4070150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08e80fb3b0 0x7f08e80fc590 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 --2- 172.21.15.153:0/3320006035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e8102f70 0x7f08e80fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 >> 172.21.15.153:0/3320006035 conn(0x7f08e8072310 msgr2=0x7f08e80f9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 shutdown_connections 2024-04-05T03:26:44.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.622+0000 7f08ef828700 1 -- 172.21.15.153:0/3320006035 wait complete. 2024-04-05T03:26:44.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 -- 172.21.15.153:0/728694028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fba90 msgr2=0x7f1654101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 --2- 172.21.15.153:0/728694028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fba90 0x7f1654101a80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1644004940 tx=0x7f1644009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 -- 172.21.15.153:0/728694028 shutdown_connections 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 --2- 172.21.15.153:0/728694028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fba90 0x7f1654101a80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 --2- 172.21.15.153:0/728694028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fb0f0 0x7f16540fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 -- 172.21.15.153:0/728694028 >> 172.21.15.153:0/728694028 conn(0x7f16540f6bd0 msgr2=0x7f16540f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.723+0000 7f165aa5a700 1 -- 172.21.15.153:0/728694028 shutdown_connections 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 -- 172.21.15.153:0/728694028 wait complete. 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 Processor -- start 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 -- start start 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fba90 0x7f1654113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654114580 con 0x7f16540fb0f0 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.724+0000 7f165aa5a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165410db80 con 0x7f16540fba90 2024-04-05T03:26:44.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60196/0 (socket says 172.21.15.153:60196) 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 -- 172.21.15.153:0/2785986212 learned_addr learned my addr 172.21.15.153:0/2785986212 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f16537fe700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fba90 0x7f1654113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 -- 172.21.15.153:0/2785986212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fba90 msgr2=0x7f1654113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fba90 0x7f1654113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 -- 172.21.15.153:0/2785986212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1644005040 con 0x7f16540fb0f0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1653fff700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f164800c830 tx=0x7f1648004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164800a760 con 0x7f16540fb0f0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165410de60 con 0x7f16540fb0f0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f164800ad90 con 0x7f16540fb0f0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16480113a0 con 0x7f16540fb0f0 2024-04-05T03:26:44.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.725+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165410e370 con 0x7f16540fb0f0 2024-04-05T03:26:44.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.726+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16540fc320 con 0x7f16540fb0f0 2024-04-05T03:26:44.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.727+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1648012400 con 0x7f16540fb0f0 2024-04-05T03:26:44.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.727+0000 7f1650ff9700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 0x7f163c070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:44.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.727+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f164808d600 con 0x7f16540fb0f0 2024-04-05T03:26:44.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.727+0000 7f16537fe700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 0x7f163c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:44.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.728+0000 7f16537fe700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 0x7f163c070150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f164400b3b0 tx=0x7f1644009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:44.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.732+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f164805a730 con 0x7f16540fb0f0 2024-04-05T03:26:44.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.936+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f165410f070 con 0x7f163c06dca0 2024-04-05T03:26:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.946+0000 7f1650ff9700 1 -- 172.21.15.153:0/2785986212 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f165410f070 con 0x7f163c06dca0 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (21m) 14m ago 32m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (18m) 14m ago 33m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (18m) 14m ago 31m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (21m) 14m ago 32m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (14m) 14m ago 26m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (15m) 14m ago 26m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (14m) 14m ago 26m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (14m) 14m ago 26m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (22m) 14m ago 34m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (22m) 14m ago 31m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (19m) 14m ago 34m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:26:44.945 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (19m) 14m ago 31m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (21m) 14m ago 32m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (21m) 14m ago 31m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (18m) 14m ago 30m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (17m) 14m ago 30m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (17m) 14m ago 29m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (16m) 14m ago 29m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (16m) 14m ago 28m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (15m) 14m ago 28m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:26:44.946 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (21m) 14m ago 30m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.948+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 msgr2=0x7f163c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.948+0000 7f165aa5a700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 0x7f163c070150 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f164400b3b0 tx=0x7f1644009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 msgr2=0x7f16541139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f164800c830 tx=0x7f1648004470 comp rx=0 tx=0).stop 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 shutdown_connections 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f163c06dca0 0x7f163c070150 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16540fb0f0 0x7f16541139f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 --2- 172.21.15.153:0/2785986212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16540fba90 0x7f1654113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 >> 172.21.15.153:0/2785986212 conn(0x7f16540f6bd0 msgr2=0x7f16540f7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:44.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 shutdown_connections 2024-04-05T03:26:44.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:44.949+0000 7f165aa5a700 1 -- 172.21.15.153:0/2785986212 wait complete. 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 -- 172.21.15.153:0/1548135281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7500fb4d0 msgr2=0x7fb7500fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 --2- 172.21.15.153:0/1548135281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7500fb4d0 0x7fb7500fb940 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb744004940 tx=0x7fb744009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 -- 172.21.15.153:0/1548135281 shutdown_connections 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 --2- 172.21.15.153:0/1548135281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb7500fb4d0 0x7fb7500fb940 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 --2- 172.21.15.153:0/1548135281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb750102f50 0x7fb750103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 -- 172.21.15.153:0/1548135281 >> 172.21.15.153:0/1548135281 conn(0x7fb7500714c0 msgr2=0x7fb7500718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:45.049 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:44 smithi153 ceph-mon[106102]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:45.049 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:44 smithi153 ceph-mon[106102]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:45.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.049+0000 7fb755220700 1 -- 172.21.15.153:0/1548135281 shutdown_connections 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 -- 172.21.15.153:0/1548135281 wait complete. 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 Processor -- start 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 -- start start 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7500fb4d0 0x7fb7501139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb750114530 con 0x7fb750102f50 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.050+0000 7fb755220700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75010db50 con 0x7fb7500fb4d0 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60210/0 (socket says 172.21.15.153:60210) 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 -- 172.21.15.153:0/3440438221 learned_addr learned my addr 172.21.15.153:0/3440438221 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74ed9d700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7500fb4d0 0x7fb7501139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 -- 172.21.15.153:0/3440438221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7500fb4d0 msgr2=0x7fb7501139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7500fb4d0 0x7fb7501139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 -- 172.21.15.153:0/3440438221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744005040 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb74e59c700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb74400b3b0 tx=0x7fb744009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74400e040 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75010ddd0 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb744013420 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb74401c630 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.051+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75010e310 con 0x7fb750102f50 2024-04-05T03:26:45.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.052+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb750105330 con 0x7fb750102f50 2024-04-05T03:26:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.053+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb744013590 con 0x7fb750102f50 2024-04-05T03:26:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.053+0000 7fb7377fe700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 0x7fb738070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.053+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb74408e760 con 0x7fb750102f50 2024-04-05T03:26:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.053+0000 7fb74ed9d700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 0x7fb738070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.054+0000 7fb74ed9d700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 0x7fb738070150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb750065bf0 tx=0x7fb740009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.058+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb744057d60 con 0x7fb750102f50 2024-04-05T03:26:45.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:44 smithi203 ceph-mon[95988]: from='client.35182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:45.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:44 smithi203 ceph-mon[95988]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.335+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb75010eff0 con 0x7fb750102f50 2024-04-05T03:26:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.336+0000 7fb7377fe700 1 -- 172.21.15.153:0/3440438221 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb74405b420 con 0x7fb750102f50 2024-04-05T03:26:45.334 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:26:45.334 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:26:45.334 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:26:45.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 msgr2=0x7fb738070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 0x7fb738070150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb750065bf0 tx=0x7fb740009040 comp rx=0 tx=0).stop 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 msgr2=0x7fb750113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb74400b3b0 tx=0x7fb744009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 shutdown_connections 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb73806dca0 0x7fb738070150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7500fb4d0 0x7fb7501139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 --2- 172.21.15.153:0/3440438221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb750102f50 0x7fb750113ee0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 >> 172.21.15.153:0/3440438221 conn(0x7fb7500714c0 msgr2=0x7fb7500fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 shutdown_connections 2024-04-05T03:26:45.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.338+0000 7fb755220700 1 -- 172.21.15.153:0/3440438221 wait complete. 2024-04-05T03:26:45.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.438+0000 7f129e92b700 1 -- 172.21.15.153:0/2549597409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 msgr2=0x7f1298102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.438+0000 7f129e92b700 1 --2- 172.21.15.153:0/2549597409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f1298102140 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1288004940 tx=0x7f1288009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- 172.21.15.153:0/2549597409 shutdown_connections 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 --2- 172.21.15.153:0/2549597409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f12980fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 --2- 172.21.15.153:0/2549597409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f1298102140 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- 172.21.15.153:0/2549597409 >> 172.21.15.153:0/2549597409 conn(0x7f1298072bc0 msgr2=0x7f1298072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:45.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- 172.21.15.153:0/2549597409 shutdown_connections 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- 172.21.15.153:0/2549597409 wait complete. 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 Processor -- start 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- start start 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f129810f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129810f650 con 0x7f1298101d70 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.439+0000 7f129e92b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129810f790 con 0x7f1298102710 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60234/0 (socket says 172.21.15.153:60234) 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 -- 172.21.15.153:0/823363202 learned_addr learned my addr 172.21.15.153:0/823363202 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f12977fe700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f129810f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 -- 172.21.15.153:0/823363202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 msgr2=0x7f129810f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f129810f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 -- 172.21.15.153:0/823363202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288005040 con 0x7f1298101d70 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f12977fe700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f129810f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1297fff700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1288009b30 tx=0x7f1288009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128800e070 con 0x7f1298101d70 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1298117b30 con 0x7f1298101d70 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1288013cd0 con 0x7f1298101d70 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128800b720 con 0x7f1298101d70 2024-04-05T03:26:45.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.440+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1298117f90 con 0x7f1298101d70 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.441+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12980ff080 con 0x7f1298101d70 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.442+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f128801cab0 con 0x7f1298101d70 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.442+0000 7f1294ff9700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 0x7f1280070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.442+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f128808ef00 con 0x7f1298101d70 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.442+0000 7f12977fe700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 0x7f1280070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.443+0000 7f12977fe700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 0x7f1280070150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f128c004600 tx=0x7f128c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.447+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1288058500 con 0x7f1298101d70 2024-04-05T03:26:45.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.686+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1298108ed0 con 0x7f1298101d70 2024-04-05T03:26:45.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.686+0000 7f1294ff9700 1 -- 172.21.15.153:0/823363202 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f128805bbc0 con 0x7f1298101d70 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:26:45.685 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:26:45.686 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:45.687 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:45.688 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:45.688 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:26:45.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 msgr2=0x7f1280070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 0x7f1280070150 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f128c004600 tx=0x7f128c00b040 comp rx=0 tx=0).stop 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 msgr2=0x7f129810eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1288009b30 tx=0x7f1288009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 shutdown_connections 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f128006dca0 0x7f1280070150 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1298101d70 0x7f129810eac0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 --2- 172.21.15.153:0/823363202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1298102710 0x7f129810f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 >> 172.21.15.153:0/823363202 conn(0x7f1298072bc0 msgr2=0x7f12980faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 shutdown_connections 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.689+0000 7f129e92b700 1 -- 172.21.15.153:0/823363202 wait complete. 2024-04-05T03:26:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:26:45.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 -- 172.21.15.153:0/448784975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd081037a0 msgr2=0x7fdd080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 --2- 172.21.15.153:0/448784975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd081037a0 0x7fdd080fce10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdd04004940 tx=0x7fdd04009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:45.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 -- 172.21.15.153:0/448784975 shutdown_connections 2024-04-05T03:26:45.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 --2- 172.21.15.153:0/448784975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd081037a0 0x7fdd080fce10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 --2- 172.21.15.153:0/448784975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd08102e00 0x7fdd081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 -- 172.21.15.153:0/448784975 >> 172.21.15.153:0/448784975 conn(0x7fdd080f89b0 msgr2=0x7fdd080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 -- 172.21.15.153:0/448784975 shutdown_connections 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.792+0000 7fdd10149700 1 -- 172.21.15.153:0/448784975 wait complete. 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 Processor -- start 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 -- start start 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd081037a0 0x7fdd081140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd08114710 con 0x7fdd08102e00 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd10149700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0810db70 con 0x7fdd081037a0 2024-04-05T03:26:45.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd0dee5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd0dee5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60254/0 (socket says 172.21.15.153:60254) 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd0dee5700 1 -- 172.21.15.153:0/161301171 learned_addr learned my addr 172.21.15.153:0/161301171 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.793+0000 7fdd0d6e4700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd081037a0 0x7fdd081140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd0dee5700 1 -- 172.21.15.153:0/161301171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd081037a0 msgr2=0x7fdd081140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd0dee5700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd081037a0 0x7fdd081140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd0dee5700 1 -- 172.21.15.153:0/161301171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04005040 con 0x7fdd08102e00 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd0dee5700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8002a50 tx=0x7fdcf80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf8007cc0 con 0x7fdd08102e00 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0810de50 con 0x7fdd08102e00 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcf800f040 con 0x7fdd08102e00 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf800da30 con 0x7fdd08102e00 2024-04-05T03:26:45.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.794+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0810e410 con 0x7fdd08102e00 2024-04-05T03:26:45.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.795+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd081052d0 con 0x7fdd08102e00 2024-04-05T03:26:45.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.796+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fdcf800db90 con 0x7fdd08102e00 2024-04-05T03:26:45.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.796+0000 7fdcfe7fc700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 0x7fdcf4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:45.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.796+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdcf808d310 con 0x7fdd08102e00 2024-04-05T03:26:45.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.796+0000 7fdd0d6e4700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 0x7fdcf4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:45.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.796+0000 7fdd0d6e4700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 0x7fdcf4070150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd0810f420 tx=0x7fdd04009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:45.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:45.801+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdcf8056890 con 0x7fdd08102e00 2024-04-05T03:26:46.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.013+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd08109ee0 con 0x7fdcf406dca0 2024-04-05T03:26:46.011 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:45 smithi153 ceph-mon[106102]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:46.011 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:45 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3440438221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:26:46.011 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:45 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/823363202' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:26:46.012 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:45 smithi153 ceph-mon[106102]: pgmap v633: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:46.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.014+0000 7fdcfe7fc700 1 -- 172.21.15.153:0/161301171 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdd08109ee0 con 0x7fdcf406dca0 2024-04-05T03:26:46.012 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:26:46.012 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:26:46.012 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:26:46.012 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:26:46.013 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:26:46.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 msgr2=0x7fdcf4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 0x7fdcf4070150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd0810f420 tx=0x7fdd04009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 msgr2=0x7fdd08113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8002a50 tx=0x7fdcf80046b0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 shutdown_connections 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdcf406dca0 0x7fdcf4070150 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd08102e00 0x7fdd08113b80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 --2- 172.21.15.153:0/161301171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd081037a0 0x7fdd081140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 >> 172.21.15.153:0/161301171 conn(0x7fdd080f89b0 msgr2=0x7fdd08102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.016+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 shutdown_connections 2024-04-05T03:26:46.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.017+0000 7fdd10149700 1 -- 172.21.15.153:0/161301171 wait complete. 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.117+0000 7f2036579700 1 -- 172.21.15.153:0/1378706005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030101cb0 msgr2=0x7f2030102080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.117+0000 7f2036579700 1 --2- 172.21.15.153:0/1378706005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030101cb0 0x7f2030102080 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2020004940 tx=0x7f2020009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 -- 172.21.15.153:0/1378706005 shutdown_connections 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 --2- 172.21.15.153:0/1378706005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030102650 0x7f20300fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 --2- 172.21.15.153:0/1378706005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030101cb0 0x7f2030102080 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 -- 172.21.15.153:0/1378706005 >> 172.21.15.153:0/1378706005 conn(0x7f20300f7860 msgr2=0x7f20300f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 -- 172.21.15.153:0/1378706005 shutdown_connections 2024-04-05T03:26:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 -- 172.21.15.153:0/1378706005 wait complete. 2024-04-05T03:26:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 Processor -- start 2024-04-05T03:26:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.118+0000 7f2036579700 1 -- start start 2024-04-05T03:26:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030101cb0 0x7f2030113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301146b0 con 0x7f2030102650 2024-04-05T03:26:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203010daa0 con 0x7f2030101cb0 2024-04-05T03:26:46.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60282/0 (socket says 172.21.15.153:60282) 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 -- 172.21.15.153:0/3775378694 learned_addr learned my addr 172.21.15.153:0/3775378694 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202ffff700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030101cb0 0x7f2030113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 -- 172.21.15.153:0/3775378694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030101cb0 msgr2=0x7f2030113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030101cb0 0x7f2030113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 -- 172.21.15.153:0/3775378694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2020005040 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202f7fe700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2024002a50 tx=0x7f2024004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2024007cc0 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f202400f040 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f202400da40 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203010dd80 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.119+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203010e340 con 0x7f2030102650 2024-04-05T03:26:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.120+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20300fd400 con 0x7f2030102650 2024-04-05T03:26:46.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.121+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f20240097c0 con 0x7f2030102650 2024-04-05T03:26:46.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.121+0000 7f202cff9700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 0x7f2018070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:26:46.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.121+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f202408d2f0 con 0x7f2030102650 2024-04-05T03:26:46.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.121+0000 7f202ffff700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 0x7f2018070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:26:46.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.122+0000 7f202ffff700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 0x7f2018070150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2020004f90 tx=0x7f2020009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:26:46.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.126+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f202405a190 con 0x7f2030102650 2024-04-05T03:26:46.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:45 smithi203 ceph-mon[95988]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:46.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:45 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3440438221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:26:46.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:45 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/823363202' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:26:46.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:45 smithi203 ceph-mon[95988]: pgmap v633: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.404+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f203010f1a0 con 0x7f2030102650 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.405+0000 7f202cff9700 1 -- 172.21.15.153:0/3775378694 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2024059fb0 con 0x7f2030102650 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:26:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 msgr2=0x7f2018070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 0x7f2018070150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2020004f90 tx=0x7f2020009e30 comp rx=0 tx=0).stop 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 msgr2=0x7f2030114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2024002a50 tx=0x7f2024004600 comp rx=0 tx=0).stop 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 shutdown_connections 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f201806dca0 0x7f2018070150 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2030101cb0 0x7f2030113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 --2- 172.21.15.153:0/3775378694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2030102650 0x7f2030114060 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:26:46.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 >> 172.21.15.153:0/3775378694 conn(0x7f20300f7860 msgr2=0x7f20300f8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:26:46.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 shutdown_connections 2024-04-05T03:26:46.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:26:46.407+0000 7f2036579700 1 -- 172.21.15.153:0/3775378694 wait complete. 2024-04-05T03:26:47.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:46 smithi203 ceph-mon[95988]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:47.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:46 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3775378694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:26:47.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:46 smithi153 ceph-mon[106102]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:26:47.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:46 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3775378694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:26:48.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:47 smithi153 ceph-mon[106102]: pgmap v634: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:48.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:47 smithi203 ceph-mon[95988]: pgmap v634: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:51.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:50 smithi203 ceph-mon[95988]: pgmap v635: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:50 smithi153 ceph-mon[106102]: pgmap v635: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:52.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:51 smithi203 ceph-mon[95988]: pgmap v636: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:52.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:51 smithi153 ceph-mon[106102]: pgmap v636: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:55.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:54 smithi203 ceph-mon[95988]: pgmap v637: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:55.073 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:54 smithi153 ceph-mon[106102]: pgmap v637: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:56.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:55 smithi203 ceph-mon[95988]: pgmap v638: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:56.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:55 smithi153 ceph-mon[106102]: pgmap v638: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:57.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:58.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:26:58.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:26:58 smithi153 ceph-mon[106102]: pgmap v639: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:26:59.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:26:58 smithi203 ceph-mon[95988]: pgmap v639: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:00 smithi203 ceph-mon[95988]: pgmap v640: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:01.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:00 smithi153 ceph-mon[106102]: pgmap v640: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:02.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:01 smithi203 ceph-mon[95988]: pgmap v641: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:02.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:01 smithi153 ceph-mon[106102]: pgmap v641: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:04.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:03 smithi203 ceph-mon[95988]: pgmap v642: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:04.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:03 smithi153 ceph-mon[106102]: pgmap v642: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:06.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:05 smithi203 ceph-mon[95988]: pgmap v643: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:06.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:05 smithi153 ceph-mon[106102]: pgmap v643: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:08.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:07 smithi203 ceph-mon[95988]: pgmap v644: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:08.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:07 smithi153 ceph-mon[106102]: pgmap v644: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:10.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:09 smithi203 ceph-mon[95988]: pgmap v645: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:10.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:09 smithi153 ceph-mon[106102]: pgmap v645: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:12.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:11 smithi203 ceph-mon[95988]: pgmap v646: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:12.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:11 smithi153 ceph-mon[106102]: pgmap v646: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:13.196 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:13.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:14.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:13 smithi203 ceph-mon[95988]: pgmap v647: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:14.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:13 smithi153 ceph-mon[106102]: pgmap v647: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:16.152 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:15 smithi203 ceph-mon[95988]: pgmap v648: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:16.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:15 smithi153 ceph-mon[106102]: pgmap v648: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:16.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 -- 172.21.15.153:0/743837881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 msgr2=0x7f9ddc0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 --2- 172.21.15.153:0/743837881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc0fcdf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0004940 tx=0x7f9dd0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:16.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 -- 172.21.15.153:0/743837881 shutdown_connections 2024-04-05T03:27:16.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 --2- 172.21.15.153:0/743837881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc0fcdf0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 --2- 172.21.15.153:0/743837881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 0x7f9ddc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 -- 172.21.15.153:0/743837881 >> 172.21.15.153:0/743837881 conn(0x7f9ddc0f8970 msgr2=0x7f9ddc0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 -- 172.21.15.153:0/743837881 shutdown_connections 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.520+0000 7f9de1946700 1 -- 172.21.15.153:0/743837881 wait complete. 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 Processor -- start 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 -- start start 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 0x7f9ddc113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc1146f0 con 0x7f9ddc103780 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9de1946700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc10dae0 con 0x7f9ddc102de0 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49980/0 (socket says 172.21.15.153:49980) 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 -- 172.21.15.153:0/2465282753 learned_addr learned my addr 172.21.15.153:0/2465282753 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:16.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9ddaffd700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 0x7f9ddc113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 -- 172.21.15.153:0/2465282753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 msgr2=0x7f9ddc113b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 0x7f9ddc113b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 -- 172.21.15.153:0/2465282753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0005040 con 0x7f9ddc103780 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.521+0000 7f9dda7fc700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009100 tx=0x7f9dd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.522+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd000e040 con 0x7f9ddc103780 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.522+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dd0009af0 con 0x7f9ddc103780 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.522+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc10dd60 con 0x7f9ddc103780 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.522+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd001c3f0 con 0x7f9ddc103780 2024-04-05T03:27:16.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.522+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc10e320 con 0x7f9ddc103780 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.523+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc0fe530 con 0x7f9ddc103780 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.524+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9dd000baf0 con 0x7f9ddc103780 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.525+0000 7f9dc37fe700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.525+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9dd008f260 con 0x7f9ddc103780 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.525+0000 7f9ddaffd700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.525+0000 7f9ddaffd700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc004eb0 tx=0x7f9dcc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:16.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.529+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9dd005c430 con 0x7f9ddc103780 2024-04-05T03:27:16.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.744+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ddc109ec0 con 0x7f9dc406dca0 2024-04-05T03:27:16.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.746+0000 7f9dc37fe700 1 -- 172.21.15.153:0/2465282753 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9ddc109ec0 con 0x7f9dc406dca0 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 msgr2=0x7f9dc4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc004eb0 tx=0x7f9dcc00b040 comp rx=0 tx=0).stop 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 msgr2=0x7f9ddc1140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009100 tx=0x7f9dd0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 shutdown_connections 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f9dc406dca0 0x7f9dc4070150 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ddc102de0 0x7f9ddc113b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 --2- 172.21.15.153:0/2465282753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ddc103780 0x7f9ddc1140a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 >> 172.21.15.153:0/2465282753 conn(0x7f9ddc0f8970 msgr2=0x7f9ddc0f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:16.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 shutdown_connections 2024-04-05T03:27:16.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.748+0000 7f9de1946700 1 -- 172.21.15.153:0/2465282753 wait complete. 2024-04-05T03:27:16.757 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:27:16.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/1273612434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac1037a0 msgr2=0x7fbcac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/1273612434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac1037a0 0x7fbcac0fce10 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004940 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:16.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/1273612434 shutdown_connections 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/1273612434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac1037a0 0x7fbcac0fce10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/1273612434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac102e00 0x7fbcac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/1273612434 >> 172.21.15.153:0/1273612434 conn(0x7fbcac0f89b0 msgr2=0x7fbcac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/1273612434 shutdown_connections 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.851+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/1273612434 wait complete. 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 Processor -- start 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 -- start start 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac1037a0 0x7fbcac113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac114510 con 0x7fbcac102e00 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcb1d7d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac10db10 con 0x7fbcac1037a0 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcab7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcab7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50000/0 (socket says 172.21.15.153:50000) 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.852+0000 7fbcab7fe700 1 -- 172.21.15.153:0/2121965680 learned_addr learned my addr 172.21.15.153:0/2121965680 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcaaffd700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac1037a0 0x7fbcac113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcab7fe700 1 -- 172.21.15.153:0/2121965680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac1037a0 msgr2=0x7fbcac113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcab7fe700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac1037a0 0x7fbcac113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcab7fe700 1 -- 172.21.15.153:0/2121965680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca0005040 con 0x7fbcac102e00 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcab7fe700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c002a50 tx=0x7fbc9c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c007cc0 con 0x7fbcac102e00 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc9c00f040 con 0x7fbcac102e00 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac10ddf0 con 0x7fbcac102e00 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c009640 con 0x7fbcac102e00 2024-04-05T03:27:16.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.853+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac10e380 con 0x7fbcac102e00 2024-04-05T03:27:16.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.854+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac1052d0 con 0x7fbcac102e00 2024-04-05T03:27:16.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.855+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbc9c0097a0 con 0x7fbcac102e00 2024-04-05T03:27:16.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.855+0000 7fbc93fff700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 0x7fbc94070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:16.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.855+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbc9c08d350 con 0x7fbcac102e00 2024-04-05T03:27:16.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.855+0000 7fbcaaffd700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 0x7fbc94070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:16.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.855+0000 7fbcaaffd700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 0x7fbc94070150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbcac10edf0 tx=0x7fbca0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:16.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:16.859+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbc9c05a400 con 0x7fbcac102e00 2024-04-05T03:27:17.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.076+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcac10f040 con 0x7fbc9406dca0 2024-04-05T03:27:17.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.077+0000 7fbc93fff700 1 -- 172.21.15.153:0/2121965680 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbcac10f040 con 0x7fbc9406dca0 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.079+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 msgr2=0x7fbc94070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.079+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 0x7fbc94070150 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbcac10edf0 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.079+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 msgr2=0x7fbcac113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.079+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c002a50 tx=0x7fbc9c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 shutdown_connections 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc9406dca0 0x7fbc94070150 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbcac102e00 0x7fbcac113980 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 --2- 172.21.15.153:0/2121965680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbcac1037a0 0x7fbcac113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 >> 172.21.15.153:0/2121965680 conn(0x7fbcac0f89b0 msgr2=0x7fbcac1021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 shutdown_connections 2024-04-05T03:27:17.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.080+0000 7fbcb1d7d700 1 -- 172.21.15.153:0/2121965680 wait complete. 2024-04-05T03:27:17.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 -- 172.21.15.153:0/2564672822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c102710 msgr2=0x7fd33c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 --2- 172.21.15.153:0/2564672822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c102710 0x7fd33c0fbd80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd338004940 tx=0x7fd338009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 -- 172.21.15.153:0/2564672822 shutdown_connections 2024-04-05T03:27:17.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 --2- 172.21.15.153:0/2564672822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c102710 0x7fd33c0fbd80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.181 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:16 smithi153 ceph-mon[106102]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:17.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 --2- 172.21.15.153:0/2564672822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c101d70 0x7fd33c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 -- 172.21.15.153:0/2564672822 >> 172.21.15.153:0/2564672822 conn(0x7fd33c072bc0 msgr2=0x7fd33c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 -- 172.21.15.153:0/2564672822 shutdown_connections 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.181+0000 7fd34409d700 1 -- 172.21.15.153:0/2564672822 wait complete. 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 Processor -- start 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 -- start start 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c102710 0x7fd33c1140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c1146f0 con 0x7fd33c101d70 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd34409d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c10dae0 con 0x7fd33c102710 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50026/0 (socket says 172.21.15.153:50026) 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 -- 172.21.15.153:0/3035855961 learned_addr learned my addr 172.21.15.153:0/3035855961 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341638700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c102710 0x7fd33c1140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 -- 172.21.15.153:0/3035855961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c102710 msgr2=0x7fd33c1140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c102710 0x7fd33c1140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 -- 172.21.15.153:0/3035855961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338005040 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.182+0000 7fd341e39700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd32c002a50 tx=0x7fd32c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.183+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32c007cc0 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.183+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd32c00f040 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.183+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c10ddc0 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.183+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32c00da30 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.183+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c10e350 con 0x7fd33c101d70 2024-04-05T03:27:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.184+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c0fd4c0 con 0x7fd33c101d70 2024-04-05T03:27:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.184+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fd32c00dbb0 con 0x7fd33c101d70 2024-04-05T03:27:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.185+0000 7fd3327fc700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 0x7fd328070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.185+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd32c08d230 con 0x7fd33c101d70 2024-04-05T03:27:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.185+0000 7fd341638700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 0x7fd328070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.185+0000 7fd341638700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 0x7fd328070150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd338004890 tx=0x7fd338009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.190+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd32c05a360 con 0x7fd33c101d70 2024-04-05T03:27:17.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:16 smithi203 ceph-mon[95988]: from='client.35210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:17.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.397+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd33c108e50 con 0x7fd32806dca0 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.407+0000 7fd3327fc700 1 -- 172.21.15.153:0/3035855961 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd33c108e50 con 0x7fd32806dca0 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (21m) 15m ago 33m 25.9M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (19m) 15m ago 34m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (19m) 14m ago 31m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (21m) 15m ago 32m 80.0M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (15m) 15m ago 27m 15.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:27:17.406 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (15m) 15m ago 27m 14.2M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (14m) 14m ago 27m 13.9M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (15m) 14m ago 27m 14.0M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (23m) 15m ago 35m 508M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (22m) 14m ago 31m 452M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (20m) 15m ago 35m 52.8M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (20m) 14m ago 31m 45.9M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (22m) 15m ago 33m 16.6M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (22m) 14m ago 31m 17.5M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (18m) 15m ago 31m 93.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (18m) 15m ago 30m 96.1M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (17m) 15m ago 30m 83.8M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (17m) 14m ago 29m 110M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (16m) 14m ago 29m 94.1M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (16m) 14m ago 28m 96.6M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:27:17.407 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (22m) 14m ago 31m 65.8M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:27:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 msgr2=0x7fd328070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 0x7fd328070150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd338004890 tx=0x7fd338009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 msgr2=0x7fd33c113b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd32c002a50 tx=0x7fd32c0046b0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 shutdown_connections 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fd32806dca0 0x7fd328070150 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd33c101d70 0x7fd33c113b60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 --2- 172.21.15.153:0/3035855961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd33c102710 0x7fd33c1140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 >> 172.21.15.153:0/3035855961 conn(0x7fd33c072bc0 msgr2=0x7fd33c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 shutdown_connections 2024-04-05T03:27:17.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.409+0000 7fd34409d700 1 -- 172.21.15.153:0/3035855961 wait complete. 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 -- 172.21.15.153:0/1662727643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4424101d10 msgr2=0x7f44241020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 --2- 172.21.15.153:0/1662727643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4424101d10 0x7f44241020e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4420004940 tx=0x7f4420009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 -- 172.21.15.153:0/1662727643 shutdown_connections 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 --2- 172.21.15.153:0/1662727643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44241026b0 0x7f44240fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 --2- 172.21.15.153:0/1662727643 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4424101d10 0x7f44241020e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 -- 172.21.15.153:0/1662727643 >> 172.21.15.153:0/1662727643 conn(0x7f44240f7880 msgr2=0x7f44240f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 -- 172.21.15.153:0/1662727643 shutdown_connections 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.509+0000 7f442b549700 1 -- 172.21.15.153:0/1662727643 wait complete. 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 Processor -- start 2024-04-05T03:27:17.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 -- start start 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 0x7f4424113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4424114ab0 con 0x7f44241026b0 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f442b549700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442410ddf0 con 0x7f4424101d10 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50044/0 (socket says 172.21.15.153:50044) 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 -- 172.21.15.153:0/4032912562 learned_addr learned my addr 172.21.15.153:0/4032912562 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f44292e5700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 0x7f4424113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 -- 172.21.15.153:0/4032912562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 msgr2=0x7f4424113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 0x7f4424113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f4428ae4700 1 -- 172.21.15.153:0/4032912562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420005040 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.510+0000 7f44292e5700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 0x7f4424113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f4428ae4700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f441400b790 tx=0x7f4414009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f441400f6f0 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442410e070 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f441400fd20 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44140103a0 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.511+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442410e630 con 0x7f44241026b0 2024-04-05T03:27:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.512+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44240fd460 con 0x7f44241026b0 2024-04-05T03:27:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.513+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4414011400 con 0x7f44241026b0 2024-04-05T03:27:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.513+0000 7f4419ffb700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 0x7f4410070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.513+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f441408d450 con 0x7f44241026b0 2024-04-05T03:27:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.513+0000 7f44292e5700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 0x7f4410070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.514+0000 7f44292e5700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 0x7f4410070160 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4420004f90 tx=0x7f4420009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.518+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f441405a500 con 0x7f44241026b0 2024-04-05T03:27:17.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.796+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f442405f5f0 con 0x7f44241026b0 2024-04-05T03:27:17.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.796+0000 7f4419ffb700 1 -- 172.21.15.153:0/4032912562 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f441405a090 con 0x7f44241026b0 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:27:17.796 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:27:17.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:27:17.797 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:27:17.797 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:27:17.797 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:27:17.797 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:27:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 msgr2=0x7f4410070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 0x7f4410070160 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4420004f90 tx=0x7f4420009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 msgr2=0x7f44241143d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f441400b790 tx=0x7f4414009220 comp rx=0 tx=0).stop 2024-04-05T03:27:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 shutdown_connections 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f441006dcb0 0x7f4410070160 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4424101d10 0x7f4424113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 --2- 172.21.15.153:0/4032912562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44241026b0 0x7f44241143d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 >> 172.21.15.153:0/4032912562 conn(0x7f44240f7880 msgr2=0x7f44240f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 shutdown_connections 2024-04-05T03:27:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.799+0000 7f442b549700 1 -- 172.21.15.153:0/4032912562 wait complete. 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- 172.21.15.153:0/4043153981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc1037a0 msgr2=0x7f77bc0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 --2- 172.21.15.153:0/4043153981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc1037a0 0x7f77bc0fce10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f77b0004940 tx=0x7f77b0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- 172.21.15.153:0/4043153981 shutdown_connections 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 --2- 172.21.15.153:0/4043153981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc1037a0 0x7f77bc0fce10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 --2- 172.21.15.153:0/4043153981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc102e00 0x7f77bc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- 172.21.15.153:0/4043153981 >> 172.21.15.153:0/4043153981 conn(0x7f77bc0f89b0 msgr2=0x7f77bc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- 172.21.15.153:0/4043153981 shutdown_connections 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- 172.21.15.153:0/4043153981 wait complete. 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 Processor -- start 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.901+0000 7f77c22b2700 1 -- start start 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77c22b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77c22b2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc1037a0 0x7f77bc113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77c22b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1145b0 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77c22b2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc10db40 con 0x7f77bc1037a0 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50060/0 (socket says 172.21.15.153:50060) 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 -- 172.21.15.153:0/447533440 learned_addr learned my addr 172.21.15.153:0/447533440 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bb7fe700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc1037a0 0x7f77bc113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 -- 172.21.15.153:0/447533440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc1037a0 msgr2=0x7f77bc113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc1037a0 0x7f77bc113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 -- 172.21.15.153:0/447533440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0005040 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77bbfff700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00b790 tx=0x7f77ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77ac00f6f0 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77ac00fd20 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc10de20 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.902+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77ac0103a0 con 0x7f77bc102e00 2024-04-05T03:27:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.903+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77bc10e3e0 con 0x7f77bc102e00 2024-04-05T03:27:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.903+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc100110 con 0x7f77bc102e00 2024-04-05T03:27:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.904+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f77ac011400 con 0x7f77bc102e00 2024-04-05T03:27:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.904+0000 7f77b8ff9700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 0x7f77a4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.904+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f77ac08d3a0 con 0x7f77bc102e00 2024-04-05T03:27:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.904+0000 7f77bb7fe700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 0x7f77a4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.905+0000 7f77bb7fe700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 0x7f77a4070150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f77bc10f230 tx=0x7f77b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:17.909+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f77ac0569a0 con 0x7f77bc102e00 2024-04-05T03:27:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:17 smithi153 ceph-mon[106102]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:17 smithi153 ceph-mon[106102]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:17 smithi153 ceph-mon[106102]: pgmap v649: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:18.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:17 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4032912562' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.147+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77bc109f50 con 0x7f77bc102e00 2024-04-05T03:27:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.147+0000 7f77b8ff9700 1 -- 172.21.15.153:0/447533440 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1879 (secure 0 0 0) 0x7f77ac05a060 con 0x7f77bc102e00 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:e36 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 27 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:27:18.147 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:11:23.582846+0000 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:failed 0,1 2024-04-05T03:27:18.148 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 0 members: 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:Standby daemons: 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{-1:44325} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:27:18.149 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{-1:44357} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:27:18.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 msgr2=0x7f77a4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 0x7f77a4070150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f77bc10f230 tx=0x7f77b0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 msgr2=0x7f77bc113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00b790 tx=0x7f77ac009250 comp rx=0 tx=0).stop 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 shutdown_connections 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f77a406dca0 0x7f77a4070150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.149+0000 7f77c22b2700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77bc102e00 0x7f77bc113990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.150+0000 7f77c22b2700 1 --2- 172.21.15.153:0/447533440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77bc1037a0 0x7f77bc113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.150+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 >> 172.21.15.153:0/447533440 conn(0x7f77bc0f89b0 msgr2=0x7f77bc0fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.150+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 shutdown_connections 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.150+0000 7f77c22b2700 1 -- 172.21.15.153:0/447533440 wait complete. 2024-04-05T03:27:18.151 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 -- 172.21.15.153:0/224145283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 msgr2=0x7f75600fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 --2- 172.21.15.153:0/224145283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f75600fe260 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7550004940 tx=0x7f7550009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 -- 172.21.15.153:0/224145283 shutdown_connections 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 --2- 172.21.15.153:0/224145283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 0x7f7560106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 --2- 172.21.15.153:0/224145283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f75600fe260 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 -- 172.21.15.153:0/224145283 >> 172.21.15.153:0/224145283 conn(0x7f7560072bc0 msgr2=0x7f7560072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 -- 172.21.15.153:0/224145283 shutdown_connections 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.252+0000 7f7565373700 1 -- 172.21.15.153:0/224145283 wait complete. 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 Processor -- start 2024-04-05T03:27:18.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 -- start start 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 0x7f75601140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560114720 con 0x7f75600fde90 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f7565373700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756010db30 con 0x7f75600fe830 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50072/0 (socket says 172.21.15.153:50072) 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 -- 172.21.15.153:0/480303364 learned_addr learned my addr 172.21.15.153:0/480303364 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:18.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755e7fc700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 0x7f75601140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 -- 172.21.15.153:0/480303364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 msgr2=0x7f75601140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 0x7f75601140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 -- 172.21.15.153:0/480303364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550005040 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f755effd700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f755000b3b0 tx=0x7f7550009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.253+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755000e040 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.254+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756010ddb0 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.254+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7550013420 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.254+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755001c5f0 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.254+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756010e2f0 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.255+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560101980 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.255+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7550013590 con 0x7f75600fde90 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.255+0000 7f75477fe700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 0x7f7548070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.256+0000 7f755e7fc700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 0x7f7548070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.256+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f755008e730 con 0x7f75600fde90 2024-04-05T03:27:18.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.256+0000 7f755e7fc700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 0x7f7548070150 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f756010f430 tx=0x7f755400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:18.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.260+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7550057d30 con 0x7f75600fde90 2024-04-05T03:27:18.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:17 smithi203 ceph-mon[95988]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:18.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:17 smithi203 ceph-mon[95988]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:18.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:17 smithi203 ceph-mon[95988]: pgmap v649: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:18.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:17 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4032912562' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:18.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.474+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7560109f00 con 0x7f754806dca0 2024-04-05T03:27:18.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.475+0000 7f75477fe700 1 -- 172.21.15.153:0/480303364 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7560109f00 con 0x7f754806dca0 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [ 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "mds", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "osd", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "crash", 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "mon" 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-05T03:27:18.475 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "16/21 daemons upgraded", 2024-04-05T03:27:18.476 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T03:27:18.476 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:27:18.476 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:27:18.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 msgr2=0x7f7548070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 0x7f7548070150 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f756010f430 tx=0x7f755400b040 comp rx=0 tx=0).stop 2024-04-05T03:27:18.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 msgr2=0x7f7560113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f755000b3b0 tx=0x7f7550009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 shutdown_connections 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f754806dca0 0x7f7548070150 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75600fde90 0x7f7560113b90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 --2- 172.21.15.153:0/480303364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75600fe830 0x7f75601140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 >> 172.21.15.153:0/480303364 conn(0x7f7560072bc0 msgr2=0x7f75600ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 shutdown_connections 2024-04-05T03:27:18.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.478+0000 7f7565373700 1 -- 172.21.15.153:0/480303364 wait complete. 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 -- 172.21.15.153:0/186068380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 msgr2=0x7f413c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 --2- 172.21.15.153:0/186068380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c1031d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f412c004940 tx=0x7f412c009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 -- 172.21.15.153:0/186068380 shutdown_connections 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 --2- 172.21.15.153:0/186068380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 0x7f413c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 --2- 172.21.15.153:0/186068380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c1031d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 -- 172.21.15.153:0/186068380 >> 172.21.15.153:0/186068380 conn(0x7f413c0f89b0 msgr2=0x7f413c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 -- 172.21.15.153:0/186068380 shutdown_connections 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.579+0000 7f41421ad700 1 -- 172.21.15.153:0/186068380 wait complete. 2024-04-05T03:27:18.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 Processor -- start 2024-04-05T03:27:18.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 -- start start 2024-04-05T03:27:18.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 0x7f413c1160b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c116700 con 0x7f413c102e00 2024-04-05T03:27:18.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f41421ad700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c10fce0 con 0x7f413c1037a0 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f413b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f413b7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50092/0 (socket says 172.21.15.153:50092) 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.580+0000 7f413b7fe700 1 -- 172.21.15.153:0/694960922 learned_addr learned my addr 172.21.15.153:0/694960922 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f413affd700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 0x7f413c1160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f413b7fe700 1 -- 172.21.15.153:0/694960922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 msgr2=0x7f413c1160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f413b7fe700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 0x7f413c1160b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f413b7fe700 1 -- 172.21.15.153:0/694960922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c005040 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f413b7fe700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f412c00b3b0 tx=0x7f412c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412c00e040 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f412c00bbc0 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c10ff60 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f412c01c3f0 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.581+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c110520 con 0x7f413c102e00 2024-04-05T03:27:18.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.582+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c107300 con 0x7f413c102e00 2024-04-05T03:27:18.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.583+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f412c00bd30 con 0x7f413c102e00 2024-04-05T03:27:18.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.583+0000 7f4123fff700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 0x7f4124070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:18.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.583+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f412c08e820 con 0x7f413c102e00 2024-04-05T03:27:18.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.583+0000 7f413affd700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 0x7f4124070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:18.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.584+0000 7f413affd700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 0x7f4124070150 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4130007fd0 tx=0x7f4130009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:18.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.588+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f412c05b8d0 con 0x7f413c102e00 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.867+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f413c109f00 con 0x7f413c102e00 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.868+0000 7f4123fff700 1 -- 172.21.15.153:0/694960922 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f412c05b460 con 0x7f413c102e00 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is degraded 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T03:27:18.867 INFO:teuthology.orchestra.run.smithi153.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 msgr2=0x7f4124070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 0x7f4124070150 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4130007fd0 tx=0x7f4130009040 comp rx=0 tx=0).stop 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 msgr2=0x7f413c115b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f412c00b3b0 tx=0x7f412c009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 shutdown_connections 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f412406dca0 0x7f4124070150 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f413c102e00 0x7f413c115b70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 --2- 172.21.15.153:0/694960922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f413c1037a0 0x7f413c1160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 >> 172.21.15.153:0/694960922 conn(0x7f413c0f89b0 msgr2=0x7f413c070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:18.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 shutdown_connections 2024-04-05T03:27:18.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:18.870+0000 7f41421ad700 1 -- 172.21.15.153:0/694960922 wait complete. 2024-04-05T03:27:19.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:18 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/447533440' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:27:19.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:18 smithi203 ceph-mon[95988]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:19.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:18 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/447533440' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:27:19.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:18 smithi153 ceph-mon[106102]: from='client.35230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:20.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:19 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/694960922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:27:20.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:19 smithi203 ceph-mon[95988]: pgmap v650: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:20.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:19 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/694960922' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:27:20.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:19 smithi153 ceph-mon[106102]: pgmap v650: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:22.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:21 smithi203 ceph-mon[95988]: pgmap v651: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:22.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:21 smithi153 ceph-mon[106102]: pgmap v651: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:23.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:23.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:23.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: pgmap v652: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:23.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:23.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all mds 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.fmpmvd"}]': finished 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.vicaky"}]': finished 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.hgbifk"}]': finished 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:27:23.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.xpteyc"}]': finished 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Standby daemon mds.cephfs.smithi153.vicaky assigned to filesystem cephfs as rank 0 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Standby daemon mds.cephfs.smithi153.fmpmvd assigned to filesystem cephfs as rank 1 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:replay,1=cephfs.smithi153.fmpmvd=up:replay} 2 up:standby 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-05T03:27:23.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:22 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: pgmap v652: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all mds 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.fmpmvd"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.fmpmvd"}]': finished 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.vicaky"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi153.vicaky"}]': finished 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.hgbifk"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.hgbifk"}]': finished 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.xpteyc"}]: dispatch 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi203.xpteyc"}]': finished 2024-04-05T03:27:23.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Standby daemon mds.cephfs.smithi153.vicaky assigned to filesystem cephfs as rank 0 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Standby daemon mds.cephfs.smithi153.fmpmvd assigned to filesystem cephfs as rank 1 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:replay,1=cephfs.smithi153.fmpmvd=up:replay} 2 up:standby 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-05T03:27:23.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:22 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:replay,1=cephfs.smithi153.fmpmvd=up:replay} 2 up:standby 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all rgw 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all iscsi 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all nfs 2024-04-05T03:27:24.296 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Setting container_image for all nvmeof 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: Upgrade: Finalizing container_image settings 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T03:27:24.297 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T03:27:24.298 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T03:27:24.299 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T03:27:24.299 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T03:27:24.299 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T03:27:24.299 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:23 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:replay,1=cephfs.smithi153.fmpmvd=up:replay} 2 up:standby 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all rgw 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all iscsi 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all nfs 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Setting container_image for all nvmeof 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:27:24.323 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: Upgrade: Finalizing container_image settings 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T03:27:24.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T03:27:24.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:23 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T03:27:25.173 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: Upgrade: Complete! 2024-04-05T03:27:25.174 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: pgmap v653: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.175 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:24 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: Upgrade: Complete! 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T03:27:25.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: pgmap v653: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail 2024-04-05T03:27:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:25.322 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:24 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:27.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:27 smithi153 ceph-mon[106102]: pgmap v654: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T03:27:27.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:27 smithi203 ceph-mon[95988]: pgmap v654: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T03:27:28.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:28 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:28 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: pgmap v655: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:30.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:30.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: pgmap v655: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:27:30.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:30 smithi203 ceph-mon[95988]: pgmap v656: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-05T03:27:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:30 smithi203 ceph-mon[95988]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-05T03:27:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:30 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:resolve 2024-04-05T03:27:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:30 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:resolve 2024-04-05T03:27:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:30 smithi203 ceph-mon[95988]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:resolve,1=cephfs.smithi153.fmpmvd=up:resolve} 2 up:standby 2024-04-05T03:27:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:30 smithi153 ceph-mon[106102]: pgmap v656: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-05T03:27:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:30 smithi153 ceph-mon[106102]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-05T03:27:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:30 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:resolve 2024-04-05T03:27:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:30 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:resolve 2024-04-05T03:27:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:30 smithi153 ceph-mon[106102]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:resolve,1=cephfs.smithi153.fmpmvd=up:resolve} 2 up:standby 2024-04-05T03:27:32.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: reconnect by client.24311 192.168.0.1:0/2642975444 after 0.000999983 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: reconnect by client.14558 192.168.0.1:0/761609335 after 0.000999983 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: reconnect by client.14558 192.168.0.1:0/761609335 after 0.000999983 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: reconnect by client.24311 192.168.0.1:0/2642975444 after 0.000999983 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:reconnect 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:reconnect 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:reconnect,1=cephfs.smithi153.fmpmvd=up:reconnect} 2 up:standby 2024-04-05T03:27:32.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:31 smithi203 ceph-mon[95988]: pgmap v657: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-05T03:27:32.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: reconnect by client.24311 192.168.0.1:0/2642975444 after 0.000999983 2024-04-05T03:27:32.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: reconnect by client.14558 192.168.0.1:0/761609335 after 0.000999983 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: reconnect by client.14558 192.168.0.1:0/761609335 after 0.000999983 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: reconnect by client.24311 192.168.0.1:0/2642975444 after 0.000999983 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:reconnect 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:reconnect 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:reconnect,1=cephfs.smithi153.fmpmvd=up:reconnect} 2 up:standby 2024-04-05T03:27:32.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:31 smithi153 ceph-mon[106102]: pgmap v657: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-05T03:27:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:rejoin 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:rejoin 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:rejoin,1=cephfs.smithi153.fmpmvd=up:rejoin} 2 up:standby 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: daemon mds.cephfs.smithi153.fmpmvd is now active in filesystem cephfs as rank 1 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: daemon mds.cephfs.smithi153.vicaky is now active in filesystem cephfs as rank 0 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: Cluster is now healthy 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:active 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:active 2024-04-05T03:27:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:32 smithi203 ceph-mon[95988]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi153.fmpmvd=up:active} 2 up:standby-replay 2024-04-05T03:27:33.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:rejoin 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:rejoin 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: fsmap cephfs:2/2 {0=cephfs.smithi153.vicaky=up:rejoin,1=cephfs.smithi153.fmpmvd=up:rejoin} 2 up:standby 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: daemon mds.cephfs.smithi153.fmpmvd is now active in filesystem cephfs as rank 1 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: daemon mds.cephfs.smithi153.vicaky is now active in filesystem cephfs as rank 0 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: Cluster is now healthy 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] up:active 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: mds.? [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] up:active 2024-04-05T03:27:33.321 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:32 smithi153 ceph-mon[106102]: fsmap cephfs:2 {0=cephfs.smithi153.vicaky=up:active,1=cephfs.smithi153.fmpmvd=up:active} 2 up:standby-replay 2024-04-05T03:27:34.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:33 smithi203 ceph-mon[95988]: pgmap v658: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-05T03:27:34.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:33 smithi153 ceph-mon[106102]: pgmap v658: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-04-05T03:27:36.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:36 smithi203 ceph-mon[95988]: pgmap v659: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 4.8 KiB/s wr, 16 op/s 2024-04-05T03:27:36.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:36 smithi153 ceph-mon[106102]: pgmap v659: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 4.8 KiB/s wr, 16 op/s 2024-04-05T03:27:38.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:38 smithi203 ceph-mon[95988]: pgmap v660: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.9 KiB/s wr, 16 op/s 2024-04-05T03:27:38.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:38 smithi153 ceph-mon[106102]: pgmap v660: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.9 KiB/s wr, 16 op/s 2024-04-05T03:27:40.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:40 smithi203 ceph-mon[95988]: pgmap v661: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.9 KiB/s wr, 16 op/s 2024-04-05T03:27:40.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:40 smithi153 ceph-mon[106102]: pgmap v661: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 4.9 KiB/s wr, 16 op/s 2024-04-05T03:27:42.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:42 smithi203 ceph-mon[95988]: pgmap v662: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8.7 KiB/s wr, 18 op/s 2024-04-05T03:27:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:42 smithi153 ceph-mon[106102]: pgmap v662: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 8.7 KiB/s wr, 18 op/s 2024-04-05T03:27:43.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:43 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:43.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:43 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:43.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:43 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:27:43.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:43 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:44.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:44 smithi203 ceph-mon[95988]: pgmap v663: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.7 KiB/s wr, 17 op/s 2024-04-05T03:27:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:44 smithi153 ceph-mon[106102]: pgmap v663: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.7 KiB/s wr, 17 op/s 2024-04-05T03:27:46.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:46 smithi203 ceph-mon[95988]: pgmap v664: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.7 KiB/s wr, 18 op/s 2024-04-05T03:27:46.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:46 smithi153 ceph-mon[106102]: pgmap v664: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.7 KiB/s wr, 18 op/s 2024-04-05T03:27:48.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:48 smithi203 ceph-mon[95988]: pgmap v665: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-04-05T03:27:48.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:48 smithi153 ceph-mon[106102]: pgmap v665: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 13 MiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 -- 172.21.15.153:0/2804987555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 msgr2=0x7f93e00fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2804987555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fb940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f93d4004940 tx=0x7f93d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 -- 172.21.15.153:0/2804987555 shutdown_connections 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2804987555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fb940 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2804987555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 0x7f93e0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 -- 172.21.15.153:0/2804987555 >> 172.21.15.153:0/2804987555 conn(0x7f93e00714c0 msgr2=0x7f93e00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 -- 172.21.15.153:0/2804987555 shutdown_connections 2024-04-05T03:27:48.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.992+0000 7f93e53ea700 1 -- 172.21.15.153:0/2804987555 wait complete. 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 Processor -- start 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 -- start start 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 0x7f93e00fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00fe600 con 0x7f93e00fb4d0 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93e53ea700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e00fd130 con 0x7f93e0102f50 2024-04-05T03:27:48.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.993+0000 7f93deffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34992/0 (socket says 172.21.15.153:34992) 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 -- 172.21.15.153:0/2585550792 learned_addr learned my addr 172.21.15.153:0/2585550792 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93de7fc700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 0x7f93e00fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 -- 172.21.15.153:0/2585550792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 msgr2=0x7f93e00fcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 0x7f93e00fcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 -- 172.21.15.153:0/2585550792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4005040 con 0x7f93e00fb4d0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93deffd700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f93d0002a80 tx=0x7f93d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d0007cc0 con 0x7f93e00fb4d0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e00fd410 con 0x7f93e00fb4d0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93d000f040 con 0x7f93e00fb4d0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.994+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e011a0f0 con 0x7f93e00fb4d0 2024-04-05T03:27:48.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.995+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93d000dad0 con 0x7f93e00fb4d0 2024-04-05T03:27:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.996+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc0052b0 con 0x7f93e00fb4d0 2024-04-05T03:27:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.997+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f93d0009810 con 0x7f93e00fb4d0 2024-04-05T03:27:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.997+0000 7f93c77fe700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 0x7f93c8070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.997+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f93d008d1b0 con 0x7f93e00fb4d0 2024-04-05T03:27:48.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.997+0000 7f93de7fc700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 0x7f93c8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:48.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:48.998+0000 7f93de7fc700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 0x7f93c8070150 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f93d400b3b0 tx=0x7f93d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:49.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.002+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f93d00567b0 con 0x7f93e00fb4d0 2024-04-05T03:27:49.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.232+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f93cc000bc0 con 0x7f93c806dca0 2024-04-05T03:27:49.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.234+0000 7f93c77fe700 1 -- 172.21.15.153:0/2585550792 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f93cc000bc0 con 0x7f93c806dca0 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 msgr2=0x7f93c8070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 0x7f93c8070150 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f93d400b3b0 tx=0x7f93d4009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 msgr2=0x7f93e00fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f93d0002a80 tx=0x7f93d0004600 comp rx=0 tx=0).stop 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 shutdown_connections 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f93c806dca0 0x7f93c8070150 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93e00fb4d0 0x7f93e00fc6b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:49.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 --2- 172.21.15.153:0/2585550792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e0102f50 0x7f93e00fcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:49.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.236+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 >> 172.21.15.153:0/2585550792 conn(0x7f93e00714c0 msgr2=0x7f93e00fa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:49.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.237+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 shutdown_connections 2024-04-05T03:27:49.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:49.237+0000 7f93e53ea700 1 -- 172.21.15.153:0/2585550792 wait complete. 2024-04-05T03:27:49.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch ps' 2024-04-05T03:27:50.189 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:27:50.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:50 smithi153 ceph-mon[106102]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:50.571 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:50 smithi153 ceph-mon[106102]: pgmap v666: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-05T03:27:50.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:50 smithi203 ceph-mon[95988]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:51.036 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:50 smithi203 ceph-mon[95988]: pgmap v666: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4148940260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 msgr2=0x7fb5080fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4148940260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb5080fb910 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc004940 tx=0x7fb4fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4148940260 shutdown_connections 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4148940260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb5080fb910 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4148940260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 0x7fb508103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4148940260 >> 172.21.15.153:0/4148940260 conn(0x7fb5080714c0 msgr2=0x7fb5080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4148940260 shutdown_connections 2024-04-05T03:27:51.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4148940260 wait complete. 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 Processor -- start 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.815+0000 7fb50d0a7700 1 -- start start 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50d0a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50d0a7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 0x7fb508114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50d0a7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508114730 con 0x7fb5080fb4a0 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50d0a7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50810da90 con 0x7fb508102e70 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35006/0 (socket says 172.21.15.153:35006) 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 -- 172.21.15.153:0/4086282656 learned_addr learned my addr 172.21.15.153:0/4086282656 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50659c700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 0x7fb508114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 -- 172.21.15.153:0/4086282656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 msgr2=0x7fb508114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 0x7fb508114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 -- 172.21.15.153:0/4086282656 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc005040 con 0x7fb5080fb4a0 2024-04-05T03:27:51.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb506d9d700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800b790 tx=0x7fb4f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f800f6f0 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4f800fd20 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f80103a0 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.816+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50810dd70 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.817+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50810e330 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.817+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb508100110 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.819+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb4f8011400 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.819+0000 7fb4ef7fe700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 0x7fb4f0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.819+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb4f808d380 con 0x7fb5080fb4a0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.819+0000 7fb50659c700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 0x7fb4f0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:51.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.820+0000 7fb50659c700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 0x7fb4f0070160 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb50810ec50 tx=0x7fb4fc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:51.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:51.824+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb4f805a4b0 con 0x7fb5080fb4a0 2024-04-05T03:27:52.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.098+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb50810f200 con 0x7fb4f006dcb0 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.116+0000 7fb4ef7fe700 1 -- 172.21.15.153:0/4086282656 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fb50810f200 con 0x7fb4f006dcb0 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (22m) 23s ago 33m 25.4M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (19m) 23s ago 34m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (19m) 24s ago 32m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (22m) 23s ago 33m 81.1M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (16m) 23s ago 27m 123M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:27:52.115 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (16m) 23s ago 28m 110M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (15m) 24s ago 27m 19.4M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (15m) 24s ago 28m 19.5M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (23m) 23s ago 35m 513M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (23m) 24s ago 32m 458M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (21m) 23s ago 36m 70.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (20m) 24s ago 32m 53.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (23m) 23s ago 33m 16.4M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (22m) 24s ago 32m 16.7M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (19m) 23s ago 31m 101M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (18m) 23s ago 31m 108M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (18m) 23s ago 30m 91.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (17m) 24s ago 30m 121M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (17m) 24s ago 29m 103M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:27:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (16m) 24s ago 29m 106M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:27:52.117 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (22m) 24s ago 32m 70.9M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:27:52.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.118+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 msgr2=0x7fb4f0070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.118+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 0x7fb4f0070160 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb50810ec50 tx=0x7fb4fc009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.118+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 msgr2=0x7fb508113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.118+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800b790 tx=0x7fb4f8009250 comp rx=0 tx=0).stop 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 shutdown_connections 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb4f006dcb0 0x7fb4f0070160 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5080fb4a0 0x7fb508113b10 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 --2- 172.21.15.153:0/4086282656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb508102e70 0x7fb508114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 >> 172.21.15.153:0/4086282656 conn(0x7fb5080714c0 msgr2=0x7fb5080fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 shutdown_connections 2024-04-05T03:27:52.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:52.119+0000 7fb50d0a7700 1 -- 172.21.15.153:0/4086282656 wait complete. 2024-04-05T03:27:52.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch upgrade status' 2024-04-05T03:27:52.768 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:52 smithi153 ceph-mon[106102]: pgmap v667: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-05T03:27:52.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:52 smithi203 ceph-mon[95988]: pgmap v667: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-04-05T03:27:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:27:53.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:53 smithi203 ceph-mon[95988]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:53.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:53 smithi153 ceph-mon[106102]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:54.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:54 smithi203 ceph-mon[95988]: pgmap v668: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:27:54.802 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:54 smithi153 ceph-mon[106102]: pgmap v668: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:27:54.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 -- 172.21.15.153:0/721021828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 msgr2=0x7f5aec106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 --2- 172.21.15.153:0/721021828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec106f00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0004940 tx=0x7f5ae0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 -- 172.21.15.153:0/721021828 shutdown_connections 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 --2- 172.21.15.153:0/721021828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec106f00 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 --2- 172.21.15.153:0/721021828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 -- 172.21.15.153:0/721021828 >> 172.21.15.153:0/721021828 conn(0x7f5aec072bc0 msgr2=0x7f5aec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 -- 172.21.15.153:0/721021828 shutdown_connections 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.805+0000 7f5af284c700 1 -- 172.21.15.153:0/721021828 wait complete. 2024-04-05T03:27:54.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 Processor -- start 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 -- start start 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1146b0 con 0x7f5aec0fe830 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5af284c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec10daa0 con 0x7f5aec0fde90 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5aeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5aeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35028/0 (socket says 172.21.15.153:35028) 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.806+0000 7f5aeb7fe700 1 -- 172.21.15.153:0/1618335204 learned_addr learned my addr 172.21.15.153:0/1618335204 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aebfff700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aeb7fe700 1 -- 172.21.15.153:0/1618335204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 msgr2=0x7f5aec113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aeb7fe700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec113b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aeb7fe700 1 -- 172.21.15.153:0/1618335204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0005040 con 0x7f5aec0fe830 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aebfff700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5aeb7fe700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0000c00 tx=0x7f5ae0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ae0021070 con 0x7f5aec0fe830 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec10dd20 con 0x7f5aec0fe830 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ae0013cd0 con 0x7f5aec0fe830 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ae000e070 con 0x7f5aec0fe830 2024-04-05T03:27:54.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.807+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec10e260 con 0x7f5aec0fe830 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.808+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec065e30 con 0x7f5aec0fe830 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.809+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f5ae001c9d0 con 0x7f5aec0fe830 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.810+0000 7f5ae8ff9700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 0x7f5ad4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.810+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5ae008ea40 con 0x7f5aec0fe830 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.810+0000 7f5aebfff700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 0x7f5ad4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:54.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.810+0000 7f5aebfff700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 0x7f5ad4070150 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5adc004600 tx=0x7f5adc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:54.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:54.815+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5ae005bad0 con 0x7f5aec0fe830 2024-04-05T03:27:55.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.036+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5aec109f80 con 0x7f5ad406dca0 2024-04-05T03:27:55.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.037+0000 7f5ae8ff9700 1 -- 172.21.15.153:0/1618335204 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f5aec109f80 con 0x7f5ad406dca0 2024-04-05T03:27:55.035 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": null, 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": false, 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "", 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": null, 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "", 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-04-05T03:27:55.036 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 msgr2=0x7f5ad4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 0x7f5ad4070150 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5adc004600 tx=0x7f5adc00b040 comp rx=0 tx=0).stop 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 msgr2=0x7f5aec114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0000c00 tx=0x7f5ae0009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 shutdown_connections 2024-04-05T03:27:55.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f5ad406dca0 0x7f5ad4070150 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aec0fde90 0x7f5aec113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 --2- 172.21.15.153:0/1618335204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5aec0fe830 0x7f5aec114060 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 >> 172.21.15.153:0/1618335204 conn(0x7f5aec072bc0 msgr2=0x7f5aec103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 shutdown_connections 2024-04-05T03:27:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:55.040+0000 7f5af284c700 1 -- 172.21.15.153:0/1618335204 wait complete. 2024-04-05T03:27:55.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph health detail' 2024-04-05T03:27:56.064 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:27:56.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:56 smithi203 ceph-mon[95988]: from='client.35246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:56.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:56 smithi203 ceph-mon[95988]: pgmap v669: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:27:56.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:56 smithi153 ceph-mon[106102]: from='client.35246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:27:56.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:56 smithi153 ceph-mon[106102]: pgmap v669: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:27:57.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 -- 172.21.15.153:0/14687375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fc4f0 msgr2=0x7f48b8109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:57.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 --2- 172.21.15.153:0/14687375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fc4f0 0x7f48b8109250 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f48ac004940 tx=0x7f48ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:57.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 -- 172.21.15.153:0/14687375 shutdown_connections 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 --2- 172.21.15.153:0/14687375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fc4f0 0x7f48b8109250 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 --2- 172.21.15.153:0/14687375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fbbe0 0x7f48b80fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 -- 172.21.15.153:0/14687375 >> 172.21.15.153:0/14687375 conn(0x7f48b80f78d0 msgr2=0x7f48b80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 -- 172.21.15.153:0/14687375 shutdown_connections 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 -- 172.21.15.153:0/14687375 wait complete. 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.686+0000 7f48bc934700 1 Processor -- start 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48bc934700 1 -- start start 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48bc934700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48bc934700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fc4f0 0x7f48b8116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48bc934700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b8116860 con 0x7f48b80fbbe0 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48bc934700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b810fc30 con 0x7f48b80fc4f0 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:57.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35044/0 (socket says 172.21.15.153:35044) 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 -- 172.21.15.153:0/1599542964 learned_addr learned my addr 172.21.15.153:0/1599542964 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b5d9b700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fc4f0 0x7f48b8116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 -- 172.21.15.153:0/1599542964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fc4f0 msgr2=0x7f48b8116210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fc4f0 0x7f48b8116210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.687+0000 7f48b659c700 1 -- 172.21.15.153:0/1599542964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac005040 con 0x7f48b80fbbe0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48b659c700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f48a8002a50 tx=0x7f48a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48a8007cc0 con 0x7f48b80fbbe0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b810ff10 con 0x7f48b80fbbe0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48a800f040 con 0x7f48b80fbbe0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48a800da30 con 0x7f48b80fbbe0 2024-04-05T03:27:57.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.688+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b81104d0 con 0x7f48b80fbbe0 2024-04-05T03:27:57.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.689+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8066710 con 0x7f48b80fbbe0 2024-04-05T03:27:57.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.690+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f48a800db90 con 0x7f48b80fbbe0 2024-04-05T03:27:57.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.690+0000 7f48a6ffd700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 0x7f48a0070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:27:57.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.690+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f48a808d310 con 0x7f48b80fbbe0 2024-04-05T03:27:57.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.690+0000 7f48b5d9b700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 0x7f48a0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:27:57.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.691+0000 7f48b5d9b700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 0x7f48a0070150 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f48b8111570 tx=0x7f48ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:27:57.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:57.695+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f48a805a3c0 con 0x7f48b80fbbe0 2024-04-05T03:27:57.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:27:58.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.066+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f48b804bed0 con 0x7f48b80fbbe0 2024-04-05T03:27:58.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.067+0000 7f48a6ffd700 1 -- 172.21.15.153:0/1599542964 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f48a8059f50 con 0x7f48b80fbbe0 2024-04-05T03:27:58.065 INFO:teuthology.orchestra.run.smithi153.stdout:HEALTH_OK 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 msgr2=0x7f48a0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 0x7f48a0070150 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f48b8111570 tx=0x7f48ac009e30 comp rx=0 tx=0).stop 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 msgr2=0x7f48b8115cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f48a8002a50 tx=0x7f48a80046b0 comp rx=0 tx=0).stop 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 shutdown_connections 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f48a006dca0 0x7f48a0070150 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:58.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f48b80fbbe0 0x7f48b8115cd0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:58.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 --2- 172.21.15.153:0/1599542964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48b80fc4f0 0x7f48b8116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:27:58.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 >> 172.21.15.153:0/1599542964 conn(0x7f48b80f78d0 msgr2=0x7f48b80f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:27:58.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 shutdown_connections 2024-04-05T03:27:58.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:27:58.070+0000 7f48bc934700 1 -- 172.21.15.153:0/1599542964 wait complete. 2024-04-05T03:27:58.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions' 2024-04-05T03:27:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:27:59.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:58 smithi203 ceph-mon[95988]: pgmap v670: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:27:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:27:58 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1599542964' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:27:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:58 smithi153 ceph-mon[106102]: pgmap v670: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:27:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:27:58 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1599542964' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- 172.21.15.153:0/3344718619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 msgr2=0x7f64580fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 --2- 172.21.15.153:0/3344718619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f64580fb910 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f644c004940 tx=0x7f644c009e30 comp rx=0 tx=0).stop 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- 172.21.15.153:0/3344718619 shutdown_connections 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 --2- 172.21.15.153:0/3344718619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f64580fb910 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 --2- 172.21.15.153:0/3344718619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 0x7f64581018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- 172.21.15.153:0/3344718619 >> 172.21.15.153:0/3344718619 conn(0x7f64580714c0 msgr2=0x7f64580718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- 172.21.15.153:0/3344718619 shutdown_connections 2024-04-05T03:28:00.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- 172.21.15.153:0/3344718619 wait complete. 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 Processor -- start 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.581+0000 7f645dc96700 1 -- start start 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f645dc96700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f645dc96700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 0x7f6458113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f645dc96700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6458114640 con 0x7f64580fb4a0 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f645dc96700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645810da30 con 0x7f64581014f0 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38882/0 (socket says 172.21.15.153:38882) 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 -- 172.21.15.153:0/488745061 learned_addr learned my addr 172.21.15.153:0/488745061 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f6456ffd700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 0x7f6458113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 -- 172.21.15.153:0/488745061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 msgr2=0x7f6458113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 0x7f6458113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 -- 172.21.15.153:0/488745061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c005040 con 0x7f64580fb4a0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.582+0000 7f64577fe700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6448002a80 tx=0x7f64480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.583+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6448007cc0 con 0x7f64580fb4a0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.583+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f644800f040 con 0x7f64580fb4a0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.583+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645810dd10 con 0x7f64580fb4a0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.583+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64480096f0 con 0x7f64580fb4a0 2024-04-05T03:28:00.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.583+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645810e2d0 con 0x7f64580fb4a0 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.584+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6458104ff0 con 0x7f64580fb4a0 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.584+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f644800d690 con 0x7f64580fb4a0 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.585+0000 7f643ffff700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 0x7f6440070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.585+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f644808cd90 con 0x7f64580fb4a0 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.585+0000 7f6456ffd700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 0x7f6440070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:28:00.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.586+0000 7f6456ffd700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 0x7f6440070150 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f644c00b3b0 tx=0x7f644c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:28:00.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.591+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6448059da0 con 0x7f64580fb4a0 2024-04-05T03:28:00.886 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:00 smithi153 ceph-mon[106102]: pgmap v671: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:00.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.945+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f645810f100 con 0x7f64580fb4a0 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.946+0000 7f643ffff700 1 -- 172.21.15.153:0/488745061 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6448059930 con 0x7f64580fb4a0 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:28:00.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:28:00.945 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:28:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 msgr2=0x7f6440070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 0x7f6440070150 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f644c00b3b0 tx=0x7f644c009e30 comp rx=0 tx=0).stop 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 msgr2=0x7f6458113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6448002a80 tx=0x7f64480046b0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 shutdown_connections 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f644006dca0 0x7f6440070150 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64580fb4a0 0x7f6458113ab0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 --2- 172.21.15.153:0/488745061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64581014f0 0x7f6458113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.948+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 >> 172.21.15.153:0/488745061 conn(0x7f64580714c0 msgr2=0x7f6458102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.949+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 shutdown_connections 2024-04-05T03:28:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:28:00.949+0000 7f645dc96700 1 -- 172.21.15.153:0/488745061 wait complete. 2024-04-05T03:28:01.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:00 smithi203 ceph-mon[95988]: pgmap v671: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:01.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-05T03:28:01.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:01 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/488745061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:28:01.868 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:28:02.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:01 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/488745061' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:28:03.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:02 smithi203 ceph-mon[95988]: pgmap v672: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:02 smithi153 ceph-mon[106102]: pgmap v672: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:03.402 INFO:teuthology.orchestra.run.smithi153.stdout:wait for servicemap items w/ changing names to refresh 2024-04-05T03:28:04.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'sleep 60' 2024-04-05T03:28:04.356 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:28:05.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:04 smithi203 ceph-mon[95988]: pgmap v673: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:04 smithi153 ceph-mon[106102]: pgmap v673: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:06 smithi203 ceph-mon[95988]: pgmap v674: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:06 smithi153 ceph-mon[106102]: pgmap v674: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:09.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:08 smithi203 ceph-mon[95988]: pgmap v675: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:08 smithi153 ceph-mon[106102]: pgmap v675: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:11.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:10 smithi203 ceph-mon[95988]: pgmap v676: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:11.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:10 smithi153 ceph-mon[106102]: pgmap v676: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:12 smithi203 ceph-mon[95988]: pgmap v677: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:12 smithi153 ceph-mon[106102]: pgmap v677: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:15.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:14 smithi203 ceph-mon[95988]: pgmap v678: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:14 smithi153 ceph-mon[106102]: pgmap v678: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:17.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:16 smithi203 ceph-mon[95988]: pgmap v679: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:17.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:16 smithi153 ceph-mon[106102]: pgmap v679: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:19.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:18 smithi203 ceph-mon[95988]: pgmap v680: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:18 smithi153 ceph-mon[106102]: pgmap v680: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:20 smithi203 ceph-mon[95988]: pgmap v681: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:21.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:20 smithi153 ceph-mon[106102]: pgmap v681: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:23.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:22 smithi203 ceph-mon[95988]: pgmap v682: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:22 smithi153 ceph-mon[106102]: pgmap v682: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:24 smithi203 ceph-mon[95988]: pgmap v683: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:24 smithi153 ceph-mon[106102]: pgmap v683: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:26 smithi203 ceph-mon[95988]: pgmap v684: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:26 smithi153 ceph-mon[106102]: pgmap v684: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:28.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:28 smithi203 ceph-mon[95988]: pgmap v685: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:28 smithi153 ceph-mon[106102]: pgmap v685: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:30.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:29 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:28:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:29 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:28:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:30 smithi203 ceph-mon[95988]: pgmap v686: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:28:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:28:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:28:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:30 smithi153 ceph-mon[106102]: pgmap v686: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:28:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:28:31.071 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:28:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:32 smithi203 ceph-mon[95988]: pgmap v687: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:32 smithi153 ceph-mon[106102]: pgmap v687: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:34 smithi203 ceph-mon[95988]: pgmap v688: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:34 smithi153 ceph-mon[106102]: pgmap v688: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:36.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:35 smithi203 ceph-mon[95988]: pgmap v689: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:36.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:35 smithi153 ceph-mon[106102]: pgmap v689: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:38.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:38 smithi203 ceph-mon[95988]: pgmap v690: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:38.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:38 smithi153 ceph-mon[106102]: pgmap v690: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:40.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:40 smithi203 ceph-mon[95988]: pgmap v691: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:40.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:40 smithi153 ceph-mon[106102]: pgmap v691: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:42.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:42 smithi203 ceph-mon[95988]: pgmap v692: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:42 smithi153 ceph-mon[106102]: pgmap v692: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:43.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:43 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:43.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:43 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:44.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:44 smithi203 ceph-mon[95988]: pgmap v693: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:44 smithi153 ceph-mon[106102]: pgmap v693: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:46.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:46 smithi203 ceph-mon[95988]: pgmap v694: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:46.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:46 smithi153 ceph-mon[106102]: pgmap v694: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:48.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:48 smithi203 ceph-mon[95988]: pgmap v695: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:48.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:48 smithi153 ceph-mon[106102]: pgmap v695: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:50.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:50 smithi203 ceph-mon[95988]: pgmap v696: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:50.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:50 smithi153 ceph-mon[106102]: pgmap v696: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:52.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:52 smithi203 ceph-mon[95988]: pgmap v697: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:52.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:52 smithi153 ceph-mon[106102]: pgmap v697: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:28:54.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:54 smithi203 ceph-mon[95988]: pgmap v698: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:54.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:54 smithi153 ceph-mon[106102]: pgmap v698: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:28:56.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:56 smithi203 ceph-mon[95988]: pgmap v699: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:56.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:56 smithi153 ceph-mon[106102]: pgmap v699: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:28:58.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:58 smithi203 ceph-mon[95988]: pgmap v700: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:58.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:28:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:28:58.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:58 smithi153 ceph-mon[106102]: pgmap v700: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:28:58.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:28:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:00.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:00 smithi203 ceph-mon[95988]: pgmap v701: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:00.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:00 smithi153 ceph-mon[106102]: pgmap v701: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:02.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:02 smithi203 ceph-mon[95988]: pgmap v702: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:02.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:02 smithi153 ceph-mon[106102]: pgmap v702: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:04.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:04 smithi203 ceph-mon[95988]: pgmap v703: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:04.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:04 smithi153 ceph-mon[106102]: pgmap v703: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:06.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:06 smithi153 ceph-mon[106102]: pgmap v704: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:06.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph orch ps' 2024-04-05T03:29:06.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:06 smithi203 ceph-mon[95988]: pgmap v704: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:06.941 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:29:08.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/4156719886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 msgr2=0x7f4c840fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/4156719886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c840fce10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4c70004940 tx=0x7f4c70009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/4156719886 shutdown_connections 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/4156719886 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c840fce10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/4156719886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 0x7f4c841031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/4156719886 >> 172.21.15.153:0/4156719886 conn(0x7f4c840f89b0 msgr2=0x7f4c840fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/4156719886 shutdown_connections 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/4156719886 wait complete. 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.614+0000 7f4c8b1d3700 1 Processor -- start 2024-04-05T03:29:08.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c8b1d3700 1 -- start start 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c8b1d3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 0x7f4c84113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c8b1d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c8b1d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c841147c0 con 0x7f4c841037a0 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c8b1d3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8410db40 con 0x7f4c84102e00 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:08.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56394/0 (socket says 172.21.15.153:56394) 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 -- 172.21.15.153:0/3630727441 learned_addr learned my addr 172.21.15.153:0/3630727441 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c88f6f700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 0x7f4c84113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 -- 172.21.15.153:0/3630727441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 msgr2=0x7f4c84113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 0x7f4c84113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.615+0000 7f4c83fff700 1 -- 172.21.15.153:0/3630727441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c70005040 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c83fff700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4c70009180 tx=0x7f4c70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7000e040 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c7000bbc0 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8410ddc0 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7001c3f0 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.616+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8410e380 con 0x7f4c841037a0 2024-04-05T03:29:08.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.617+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c84100110 con 0x7f4c841037a0 2024-04-05T03:29:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.618+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f4c7000bd30 con 0x7f4c841037a0 2024-04-05T03:29:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.618+0000 7f4c817fa700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 0x7f4c74070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.618+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4c7008e8e0 con 0x7f4c841037a0 2024-04-05T03:29:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.618+0000 7f4c88f6f700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 0x7f4c74070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:08.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.619+0000 7f4c88f6f700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 0x7f4c74070150 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4c78006fd0 tx=0x7f4c78008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:08.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.627+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4c70057ee0 con 0x7f4c841037a0 2024-04-05T03:29:08.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:08 smithi203 ceph-mon[95988]: pgmap v705: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:08.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:08 smithi153 ceph-mon[106102]: pgmap v705: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:08.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.845+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 --> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c84109f50 con 0x7f4c7406dca0 2024-04-05T03:29:08.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.855+0000 7f4c817fa700 1 -- 172.21.15.153:0/3630727441 <== mgr.34104 v2:172.21.15.153:6800/3734499116 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4c84109f50 con 0x7f4c7406dca0 2024-04-05T03:29:08.853 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (23m) 100s ago 34m 25.4M - 0.25.0 c8568f914cd2 7eefdb3664d8 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (21m) 100s ago 35m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 e1c5da4a4a64 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi203 smithi203 running (21m) 101s ago 33m 6660k - 19.0.0-2013-g0028525c d8c35eaf3c25 5054bbe01960 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (23m) 100s ago 34m 81.1M - 9.4.12 2bacad6d85d8 94a23d16813d 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.fmpmvd smithi153 running (17m) 100s ago 29m 123M - 19.0.0-2013-g0028525c d8c35eaf3c25 76fa98864f86 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.vicaky smithi153 running (17m) 100s ago 29m 110M - 19.0.0-2013-g0028525c d8c35eaf3c25 efa706f983f4 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.hgbifk smithi203 running (16m) 101s ago 29m 19.4M - 19.0.0-2013-g0028525c d8c35eaf3c25 86473397e5a9 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi203.xpteyc smithi203 running (17m) 101s ago 29m 19.5M - 19.0.0-2013-g0028525c d8c35eaf3c25 f6e8ff3e92a0 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.npzyqy smithi153 *:8443,9283,8765 running (25m) 100s ago 37m 513M - 19.0.0-2013-g0028525c d8c35eaf3c25 c224ad98d658 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi203.iakodz smithi203 *:8443,9283,8765 running (24m) 101s ago 33m 458M - 19.0.0-2013-g0028525c d8c35eaf3c25 85612ae4feb6 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (22m) 100s ago 37m 70.5M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 26862dff0163 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi203 smithi203 running (21m) 101s ago 33m 53.4M 2048M 19.0.0-2013-g0028525c d8c35eaf3c25 1cf9e0034643 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (24m) 100s ago 35m 16.4M - 1.5.0 0da6a335fe13 1a3a1e424a47 2024-04-05T03:29:08.854 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi203 smithi203 *:9100 running (24m) 101s ago 33m 16.7M - 1.5.0 0da6a335fe13 b99aa4e4f2fd 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (20m) 100s ago 33m 101M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 81dada4d04a0 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (20m) 100s ago 32m 108M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 4ca694768d37 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (19m) 100s ago 32m 91.4M 1898M 19.0.0-2013-g0028525c d8c35eaf3c25 fc7ea381564e 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi203 running (19m) 101s ago 31m 121M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 3c8b7968d2cb 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi203 running (18m) 101s ago 31m 103M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 57d855c137b5 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi203 running (17m) 101s ago 30m 106M 2240M 19.0.0-2013-g0028525c d8c35eaf3c25 750dd619bf46 2024-04-05T03:29:08.855 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi203 smithi203 *:9095 running (23m) 101s ago 33m 70.9M - 2.43.0 a07b618ecd1d 3f45547900f0 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 msgr2=0x7f4c74070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 0x7f4c74070150 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4c78006fd0 tx=0x7f4c78008040 comp rx=0 tx=0).stop 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 msgr2=0x7f4c841140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4c70009180 tx=0x7f4c70009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 shutdown_connections 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f4c7406dca0 0x7f4c74070150 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c84102e00 0x7f4c84113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 --2- 172.21.15.153:0/3630727441 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c841037a0 0x7f4c841140e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 >> 172.21.15.153:0/3630727441 conn(0x7f4c840f89b0 msgr2=0x7f4c840fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:08.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 shutdown_connections 2024-04-05T03:29:08.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:08.858+0000 7f4c8b1d3700 1 -- 172.21.15.153:0/3630727441 wait complete. 2024-04-05T03:29:09.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions' 2024-04-05T03:29:09.817 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:29:10.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:10 smithi203 ceph-mon[95988]: from='client.35258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:29:10.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:10 smithi203 ceph-mon[95988]: pgmap v706: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:10.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:10 smithi153 ceph-mon[106102]: from='client.35258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T03:29:10.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:10 smithi153 ceph-mon[106102]: pgmap v706: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 -- 172.21.15.153:0/2620417197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c101d70 msgr2=0x7f1a5c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 --2- 172.21.15.153:0/2620417197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c101d70 0x7f1a5c102140 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004940 tx=0x7f1a4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 -- 172.21.15.153:0/2620417197 shutdown_connections 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 --2- 172.21.15.153:0/2620417197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c102710 0x7f1a5c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 --2- 172.21.15.153:0/2620417197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c101d70 0x7f1a5c102140 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 -- 172.21.15.153:0/2620417197 >> 172.21.15.153:0/2620417197 conn(0x7f1a5c072bc0 msgr2=0x7f1a5c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 -- 172.21.15.153:0/2620417197 shutdown_connections 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.425+0000 7f1a6203d700 1 -- 172.21.15.153:0/2620417197 wait complete. 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 Processor -- start 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 -- start start 2024-04-05T03:29:11.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c101d70 0x7f1a5c115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c116910 con 0x7f1a5c102710 2024-04-05T03:29:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a6203d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c10fce0 con 0x7f1a5c101d70 2024-04-05T03:29:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a5affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:11.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a5affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56416/0 (socket says 172.21.15.153:56416) 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.426+0000 7f1a5affd700 1 -- 172.21.15.153:0/815503225 learned_addr learned my addr 172.21.15.153:0/815503225 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a5b7fe700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c101d70 0x7f1a5c115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a5affd700 1 -- 172.21.15.153:0/815503225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c101d70 msgr2=0x7f1a5c115d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a5affd700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c101d70 0x7f1a5c115d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a5affd700 1 -- 172.21.15.153:0/815503225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c005040 con 0x7f1a5c102710 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a5affd700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1a50004e70 tx=0x7f1a50004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a50004ea0 con 0x7f1a5c102710 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c10ff60 con 0x7f1a5c102710 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a50016e20 con 0x7f1a5c102710 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5000d5c0 con 0x7f1a5c102710 2024-04-05T03:29:11.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.427+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c110520 con 0x7f1a5c102710 2024-04-05T03:29:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.429+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c0fd4c0 con 0x7f1a5c102710 2024-04-05T03:29:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.429+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f1a5001c070 con 0x7f1a5c102710 2024-04-05T03:29:11.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.429+0000 7f1a43fff700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 0x7f1a44070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.430+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1a50095940 con 0x7f1a5c102710 2024-04-05T03:29:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.430+0000 7f1a5b7fe700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 0x7f1a44070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.430+0000 7f1a5b7fe700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 0x7f1a44070150 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004f90 tx=0x7f1a4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:11.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.434+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1a500629f0 con 0x7f1a5c102710 2024-04-05T03:29:11.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.723+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a5c04bed0 con 0x7f1a5c102710 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.724+0000 7f1a43fff700 1 -- 172.21.15.153:0/815503225 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f1a50062580 con 0x7f1a5c102710 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 2 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:29:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 6 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 4 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)": 14 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-05T03:29:11.723 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 msgr2=0x7f1a44070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 0x7f1a44070150 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004f90 tx=0x7f1a4c009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 msgr2=0x7f1a5c1162c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1a50004e70 tx=0x7f1a50004600 comp rx=0 tx=0).stop 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 shutdown_connections 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f1a4406dca0 0x7f1a44070150 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a5c101d70 0x7f1a5c115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 --2- 172.21.15.153:0/815503225 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a5c102710 0x7f1a5c1162c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:11.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 >> 172.21.15.153:0/815503225 conn(0x7f1a5c072bc0 msgr2=0x7f1a5c065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:11.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 shutdown_connections 2024-04-05T03:29:11.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:11.727+0000 7f1a6203d700 1 -- 172.21.15.153:0/815503225 wait complete. 2024-04-05T03:29:12.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-05T03:29:12.646 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:29:12.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:12 smithi153 ceph-mon[106102]: pgmap v707: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:12.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:12 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/815503225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:12.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:12 smithi203 ceph-mon[95988]: pgmap v707: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:12.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:12 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/815503225' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:13.789 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:13 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:13.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:13 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 -- 172.21.15.153:0/1678331447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18101d70 msgr2=0x7fbc18102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 --2- 172.21.15.153:0/1678331447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18101d70 0x7fbc18102140 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbc08004940 tx=0x7fbc08009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 -- 172.21.15.153:0/1678331447 shutdown_connections 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 --2- 172.21.15.153:0/1678331447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18102710 0x7fbc180fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 --2- 172.21.15.153:0/1678331447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18101d70 0x7fbc18102140 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 -- 172.21.15.153:0/1678331447 >> 172.21.15.153:0/1678331447 conn(0x7fbc18072bc0 msgr2=0x7fbc18072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.221+0000 7fbc1e011700 1 -- 172.21.15.153:0/1678331447 shutdown_connections 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.222+0000 7fbc1e011700 1 -- 172.21.15.153:0/1678331447 wait complete. 2024-04-05T03:29:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.222+0000 7fbc1e011700 1 Processor -- start 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.222+0000 7fbc1e011700 1 -- start start 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc1e011700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18101d70 0x7fbc1810c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc1e011700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc1e011700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1810d350 con 0x7fbc18102710 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc1e011700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1810d4c0 con 0x7fbc18101d70 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56430/0 (socket says 172.21.15.153:56430) 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 -- 172.21.15.153:0/2586511161 learned_addr learned my addr 172.21.15.153:0/2586511161 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc177fe700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18101d70 0x7fbc1810c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 -- 172.21.15.153:0/2586511161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18101d70 msgr2=0x7fbc1810c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18101d70 0x7fbc1810c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 -- 172.21.15.153:0/2586511161 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08005040 con 0x7fbc18102710 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.223+0000 7fbc16ffd700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c800 tx=0x7fbc0c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.224+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc0c0106f0 con 0x7fbc18102710 2024-04-05T03:29:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.224+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc18119f40 con 0x7fbc18102710 2024-04-05T03:29:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.224+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc0c010d20 con 0x7fbc18102710 2024-04-05T03:29:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.224+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc0c0113a0 con 0x7fbc18102710 2024-04-05T03:29:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.224+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1811a310 con 0x7fbc18102710 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.225+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc180ff080 con 0x7fbc18102710 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.225+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbc0c012400 con 0x7fbc18102710 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.226+0000 7fbbfffff700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 0x7fbc00070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.226+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbc0c08d430 con 0x7fbc18102710 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.226+0000 7fbc177fe700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 0x7fbc00070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.227+0000 7fbc177fe700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 0x7fbc00070150 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbc08004f90 tx=0x7fbc08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.231+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbc0c05a560 con 0x7fbc18102710 2024-04-05T03:29:14.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.537+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbc18106760 con 0x7fbc18102710 2024-04-05T03:29:14.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.537+0000 7fbbfffff700 1 -- 172.21.15.153:0/2586511161 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fbc0c05a0f0 con 0x7fbc18102710 2024-04-05T03:29:14.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 msgr2=0x7fbc00070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 0x7fbc00070150 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbc08004f90 tx=0x7fbc08009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 msgr2=0x7fbc1810ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c800 tx=0x7fbc0c004470 comp rx=0 tx=0).stop 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 shutdown_connections 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbc0006dca0 0x7fbc00070150 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc18101d70 0x7fbc1810c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 --2- 172.21.15.153:0/2586511161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc18102710 0x7fbc1810ce10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.539+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 >> 172.21.15.153:0/2586511161 conn(0x7fbc18072bc0 msgr2=0x7fbc180fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.540+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 shutdown_connections 2024-04-05T03:29:14.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:14.540+0000 7fbc1e011700 1 -- 172.21.15.153:0/2586511161 wait complete. 2024-04-05T03:29:14.549 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-04-05T03:29:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:14 smithi203 ceph-mon[95988]: pgmap v708: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:14.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:14 smithi153 ceph-mon[106102]: pgmap v708: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:15.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -e sha1=0028525c15e19309c20289c609e4fb9d0b12dd06 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-05T03:29:15.425 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:29:15.762 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:15 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2586511161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:15.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:15 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2586511161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:16.729 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:16 smithi153 ceph-mon[106102]: pgmap v709: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:16.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:16 smithi203 ceph-mon[95988]: pgmap v709: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 -- 172.21.15.153:0/4081797687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34102710 msgr2=0x7fef340fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/4081797687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34102710 0x7fef340fbd80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fef24004940 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 -- 172.21.15.153:0/4081797687 shutdown_connections 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/4081797687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34102710 0x7fef340fbd80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/4081797687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef34101d70 0x7fef34102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 -- 172.21.15.153:0/4081797687 >> 172.21.15.153:0/4081797687 conn(0x7fef34072bc0 msgr2=0x7fef34072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 -- 172.21.15.153:0/4081797687 shutdown_connections 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.971+0000 7fef3b4fc700 1 -- 172.21.15.153:0/4081797687 wait complete. 2024-04-05T03:29:16.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 Processor -- start 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 -- start start 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef34102710 0x7fef3410f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3410f760 con 0x7fef34101d70 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.972+0000 7fef3b4fc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3410f8a0 con 0x7fef34102710 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56438/0 (socket says 172.21.15.153:56438) 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 -- 172.21.15.153:0/330197995 learned_addr learned my addr 172.21.15.153:0/330197995 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 -- 172.21.15.153:0/330197995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef34102710 msgr2=0x7fef3410f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef34102710 0x7fef3410f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 -- 172.21.15.153:0/330197995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24005040 con 0x7fef34101d70 2024-04-05T03:29:16.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef39298700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef30002a50 tx=0x7fef300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef30007cc0 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef34117b90 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef3000f040 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.973+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef30009640 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.974+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef34118120 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.975+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef34104240 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.975+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fef3000d690 con 0x7fef34101d70 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.976+0000 7fef29ffb700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 0x7fef20070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:16.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.976+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fef3008d260 con 0x7fef34101d70 2024-04-05T03:29:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.976+0000 7fef38a97700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 0x7fef20070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:16.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.976+0000 7fef38a97700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 0x7fef20070150 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fef24004f90 tx=0x7fef24009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:16.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:16.980+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fef30056860 con 0x7fef34101d70 2024-04-05T03:29:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.268+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fef34108e50 con 0x7fef34101d70 2024-04-05T03:29:17.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.268+0000 7fef29ffb700 1 -- 172.21.15.153:0/330197995 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fef30059f20 con 0x7fef34101d70 2024-04-05T03:29:17.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 msgr2=0x7fef20070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 0x7fef20070150 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fef24004f90 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 msgr2=0x7fef3410ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fef30002a50 tx=0x7fef300046b0 comp rx=0 tx=0).stop 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 shutdown_connections 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fef2006dca0 0x7fef20070150 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef34101d70 0x7fef3410ece0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 --2- 172.21.15.153:0/330197995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef34102710 0x7fef3410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 >> 172.21.15.153:0/330197995 conn(0x7fef34072bc0 msgr2=0x7fef340fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 shutdown_connections 2024-04-05T03:29:17.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:17.271+0000 7fef3b4fc700 1 -- 172.21.15.153:0/330197995 wait complete. 2024-04-05T03:29:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 19.0.0-2013-g0028525c (0028525c15e19309c20289c609e4fb9d0b12dd06) squid (dev)" 2024-04-05T03:29:17.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:17 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/330197995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:17.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:17 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/330197995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T03:29:17.920 DEBUG:teuthology.parallel:result is None 2024-04-05T03:29:17.921 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T03:29:17.932 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:29:17.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T03:29:18.203 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/mon.smithi153/config 2024-04-05T03:29:18.578 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:18 smithi153 ceph-mon[106102]: pgmap v710: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:18.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:18 smithi203 ceph-mon[95988]: pgmap v710: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.820+0000 7f569b462700 1 -- 172.21.15.153:0/3411043264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 msgr2=0x7f56940fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.820+0000 7f569b462700 1 --2- 172.21.15.153:0/3411043264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56940fce30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5684004940 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- 172.21.15.153:0/3411043264 shutdown_connections 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 --2- 172.21.15.153:0/3411043264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56940fce30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 --2- 172.21.15.153:0/3411043264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 0x7f56941031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- 172.21.15.153:0/3411043264 >> 172.21.15.153:0/3411043264 conn(0x7f56940f8970 msgr2=0x7f56940fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- 172.21.15.153:0/3411043264 shutdown_connections 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- 172.21.15.153:0/3411043264 wait complete. 2024-04-05T03:29:19.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 Processor -- start 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- start start 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 0x7f5694113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5694114710 con 0x7f56941037c0 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.821+0000 7f569b462700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569410db70 con 0x7f5694102e20 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41672/0 (socket says 172.21.15.153:41672) 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 -- 172.21.15.153:0/3159109956 learned_addr learned my addr 172.21.15.153:0/3159109956 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56991fe700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 0x7f5694113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 -- 172.21.15.153:0/3159109956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 msgr2=0x7f5694113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 0x7f5694113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 -- 172.21.15.153:0/3159109956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f56941037c0 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f56989fd700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5684009100 tx=0x7f5684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568400e040 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5684009af0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569410ddf0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.822+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f568401c3f0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.823+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569410e3b0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.824+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56941052f0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.824+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f568400baf0 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.825+0000 7f5689ffb700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 0x7f5680070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.825+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f568408e700 con 0x7f56941037c0 2024-04-05T03:29:19.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.825+0000 7f56991fe700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 0x7f5680070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:19.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.825+0000 7f56991fe700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 0x7f5680070150 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5690004ed0 tx=0x7f569000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:19.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:19.830+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5684057da0 con 0x7f56941037c0 2024-04-05T03:29:20.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.118+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f569410f1d0 con 0x7f56941037c0 2024-04-05T03:29:20.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.118+0000 7f5689ffb700 1 -- 172.21.15.153:0/3159109956 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 42 v42) v1 ==== 76+0+1973 (secure 0 0 0) 0x7f568405b460 con 0x7f56941037c0 2024-04-05T03:29:20.116 INFO:teuthology.orchestra.run.smithi153.stdout:e42 2024-04-05T03:29:20.116 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 42 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-04-05T02:59:41.656590+0000 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-04-05T03:27:32.727469+0000 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-04-05T03:29:20.117 INFO:teuthology.orchestra.run.smithi153.stdout:max_xattr_size 65536 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 80 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=34388,1=34420} 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-04-05T03:29:20.118 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:bal_rank_mask -1 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:qdb_cluster leader: 34388 members: 34388,34420 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.vicaky{0:34388} state up:active seq 244 join_fscid=1 addr [v2:172.21.15.153:6826/1501229343,v1:172.21.15.153:6827/1501229343] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.xpteyc{0:44325} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.203:6824/185099298,v1:172.21.15.203:6825/185099298] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.fmpmvd{1:34420} state up:active seq 239 join_fscid=1 addr [v2:172.21.15.153:6828/2545565132,v1:172.21.15.153:6829/2545565132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi203.hgbifk{1:44357} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.203:6826/3079507818,v1:172.21.15.203:6827/3079507818] compat {c=[1],r=[1],i=[1fff]}] 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:20.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 msgr2=0x7f5680070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 0x7f5680070150 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5690004ed0 tx=0x7f569000b040 comp rx=0 tx=0).stop 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 msgr2=0x7f56941140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5684009100 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 shutdown_connections 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f568006dca0 0x7f5680070150 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5694102e20 0x7f5694113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 --2- 172.21.15.153:0/3159109956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56941037c0 0x7f56941140c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 >> 172.21.15.153:0/3159109956 conn(0x7f56940f8970 msgr2=0x7f5694102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 shutdown_connections 2024-04-05T03:29:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.121+0000 7f569b462700 1 -- 172.21.15.153:0/3159109956 wait complete. 2024-04-05T03:29:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 42 2024-04-05T03:29:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:20 smithi153 ceph-mon[106102]: pgmap v711: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:20.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:20 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3159109956' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:29:20.758 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-05T03:29:20.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T03:29:20.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:20 smithi203 ceph-mon[95988]: pgmap v711: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:20.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:20 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3159109956' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 -- 172.21.15.153:0/3163791946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 msgr2=0x7f2704101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 --2- 172.21.15.153:0/3163791946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f2704101510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f26f40048d0 tx=0x7f26f4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 -- 172.21.15.153:0/3163791946 shutdown_connections 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 --2- 172.21.15.153:0/3163791946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 0x7f27040fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 --2- 172.21.15.153:0/3163791946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f2704101510 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 -- 172.21.15.153:0/3163791946 >> 172.21.15.153:0/3163791946 conn(0x7f2704071130 msgr2=0x7f2704071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 -- 172.21.15.153:0/3163791946 shutdown_connections 2024-04-05T03:29:20.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 -- 172.21.15.153:0/3163791946 wait complete. 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.906+0000 7f2708b5d640 1 Processor -- start 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 -- start start 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 0x7f270410deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270410e930 con 0x7f2704101140 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270410eaa0 con 0x7f27040fb110 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41692/0 (socket says 172.21.15.153:41692) 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 -- 172.21.15.153:0/1591110086 learned_addr learned my addr 172.21.15.153:0/1591110086 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f270259c640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 0x7f270410deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 -- 172.21.15.153:0/1591110086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 msgr2=0x7f270410deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 0x7f270410deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:20.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 -- 172.21.15.153:0/1591110086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4005040 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2701d9b640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f26f800b790 tx=0x7f26f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f800c700 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26f800cd30 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.907+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2704116f20 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.908+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f8013370 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.908+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27041172a0 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.908+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27040fc800 con 0x7f2704101140 2024-04-05T03:29:20.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.909+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f26f8012070 con 0x7f2704101140 2024-04-05T03:29:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.909+0000 7f26f2ffd640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 0x7f26ec0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.909+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26f808d0b0 con 0x7f2704101140 2024-04-05T03:29:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.910+0000 7f270259c640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 0x7f26ec0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:20.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.910+0000 7f270259c640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 0x7f26ec0700c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009fa0 tx=0x7f26f4009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:20.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:20.914+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f26f8056630 con 0x7f2704101140 2024-04-05T03:29:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.095+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2704108310 con 0x7f2704101140 2024-04-05T03:29:21.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.096+0000 7f26f2ffd640 1 -- 172.21.15.153:0/1591110086 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16727 (secure 0 0 0) 0x7f26f8059cf0 con 0x7f2704101140 2024-04-05T03:29:21.095 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:21.095 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":84,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T03:12:17.587040+0000","last_up_change":"2024-04-05T03:11:21.401777+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T02:59:40.392548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T02:59:40.884499+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6803","nonce":190885135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6805","nonce":190885135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6809","nonce":190885135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6807","nonce":190885135}]},"public_addr":"172.21.15.153:6803/190885135","cluster_addr":"172.21.15.153:6805/190885135","heartbeat_back_addr":"172.21.15.153:6809/190885135","heartbeat_front_addr":"172.21.15.153:6807/190885135","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6811","nonce":2271479340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6813","nonce":2271479340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6817","nonce":2271479340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6815","nonce":2271479340}]},"public_addr":"172.21.15.153:6811/2271479340","cluster_addr":"172.21.15.153:6813/2271479340","heartbeat_back_addr":"172.21.15.153:6817/2271479340","heartbeat_front_addr":"172.21.15.153:6815/2271479340","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6819","nonce":2475488589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6821","nonce":2475488589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6825","nonce":2475488589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6823","nonce":2475488589}]},"public_addr":"172.21.15.153:6819/2475488589","cluster_addr":"172.21.15.153:6821/2475488589","heartbeat_back_addr":"172.21.15.153:6825/2475488589","heartbeat_front_addr":"172.21.15.153:6823/2475488589","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6801","nonce":2106212898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6803","nonce":2106212898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6807","nonce":2106212898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6805","nonce":2106212898}]},"public_addr":"172.21.15.203:6801/2106212898","cluster_addr":"172.21.15.203:6803/2106212898","heartbeat_back_addr":"172.21.15.203:6807/2106212898","heartbeat_front_addr":"172.21.15.203:6805/2106212898","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6809","nonce":2710960120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6811","nonce":2710960120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6815","nonce":2710960120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6813","nonce":2710960120}]},"public_addr":"172.21.15.203:6809/2710960120","cluster_addr":"172.21.15.203:6811/2710960120","heartbeat_back_addr":"172.21.15.203:6815/2710960120","heartbeat_front_addr":"172.21.15.203:6813/2710960120","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6817","nonce":2060217398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6819","nonce":2060217398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6823","nonce":2060217398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6821","nonce":2060217398}]},"public_addr":"172.21.15.203:6817/2060217398","cluster_addr":"172.21.15.203:6819/2060217398","heartbeat_back_addr":"172.21.15.203:6823/2060217398","heartbeat_front_addr":"172.21.15.203:6821/2060217398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6829/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.203:0/3454911926":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6826/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/2095582209":"2024-04-06T03:04:07.806216+0000","172.21.15.203:6825/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:6827/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1792434239":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/3744404984":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/4110399623":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:6824/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:0/2060103876":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.203:6828/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6828/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.153:0/971996676":"2024-04-06T03:02:35.649226+0000","172.21.15.203:6829/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3393095164":"2024-04-06T03:02:35.649226+0000","172.21.15.203:0/3658559261":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6800/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/3278817285":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/220940347":"2024-04-06T03:02:35.649226+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/2297491169":"2024-04-06T03:06:57.410816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T03:29:21.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 msgr2=0x7f26ec0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 0x7f26ec0700c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009fa0 tx=0x7f26f4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 msgr2=0x7f270410e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f26f800b790 tx=0x7f26f80091d0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 shutdown_connections 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f26ec06dc10 0x7f26ec0700c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2704101140 0x7f270410e3f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 --2- 172.21.15.153:0/1591110086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27040fb110 0x7f270410deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 >> 172.21.15.153:0/1591110086 conn(0x7f2704071130 msgr2=0x7f27040f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 shutdown_connections 2024-04-05T03:29:21.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.098+0000 7f2708b5d640 1 -- 172.21.15.153:0/1591110086 wait complete. 2024-04-05T03:29:21.108 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T02:57:13.688102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T02:59:40.392548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T02:59:40.884499+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T03:29:21.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 -- 172.21.15.153:0/3628321308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 msgr2=0x7f7adc101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 --2- 172.21.15.153:0/3628321308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc101510 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0048d0 tx=0x7f7acc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 -- 172.21.15.153:0/3628321308 shutdown_connections 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 --2- 172.21.15.153:0/3628321308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 0x7f7adc0fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 --2- 172.21.15.153:0/3628321308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc101510 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 -- 172.21.15.153:0/3628321308 >> 172.21.15.153:0/3628321308 conn(0x7f7adc0710b0 msgr2=0x7f7adc0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 -- 172.21.15.153:0/3628321308 shutdown_connections 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.212+0000 7f7ae299d640 1 -- 172.21.15.153:0/3628321308 wait complete. 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 Processor -- start 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 -- start start 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 0x7f7adc111be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc112720 con 0x7f7adc101140 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7ae299d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc10bd70 con 0x7f7adc0fb110 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7adb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7adb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41696/0 (socket says 172.21.15.153:41696) 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.213+0000 7f7adb7fe640 1 -- 172.21.15.153:0/4013129913 learned_addr learned my addr 172.21.15.153:0/4013129913 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7adbfff640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 0x7f7adc111be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7adb7fe640 1 -- 172.21.15.153:0/4013129913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 msgr2=0x7f7adc111be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7adb7fe640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 0x7f7adc111be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7adb7fe640 1 -- 172.21.15.153:0/4013129913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc005040 con 0x7f7adc101140 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7adb7fe640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0002a50 tx=0x7f7ad0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad0007c30 con 0x7f7adc101140 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc10c050 con 0x7f7adc101140 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ad000f040 con 0x7f7adc101140 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad000db50 con 0x7f7adc101140 2024-04-05T03:29:21.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.214+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc10c570 con 0x7f7adc101140 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.215+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc0fc800 con 0x7f7adc101140 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.215+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7ad001c400 con 0x7f7adc101140 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.216+0000 7f7ad8ff9640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 0x7f7ac40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.216+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7ad008ce90 con 0x7f7adc101140 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.216+0000 7f7adbfff640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 0x7f7ac40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.217+0000 7f7adbfff640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 0x7f7ac40700c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009990 tx=0x7f7acc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.221+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7ad0056410 con 0x7f7adc101140 2024-04-05T03:29:21.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.401+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7adc108220 con 0x7f7adc101140 2024-04-05T03:29:21.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.401+0000 7f7ad8ff9640 1 -- 172.21.15.153:0/4013129913 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ad0059ad0 con 0x7f7adc101140 2024-04-05T03:29:21.399 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.403+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 msgr2=0x7f7ac40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.403+0000 7f7ae299d640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 0x7f7ac40700c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009990 tx=0x7f7acc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.403+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 msgr2=0x7f7adc112120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.403+0000 7f7ae299d640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0002a50 tx=0x7f7ad0004600 comp rx=0 tx=0).stop 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 shutdown_connections 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f7ac406dc10 0x7f7ac40700c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7adc101140 0x7f7adc112120 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 --2- 172.21.15.153:0/4013129913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7adc0fb110 0x7f7adc111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 >> 172.21.15.153:0/4013129913 conn(0x7f7adc0710b0 msgr2=0x7f7adc0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 shutdown_connections 2024-04-05T03:29:21.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.404+0000 7f7ae299d640 1 -- 172.21.15.153:0/4013129913 wait complete. 2024-04-05T03:29:21.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 -- 172.21.15.153:0/3524517879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 msgr2=0x7f36c4102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 --2- 172.21.15.153:0/3524517879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c4102580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f36b40048d0 tx=0x7f36b4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 -- 172.21.15.153:0/3524517879 shutdown_connections 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 --2- 172.21.15.153:0/3524517879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 0x7f36c40fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 --2- 172.21.15.153:0/3524517879 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c4102580 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 -- 172.21.15.153:0/3524517879 >> 172.21.15.153:0/3524517879 conn(0x7f36c40f7dc0 msgr2=0x7f36c40fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 -- 172.21.15.153:0/3524517879 shutdown_connections 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.523+0000 7f36c902e640 1 -- 172.21.15.153:0/3524517879 wait complete. 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 Processor -- start 2024-04-05T03:29:21.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 -- start start 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 0x7f36c406fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c40716b0 con 0x7f36c41021b0 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c902e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4070250 con 0x7f36c4102b50 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c2d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41728/0 (socket says 172.21.15.153:41728) 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36c2d9d640 1 -- 172.21.15.153:0/4027084707 learned_addr learned my addr 172.21.15.153:0/4027084707 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.524+0000 7f36ba59c640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 0x7f36c406fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c2d9d640 1 -- 172.21.15.153:0/4027084707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 msgr2=0x7f36c406fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c2d9d640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 0x7f36c406fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c2d9d640 1 -- 172.21.15.153:0/4027084707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4005040 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c2d9d640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004930 tx=0x7f36b4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b40094c0 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c40704d0 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36b4032e50 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b40433b0 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.525+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c41193b0 con 0x7f36c41021b0 2024-04-05T03:29:21.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.526+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c40fd920 con 0x7f36c41021b0 2024-04-05T03:29:21.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.527+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f36b402f070 con 0x7f36c41021b0 2024-04-05T03:29:21.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.527+0000 7f36bbfff640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 0x7f36b00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.527+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f36b40af310 con 0x7f36c41021b0 2024-04-05T03:29:21.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.527+0000 7f36ba59c640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 0x7f36b00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.528+0000 7f36ba59c640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 0x7f36b00700c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36c4070f00 tx=0x7f36ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.536+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f36b4078940 con 0x7f36c41021b0 2024-04-05T03:29:21.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.718+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f36c404bcc0 con 0x7f36c41021b0 2024-04-05T03:29:21.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.718+0000 7f36bbfff640 1 -- 172.21.15.153:0/4027084707 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f36b407c000 con 0x7f36c41021b0 2024-04-05T03:29:21.716 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:21.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 msgr2=0x7f36b00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 0x7f36b00700c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f36c4070f00 tx=0x7f36ac00b040 comp rx=0 tx=0).stop 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 msgr2=0x7f36c406f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004930 tx=0x7f36b4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 shutdown_connections 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f36b006dc10 0x7f36b00700c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36c4102b50 0x7f36c406fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 --2- 172.21.15.153:0/4027084707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36c41021b0 0x7f36c406f7d0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 >> 172.21.15.153:0/4027084707 conn(0x7f36c40f7dc0 msgr2=0x7f36c40f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 shutdown_connections 2024-04-05T03:29:21.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.720+0000 7f36c902e640 1 -- 172.21.15.153:0/4027084707 wait complete. 2024-04-05T03:29:21.728 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:21 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1591110086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:21.728 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:21 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4013129913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:21.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T03:29:21.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:21 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1591110086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:21.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:21 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4013129913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 -- 172.21.15.153:0/1243033234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 msgr2=0x7fde840fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 --2- 172.21.15.153:0/1243033234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde840fb9e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0048d0 tx=0x7fde6c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 -- 172.21.15.153:0/1243033234 shutdown_connections 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 --2- 172.21.15.153:0/1243033234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde840fb9e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 --2- 172.21.15.153:0/1243033234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde841014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 -- 172.21.15.153:0/1243033234 >> 172.21.15.153:0/1243033234 conn(0x7fde840708f0 msgr2=0x7fde84070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 -- 172.21.15.153:0/1243033234 shutdown_connections 2024-04-05T03:29:21.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.839+0000 7fde89225640 1 -- 172.21.15.153:0/1243033234 wait complete. 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 Processor -- start 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 -- start start 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde84112e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde84113950 con 0x7fde84101a90 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde89225640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8410cff0 con 0x7fde841010f0 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41746/0 (socket says 172.21.15.153:41746) 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 -- 172.21.15.153:0/1593587920 learned_addr learned my addr 172.21.15.153:0/1593587920 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde82d9d640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde84112e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 -- 172.21.15.153:0/1593587920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 msgr2=0x7fde84112e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde84112e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde7a59c640 1 -- 172.21.15.153:0/1593587920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c005040 con 0x7fde84101a90 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.840+0000 7fde82d9d640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde84112e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:29:21.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde7a59c640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fde6c02be70 tx=0x7fde6c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c0094c0 con 0x7fde84101a90 2024-04-05T03:29:21.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8410d270 con 0x7fde84101a90 2024-04-05T03:29:21.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde6c032e70 con 0x7fde84101a90 2024-04-05T03:29:21.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8410d730 con 0x7fde84101a90 2024-04-05T03:29:21.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.841+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6c0433b0 con 0x7fde84101a90 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.842+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde840fc9f0 con 0x7fde84101a90 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.842+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fde6c02f070 con 0x7fde84101a90 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.843+0000 7fde7bfff640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 0x7fde700700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.843+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fde6c0af360 con 0x7fde84101a90 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.843+0000 7fde82d9d640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 0x7fde700700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:21.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.843+0000 7fde82d9d640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 0x7fde700700d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde74009fd0 tx=0x7fde74009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:21.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:21.849+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fde6c0788e0 con 0x7fde84101a90 2024-04-05T03:29:22.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.028+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fde84108270 con 0x7fde84101a90 2024-04-05T03:29:22.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.029+0000 7fde7bfff640 1 -- 172.21.15.153:0/1593587920 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fde6c07bfa0 con 0x7fde84101a90 2024-04-05T03:29:22.027 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 msgr2=0x7fde700700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 0x7fde700700d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fde74009fd0 tx=0x7fde74009e50 comp rx=0 tx=0).stop 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 msgr2=0x7fde84113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fde6c02be70 tx=0x7fde6c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 shutdown_connections 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fde7006dc20 0x7fde700700d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde84101a90 0x7fde84113350 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 --2- 172.21.15.153:0/1593587920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde841010f0 0x7fde84112e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 >> 172.21.15.153:0/1593587920 conn(0x7fde840708f0 msgr2=0x7fde84071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 shutdown_connections 2024-04-05T03:29:22.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.031+0000 7fde89225640 1 -- 172.21.15.153:0/1593587920 wait complete. 2024-04-05T03:29:22.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T03:29:22.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 -- 172.21.15.153:0/3584999564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 msgr2=0x7f2f00101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 --2- 172.21.15.153:0/3584999564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00101510 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00048d0 tx=0x7f2ef0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 -- 172.21.15.153:0/3584999564 shutdown_connections 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 --2- 172.21.15.153:0/3584999564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 0x7f2f000fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 --2- 172.21.15.153:0/3584999564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00101510 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 -- 172.21.15.153:0/3584999564 >> 172.21.15.153:0/3584999564 conn(0x7f2f00071130 msgr2=0x7f2f00071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 -- 172.21.15.153:0/3584999564 shutdown_connections 2024-04-05T03:29:22.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.145+0000 7f2f0519d640 1 -- 172.21.15.153:0/3584999564 wait complete. 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 Processor -- start 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 -- start start 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 0x7f2f00111bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00112710 con 0x7f2f00101140 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.146+0000 7f2f0519d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0010bd60 con 0x7f2f000fb110 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41752/0 (socket says 172.21.15.153:41752) 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 -- 172.21.15.153:0/769782010 learned_addr learned my addr 172.21.15.153:0/769782010 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efed9d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 0x7f2f00111bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 -- 172.21.15.153:0/769782010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 msgr2=0x7f2f00111bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 0x7f2f00111bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 -- 172.21.15.153:0/769782010 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0005040 con 0x7f2f00101140 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2efe59c640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400eb70 tx=0x7f2ef400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4007c30 con 0x7f2f00101140 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ef400f040 con 0x7f2f00101140 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0010bfe0 con 0x7f2f00101140 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef40095b0 con 0x7f2f00101140 2024-04-05T03:29:22.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.147+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0010c500 con 0x7f2f00101140 2024-04-05T03:29:22.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.149+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f000fc800 con 0x7f2f00101140 2024-04-05T03:29:22.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.149+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f2ef4009710 con 0x7f2f00101140 2024-04-05T03:29:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.149+0000 7f2ee77fe640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 0x7f2ee80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.149+0000 7f2efed9d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 0x7f2ee80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.149+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2ef408d1a0 con 0x7f2f00101140 2024-04-05T03:29:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.150+0000 7f2efed9d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 0x7f2ee80700c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00043d0 tx=0x7f2ef0009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.155+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2ef4056850 con 0x7f2f00101140 2024-04-05T03:29:22.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.373+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f00108220 con 0x7f2f00101140 2024-04-05T03:29:22.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.374+0000 7f2ee77fe640 1 -- 172.21.15.153:0/769782010 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5257 (secure 0 0 0) 0x7f2ef4059f10 con 0x7f2f00101140 2024-04-05T03:29:22.372 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:22.372 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:32.727469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:active","state_seq":244,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:active","state_seq":239,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44325":{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44357":{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34388,"qdb_cluster":[34388,34420]},"id":1}]} 2024-04-05T03:29:22.372 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 42 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.375+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 msgr2=0x7f2ee80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 0x7f2ee80700c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00043d0 tx=0x7f2ef0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 msgr2=0x7f2f00112110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400eb70 tx=0x7f2ef400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 shutdown_connections 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f2ee806dc10 0x7f2ee80700c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f00101140 0x7f2f00112110 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 --2- 172.21.15.153:0/769782010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f000fb110 0x7f2f00111bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 >> 172.21.15.153:0/769782010 conn(0x7f2f00071130 msgr2=0x7f2f000f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 shutdown_connections 2024-04-05T03:29:22.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.376+0000 7f2f0519d640 1 -- 172.21.15.153:0/769782010 wait complete. 2024-04-05T03:29:22.385 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 21, 'max_mds': 2, 'flags': 50} 2024-04-05T03:29:22.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 -- 172.21.15.153:0/1555550178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2064101140 msgr2=0x7f2064101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 --2- 172.21.15.153:0/1555550178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2064101140 0x7f2064101510 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f20600048d0 tx=0x7f2060009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 -- 172.21.15.153:0/1555550178 shutdown_connections 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 --2- 172.21.15.153:0/1555550178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20640fb110 0x7f20640fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 --2- 172.21.15.153:0/1555550178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2064101140 0x7f2064101510 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 -- 172.21.15.153:0/1555550178 >> 172.21.15.153:0/1555550178 conn(0x7f20640710b0 msgr2=0x7f20640714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 -- 172.21.15.153:0/1555550178 shutdown_connections 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.491+0000 7f206b55d640 1 -- 172.21.15.153:0/1555550178 wait complete. 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 Processor -- start 2024-04-05T03:29:22.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 -- start start 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2064101140 0x7f206410e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2064065330 con 0x7f20640fb110 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f206b55d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206410e910 con 0x7f2064101140 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f20692f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f20692f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41766/0 (socket says 172.21.15.153:41766) 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.492+0000 7f20692f9640 1 -- 172.21.15.153:0/2142242782 learned_addr learned my addr 172.21.15.153:0/2142242782 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f2068af8640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2064101140 0x7f206410e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f20692f9640 1 -- 172.21.15.153:0/2142242782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2064101140 msgr2=0x7f206410e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f20692f9640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2064101140 0x7f206410e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f20692f9640 1 -- 172.21.15.153:0/2142242782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2060005040 con 0x7f20640fb110 2024-04-05T03:29:22.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f20692f9640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f20600041d0 tx=0x7f2060009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20600092e0 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206410eb90 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2060009440 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2060043640 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.493+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2064117120 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.494+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20640fc800 con 0x7f20640fb110 2024-04-05T03:29:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.495+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f206003b9f0 con 0x7f20640fb110 2024-04-05T03:29:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.495+0000 7f2059ffb640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 0x7f20500700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.496+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f20600b0350 con 0x7f20640fb110 2024-04-05T03:29:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.496+0000 7f2068af8640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 0x7f20500700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.496+0000 7f2068af8640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 0x7f20500700c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2064064b80 tx=0x7f2054009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.503+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2060079950 con 0x7f20640fb110 2024-04-05T03:29:22.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.722+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f206404bcc0 con 0x7f20640fb110 2024-04-05T03:29:22.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.723+0000 7f2059ffb640 1 -- 172.21.15.153:0/2142242782 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v42) v1 ==== 107+0+4898 (secure 0 0 0) 0x7f206007d010 con 0x7f20640fb110 2024-04-05T03:29:22.721 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:22.721 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:01:52.770007+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14518":{"gid":14518,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/1440749410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1440749410},{"type":"v1","addr":"172.21.15.153:6827","nonce":1440749410}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.203:6827/215598905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":215598905},{"type":"v1","addr":"172.21.15.203:6827","nonce":215598905}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:22.721 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 22 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 msgr2=0x7f20500700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 0x7f20500700c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2064064b80 tx=0x7f2054009040 comp rx=0 tx=0).stop 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 msgr2=0x7f206410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f20600041d0 tx=0x7f2060009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 shutdown_connections 2024-04-05T03:29:22.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f205006dc10 0x7f20500700c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2064101140 0x7f206410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 --2- 172.21.15.153:0/2142242782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20640fb110 0x7f206410de60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 >> 172.21.15.153:0/2142242782 conn(0x7f20640710b0 msgr2=0x7f20640f9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 shutdown_connections 2024-04-05T03:29:22.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.726+0000 7f206b55d640 1 -- 172.21.15.153:0/2142242782 wait complete. 2024-04-05T03:29:22.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:22 smithi153 ceph-mon[106102]: pgmap v712: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:22.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4027084707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:22.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1593587920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:22.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:22 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/769782010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:29:22.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-05T03:29:22.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:22 smithi203 ceph-mon[95988]: pgmap v712: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:22.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4027084707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:22.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1593587920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:22.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:22 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/769782010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- 172.21.15.153:0/226356716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 msgr2=0x7f508c102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 --2- 172.21.15.153:0/226356716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c102590 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f50880048d0 tx=0x7f5088009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- 172.21.15.153:0/226356716 shutdown_connections 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 --2- 172.21.15.153:0/226356716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 0x7f508c0647e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 --2- 172.21.15.153:0/226356716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c102590 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- 172.21.15.153:0/226356716 >> 172.21.15.153:0/226356716 conn(0x7f508c071120 msgr2=0x7f508c071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- 172.21.15.153:0/226356716 shutdown_connections 2024-04-05T03:29:22.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- 172.21.15.153:0/226356716 wait complete. 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 Processor -- start 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 -- start start 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 0x7f508c0fb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.858+0000 7f509335d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f509335d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0fd0b0 con 0x7f508c1021c0 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f509335d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0fbc50 con 0x7f508c064370 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41776/0 (socket says 172.21.15.153:41776) 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 -- 172.21.15.153:0/472900438 learned_addr learned my addr 172.21.15.153:0/472900438 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50910f9640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 0x7f508c0fb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 -- 172.21.15.153:0/472900438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 msgr2=0x7f508c0fb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 0x7f508c0fb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 -- 172.21.15.153:0/472900438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088005040 con 0x7f508c1021c0 2024-04-05T03:29:22.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f50908f8640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f507c002a50 tx=0x7f507c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507c007c30 con 0x7f508c1021c0 2024-04-05T03:29:22.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f507c00f040 con 0x7f508c1021c0 2024-04-05T03:29:22.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507c00db60 con 0x7f508c1021c0 2024-04-05T03:29:22.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.859+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c0fbf30 con 0x7f508c1021c0 2024-04-05T03:29:22.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.860+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c1193e0 con 0x7f508c1021c0 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.861+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508c0fd8b0 con 0x7f508c1021c0 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.861+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f507c00dcc0 con 0x7f508c1021c0 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.862+0000 7f5081ffb640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 0x7f50780700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.862+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f507c08d400 con 0x7f508c1021c0 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.862+0000 7f50910f9640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 0x7f50780700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:22.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.862+0000 7f50910f9640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 0x7f50780700d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50880048a0 tx=0x7f5088009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:22.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:22.871+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f507c056a00 con 0x7f508c1021c0 2024-04-05T03:29:23.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.238+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f508c04bcc0 con 0x7f508c1021c0 2024-04-05T03:29:23.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.239+0000 7f5081ffb640 1 -- 172.21.15.153:0/472900438 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v42) v1 ==== 107+0+4899 (secure 0 0 0) 0x7f507c05a0c0 con 0x7f508c1021c0 2024-04-05T03:29:23.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:23.237 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:03:29.112774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14518":{"gid":14518,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/1440749410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1440749410},{"type":"v1","addr":"172.21.15.153:6827","nonce":1440749410}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.203:6827/215598905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":215598905},{"type":"v1","addr":"172.21.15.203:6827","nonce":215598905}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:23.237 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 23 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.241+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 msgr2=0x7f50780700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.241+0000 7f509335d640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 0x7f50780700d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f50880048a0 tx=0x7f5088009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.241+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 msgr2=0x7f508c0fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f507c002a50 tx=0x7f507c004600 comp rx=0 tx=0).stop 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 shutdown_connections 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f507806dc20 0x7f50780700d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f508c1021c0 0x7f508c0fb710 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 --2- 172.21.15.153:0/472900438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f508c064370 0x7f508c0fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 >> 172.21.15.153:0/472900438 conn(0x7f508c071120 msgr2=0x7f508c0fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:23.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 shutdown_connections 2024-04-05T03:29:23.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.242+0000 7f509335d640 1 -- 172.21.15.153:0/472900438 wait complete. 2024-04-05T03:29:23.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 -- 172.21.15.153:0/1010850606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a0102240 msgr2=0x7f10a0102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/1010850606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a0102240 0x7f10a0102610 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f10940048d0 tx=0x7f1094009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 -- 172.21.15.153:0/1010850606 shutdown_connections 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/1010850606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a00643a0 0x7f10a00647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/1010850606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a0102240 0x7f10a0102610 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 -- 172.21.15.153:0/1010850606 >> 172.21.15.153:0/1010850606 conn(0x7f10a00710b0 msgr2=0x7f10a00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 -- 172.21.15.153:0/1010850606 shutdown_connections 2024-04-05T03:29:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.369+0000 7f10a6f9d640 1 -- 172.21.15.153:0/1010850606 wait complete. 2024-04-05T03:29:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.370+0000 7f10a6f9d640 1 Processor -- start 2024-04-05T03:29:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.370+0000 7f10a6f9d640 1 -- start start 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 0x7f10a0113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0113810 con 0x7f10a00643a0 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a010ce60 con 0x7f10a0102240 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41798/0 (socket says 172.21.15.153:41798) 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 -- 172.21.15.153:0/2752260931 learned_addr learned my addr 172.21.15.153:0/2752260931 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f109ffff640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 0x7f10a0113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 -- 172.21.15.153:0/2752260931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 msgr2=0x7f10a0113210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 0x7f10a0113210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 -- 172.21.15.153:0/2752260931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1094005040 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f109ffff640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 0x7f10a0113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a4d39640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1094004930 tx=0x7f1094009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10940094c0 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a010d0e0 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1094032e50 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10940433b0 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.371+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a010d5a0 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.372+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a00fd940 con 0x7f10a00643a0 2024-04-05T03:29:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.373+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f109402f070 con 0x7f10a00643a0 2024-04-05T03:29:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.374+0000 7f109d7fa640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 0x7f10900700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.374+0000 7f109ffff640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 0x7f10900700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.374+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f10940af410 con 0x7f10a00643a0 2024-04-05T03:29:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.374+0000 7f109ffff640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 0x7f10900700c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f108c007fd0 tx=0x7f108c009e90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.382+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1094078990 con 0x7f10a00643a0 2024-04-05T03:29:23.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.612+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f10a004bcc0 con 0x7f10a00643a0 2024-04-05T03:29:23.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.613+0000 7f109d7fa640 1 -- 172.21.15.153:0/2752260931 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v42) v1 ==== 107+0+4898 (secure 0 0 0) 0x7f109407c050 con 0x7f10a00643a0 2024-04-05T03:29:23.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:23.611 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:06:07.819154+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14518":{"gid":14518,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/1440749410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1440749410},{"type":"v1","addr":"172.21.15.153:6827","nonce":1440749410}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.203:6827/215598905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":215598905},{"type":"v1","addr":"172.21.15.203:6827","nonce":215598905}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 24 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 msgr2=0x7f10900700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 0x7f10900700c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f108c007fd0 tx=0x7f108c009e90 comp rx=0 tx=0).stop 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 msgr2=0x7f10a0112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f1094004930 tx=0x7f1094009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 shutdown_connections 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f109006dc10 0x7f10900700c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10a0102240 0x7f10a0113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 --2- 172.21.15.153:0/2752260931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10a00643a0 0x7f10a0112cd0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 >> 172.21.15.153:0/2752260931 conn(0x7f10a00710b0 msgr2=0x7f10a00f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 shutdown_connections 2024-04-05T03:29:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.615+0000 7f10a6f9d640 1 -- 172.21.15.153:0/2752260931 wait complete. 2024-04-05T03:29:23.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/2082072361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 msgr2=0x7fe9ec100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/2082072361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec100c90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0048d0 tx=0x7fe9dc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/2082072361 shutdown_connections 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/2082072361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 0x7fe9ec0fad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/2082072361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec100c90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/2082072361 >> 172.21.15.153:0/2082072361 conn(0x7fe9ec0708f0 msgr2=0x7fe9ec070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:23.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/2082072361 shutdown_connections 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.726+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/2082072361 wait complete. 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 Processor -- start 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 -- start start 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 0x7fe9ec10df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec1065a0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9f1d9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec10e980 con 0x7fe9ec0fa8d0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41828/0 (socket says 172.21.15.153:41828) 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 -- 172.21.15.153:0/1407821328 learned_addr learned my addr 172.21.15.153:0/1407821328 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9eb7fe640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 0x7fe9ec10df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 -- 172.21.15.153:0/1407821328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 msgr2=0x7fe9ec10df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 0x7fe9ec10df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.727+0000 7fe9e3fff640 1 -- 172.21.15.153:0/1407821328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc005040 con 0x7fe9ec1008c0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9e3fff640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400eb70 tx=0x7fe9d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:23.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d4007c30 con 0x7fe9ec1008c0 2024-04-05T03:29:23.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d400f040 con 0x7fe9ec1008c0 2024-04-05T03:29:23.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec10ec00 con 0x7fe9ec1008c0 2024-04-05T03:29:23.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d40095b0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.728+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec1171c0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.729+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec0fbfe0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.730+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe9d4009710 con 0x7fe9ec1008c0 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.730+0000 7fe9e8ff9640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 0x7fe9d80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.730+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe9d408d6d0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.730+0000 7fe9eb7fe640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 0x7fe9d80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:23.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.731+0000 7fe9eb7fe640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 0x7fe9d80700c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc003080 tx=0x7fe9dc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:23.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.735+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe9d4056cd0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:23 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2142242782' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-05T03:29:23.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:23 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/472900438' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-05T03:29:23.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:23 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2142242782' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-05T03:29:23.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:23 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/472900438' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-05T03:29:23.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.980+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fe9ec04bcc0 con 0x7fe9ec1008c0 2024-04-05T03:29:23.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.980+0000 7fe9e8ff9640 1 -- 172.21.15.153:0/1407821328 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v42) v1 ==== 107+0+4897 (secure 0 0 0) 0x7fe9d405a390 con 0x7fe9ec1008c0 2024-04-05T03:29:23.979 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:23.979 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:06:24.861523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14518":{"gid":14518,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6827/1440749410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1440749410},{"type":"v1","addr":"172.21.15.153:6827","nonce":1440749410}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24271":{"gid":24271,"name":"cephfs.smithi203.hgbifk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.203:6827/215598905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":215598905},{"type":"v1","addr":"172.21.15.203:6827","nonce":215598905}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:23.979 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 25 2024-04-05T03:29:23.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 msgr2=0x7fe9d80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 0x7fe9d80700c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc003080 tx=0x7fe9dc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 msgr2=0x7fe9ec10e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:23.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400eb70 tx=0x7fe9d400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:23.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 shutdown_connections 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe9d806dc10 0x7fe9d80700c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe9ec1008c0 0x7fe9ec10e440 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 --2- 172.21.15.153:0/1407821328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9ec0fa8d0 0x7fe9ec10df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 >> 172.21.15.153:0/1407821328 conn(0x7fe9ec0708f0 msgr2=0x7fe9ec0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 shutdown_connections 2024-04-05T03:29:23.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:23.983+0000 7fe9f1d9d640 1 -- 172.21.15.153:0/1407821328 wait complete. 2024-04-05T03:29:23.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 -- 172.21.15.153:0/3955560346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc0fb110 msgr2=0x7ff5dc0fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3955560346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc0fb560 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00048d0 tx=0x7ff5d0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 -- 172.21.15.153:0/3955560346 shutdown_connections 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3955560346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc0fb560 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3955560346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc101140 0x7ff5dc101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 -- 172.21.15.153:0/3955560346 >> 172.21.15.153:0/3955560346 conn(0x7ff5dc0710b0 msgr2=0x7ff5dc0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 -- 172.21.15.153:0/3955560346 shutdown_connections 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.109+0000 7ff5e219d640 1 -- 172.21.15.153:0/3955560346 wait complete. 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 Processor -- start 2024-04-05T03:29:24.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 -- start start 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc115a00 con 0x7ff5dc101140 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5e219d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc115b70 con 0x7ff5dc0fb110 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5daffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5daffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41850/0 (socket says 172.21.15.153:41850) 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.110+0000 7ff5daffd640 1 -- 172.21.15.153:0/3605208059 learned_addr learned my addr 172.21.15.153:0/3605208059 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5db7fe640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5daffd640 1 -- 172.21.15.153:0/3605208059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc0fb110 msgr2=0x7ff5dc114ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5daffd640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc114ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5daffd640 1 -- 172.21.15.153:0/3605208059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0005040 con 0x7ff5dc101140 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5daffd640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00041d0 tx=0x7ff5d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d0030870 con 0x7ff5dc101140 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5d00092e0 con 0x7ff5dc101140 2024-04-05T03:29:24.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc10f1a0 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d0043400 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.111+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc10f6c0 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.112+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc0fc800 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.113+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff5d003b420 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.113+0000 7ff5c3fff640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 0x7ff5c40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.113+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff5d00b0380 con 0x7ff5dc101140 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.113+0000 7ff5db7fe640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 0x7ff5c40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.113+0000 7ff5db7fe640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 0x7ff5c40700c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009fd0 tx=0x7ff5cc009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.118+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff5d0079980 con 0x7ff5dc101140 2024-04-05T03:29:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.336+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7ff5dc108200 con 0x7ff5dc101140 2024-04-05T03:29:24.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.336+0000 7ff5c3fff640 1 -- 172.21.15.153:0/3605208059 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v42) v1 ==== 107+0+3337 (secure 0 0 0) 0x7ff5d007d040 con 0x7ff5dc101140 2024-04-05T03:29:24.335 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:24.335 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.421619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14506,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14506":{"gid":14506,"name":"cephfs.smithi153.fmpmvd","rank":0,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/1509596739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1509596739},{"type":"v1","addr":"172.21.15.153:6829","nonce":1509596739}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi203.xpteyc","rank":1,"incarnation":8,"state":"up:active","state_seq":4,"addr":"172.21.15.203:6825/1771218289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1771218289},{"type":"v1","addr":"172.21.15.203:6825","nonce":1771218289}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14506,"qdb_cluster":[14506,24261]},"id":1}]} 2024-04-05T03:29:24.335 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 26 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 msgr2=0x7ff5c40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 0x7ff5c40700c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009fd0 tx=0x7ff5cc009e50 comp rx=0 tx=0).stop 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 msgr2=0x7ff5dc115400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00041d0 tx=0x7ff5d0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 shutdown_connections 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff5c406dc10 0x7ff5c40700c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5dc101140 0x7ff5dc115400 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 --2- 172.21.15.153:0/3605208059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc0fb110 0x7ff5dc114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 >> 172.21.15.153:0/3605208059 conn(0x7ff5dc0710b0 msgr2=0x7ff5dc064370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 shutdown_connections 2024-04-05T03:29:24.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.339+0000 7ff5e219d640 1 -- 172.21.15.153:0/3605208059 wait complete. 2024-04-05T03:29:24.348 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-05T03:29:24.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-05T03:29:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2611070750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f40643b0 msgr2=0x7fe0f4064820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2611070750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f40643b0 0x7fe0f4064820 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40048d0 tx=0x7fe0e4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2611070750 shutdown_connections 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2611070750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f40643b0 0x7fe0f4064820 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2611070750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f4102020 0x7fe0f41023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2611070750 >> 172.21.15.153:0/2611070750 conn(0x7fe0f4071e80 msgr2=0x7fe0f4072280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2611070750 shutdown_connections 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.451+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2611070750 wait complete. 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 Processor -- start 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 -- start start 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f40643b0 0x7fe0f4112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f41136a0 con 0x7fe0f4102020 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0fa99e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f410ccf0 con 0x7fe0f40643b0 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41882/0 (socket says 172.21.15.153:41882) 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 -- 172.21.15.153:0/2278396593 learned_addr learned my addr 172.21.15.153:0/2278396593 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f999c640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f40643b0 0x7fe0f4112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 -- 172.21.15.153:0/2278396593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f40643b0 msgr2=0x7fe0f4112b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f40643b0 0x7fe0f4112b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.452+0000 7fe0f919b640 1 -- 172.21.15.153:0/2278396593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e4005040 con 0x7fe0f4102020 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0f919b640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4034040 tx=0x7fe0e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e40092b0 con 0x7fe0f4102020 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0e403b400 con 0x7fe0f4102020 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f410cf70 con 0x7fe0f4102020 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e40435d0 con 0x7fe0f4102020 2024-04-05T03:29:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.453+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f410d430 con 0x7fe0f4102020 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.454+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f40fd790 con 0x7fe0f4102020 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.454+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fe0e402f070 con 0x7fe0f4102020 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.455+0000 7fe0ea7fc640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 0x7fe0e00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.455+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe0e40af310 con 0x7fe0f4102020 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.455+0000 7fe0f999c640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 0x7fe0e00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.455+0000 7fe0f999c640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 0x7fe0e00700c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0004a70 tx=0x7fe0f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.459+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe0e4078890 con 0x7fe0f4102020 2024-04-05T03:29:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.730+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fe0f4109290 con 0x7fe0f4102020 2024-04-05T03:29:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.730+0000 7fe0ea7fc640 1 -- 172.21.15.153:0/2278396593 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v42) v1 ==== 107+0+3298 (secure 0 0 0) 0x7fe0e403fd00 con 0x7fe0f4102020 2024-04-05T03:29:24.728 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:24.729 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/646956490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":646956490},{"type":"v1","addr":"172.21.15.153:6827","nonce":646956490}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 27 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 msgr2=0x7fe0e00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 0x7fe0e00700c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0004a70 tx=0x7fe0f000b040 comp rx=0 tx=0).stop 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 msgr2=0x7fe0f41130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4034040 tx=0x7fe0e4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 shutdown_connections 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.732+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fe0e006dc10 0x7fe0e00700c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.733+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0f4102020 0x7fe0f41130a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.733+0000 7fe0fa99e640 1 --2- 172.21.15.153:0/2278396593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0f40643b0 0x7fe0f4112b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.733+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 >> 172.21.15.153:0/2278396593 conn(0x7fe0f4071e80 msgr2=0x7fe0f40fa120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.733+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 shutdown_connections 2024-04-05T03:29:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.733+0000 7fe0fa99e640 1 -- 172.21.15.153:0/2278396593 wait complete. 2024-04-05T03:29:24.740 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:24 smithi153 ceph-mon[106102]: pgmap v713: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:24.740 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:24 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2752260931' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-05T03:29:24.740 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:24 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1407821328' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-05T03:29:24.740 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:24 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3605208059' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-05T03:29:24.741 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:24.742 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-05T03:29:24.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-05T03:29:24.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:24 smithi203 ceph-mon[95988]: pgmap v713: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:24.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:24 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2752260931' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-05T03:29:24.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:24 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1407821328' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-05T03:29:24.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:24 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3605208059' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-05T03:29:24.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 -- 172.21.15.153:0/2906227745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 msgr2=0x7efee4102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 --2- 172.21.15.153:0/2906227745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4102580 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7efed40048d0 tx=0x7efed4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 -- 172.21.15.153:0/2906227745 shutdown_connections 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 --2- 172.21.15.153:0/2906227745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 0x7efee40fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 --2- 172.21.15.153:0/2906227745 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4102580 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 -- 172.21.15.153:0/2906227745 >> 172.21.15.153:0/2906227745 conn(0x7efee40f7dc0 msgr2=0x7efee40fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 -- 172.21.15.153:0/2906227745 shutdown_connections 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.847+0000 7efeea6ae640 1 -- 172.21.15.153:0/2906227745 wait complete. 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 Processor -- start 2024-04-05T03:29:24.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 -- start start 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 0x7efee4113220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee4113820 con 0x7efee41021b0 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efeea6ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee410ce70 con 0x7efee4102b50 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41900/0 (socket says 172.21.15.153:41900) 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 -- 172.21.15.153:0/261261198 learned_addr learned my addr 172.21.15.153:0/261261198 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:24.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee37fe640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 0x7efee4113220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 -- 172.21.15.153:0/261261198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 msgr2=0x7efee4113220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 0x7efee4113220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.848+0000 7efee3fff640 1 -- 172.21.15.153:0/261261198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4005040 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efee3fff640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7efed4004930 tx=0x7efed4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efed40094c0 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee410d0f0 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efed4032e50 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efed40433b0 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.849+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee410d5b0 con 0x7efee41021b0 2024-04-05T03:29:24.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.850+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee40fd920 con 0x7efee41021b0 2024-04-05T03:29:24.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.851+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efed402f070 con 0x7efee41021b0 2024-04-05T03:29:24.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.851+0000 7efee0ff9640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 0x7efecc0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:24.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.851+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efed40af5a0 con 0x7efee41021b0 2024-04-05T03:29:24.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.851+0000 7efee37fe640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 0x7efecc0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:24.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.852+0000 7efee37fe640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 0x7efecc0700c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efee410e5a0 tx=0x7efed8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:24.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:24.856+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efed4078b20 con 0x7efee41021b0 2024-04-05T03:29:25.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.105+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7efee410e350 con 0x7efee41021b0 2024-04-05T03:29:25.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.106+0000 7efee0ff9640 1 -- 172.21.15.153:0/261261198 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v42) v1 ==== 107+0+4862 (secure 0 0 0) 0x7efed407c1e0 con 0x7efee41021b0 2024-04-05T03:29:25.104 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:25.104 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/1079014242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1079014242},{"type":"v1","addr":"172.21.15.153:6829","nonce":1079014242}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44291,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/646956490","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":646956490},{"type":"v1","addr":"172.21.15.153:6827","nonce":646956490}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44297,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/4013577824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":4013577824},{"type":"v1","addr":"172.21.15.203:6825","nonce":4013577824}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:25.104 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 28 2024-04-05T03:29:25.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 msgr2=0x7efecc0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 0x7efecc0700c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efee410e5a0 tx=0x7efed8009040 comp rx=0 tx=0).stop 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 msgr2=0x7efee4112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7efed4004930 tx=0x7efed4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 shutdown_connections 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efecc06dc10 0x7efecc0700c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efee4102b50 0x7efee4113220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 --2- 172.21.15.153:0/261261198 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efee41021b0 0x7efee4112ce0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 >> 172.21.15.153:0/261261198 conn(0x7efee40f7dc0 msgr2=0x7efee40f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 shutdown_connections 2024-04-05T03:29:25.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.108+0000 7efeea6ae640 1 -- 172.21.15.153:0/261261198 wait complete. 2024-04-05T03:29:25.117 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:25.117 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-05T03:29:25.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 -- 172.21.15.153:0/822594906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac102240 msgr2=0x7fb4ac102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 --2- 172.21.15.153:0/822594906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac102240 0x7fb4ac102610 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80048d0 tx=0x7fb4a8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 -- 172.21.15.153:0/822594906 shutdown_connections 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 --2- 172.21.15.153:0/822594906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 --2- 172.21.15.153:0/822594906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac102240 0x7fb4ac102610 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.221+0000 7fb4b355d640 1 -- 172.21.15.153:0/822594906 >> 172.21.15.153:0/822594906 conn(0x7fb4ac0710b0 msgr2=0x7fb4ac0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.222+0000 7fb4b355d640 1 -- 172.21.15.153:0/822594906 shutdown_connections 2024-04-05T03:29:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.222+0000 7fb4b355d640 1 -- 172.21.15.153:0/822594906 wait complete. 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.222+0000 7fb4b355d640 1 Processor -- start 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.222+0000 7fb4b355d640 1 -- start start 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.222+0000 7fb4b355d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b355d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac102240 0x7fb4ac0fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b355d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0fd0c0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b355d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0fbc60 con 0x7fb4ac102240 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41926/0 (socket says 172.21.15.153:41926) 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 -- 172.21.15.153:0/3163055365 learned_addr learned my addr 172.21.15.153:0/3163055365 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b0af8640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac102240 0x7fb4ac0fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 -- 172.21.15.153:0/3163055365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac102240 msgr2=0x7fb4ac0fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac102240 0x7fb4ac0fb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 -- 172.21.15.153:0/3163055365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8005040 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b12f9640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8004930 tx=0x7fb4a8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a80094c0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac0fbee0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4a8032e50 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.223+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a80433b0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.224+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac1193a0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.225+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac0fd940 con 0x7fb4ac0643a0 2024-04-05T03:29:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.225+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fb4a802f070 con 0x7fb4ac0643a0 2024-04-05T03:29:25.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.225+0000 7fb4a1ffb640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 0x7fb4980700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.225+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb4a80af560 con 0x7fb4ac0643a0 2024-04-05T03:29:25.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.226+0000 7fb4b0af8640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 0x7fb4980700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.226+0000 7fb4b0af8640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 0x7fb4980700c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0fc910 tx=0x7fb49c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.230+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb4a8078ae0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.453+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fb4ac119530 con 0x7fb4ac0643a0 2024-04-05T03:29:25.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.453+0000 7fb4a1ffb640 1 -- 172.21.15.153:0/3163055365 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v42) v1 ==== 107+0+4083 (secure 0 0 0) 0x7fb4a807c1a0 con 0x7fb4ac0643a0 2024-04-05T03:29:25.451 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:25.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/1079014242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1079014242},{"type":"v1","addr":"172.21.15.153:6829","nonce":1079014242}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44297,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/4013577824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":4013577824},{"type":"v1","addr":"172.21.15.203:6825","nonce":4013577824}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:25.452 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 29 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 msgr2=0x7fb4980700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 0x7fb4980700c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0fc910 tx=0x7fb49c009040 comp rx=0 tx=0).stop 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 msgr2=0x7fb4ac0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8004930 tx=0x7fb4a8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 shutdown_connections 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fb49806dc10 0x7fb4980700c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4ac102240 0x7fb4ac0fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 --2- 172.21.15.153:0/3163055365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4ac0643a0 0x7fb4ac0fb1e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 >> 172.21.15.153:0/3163055365 conn(0x7fb4ac0710b0 msgr2=0x7fb4ac0f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 shutdown_connections 2024-04-05T03:29:25.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.456+0000 7fb4b355d640 1 -- 172.21.15.153:0/3163055365 wait complete. 2024-04-05T03:29:25.465 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:25.465 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-05T03:29:25.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-05T03:29:25.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 -- 172.21.15.153:0/2555217277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 msgr2=0x7f91380647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 --2- 172.21.15.153:0/2555217277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f91380647d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f91280048d0 tx=0x7f9128009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 -- 172.21.15.153:0/2555217277 shutdown_connections 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 --2- 172.21.15.153:0/2555217277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f91380647d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 --2- 172.21.15.153:0/2555217277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 0x7f9138102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.581+0000 7f913fe6e640 1 -- 172.21.15.153:0/2555217277 >> 172.21.15.153:0/2555217277 conn(0x7f91380710b0 msgr2=0x7f91380714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 -- 172.21.15.153:0/2555217277 shutdown_connections 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 -- 172.21.15.153:0/2555217277 wait complete. 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 Processor -- start 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 -- start start 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 0x7f9138113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.582+0000 7f913fe6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9138113810 con 0x7f91380643a0 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913fe6e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913810ce60 con 0x7f9138102240 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41948/0 (socket says 172.21.15.153:41948) 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 -- 172.21.15.153:0/713541241 learned_addr learned my addr 172.21.15.153:0/713541241 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913d409640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 0x7f9138113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 -- 172.21.15.153:0/713541241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 msgr2=0x7f9138113210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 0x7f9138113210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 -- 172.21.15.153:0/713541241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128005040 con 0x7f91380643a0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913dc0a640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f913400eb70 tx=0x7f913400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9134007c30 con 0x7f91380643a0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913810d0e0 con 0x7f91380643a0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f913400f040 con 0x7f91380643a0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91340095b0 con 0x7f91380643a0 2024-04-05T03:29:25.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.583+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913810d5d0 con 0x7f91380643a0 2024-04-05T03:29:25.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.584+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91380fd940 con 0x7f91380643a0 2024-04-05T03:29:25.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.585+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f9134009710 con 0x7f91380643a0 2024-04-05T03:29:25.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.586+0000 7f912e7fc640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 0x7f91240700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.586+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f913408d740 con 0x7f91380643a0 2024-04-05T03:29:25.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.586+0000 7f913d409640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 0x7f91240700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.586+0000 7f913d409640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 0x7f91240700c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f91380fafe0 tx=0x7f9128009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.590+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9134056d40 con 0x7f91380643a0 2024-04-05T03:29:25.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:25 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2278396593' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-05T03:29:25.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:25 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/261261198' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-05T03:29:25.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:25 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3163055365' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-05T03:29:25.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.815+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f913804bd40 con 0x7f91380643a0 2024-04-05T03:29:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.816+0000 7f912e7fc640 1 -- 172.21.15.153:0/713541241 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v42) v1 ==== 107+0+4933 (secure 0 0 0) 0x7f913405a400 con 0x7f91380643a0 2024-04-05T03:29:25.814 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:25.814 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34384,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/1079014242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1079014242},{"type":"v1","addr":"172.21.15.153:6829","nonce":1079014242}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44297,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/4013577824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":4013577824},{"type":"v1","addr":"172.21.15.203:6825","nonce":4013577824}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:25.814 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 30 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 msgr2=0x7f91240700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 0x7f91240700c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f91380fafe0 tx=0x7f9128009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 msgr2=0x7f9138112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f913400eb70 tx=0x7f913400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 shutdown_connections 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f912406dc10 0x7f91240700c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9138102240 0x7f9138113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 --2- 172.21.15.153:0/713541241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91380643a0 0x7f9138112cd0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 >> 172.21.15.153:0/713541241 conn(0x7f91380710b0 msgr2=0x7f91380f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 shutdown_connections 2024-04-05T03:29:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.818+0000 7f913fe6e640 1 -- 172.21.15.153:0/713541241 wait complete. 2024-04-05T03:29:25.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:25 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2278396593' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-05T03:29:25.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:25 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/261261198' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-05T03:29:25.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:25 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3163055365' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-05T03:29:25.830 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:25.830 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-05T03:29:25.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-05T03:29:25.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 -- 172.21.15.153:0/510361070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102200 msgr2=0x7fdb5c1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/510361070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102200 0x7fdb5c1025d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0048d0 tx=0x7fdb4c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 -- 172.21.15.153:0/510361070 shutdown_connections 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/510361070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/510361070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102200 0x7fdb5c1025d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.932+0000 7fdb62b9d640 1 -- 172.21.15.153:0/510361070 >> 172.21.15.153:0/510361070 conn(0x7fdb5c0708f0 msgr2=0x7fdb5c070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 -- 172.21.15.153:0/510361070 shutdown_connections 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 -- 172.21.15.153:0/510361070 wait complete. 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 Processor -- start 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 -- start start 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102200 0x7fdb5c0fbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0fd990 con 0x7fdb5c102ba0 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.933+0000 7fdb62b9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0fc530 con 0x7fdb5c102200 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41962/0 (socket says 172.21.15.153:41962) 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 -- 172.21.15.153:0/1906702180 learned_addr learned my addr 172.21.15.153:0/1906702180 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:25.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb60939640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102200 0x7fdb5c0fbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 -- 172.21.15.153:0/1906702180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102200 msgr2=0x7fdb5c0fbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102200 0x7fdb5c0fbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 -- 172.21.15.153:0/1906702180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c005040 con 0x7fdb5c102ba0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb5bfff640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400eb40 tx=0x7fdb4400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb44007c30 con 0x7fdb5c102ba0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb4400f040 con 0x7fdb5c102ba0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0fc7b0 con 0x7fdb5c102ba0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb44009620 con 0x7fdb5c102ba0 2024-04-05T03:29:25.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.934+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c119500 con 0x7fdb5c102ba0 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.935+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0652a0 con 0x7fdb5c102ba0 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.936+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fdb44009780 con 0x7fdb5c102ba0 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.936+0000 7fdb597fa640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 0x7fdb480700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.936+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fdb4408d1f0 con 0x7fdb5c102ba0 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.936+0000 7fdb60939640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 0x7fdb480700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:25.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.937+0000 7fdb60939640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 0x7fdb480700c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c031870 tx=0x7fdb4c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:25.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:25.941+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdb440567f0 con 0x7fdb5c102ba0 2024-04-05T03:29:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.172+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fdb5c04bcc0 con 0x7fdb5c102ba0 2024-04-05T03:29:26.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.172+0000 7fdb597fa640 1 -- 172.21.15.153:0/1906702180 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v42) v1 ==== 107+0+4151 (secure 0 0 0) 0x7fdb5c04bcc0 con 0x7fdb5c102ba0 2024-04-05T03:29:26.170 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:26.171 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44297,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/4013577824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":4013577824},{"type":"v1","addr":"172.21.15.203:6825","nonce":4013577824}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:26.171 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 31 2024-04-05T03:29:26.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 msgr2=0x7fdb480700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 0x7fdb480700c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c031870 tx=0x7fdb4c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 msgr2=0x7fdb5c0fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400eb40 tx=0x7fdb4400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 shutdown_connections 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fdb4806dc10 0x7fdb480700c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb5c102ba0 0x7fdb5c0fbff0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 --2- 172.21.15.153:0/1906702180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb5c102200 0x7fdb5c0fbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.174+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 >> 172.21.15.153:0/1906702180 conn(0x7fdb5c0708f0 msgr2=0x7fdb5c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.175+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 shutdown_connections 2024-04-05T03:29:26.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.175+0000 7fdb62b9d640 1 -- 172.21.15.153:0/1906702180 wait complete. 2024-04-05T03:29:26.184 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:26.184 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-05T03:29:26.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 -- 172.21.15.153:0/2370299347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e8104d90 msgr2=0x7f55e80fe420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 --2- 172.21.15.153:0/2370299347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e8104d90 0x7f55e80fe420 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f55d80048d0 tx=0x7f55d8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 -- 172.21.15.153:0/2370299347 shutdown_connections 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 --2- 172.21.15.153:0/2370299347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e8104d90 0x7f55e80fe420 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 --2- 172.21.15.153:0/2370299347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e81043f0 0x7f55e81047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 -- 172.21.15.153:0/2370299347 >> 172.21.15.153:0/2370299347 conn(0x7f55e80f9fc0 msgr2=0x7f55e80fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 -- 172.21.15.153:0/2370299347 shutdown_connections 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.286+0000 7f55ef86e640 1 -- 172.21.15.153:0/2370299347 wait complete. 2024-04-05T03:29:26.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 Processor -- start 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 -- start start 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e8104d90 0x7f55e8115410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e8115a10 con 0x7f55e81043f0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ef86e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e810f060 con 0x7f55e8104d90 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41970/0 (socket says 172.21.15.153:41970) 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 -- 172.21.15.153:0/3080430975 learned_addr learned my addr 172.21.15.153:0/3080430975 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 -- 172.21.15.153:0/3080430975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e8104d90 msgr2=0x7f55e8115410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e8104d90 0x7f55e8115410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.287+0000 7f55ed60a640 1 -- 172.21.15.153:0/3080430975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d8005040 con 0x7f55e81043f0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ed60a640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55e4002a50 tx=0x7f55e4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e4007c30 con 0x7f55e81043f0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55e400f040 con 0x7f55e81043f0 2024-04-05T03:29:26.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55e400db50 con 0x7f55e81043f0 2024-04-05T03:29:26.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e810f340 con 0x7f55e81043f0 2024-04-05T03:29:26.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.288+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e810f860 con 0x7f55e81043f0 2024-04-05T03:29:26.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.289+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e80ffb60 con 0x7f55e81043f0 2024-04-05T03:29:26.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.290+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f55e400dcf0 con 0x7f55e81043f0 2024-04-05T03:29:26.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.290+0000 7f55ddffb640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 0x7f55d40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.290+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f55e408cdc0 con 0x7f55e81043f0 2024-04-05T03:29:26.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.290+0000 7f55ece09640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 0x7f55d40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:26.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.291+0000 7f55ece09640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 0x7f55d40700c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f55d80043d0 tx=0x7f55d8009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:26.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.296+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f55e40563c0 con 0x7f55e81043f0 2024-04-05T03:29:26.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.539+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f55e804bcc0 con 0x7f55e81043f0 2024-04-05T03:29:26.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.540+0000 7f55ddffb640 1 -- 172.21.15.153:0/3080430975 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v42) v1 ==== 107+0+5001 (secure 0 0 0) 0x7f55e4059a80 con 0x7f55e81043f0 2024-04-05T03:29:26.538 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:26.538 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44297,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/4013577824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":4013577824},{"type":"v1","addr":"172.21.15.203:6825","nonce":4013577824}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:26.539 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 32 2024-04-05T03:29:26.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 msgr2=0x7f55d40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 0x7f55d40700c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f55d80043d0 tx=0x7f55d8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 msgr2=0x7f55e8114ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55e4002a50 tx=0x7f55e4004600 comp rx=0 tx=0).stop 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 shutdown_connections 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f55d406dc10 0x7f55d40700c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e8104d90 0x7f55e8115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 --2- 172.21.15.153:0/3080430975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55e81043f0 0x7f55e8114ed0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 >> 172.21.15.153:0/3080430975 conn(0x7f55e80f9fc0 msgr2=0x7f55e80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 shutdown_connections 2024-04-05T03:29:26.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.542+0000 7f55ef86e640 1 -- 172.21.15.153:0/3080430975 wait complete. 2024-04-05T03:29:26.551 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:26.551 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-05T03:29:26.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 -- 172.21.15.153:0/220552539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0544102240 msgr2=0x7f0544102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 --2- 172.21.15.153:0/220552539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0544102240 0x7f0544102610 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f05340048d0 tx=0x7f0534009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 -- 172.21.15.153:0/220552539 shutdown_connections 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 --2- 172.21.15.153:0/220552539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05440643a0 0x7f05440647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 --2- 172.21.15.153:0/220552539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0544102240 0x7f0544102610 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 -- 172.21.15.153:0/220552539 >> 172.21.15.153:0/220552539 conn(0x7f05440710b0 msgr2=0x7f05440714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 -- 172.21.15.153:0/220552539 shutdown_connections 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.655+0000 7f054a19d640 1 -- 172.21.15.153:0/220552539 wait complete. 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 Processor -- start 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 -- start start 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0544102240 0x7f0544113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544113810 con 0x7f05440643a0 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f054a19d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054410ce60 con 0x7f0544102240 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f05437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f05437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41980/0 (socket says 172.21.15.153:41980) 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.656+0000 7f05437fe640 1 -- 172.21.15.153:0/1580493023 learned_addr learned my addr 172.21.15.153:0/1580493023 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f0542ffd640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0544102240 0x7f0544113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f05437fe640 1 -- 172.21.15.153:0/1580493023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0544102240 msgr2=0x7f0544113210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f05437fe640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0544102240 0x7f0544113210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:26.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f05437fe640 1 -- 172.21.15.153:0/1580493023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534005040 con 0x7f05440643a0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f05437fe640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0534004930 tx=0x7f0534009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05340094c0 con 0x7f05440643a0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054410d0e0 con 0x7f05440643a0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0534032e50 con 0x7f05440643a0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05340433b0 con 0x7f05440643a0 2024-04-05T03:29:26.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.657+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054410d5a0 con 0x7f05440643a0 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.658+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05440fd940 con 0x7f05440643a0 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.659+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f053402f070 con 0x7f05440643a0 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.659+0000 7f052bfff640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 0x7f052c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.659+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f05340af5a0 con 0x7f05440643a0 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.659+0000 7f0542ffd640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 0x7f052c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:26.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.660+0000 7f0542ffd640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 0x7f052c0700c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f054410e590 tx=0x7f0538009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:26.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:26.664+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0534078b20 con 0x7f05440643a0 2024-04-05T03:29:26.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:26 smithi203 ceph-mon[95988]: pgmap v714: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:26.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:26 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/713541241' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-05T03:29:26.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:26 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1906702180' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-05T03:29:26.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:26 smithi153 ceph-mon[106102]: pgmap v714: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:26.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:26 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/713541241' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-05T03:29:26.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:26 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1906702180' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-05T03:29:27.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.033+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f054404bcc0 con 0x7f05440643a0 2024-04-05T03:29:27.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.034+0000 7f052bfff640 1 -- 172.21.15.153:0/1580493023 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v42) v1 ==== 107+0+4219 (secure 0 0 0) 0x7f053407c1e0 con 0x7f05440643a0 2024-04-05T03:29:27.032 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:27.032 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:27.033 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 33 2024-04-05T03:29:27.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 msgr2=0x7f052c0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 0x7f052c0700c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f054410e590 tx=0x7f0538009040 comp rx=0 tx=0).stop 2024-04-05T03:29:27.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 msgr2=0x7f0544112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0534004930 tx=0x7f0534009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 shutdown_connections 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f052c06dc10 0x7f052c0700c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0544102240 0x7f0544113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 --2- 172.21.15.153:0/1580493023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05440643a0 0x7f0544112cd0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 >> 172.21.15.153:0/1580493023 conn(0x7f05440710b0 msgr2=0x7f05440f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 shutdown_connections 2024-04-05T03:29:27.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.036+0000 7f054a19d640 1 -- 172.21.15.153:0/1580493023 wait complete. 2024-04-05T03:29:27.045 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:27.045 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-05T03:29:27.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-05T03:29:27.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 -- 172.21.15.153:0/3642761478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf14102240 msgr2=0x7faf14102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 --2- 172.21.15.153:0/3642761478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf14102240 0x7faf14102610 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faf040048d0 tx=0x7faf04009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 -- 172.21.15.153:0/3642761478 shutdown_connections 2024-04-05T03:29:27.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 --2- 172.21.15.153:0/3642761478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf140643a0 0x7faf140647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 --2- 172.21.15.153:0/3642761478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf14102240 0x7faf14102610 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 -- 172.21.15.153:0/3642761478 >> 172.21.15.153:0/3642761478 conn(0x7faf140710b0 msgr2=0x7faf140714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 -- 172.21.15.153:0/3642761478 shutdown_connections 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.156+0000 7faf188ae640 1 -- 172.21.15.153:0/3642761478 wait complete. 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 Processor -- start 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 -- start start 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf14102240 0x7faf140fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140fd150 con 0x7faf140643a0 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf188ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf140fbc60 con 0x7faf14102240 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41990/0 (socket says 172.21.15.153:41990) 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 -- 172.21.15.153:0/2929408078 learned_addr learned my addr 172.21.15.153:0/2929408078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf11d9b640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf14102240 0x7faf140fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 -- 172.21.15.153:0/2929408078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf14102240 msgr2=0x7faf140fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf14102240 0x7faf140fb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.157+0000 7faf1259c640 1 -- 172.21.15.153:0/2929408078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04005040 con 0x7faf140643a0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf1259c640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7faf04004930 tx=0x7faf04009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf040094c0 con 0x7faf140643a0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf04032e50 con 0x7faf140643a0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf040433b0 con 0x7faf140643a0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf140fbe80 con 0x7faf140643a0 2024-04-05T03:29:27.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.158+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf141193f0 con 0x7faf140643a0 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.159+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf140fd940 con 0x7faf140643a0 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.159+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7faf0402f070 con 0x7faf140643a0 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.160+0000 7faf02ffd640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 0x7faefc0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.160+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7faf040af540 con 0x7faf140643a0 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.160+0000 7faf11d9b640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 0x7faefc0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.160+0000 7faf11d9b640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 0x7faefc0700c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7faf08006fd0 tx=0x7faf08008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.169+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faf04078ac0 con 0x7faf140643a0 2024-04-05T03:29:27.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.416+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7faf1404bcc0 con 0x7faf140643a0 2024-04-05T03:29:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.417+0000 7faf02ffd640 1 -- 172.21.15.153:0/2929408078 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v42) v1 ==== 107+0+5066 (secure 0 0 0) 0x7faf0407c180 con 0x7faf140643a0 2024-04-05T03:29:27.415 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:27.415 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44289,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/2394111038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2394111038},{"type":"v1","addr":"172.21.15.203:6827","nonce":2394111038}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:27.415 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 34 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.419+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 msgr2=0x7faefc0700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.419+0000 7faf188ae640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 0x7faefc0700c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7faf08006fd0 tx=0x7faf08008040 comp rx=0 tx=0).stop 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.419+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 msgr2=0x7faf140fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.419+0000 7faf188ae640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7faf04004930 tx=0x7faf04009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 shutdown_connections 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7faefc06dc10 0x7faefc0700c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf14102240 0x7faf140fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 --2- 172.21.15.153:0/2929408078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf140643a0 0x7faf140fb1e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 >> 172.21.15.153:0/2929408078 conn(0x7faf140710b0 msgr2=0x7faf140f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 shutdown_connections 2024-04-05T03:29:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.420+0000 7faf188ae640 1 -- 172.21.15.153:0/2929408078 wait complete. 2024-04-05T03:29:27.429 DEBUG:tasks.fs:max_mds not reduced in epoch 34 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:27.429 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-05T03:29:27.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- 172.21.15.153:0/2863475598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fb110 msgr2=0x7ff3c80fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/2863475598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fb110 0x7ff3c80fb560 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0048d0 tx=0x7ff3bc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- 172.21.15.153:0/2863475598 shutdown_connections 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/2863475598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fb110 0x7ff3c80fb560 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/2863475598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c8101140 0x7ff3c8101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- 172.21.15.153:0/2863475598 >> 172.21.15.153:0/2863475598 conn(0x7ff3c80710b0 msgr2=0x7ff3c80714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- 172.21.15.153:0/2863475598 shutdown_connections 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- 172.21.15.153:0/2863475598 wait complete. 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 Processor -- start 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.543+0000 7ff3ce19d640 1 -- start start 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3ce19d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c80fb110 0x7ff3c8112cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3ce19d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3ce19d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8113800 con 0x7ff3c8101140 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3ce19d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c810ce50 con 0x7ff3c80fb110 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41994/0 (socket says 172.21.15.153:41994) 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/1052966239 learned_addr learned my addr 172.21.15.153:0/1052966239 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c77fe640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c80fb110 0x7ff3c8112cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/1052966239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c80fb110 msgr2=0x7ff3c8112cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c80fb110 0x7ff3c8112cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/1052966239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc005040 con 0x7ff3c8101140 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3c6ffd640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0043d0 tx=0x7ff3bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3bc030840 con 0x7ff3c8101140 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.544+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c810d0d0 con 0x7ff3c8101140 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.545+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3bc0092e0 con 0x7ff3c8101140 2024-04-05T03:29:27.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.545+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3bc043640 con 0x7ff3c8101140 2024-04-05T03:29:27.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.545+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c810d5f0 con 0x7ff3c8101140 2024-04-05T03:29:27.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.546+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c80fc800 con 0x7ff3c8101140 2024-04-05T03:29:27.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.546+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7ff3bc009450 con 0x7ff3c8101140 2024-04-05T03:29:27.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.547+0000 7ff3affff640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 0x7ff3b00700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.547+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff3bc0b0310 con 0x7ff3c8101140 2024-04-05T03:29:27.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.547+0000 7ff3c77fe640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 0x7ff3b00700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.547+0000 7ff3c77fe640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 0x7ff3b00700d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009fd0 tx=0x7ff3b800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.553+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff3bc0798f0 con 0x7ff3c8101140 2024-04-05T03:29:27.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:27 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3080430975' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-05T03:29:27.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:27 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1580493023' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-05T03:29:27.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:27 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2929408078' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-05T03:29:27.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:27.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:27 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3080430975' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-05T03:29:27.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:27 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1580493023' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-05T03:29:27.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:27 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2929408078' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-05T03:29:27.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.853+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7ff3c8108260 con 0x7ff3c8101140 2024-04-05T03:29:27.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.854+0000 7ff3affff640 1 -- 172.21.15.153:0/1052966239 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v42) v1 ==== 107+0+4284 (secure 0 0 0) 0x7ff3bc07cfb0 con 0x7ff3c8101140 2024-04-05T03:29:27.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:27.852 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:27.852 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 35 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 msgr2=0x7ff3b00700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 0x7ff3b00700d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009fd0 tx=0x7ff3b800b040 comp rx=0 tx=0).stop 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 msgr2=0x7ff3c8113200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0043d0 tx=0x7ff3bc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 shutdown_connections 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7ff3b006dc20 0x7ff3b00700d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8101140 0x7ff3c8113200 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 --2- 172.21.15.153:0/1052966239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c80fb110 0x7ff3c8112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 >> 172.21.15.153:0/1052966239 conn(0x7ff3c80710b0 msgr2=0x7ff3c80f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 shutdown_connections 2024-04-05T03:29:27.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.856+0000 7ff3ce19d640 1 -- 172.21.15.153:0/1052966239 wait complete. 2024-04-05T03:29:27.865 DEBUG:tasks.fs:max_mds not reduced in epoch 35 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:27.865 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-04-05T03:29:27.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 -- 172.21.15.153:0/711099965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 msgr2=0x7f66080647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 --2- 172.21.15.153:0/711099965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f66080647d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f66040048d0 tx=0x7f6604009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 -- 172.21.15.153:0/711099965 shutdown_connections 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 --2- 172.21.15.153:0/711099965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f66080647d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 --2- 172.21.15.153:0/711099965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 0x7f6608102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 -- 172.21.15.153:0/711099965 >> 172.21.15.153:0/711099965 conn(0x7f66080710b0 msgr2=0x7f66080714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 -- 172.21.15.153:0/711099965 shutdown_connections 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.981+0000 7f660fe25640 1 -- 172.21.15.153:0/711099965 wait complete. 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 Processor -- start 2024-04-05T03:29:27.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 -- start start 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 0x7f6608113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6608113810 con 0x7f66080643a0 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660fe25640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660810ce60 con 0x7f6608102240 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42014/0 (socket says 172.21.15.153:42014) 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 -- 172.21.15.153:0/834673732 learned_addr learned my addr 172.21.15.153:0/834673732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660d3c0640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 0x7f6608113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 -- 172.21.15.153:0/834673732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 msgr2=0x7f6608113210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 0x7f6608113210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.982+0000 7f660dbc1640 1 -- 172.21.15.153:0/834673732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604005040 con 0x7f66080643a0 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f660dbc1640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f65f8002a50 tx=0x7f65f8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65f8007c30 con 0x7f66080643a0 2024-04-05T03:29:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65f800f040 con 0x7f66080643a0 2024-04-05T03:29:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660810d140 con 0x7f66080643a0 2024-04-05T03:29:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65f800db50 con 0x7f66080643a0 2024-04-05T03:29:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.983+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660810d630 con 0x7f66080643a0 2024-04-05T03:29:27.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.984+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66080fd940 con 0x7f66080643a0 2024-04-05T03:29:27.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.984+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f65f801c400 con 0x7f66080643a0 2024-04-05T03:29:27.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.985+0000 7f65fe7fc640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 0x7f65f40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:27.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.985+0000 7f660d3c0640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 0x7f65f40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:27.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.985+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f65f808cde0 con 0x7f66080643a0 2024-04-05T03:29:27.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.985+0000 7f660d3c0640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 0x7f65f40700c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f66080fafe0 tx=0x7f6604009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:27.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:27.989+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f65f8056410 con 0x7f66080643a0 2024-04-05T03:29:28.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.217+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f660810e340 con 0x7f66080643a0 2024-04-05T03:29:28.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.217+0000 7f65fe7fc640 1 -- 172.21.15.153:0/834673732 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v42) v1 ==== 107+0+5134 (secure 0 0 0) 0x7f65f8059ad0 con 0x7f66080643a0 2024-04-05T03:29:28.216 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:28.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":27,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:11:23.582846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:28.216 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 36 2024-04-05T03:29:28.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 msgr2=0x7f65f40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 0x7f65f40700c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f66080fafe0 tx=0x7f6604009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 msgr2=0x7f6608112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f65f8002a50 tx=0x7f65f8004600 comp rx=0 tx=0).stop 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 shutdown_connections 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f65f406dc10 0x7f65f40700c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6608102240 0x7f6608113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 --2- 172.21.15.153:0/834673732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66080643a0 0x7f6608112cd0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 >> 172.21.15.153:0/834673732 conn(0x7f66080710b0 msgr2=0x7f66080f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 shutdown_connections 2024-04-05T03:29:28.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.220+0000 7f660fe25640 1 -- 172.21.15.153:0/834673732 wait complete. 2024-04-05T03:29:28.230 DEBUG:tasks.fs:max_mds not reduced in epoch 36 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T03:29:28.230 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 36 2024-04-05T03:29:28.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-05T03:29:28.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 -- 172.21.15.153:0/3012399269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b80fb110 msgr2=0x7fa3b80fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/3012399269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b80fb110 0x7fa3b80fb560 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40048d0 tx=0x7fa3a4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 -- 172.21.15.153:0/3012399269 shutdown_connections 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/3012399269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b80fb110 0x7fa3b80fb560 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/3012399269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b8101140 0x7fa3b8101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.356+0000 7fa3bef5d640 1 -- 172.21.15.153:0/3012399269 >> 172.21.15.153:0/3012399269 conn(0x7fa3b80710b0 msgr2=0x7fa3b80714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 -- 172.21.15.153:0/3012399269 shutdown_connections 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 -- 172.21.15.153:0/3012399269 wait complete. 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 Processor -- start 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 -- start start 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 0x7fa3b810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8065310 con 0x7fa3b8101140 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.357+0000 7fa3bef5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b810e930 con 0x7fa3b80fb110 2024-04-05T03:29:28.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42034/0 (socket says 172.21.15.153:42034) 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 -- 172.21.15.153:0/4077529716 learned_addr learned my addr 172.21.15.153:0/4077529716 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3bccf9640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 0x7fa3b810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 -- 172.21.15.153:0/4077529716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 msgr2=0x7fa3b810deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 0x7fa3b810deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 -- 172.21.15.153:0/4077529716 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4005040 con 0x7fa3b8101140 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3bccf9640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 0x7fa3b810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b7fff640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40041d0 tx=0x7fa3a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a4030870 con 0x7fa3b8101140 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3a40092e0 con 0x7fa3b8101140 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a4043400 con 0x7fa3b8101140 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b810ebb0 con 0x7fa3b8101140 2024-04-05T03:29:28.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.358+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b8117170 con 0x7fa3b8101140 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.360+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b80fc800 con 0x7fa3b8101140 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.360+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa3a403b420 con 0x7fa3b8101140 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.360+0000 7fa3b57fa640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 0x7fa3a80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.360+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3a40b0290 con 0x7fa3b8101140 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.360+0000 7fa3bccf9640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 0x7fa3a80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.361+0000 7fa3bccf9640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 0x7fa3a80700c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac004d60 tx=0x7fa3ac00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:28.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.365+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3a4079930 con 0x7fa3b8101140 2024-04-05T03:29:28.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.606+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7fa3b804bcc0 con 0x7fa3b8101140 2024-04-05T03:29:28.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.607+0000 7fa3b57fa640 1 -- 172.21.15.153:0/4077529716 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v42) v1 ==== 107+0+5224 (secure 0 0 0) 0x7fa3a407cff0 con 0x7fa3b8101140 2024-04-05T03:29:28.605 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:28.605 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:22.725206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:28.605 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 37 2024-04-05T03:29:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 msgr2=0x7fa3a80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 0x7fa3a80700c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac004d60 tx=0x7fa3ac00b040 comp rx=0 tx=0).stop 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 msgr2=0x7fa3b810e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40041d0 tx=0x7fa3a4009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 shutdown_connections 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa3a806dc10 0x7fa3a80700c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3b8101140 0x7fa3b810e3f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 --2- 172.21.15.153:0/4077529716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3b80fb110 0x7fa3b810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 >> 172.21.15.153:0/4077529716 conn(0x7fa3b80710b0 msgr2=0x7fa3b80f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 shutdown_connections 2024-04-05T03:29:28.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.609+0000 7fa3bef5d640 1 -- 172.21.15.153:0/4077529716 wait complete. 2024-04-05T03:29:28.618 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 37 2024-04-05T03:29:28.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-05T03:29:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 -- 172.21.15.153:0/2088967738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fd420 msgr2=0x7efe180fd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/2088967738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fd420 0x7efe180fd7f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7efe08009c50 tx=0x7efe08004d10 comp rx=0 tx=0).stop 2024-04-05T03:29:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 -- 172.21.15.153:0/2088967738 shutdown_connections 2024-04-05T03:29:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/2088967738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fdd30 0x7efe18106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/2088967738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fd420 0x7efe180fd7f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 -- 172.21.15.153:0/2088967738 >> 172.21.15.153:0/2088967738 conn(0x7efe180f6ea0 msgr2=0x7efe180f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 -- 172.21.15.153:0/2088967738 shutdown_connections 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.728+0000 7efe1ce2e640 1 -- 172.21.15.153:0/2088967738 wait complete. 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 Processor -- start 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 -- start start 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fd420 0x7efe18112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe18113a70 con 0x7efe180fdd30 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1ce2e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1810d0c0 con 0x7efe180fd420 2024-04-05T03:29:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43174/0 (socket says 172.21.15.153:43174) 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 -- 172.21.15.153:0/366074098 learned_addr learned my addr 172.21.15.153:0/366074098 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe1659c640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fd420 0x7efe18112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 -- 172.21.15.153:0/366074098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fd420 msgr2=0x7efe18112f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fd420 0x7efe18112f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 -- 172.21.15.153:0/366074098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080098b0 con 0x7efe180fdd30 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.729+0000 7efe15d9b640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c830 tx=0x7efe0c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.730+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe0c00a850 con 0x7efe180fdd30 2024-04-05T03:29:28.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.730+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1810d3a0 con 0x7efe180fdd30 2024-04-05T03:29:28.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.730+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe0c00ae80 con 0x7efe180fdd30 2024-04-05T03:29:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.730+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe0c012ea0 con 0x7efe180fdd30 2024-04-05T03:29:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.730+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1810d8c0 con 0x7efe180fdd30 2024-04-05T03:29:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.731+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe18065d10 con 0x7efe180fdd30 2024-04-05T03:29:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.731+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efe0c012400 con 0x7efe180fdd30 2024-04-05T03:29:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.732+0000 7efe06ffd640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 0x7efe000700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:28.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.732+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe0c08d750 con 0x7efe180fdd30 2024-04-05T03:29:28.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.732+0000 7efe1659c640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 0x7efe000700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:28.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.732+0000 7efe1659c640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 0x7efe000700c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efe08009c20 tx=0x7efe08037040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.739+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efe0c056d50 con 0x7efe180fdd30 2024-04-05T03:29:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:28 smithi203 ceph-mon[95988]: pgmap v715: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:28 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1052966239' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-05T03:29:28.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:28 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/834673732' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-05T03:29:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:28 smithi153 ceph-mon[106102]: pgmap v715: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:28 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1052966239' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-05T03:29:28.821 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:28 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/834673732' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-05T03:29:28.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.979+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7efe1804bcc0 con 0x7efe180fdd30 2024-04-05T03:29:28.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.979+0000 7efe06ffd640 1 -- 172.21.15.153:0/366074098 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v42) v1 ==== 107+0+5223 (secure 0 0 0) 0x7efe0c0126b0 con 0x7efe180fdd30 2024-04-05T03:29:28.978 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:28.978 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:22.731644+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:28.978 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 38 2024-04-05T03:29:28.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 msgr2=0x7efe000700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 0x7efe000700c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efe08009c20 tx=0x7efe08037040 comp rx=0 tx=0).stop 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 msgr2=0x7efe18113470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c830 tx=0x7efe0c004430 comp rx=0 tx=0).stop 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 shutdown_connections 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efe0006dc10 0x7efe000700c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe180fdd30 0x7efe18113470 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 --2- 172.21.15.153:0/366074098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe180fd420 0x7efe18112f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 >> 172.21.15.153:0/366074098 conn(0x7efe180f6ea0 msgr2=0x7efe180f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 shutdown_connections 2024-04-05T03:29:28.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:28.982+0000 7efe1ce2e640 1 -- 172.21.15.153:0/366074098 wait complete. 2024-04-05T03:29:28.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.106+0000 7efebaf5d640 1 -- 172.21.15.153:0/1772514271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40643a0 msgr2=0x7efeb40647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.106+0000 7efebaf5d640 1 --2- 172.21.15.153:0/1772514271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40643a0 0x7efeb40647d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efea00048d0 tx=0x7efea0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- 172.21.15.153:0/1772514271 shutdown_connections 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 --2- 172.21.15.153:0/1772514271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb40643a0 0x7efeb40647d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 --2- 172.21.15.153:0/1772514271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4102240 0x7efeb4102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- 172.21.15.153:0/1772514271 >> 172.21.15.153:0/1772514271 conn(0x7efeb40710b0 msgr2=0x7efeb40714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- 172.21.15.153:0/1772514271 shutdown_connections 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- 172.21.15.153:0/1772514271 wait complete. 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 Processor -- start 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- start start 2024-04-05T03:29:29.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb40643a0 0x7efeb4112cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb4113800 con 0x7efeb4102240 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.107+0000 7efebaf5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb410ce50 con 0x7efeb40643a0 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43184/0 (socket says 172.21.15.153:43184) 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 -- 172.21.15.153:0/2841204182 learned_addr learned my addr 172.21.15.153:0/2841204182 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb8cf9640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb40643a0 0x7efeb4112cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 -- 172.21.15.153:0/2841204182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb40643a0 msgr2=0x7efeb4112cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb40643a0 0x7efeb4112cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 -- 172.21.15.153:0/2841204182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0005040 con 0x7efeb4102240 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb3fff640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efea0004930 tx=0x7efea0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea00094c0 con 0x7efeb4102240 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb410d0d0 con 0x7efeb4102240 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efea0032e50 con 0x7efeb4102240 2024-04-05T03:29:29.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea00433b0 con 0x7efeb4102240 2024-04-05T03:29:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.108+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb410d590 con 0x7efeb4102240 2024-04-05T03:29:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.110+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb40fd940 con 0x7efeb4102240 2024-04-05T03:29:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.110+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7efea002f070 con 0x7efeb4102240 2024-04-05T03:29:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.111+0000 7efeb17fa640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 0x7efea40700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.111+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efea00af330 con 0x7efeb4102240 2024-04-05T03:29:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.111+0000 7efeb8cf9640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 0x7efea40700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.111+0000 7efeb8cf9640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 0x7efea40700c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7efea8009fd0 tx=0x7efea800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.120+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efea0078960 con 0x7efeb4102240 2024-04-05T03:29:29.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.336+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7efeb404bcc0 con 0x7efeb4102240 2024-04-05T03:29:29.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.337+0000 7efeb17fa640 1 -- 172.21.15.153:0/2841204182 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v42) v1 ==== 107+0+5229 (secure 0 0 0) 0x7efea007c020 con 0x7efeb4102240 2024-04-05T03:29:29.335 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:29.335 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:29.266116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:resolve","state_seq":241,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:resolve","state_seq":236,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:29.335 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 39 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 msgr2=0x7efea40700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 0x7efea40700c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7efea8009fd0 tx=0x7efea800b040 comp rx=0 tx=0).stop 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 msgr2=0x7efeb4113200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efea0004930 tx=0x7efea0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 shutdown_connections 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7efea406dc10 0x7efea40700c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efeb4102240 0x7efeb4113200 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 --2- 172.21.15.153:0/2841204182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb40643a0 0x7efeb4112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 >> 172.21.15.153:0/2841204182 conn(0x7efeb40710b0 msgr2=0x7efeb40f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:29.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 shutdown_connections 2024-04-05T03:29:29.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.339+0000 7efebaf5d640 1 -- 172.21.15.153:0/2841204182 wait complete. 2024-04-05T03:29:29.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-04-05T03:29:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 -- 172.21.15.153:0/1632499607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 msgr2=0x7f19dc0647f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 --2- 172.21.15.153:0/1632499607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc0647f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0048d0 tx=0x7f19cc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 -- 172.21.15.153:0/1632499607 shutdown_connections 2024-04-05T03:29:29.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 --2- 172.21.15.153:0/1632499607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc0647f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 --2- 172.21.15.153:0/1632499607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 -- 172.21.15.153:0/1632499607 >> 172.21.15.153:0/1632499607 conn(0x7f19dc0710b0 msgr2=0x7f19dc0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 -- 172.21.15.153:0/1632499607 shutdown_connections 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.456+0000 7f19e379d640 1 -- 172.21.15.153:0/1632499607 wait complete. 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 Processor -- start 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 -- start start 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1131d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc1137d0 con 0x7f19dc0643a0 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e379d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc10ce20 con 0x7f19dc102210 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e1539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e1539640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43196/0 (socket says 172.21.15.153:43196) 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e1539640 1 -- 172.21.15.153:0/1357869969 learned_addr learned my addr 172.21.15.153:0/1357869969 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.457+0000 7f19e0d38640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e1539640 1 -- 172.21.15.153:0/1357869969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 msgr2=0x7f19dc1131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e1539640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e1539640 1 -- 172.21.15.153:0/1357869969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc005040 con 0x7f19dc0643a0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e0d38640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e1539640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f19d8002a50 tx=0x7f19d8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d8007c30 con 0x7f19dc0643a0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19d800f040 con 0x7f19dc0643a0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc10d100 con 0x7f19dc0643a0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d800dbe0 con 0x7f19dc0643a0 2024-04-05T03:29:29.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.458+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc10d620 con 0x7f19dc0643a0 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.459+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f19d800dd40 con 0x7f19dc0643a0 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.459+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0fd910 con 0x7f19dc0643a0 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.460+0000 7f19d1ffb640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 0x7f19c80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.460+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f19d808d4c0 con 0x7f19dc0643a0 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.460+0000 7f19e0d38640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 0x7f19c80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.460+0000 7f19e0d38640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 0x7f19c80700c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000c00 tx=0x7f19cc009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.469+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f19d8056ac0 con 0x7f19dc0643a0 2024-04-05T03:29:29.642 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:29 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/4077529716' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-05T03:29:29.643 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:29 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/366074098' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-05T03:29:29.643 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:29 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2841204182' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-05T03:29:29.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.755+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7f19dc04bcc0 con 0x7f19dc0643a0 2024-04-05T03:29:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.756+0000 7f19d1ffb640 1 -- 172.21.15.153:0/1357869969 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v42) v1 ==== 107+0+5233 (secure 0 0 0) 0x7f19dc04bcc0 con 0x7f19dc0643a0 2024-04-05T03:29:29.754 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:29.754 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:29.875011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:reconnect","state_seq":242,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:reconnect","state_seq":237,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:29.754 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 40 2024-04-05T03:29:29.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 msgr2=0x7f19c80700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 0x7f19c80700c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000c00 tx=0x7f19cc009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 msgr2=0x7f19dc112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f19d8002a50 tx=0x7f19d8004600 comp rx=0 tx=0).stop 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 shutdown_connections 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f19c806dc10 0x7f19c80700c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19dc102210 0x7f19dc1131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 --2- 172.21.15.153:0/1357869969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19dc0643a0 0x7f19dc112c90 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 >> 172.21.15.153:0/1357869969 conn(0x7f19dc0710b0 msgr2=0x7f19dc0f9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 shutdown_connections 2024-04-05T03:29:29.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.758+0000 7f19e379d640 1 -- 172.21.15.153:0/1357869969 wait complete. 2024-04-05T03:29:29.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 -- 172.21.15.153:0/2436374082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 msgr2=0x7fc2f80fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/2436374082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f80fb560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0048d0 tx=0x7fc2ec009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 -- 172.21.15.153:0/2436374082 shutdown_connections 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/2436374082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f80fb560 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/2436374082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 -- 172.21.15.153:0/2436374082 >> 172.21.15.153:0/2436374082 conn(0x7fc2f80710b0 msgr2=0x7fc2f80714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 -- 172.21.15.153:0/2436374082 shutdown_connections 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.873+0000 7fc2fe79d640 1 -- 172.21.15.153:0/2436374082 wait complete. 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 Processor -- start 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 -- start start 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8113230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8113830 con 0x7fc2f80fb110 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2fe79d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f81139a0 con 0x7fc2f8101140 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43214/0 (socket says 172.21.15.153:43214) 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2f7fff640 1 -- 172.21.15.153:0/1992867078 learned_addr learned my addr 172.21.15.153:0/1992867078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.874+0000 7fc2f77fe640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8113230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f7fff640 1 -- 172.21.15.153:0/1992867078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 msgr2=0x7fc2f8113230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:29.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f7fff640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8113230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f7fff640 1 -- 172.21.15.153:0/1992867078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec005040 con 0x7fc2f80fb110 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f77fe640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f7fff640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c790 tx=0x7fc2e80043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e800a820 con 0x7fc2f80fb110 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f810d030 con 0x7fc2f80fb110 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e800ae50 con 0x7fc2f80fb110 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e80115e0 con 0x7fc2f80fb110 2024-04-05T03:29:29.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.875+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f810d550 con 0x7fc2f80fb110 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.876+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f80fc800 con 0x7fc2f80fb110 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.877+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fc2e8012400 con 0x7fc2f80fb110 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.877+0000 7fc2f4ff9640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 0x7fc2e00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.877+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc2e808d730 con 0x7fc2f80fb110 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.877+0000 7fc2f77fe640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 0x7fc2e00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:29.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.878+0000 7fc2f77fe640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 0x7fc2e00700c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0043d0 tx=0x7fc2ec009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:29.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:29.882+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc2e8056d30 con 0x7fc2f80fb110 2024-04-05T03:29:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:29 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/4077529716' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-05T03:29:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:29 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/366074098' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-05T03:29:30.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:29 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2841204182' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-05T03:29:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.111+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7fc2f810d7d0 con 0x7fc2f80fb110 2024-04-05T03:29:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.112+0000 7fc2f4ff9640 1 -- 172.21.15.153:0/1992867078 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v42) v1 ==== 107+0+5227 (secure 0 0 0) 0x7fc2e805a3f0 con 0x7fc2f80fb110 2024-04-05T03:29:30.110 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:30.110 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44325,"name":"cephfs.smithi203.xpteyc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6825/185099298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":185099298},{"type":"v1","addr":"172.21.15.203:6825","nonce":185099298}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":44357,"name":"cephfs.smithi203.hgbifk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6827/3079507818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3079507818},{"type":"v1","addr":"172.21.15.203:6827","nonce":3079507818}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T02:59:41.656590+0000","modified":"2024-04-05T03:27:30.849200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi153.vicaky","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":243,"addr":"172.21.15.153:6827/1501229343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1501229343},{"type":"v1","addr":"172.21.15.153:6827","nonce":1501229343}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi153.fmpmvd","rank":1,"incarnation":37,"state":"up:rejoin","state_seq":238,"addr":"172.21.15.153:6829/2545565132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2545565132},{"type":"v1","addr":"172.21.15.153:6829","nonce":2545565132}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-05T03:29:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 41 2024-04-05T03:29:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 msgr2=0x7fc2e00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 0x7fc2e00700c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0043d0 tx=0x7fc2ec009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 msgr2=0x7fc2f8112cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c790 tx=0x7fc2e80043c0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 shutdown_connections 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fc2e006dc10 0x7fc2e00700c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2f8101140 0x7fc2f8113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 --2- 172.21.15.153:0/1992867078 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2f80fb110 0x7fc2f8112cf0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 >> 172.21.15.153:0/1992867078 conn(0x7fc2f80710b0 msgr2=0x7fc2f8064370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 shutdown_connections 2024-04-05T03:29:30.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.114+0000 7fc2fe79d640 1 -- 172.21.15.153:0/1992867078 wait complete. 2024-04-05T03:29:30.123 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-05T03:29:30.135 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-05T03:29:30.135 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T03:29:30.135 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T03:29:30.152 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T03:29:30.152 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T03:29:30.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 -- 172.21.15.153:0/3862071624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3754101140 msgr2=0x7f3754101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 --2- 172.21.15.153:0/3862071624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3754101140 0x7f3754101510 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f37480048d0 tx=0x7f3748009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 -- 172.21.15.153:0/3862071624 shutdown_connections 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 --2- 172.21.15.153:0/3862071624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37540fb110 0x7f37540fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 --2- 172.21.15.153:0/3862071624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3754101140 0x7f3754101510 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 -- 172.21.15.153:0/3862071624 >> 172.21.15.153:0/3862071624 conn(0x7f3754071130 msgr2=0x7f3754071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.374+0000 7f375b22e640 1 -- 172.21.15.153:0/3862071624 shutdown_connections 2024-04-05T03:29:30.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 -- 172.21.15.153:0/3862071624 wait complete. 2024-04-05T03:29:30.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 Processor -- start 2024-04-05T03:29:30.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 -- start start 2024-04-05T03:29:30.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3754101140 0x7f37541120d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37541126d0 con 0x7f37540fb110 2024-04-05T03:29:30.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.375+0000 7f375b22e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375410bd20 con 0x7f3754101140 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43226/0 (socket says 172.21.15.153:43226) 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 -- 172.21.15.153:0/3657443971 learned_addr learned my addr 172.21.15.153:0/3657443971 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3753fff640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3754101140 0x7f37541120d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 -- 172.21.15.153:0/3657443971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3754101140 msgr2=0x7f37541120d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3754101140 0x7f37541120d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 -- 172.21.15.153:0/3657443971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3748005040 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f3758fca640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f37480043d0 tx=0x7f3748009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3748030840 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37480092e0 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375410bfa0 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3748043640 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.376+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375410c4c0 con 0x7f37540fb110 2024-04-05T03:29:30.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.378+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37540fc800 con 0x7f37540fb110 2024-04-05T03:29:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.378+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3748009450 con 0x7f37540fb110 2024-04-05T03:29:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.379+0000 7f37517fa640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 0x7f37440700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.379+0000 7f3753fff640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 0x7f37440700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.379+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f37480b0340 con 0x7f37540fb110 2024-04-05T03:29:30.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.379+0000 7f3753fff640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 0x7f37440700c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3740009fd0 tx=0x7f3740009e50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:30.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.387+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3748079940 con 0x7f37540fb110 2024-04-05T03:29:30.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.663+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f375410d200 con 0x7f37540fb110 2024-04-05T03:29:30.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.664+0000 7f37517fa640 1 -- 172.21.15.153:0/3657443971 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16727 (secure 0 0 0) 0x7f374807d000 con 0x7f37540fb110 2024-04-05T03:29:30.662 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:30.663 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":84,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T03:12:17.587040+0000","last_up_change":"2024-04-05T03:11:21.401777+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T02:59:40.392548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T02:59:40.884499+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6803","nonce":190885135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6805","nonce":190885135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6809","nonce":190885135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6807","nonce":190885135}]},"public_addr":"172.21.15.153:6803/190885135","cluster_addr":"172.21.15.153:6805/190885135","heartbeat_back_addr":"172.21.15.153:6809/190885135","heartbeat_front_addr":"172.21.15.153:6807/190885135","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6811","nonce":2271479340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6813","nonce":2271479340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6817","nonce":2271479340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6815","nonce":2271479340}]},"public_addr":"172.21.15.153:6811/2271479340","cluster_addr":"172.21.15.153:6813/2271479340","heartbeat_back_addr":"172.21.15.153:6817/2271479340","heartbeat_front_addr":"172.21.15.153:6815/2271479340","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6819","nonce":2475488589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6821","nonce":2475488589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6825","nonce":2475488589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6823","nonce":2475488589}]},"public_addr":"172.21.15.153:6819/2475488589","cluster_addr":"172.21.15.153:6821/2475488589","heartbeat_back_addr":"172.21.15.153:6825/2475488589","heartbeat_front_addr":"172.21.15.153:6823/2475488589","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6801","nonce":2106212898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6803","nonce":2106212898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6807","nonce":2106212898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6805","nonce":2106212898}]},"public_addr":"172.21.15.203:6801/2106212898","cluster_addr":"172.21.15.203:6803/2106212898","heartbeat_back_addr":"172.21.15.203:6807/2106212898","heartbeat_front_addr":"172.21.15.203:6805/2106212898","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6809","nonce":2710960120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6811","nonce":2710960120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6815","nonce":2710960120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6813","nonce":2710960120}]},"public_addr":"172.21.15.203:6809/2710960120","cluster_addr":"172.21.15.203:6811/2710960120","heartbeat_back_addr":"172.21.15.203:6815/2710960120","heartbeat_front_addr":"172.21.15.203:6813/2710960120","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6817","nonce":2060217398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6819","nonce":2060217398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6823","nonce":2060217398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6821","nonce":2060217398}]},"public_addr":"172.21.15.203:6817/2060217398","cluster_addr":"172.21.15.203:6819/2060217398","heartbeat_back_addr":"172.21.15.203:6823/2060217398","heartbeat_front_addr":"172.21.15.203:6821/2060217398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6829/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.203:0/3454911926":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6826/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/2095582209":"2024-04-06T03:04:07.806216+0000","172.21.15.203:6825/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:6827/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1792434239":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/3744404984":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/4110399623":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:6824/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:0/2060103876":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.203:6828/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6828/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.153:0/971996676":"2024-04-06T03:02:35.649226+0000","172.21.15.203:6829/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3393095164":"2024-04-06T03:02:35.649226+0000","172.21.15.203:0/3658559261":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6800/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/3278817285":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/220940347":"2024-04-06T03:02:35.649226+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/2297491169":"2024-04-06T03:06:57.410816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 msgr2=0x7f37440700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 0x7f37440700c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3740009fd0 tx=0x7f3740009e50 comp rx=0 tx=0).stop 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 msgr2=0x7f3754111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f37480043d0 tx=0x7f3748009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 shutdown_connections 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f374406dc10 0x7f37440700c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3754101140 0x7f37541120d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 --2- 172.21.15.153:0/3657443971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37540fb110 0x7f3754111b90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 >> 172.21.15.153:0/3657443971 conn(0x7f3754071130 msgr2=0x7f37540f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 shutdown_connections 2024-04-05T03:29:30.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.667+0000 7f375b22e640 1 -- 172.21.15.153:0/3657443971 wait complete. 2024-04-05T03:29:30.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T02:57:13.688102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T02:59:40.392548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T02:59:40.884499+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T03:29:30.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T03:29:30.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:30 smithi203 ceph-mon[95988]: pgmap v716: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:30.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:30 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1357869969' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-05T03:29:30.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:30 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1992867078' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-05T03:29:30.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:30 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 -- 172.21.15.153:0/3197779407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 msgr2=0x7fbed0101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 --2- 172.21.15.153:0/3197779407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0101510 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fbec00048d0 tx=0x7fbec0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 -- 172.21.15.153:0/3197779407 shutdown_connections 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 --2- 172.21.15.153:0/3197779407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed00fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 --2- 172.21.15.153:0/3197779407 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0101510 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 -- 172.21.15.153:0/3197779407 >> 172.21.15.153:0/3197779407 conn(0x7fbed00710b0 msgr2=0x7fbed00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 -- 172.21.15.153:0/3197779407 shutdown_connections 2024-04-05T03:29:30.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 -- 172.21.15.153:0/3197779407 wait complete. 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.800+0000 7fbed666e640 1 Processor -- start 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbed666e640 1 -- start start 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbed666e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed0111be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbed666e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbed666e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed0112720 con 0x7fbed0101140 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbed666e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed010bd70 con 0x7fbed00fb110 2024-04-05T03:29:30.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed0111be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43248/0 (socket says 172.21.15.153:43248) 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecf7fe640 1 -- 172.21.15.153:0/1291164580 learned_addr learned my addr 172.21.15.153:0/1291164580 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecffff640 1 -- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 msgr2=0x7fbed0112120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecffff640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecffff640 1 -- 172.21.15.153:0/1291164580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0005040 con 0x7fbed00fb110 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.801+0000 7fbecf7fe640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbecffff640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed0111be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbec00048d0 tx=0x7fbec0009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbec0004290 con 0x7fbed00fb110 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed010bff0 con 0x7fbed00fb110 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbec0030840 con 0x7fbed00fb110 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbec0043400 con 0x7fbed00fb110 2024-04-05T03:29:30.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.802+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed010c510 con 0x7fbed00fb110 2024-04-05T03:29:30.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.803+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed00fc800 con 0x7fbed00fb110 2024-04-05T03:29:30.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.803+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fbec00092e0 con 0x7fbed00fb110 2024-04-05T03:29:30.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.804+0000 7fbeccff9640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 0x7fbeb80700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:30.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.804+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbec00b00c0 con 0x7fbed00fb110 2024-04-05T03:29:30.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.804+0000 7fbecf7fe640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 0x7fbeb80700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:30.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.804+0000 7fbecf7fe640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 0x7fbeb80700d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbed010d4a0 tx=0x7fbec400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:30.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:30.809+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbec00796c0 con 0x7fbed00fb110 2024-04-05T03:29:30.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:30 smithi153 ceph-mon[106102]: pgmap v716: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:30.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:30 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1357869969' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-05T03:29:30.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:30 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1992867078' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-05T03:29:30.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:30 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:29:31.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.069+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbed004bcc0 con 0x7fbed00fb110 2024-04-05T03:29:31.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.070+0000 7fbeccff9640 1 -- 172.21.15.153:0/1291164580 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbec007cd80 con 0x7fbed00fb110 2024-04-05T03:29:31.068 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.072+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 msgr2=0x7fbeb80700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.072+0000 7fbed666e640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 0x7fbeb80700d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbed010d4a0 tx=0x7fbec400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.072+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 msgr2=0x7fbed0111be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.072+0000 7fbed666e640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed0111be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbec00048d0 tx=0x7fbec0009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 shutdown_connections 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fbeb806dc20 0x7fbeb80700d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbed0101140 0x7fbed0112120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 --2- 172.21.15.153:0/1291164580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed00fb110 0x7fbed0111be0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 >> 172.21.15.153:0/1291164580 conn(0x7fbed00710b0 msgr2=0x7fbed00f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 shutdown_connections 2024-04-05T03:29:31.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.073+0000 7fbed666e640 1 -- 172.21.15.153:0/1291164580 wait complete. 2024-04-05T03:29:31.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T03:29:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 -- 172.21.15.153:0/1038996401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4730101140 msgr2=0x7f4730101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 --2- 172.21.15.153:0/1038996401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4730101140 0x7f4730101510 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f47280048d0 tx=0x7f4728009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 -- 172.21.15.153:0/1038996401 shutdown_connections 2024-04-05T03:29:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 --2- 172.21.15.153:0/1038996401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47300fb110 0x7f47300fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 --2- 172.21.15.153:0/1038996401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4730101140 0x7f4730101510 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 -- 172.21.15.153:0/1038996401 >> 172.21.15.153:0/1038996401 conn(0x7f47300710b0 msgr2=0x7f47300714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 -- 172.21.15.153:0/1038996401 shutdown_connections 2024-04-05T03:29:31.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.205+0000 7f473799d640 1 -- 172.21.15.153:0/1038996401 wait complete. 2024-04-05T03:29:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 Processor -- start 2024-04-05T03:29:31.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 -- start start 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730101140 0x7f473010e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4730065330 con 0x7f47300fb110 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f473799d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473010e920 con 0x7f4730101140 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43264/0 (socket says 172.21.15.153:43264) 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 -- 172.21.15.153:0/3348052288 learned_addr learned my addr 172.21.15.153:0/3348052288 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4734f38640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730101140 0x7f473010e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 -- 172.21.15.153:0/3348052288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730101140 msgr2=0x7f473010e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730101140 0x7f473010e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 -- 172.21.15.153:0/3348052288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4728005040 con 0x7f47300fb110 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.206+0000 7f4735739640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f47280041d0 tx=0x7f4728009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.207+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4728030870 con 0x7f47300fb110 2024-04-05T03:29:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.207+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47280092e0 con 0x7f47300fb110 2024-04-05T03:29:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.207+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473010eba0 con 0x7f47300fb110 2024-04-05T03:29:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.207+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4728043400 con 0x7f47300fb110 2024-04-05T03:29:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.207+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4730117130 con 0x7f47300fb110 2024-04-05T03:29:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.208+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47300fc800 con 0x7f47300fb110 2024-04-05T03:29:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.208+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f472803b420 con 0x7f47300fb110 2024-04-05T03:29:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.209+0000 7f472dffb640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 0x7f47180700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.209+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f47280b0210 con 0x7f47300fb110 2024-04-05T03:29:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.209+0000 7f4734f38640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 0x7f47180700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.209+0000 7f4734f38640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 0x7f47180700c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4730064b80 tx=0x7f4724009ed0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.214+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f47280798b0 con 0x7f47300fb110 2024-04-05T03:29:31.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.394+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f473004bcc0 con 0x7f47300fb110 2024-04-05T03:29:31.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.395+0000 7f472dffb640 1 -- 172.21.15.153:0/3348052288 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f472807cf70 con 0x7f47300fb110 2024-04-05T03:29:31.393 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 msgr2=0x7f47180700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 0x7f47180700c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4730064b80 tx=0x7f4724009ed0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 msgr2=0x7f473010de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f47280041d0 tx=0x7f4728009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 shutdown_connections 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f471806dc10 0x7f47180700c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4730101140 0x7f473010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 --2- 172.21.15.153:0/3348052288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47300fb110 0x7f473010de70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 >> 172.21.15.153:0/3348052288 conn(0x7f47300710b0 msgr2=0x7f47300f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 shutdown_connections 2024-04-05T03:29:31.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.397+0000 7f473799d640 1 -- 172.21.15.153:0/3348052288 wait complete. 2024-04-05T03:29:31.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T03:29:31.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/1954107077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 msgr2=0x7f8b880fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/1954107077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b880fb9b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8b700048d0 tx=0x7f8b70009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/1954107077 shutdown_connections 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/1954107077 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b880fb9b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/1954107077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 0x7f8b88101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/1954107077 >> 172.21.15.153:0/1954107077 conn(0x7f8b88070880 msgr2=0x7f8b88070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/1954107077 shutdown_connections 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/1954107077 wait complete. 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 Processor -- start 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.512+0000 7f8b8ed9d640 1 -- start start 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b8ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 0x7f8b88101f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b8ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b8ed9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88103e00 con 0x7f8b88101a40 2024-04-05T03:29:31.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b8ed9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b881029a0 con 0x7f8b88101130 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43286/0 (socket says 172.21.15.153:43286) 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 -- 172.21.15.153:0/2547129875 learned_addr learned my addr 172.21.15.153:0/2547129875 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b8cb39640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 0x7f8b88101f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 -- 172.21.15.153:0/2547129875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 msgr2=0x7f8b88101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 0x7f8b88101f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 -- 172.21.15.153:0/2547129875 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b70005040 con 0x7f8b88101a40 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b87fff640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8b700043d0 tx=0x7f8b70009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.513+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b70030870 con 0x7f8b88101a40 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.514+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b700092e0 con 0x7f8b88101a40 2024-04-05T03:29:31.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.514+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b88102c20 con 0x7f8b88101a40 2024-04-05T03:29:31.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.514+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b70043400 con 0x7f8b88101a40 2024-04-05T03:29:31.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.514+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b881194a0 con 0x7f8b88101a40 2024-04-05T03:29:31.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.515+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b880fc9c0 con 0x7f8b88101a40 2024-04-05T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.515+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8b7003b420 con 0x7f8b88101a40 2024-04-05T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.516+0000 7f8b857fa640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 0x7f8b740700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.516+0000 7f8b8cb39640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 0x7f8b740700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.516+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b700b0220 con 0x7f8b88101a40 2024-04-05T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.516+0000 7f8b8cb39640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 0x7f8b740700c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8b80007fd0 tx=0x7f8b80009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.520+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8b70079820 con 0x7f8b88101a40 2024-04-05T03:29:31.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.734+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8b8804bcc0 con 0x7f8b88101a40 2024-04-05T03:29:31.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.734+0000 7f8b857fa640 1 -- 172.21.15.153:0/2547129875 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8b8804bcc0 con 0x7f8b88101a40 2024-04-05T03:29:31.732 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:31.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 msgr2=0x7f8b740700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 0x7f8b740700c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8b80007fd0 tx=0x7f8b80009040 comp rx=0 tx=0).stop 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 msgr2=0x7f8b88102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8b700043d0 tx=0x7f8b70009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 shutdown_connections 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b7406dc10 0x7f8b740700c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b88101a40 0x7f8b88102460 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 --2- 172.21.15.153:0/2547129875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b88101130 0x7f8b88101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 >> 172.21.15.153:0/2547129875 conn(0x7f8b88070880 msgr2=0x7f8b88071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 shutdown_connections 2024-04-05T03:29:31.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.736+0000 7f8b8ed9d640 1 -- 172.21.15.153:0/2547129875 wait complete. 2024-04-05T03:29:31.744 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3657443971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:31.744 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1291164580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:31.744 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:29:31.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:29:31.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:29:31.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:31 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3348052288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:31.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 -- 172.21.15.153:0/1133926512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 msgr2=0x7f8b680fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 --2- 172.21.15.153:0/1133926512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b680fb560 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c0048d0 tx=0x7f8b5c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 -- 172.21.15.153:0/1133926512 shutdown_connections 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 --2- 172.21.15.153:0/1133926512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b680fb560 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 --2- 172.21.15.153:0/1133926512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 -- 172.21.15.153:0/1133926512 >> 172.21.15.153:0/1133926512 conn(0x7f8b680710b0 msgr2=0x7f8b680714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 -- 172.21.15.153:0/1133926512 shutdown_connections 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.850+0000 7f8b6de58640 1 -- 172.21.15.153:0/1133926512 wait complete. 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 Processor -- start 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 -- start start 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68112140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68112740 con 0x7f8b680fb110 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b6de58640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6810bd90 con 0x7f8b68101140 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.851+0000 7f8b677fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43300/0 (socket says 172.21.15.153:43300) 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 -- 172.21.15.153:0/3484036726 learned_addr learned my addr 172.21.15.153:0/3484036726 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b66ffd640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68112140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 -- 172.21.15.153:0/3484036726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 msgr2=0x7f8b68112140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68112140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 -- 172.21.15.153:0/3484036726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c005040 con 0x7f8b680fb110 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b66ffd640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68112140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b677fe640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800cb10 tx=0x7f8b58007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b58004200 con 0x7f8b680fb110 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b5800acf0 con 0x7f8b680fb110 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6810c070 con 0x7f8b680fb110 2024-04-05T03:29:31.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b58003e40 con 0x7f8b680fb110 2024-04-05T03:29:31.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.852+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6810c590 con 0x7f8b680fb110 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.854+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b680fc800 con 0x7f8b680fb110 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.854+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f8b5800a820 con 0x7f8b680fb110 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.854+0000 7f8b4ffff640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 0x7f8b500700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.855+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b5808dc00 con 0x7f8b680fb110 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.855+0000 7f8b66ffd640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 0x7f8b500700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:31.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.855+0000 7f8b66ffd640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 0x7f8b500700c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009fa0 tx=0x7f8b5c009da0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:31.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:31.862+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8b58093020 con 0x7f8b680fb110 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3657443971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1291164580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:29:32.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:31 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3348052288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.102+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8b68108220 con 0x7f8b680fb110 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.103+0000 7f8b4ffff640 1 -- 172.21.15.153:0/3484036726 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2347 (secure 0 0 0) 0x7f8b58016940 con 0x7f8b680fb110 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6829/1509596739 2024-04-06T03:11:23.582651+0000 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3454911926 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1531431225 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1352217920 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6826/2246788375 2024-04-06T02:59:57.435445+0000 2024-04-05T03:29:32.101 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/437915692 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/663556540 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1804478369 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/458993476 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2873889549 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/2095582209 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6825/1771218289 2024-04-06T03:11:23.582665+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6827/2246788375 2024-04-06T02:59:57.435445+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/458993476 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1792434239 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2293398789 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/1535236796 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3744404984 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/4110399623 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.102 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/1535236796 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6824/1771218289 2024-04-06T03:11:23.582665+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2060103876 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3064925142 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6828/2106885355 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6828/1509596739 2024-04-06T03:11:23.582651+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/971996676 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6829/2106885355 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/499404754 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/4288916206 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/2121051366 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/2121051366 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3429410039 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3393095164 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3658559261 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/663556540 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.103 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3278817285 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.104 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/220940347 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:32.104 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/4288916206 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:32.104 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2297491169 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:listed 39 entries 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 msgr2=0x7f8b500700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 0x7f8b500700c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009fa0 tx=0x7f8b5c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 msgr2=0x7f8b68111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800cb10 tx=0x7f8b58007ee0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 shutdown_connections 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f8b5006dc10 0x7f8b500700c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101140 0x7f8b68112140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 --2- 172.21.15.153:0/3484036726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b680fb110 0x7f8b68111c00 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 >> 172.21.15.153:0/3484036726 conn(0x7f8b680710b0 msgr2=0x7f8b680f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 shutdown_connections 2024-04-05T03:29:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.105+0000 7f8b6de58640 1 -- 172.21.15.153:0/3484036726 wait complete. 2024-04-05T03:29:32.116 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-05T03:29:32.116 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T03:29:32.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 -- 172.21.15.153:0/1299418118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7580fb110 msgr2=0x7fa7580fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 --2- 172.21.15.153:0/1299418118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7580fb110 0x7fa7580fb560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0048d0 tx=0x7fa74c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 -- 172.21.15.153:0/1299418118 shutdown_connections 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 --2- 172.21.15.153:0/1299418118 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa7580fb110 0x7fa7580fb560 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 --2- 172.21.15.153:0/1299418118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa758101140 0x7fa758101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 -- 172.21.15.153:0/1299418118 >> 172.21.15.153:0/1299418118 conn(0x7fa758071130 msgr2=0x7fa758071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 -- 172.21.15.153:0/1299418118 shutdown_connections 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.276+0000 7fa75e46e640 1 -- 172.21.15.153:0/1299418118 wait complete. 2024-04-05T03:29:32.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 Processor -- start 2024-04-05T03:29:32.276 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 -- start start 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7580fb110 0x7fa75810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa758065330 con 0x7fa758101140 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.277+0000 7fa75e46e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75810e900 con 0x7fa7580fb110 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43308/0 (socket says 172.21.15.153:43308) 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 -- 172.21.15.153:0/3808726955 learned_addr learned my addr 172.21.15.153:0/3808726955 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa757fff640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7580fb110 0x7fa75810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 -- 172.21.15.153:0/3808726955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7580fb110 msgr2=0x7fa75810de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7580fb110 0x7fa75810de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 -- 172.21.15.153:0/3808726955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c005040 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa7577fe640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0041d0 tx=0x7fa74c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa74c0092e0 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa74c009440 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75810eb80 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.278+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa74c043640 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.279+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa758117110 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.279+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7580fc800 con 0x7fa758101140 2024-04-05T03:29:32.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.280+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7fa74c03b980 con 0x7fa758101140 2024-04-05T03:29:32.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.281+0000 7fa754ff9640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 0x7fa7400700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.281+0000 7fa757fff640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 0x7fa7400700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.281+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa74c0b02b0 con 0x7fa758101140 2024-04-05T03:29:32.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.281+0000 7fa757fff640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 0x7fa7400700c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa748007fd0 tx=0x7fa748009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.285+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa74c0798b0 con 0x7fa758101140 2024-04-05T03:29:32.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.484+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa75804bcc0 con 0x7fa758101140 2024-04-05T03:29:32.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.485+0000 7fa754ff9640 1 -- 172.21.15.153:0/3808726955 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16727 (secure 0 0 0) 0x7fa74c07cf70 con 0x7fa758101140 2024-04-05T03:29:32.483 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-05T03:29:32.484 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":84,"fsid":"3fa69b9e-f2f7-11ee-b649-cb9ed24678a4","created":"2024-04-05T02:51:51.789741+0000","modified":"2024-04-05T03:12:17.587040+0000","last_up_change":"2024-04-05T03:11:21.401777+0000","last_in_change":"2024-04-05T02:58:09.409380+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T02:57:13.688102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T02:59:40.392548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T02:59:40.884499+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3638168-6ff3-4751-b73c-032b22097ffa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6803","nonce":190885135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6805","nonce":190885135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6809","nonce":190885135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":190885135},{"type":"v1","addr":"172.21.15.153:6807","nonce":190885135}]},"public_addr":"172.21.15.153:6803/190885135","cluster_addr":"172.21.15.153:6805/190885135","heartbeat_back_addr":"172.21.15.153:6809/190885135","heartbeat_front_addr":"172.21.15.153:6807/190885135","state":["exists","up"]},{"osd":1,"uuid":"5f4fcb24-415c-494b-b034-5ca7e7472e4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6811","nonce":2271479340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6813","nonce":2271479340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6817","nonce":2271479340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2271479340},{"type":"v1","addr":"172.21.15.153:6815","nonce":2271479340}]},"public_addr":"172.21.15.153:6811/2271479340","cluster_addr":"172.21.15.153:6813/2271479340","heartbeat_back_addr":"172.21.15.153:6817/2271479340","heartbeat_front_addr":"172.21.15.153:6815/2271479340","state":["exists","up"]},{"osd":2,"uuid":"d951e5ca-f53c-4d05-94c4-f732a3910a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6819","nonce":2475488589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6821","nonce":2475488589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6825","nonce":2475488589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2475488589},{"type":"v1","addr":"172.21.15.153:6823","nonce":2475488589}]},"public_addr":"172.21.15.153:6819/2475488589","cluster_addr":"172.21.15.153:6821/2475488589","heartbeat_back_addr":"172.21.15.153:6825/2475488589","heartbeat_front_addr":"172.21.15.153:6823/2475488589","state":["exists","up"]},{"osd":3,"uuid":"5f2ae843-b4ae-481d-9ade-a3afcbf0d2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6801","nonce":2106212898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6803","nonce":2106212898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6807","nonce":2106212898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2106212898},{"type":"v1","addr":"172.21.15.203:6805","nonce":2106212898}]},"public_addr":"172.21.15.203:6801/2106212898","cluster_addr":"172.21.15.203:6803/2106212898","heartbeat_back_addr":"172.21.15.203:6807/2106212898","heartbeat_front_addr":"172.21.15.203:6805/2106212898","state":["exists","up"]},{"osd":4,"uuid":"e2b77994-438d-4c3b-8fb7-44217f7b3b44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6809","nonce":2710960120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6811","nonce":2710960120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6815","nonce":2710960120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2710960120},{"type":"v1","addr":"172.21.15.203:6813","nonce":2710960120}]},"public_addr":"172.21.15.203:6809/2710960120","cluster_addr":"172.21.15.203:6811/2710960120","heartbeat_back_addr":"172.21.15.203:6815/2710960120","heartbeat_front_addr":"172.21.15.203:6813/2710960120","state":["exists","up"]},{"osd":5,"uuid":"7f8e3fba-d5ed-4ce5-882a-9f5aeb8a2708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6817","nonce":2060217398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6819","nonce":2060217398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6823","nonce":2060217398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2060217398},{"type":"v1","addr":"172.21.15.203:6821","nonce":2060217398}]},"public_addr":"172.21.15.203:6817/2060217398","cluster_addr":"172.21.15.203:6819/2060217398","heartbeat_back_addr":"172.21.15.203:6823/2060217398","heartbeat_front_addr":"172.21.15.203:6821/2060217398","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:11.093919+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:56:44.241156+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:11.362277+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:57:34.711294+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:00.495278+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T02:58:26.801248+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:6829/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.203:0/3454911926":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/1531431225":"2024-04-06T02:52:55.613050+0000","172.21.15.153:0/1352217920":"2024-04-06T02:52:55.613050+0000","172.21.15.153:6826/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:0/437915692":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/1804478369":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6801/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/2873889549":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/2095582209":"2024-04-06T03:04:07.806216+0000","172.21.15.203:6825/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:6827/2246788375":"2024-04-06T02:59:57.435445+0000","172.21.15.153:6800/458993476":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/1792434239":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/2293398789":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6800/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:0/3744404984":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/4110399623":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/1535236796":"2024-04-06T02:52:55.613050+0000","172.21.15.203:6824/1771218289":"2024-04-06T03:11:23.582665+0000","172.21.15.153:0/2060103876":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3064925142":"2024-04-06T02:55:13.379556+0000","172.21.15.203:6828/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6828/1509596739":"2024-04-06T03:11:23.582651+0000","172.21.15.153:0/971996676":"2024-04-06T03:02:35.649226+0000","172.21.15.203:6829/2106885355":"2024-04-06T03:04:07.806216+0000","172.21.15.153:0/499404754":"2024-04-06T02:52:24.549449+0000","172.21.15.153:6801/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:6800/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:6801/2121051366":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/3429410039":"2024-04-06T02:52:24.549449+0000","172.21.15.153:0/3393095164":"2024-04-06T03:02:35.649226+0000","172.21.15.203:0/3658559261":"2024-04-06T03:04:07.806216+0000","172.21.15.153:6800/663556540":"2024-04-06T03:02:35.649226+0000","172.21.15.153:0/3278817285":"2024-04-06T03:06:57.410816+0000","172.21.15.153:0/220940347":"2024-04-06T03:02:35.649226+0000","172.21.15.153:6800/4288916206":"2024-04-06T02:55:13.379556+0000","172.21.15.153:0/2297491169":"2024-04-06T03:06:57.410816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.487+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 msgr2=0x7fa7400700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.487+0000 7fa75e46e640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 0x7fa7400700c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa748007fd0 tx=0x7fa748009040 comp rx=0 tx=0).stop 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.487+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 msgr2=0x7fa75810e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.487+0000 7fa75e46e640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0041d0 tx=0x7fa74c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 shutdown_connections 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7fa74006dc10 0x7fa7400700c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa758101140 0x7fa75810e390 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 --2- 172.21.15.153:0/3808726955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7580fb110 0x7fa75810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 >> 172.21.15.153:0/3808726955 conn(0x7fa758071130 msgr2=0x7fa7580f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 shutdown_connections 2024-04-05T03:29:32.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.488+0000 7fa75e46e640 1 -- 172.21.15.153:0/3808726955 wait complete. 2024-04-05T03:29:32.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T02:57:13.688102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T02:59:40.392548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T02:59:40.884499+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T03:29:32.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T03:29:32.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 -- 172.21.15.153:0/4207588813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c0fb110 msgr2=0x7f791c0fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 --2- 172.21.15.153:0/4207588813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c0fb110 0x7f791c0fb560 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f79040048d0 tx=0x7f7904009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 -- 172.21.15.153:0/4207588813 shutdown_connections 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 --2- 172.21.15.153:0/4207588813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c0fb110 0x7f791c0fb560 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 --2- 172.21.15.153:0/4207588813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c101140 0x7f791c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 -- 172.21.15.153:0/4207588813 >> 172.21.15.153:0/4207588813 conn(0x7f791c0710b0 msgr2=0x7f791c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 -- 172.21.15.153:0/4207588813 shutdown_connections 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.628+0000 7f7920aae640 1 -- 172.21.15.153:0/4207588813 wait complete. 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 Processor -- start 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 -- start start 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c0fb110 0x7f791c111be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c112720 con 0x7f791c101140 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7920aae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c10bd70 con 0x7f791c0fb110 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7911d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7911d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43328/0 (socket says 172.21.15.153:43328) 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.629+0000 7f7911d9b640 1 -- 172.21.15.153:0/165033521 learned_addr learned my addr 172.21.15.153:0/165033521 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f791a59c640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c0fb110 0x7f791c111be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7911d9b640 1 -- 172.21.15.153:0/165033521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c0fb110 msgr2=0x7f791c111be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7911d9b640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c0fb110 0x7f791c111be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7911d9b640 1 -- 172.21.15.153:0/165033521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904005040 con 0x7f791c101140 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7911d9b640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f79040043d0 tx=0x7f7904009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7904030840 con 0x7f791c101140 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c10bff0 con 0x7f791c101140 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79040092e0 con 0x7f791c101140 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7904043640 con 0x7f791c101140 2024-04-05T03:29:32.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.630+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c10c510 con 0x7f791c101140 2024-04-05T03:29:32.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.632+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c0fc800 con 0x7f791c101140 2024-04-05T03:29:32.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.632+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f7904009450 con 0x7f791c101140 2024-04-05T03:29:32.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.633+0000 7f79137fe640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 0x7f79080700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.633+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f79040b02b0 con 0x7f791c101140 2024-04-05T03:29:32.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.633+0000 7f791a59c640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 0x7f79080700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.633+0000 7f791a59c640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 0x7f79080700d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f790c004eb0 tx=0x7f790c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.637+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f79040798b0 con 0x7f791c101140 2024-04-05T03:29:32.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.817+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f791c108200 con 0x7f791c101140 2024-04-05T03:29:32.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.817+0000 7f79137fe640 1 -- 172.21.15.153:0/165033521 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f790407cf70 con 0x7f791c101140 2024-04-05T03:29:32.816 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 msgr2=0x7f79080700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 0x7f79080700d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f790c004eb0 tx=0x7f790c00b040 comp rx=0 tx=0).stop 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 msgr2=0x7f791c112120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f79040043d0 tx=0x7f7904009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 shutdown_connections 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f790806dc20 0x7f79080700d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f791c101140 0x7f791c112120 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 --2- 172.21.15.153:0/165033521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f791c0fb110 0x7f791c111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.819+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 >> 172.21.15.153:0/165033521 conn(0x7f791c0710b0 msgr2=0x7f791c0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.820+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 shutdown_connections 2024-04-05T03:29:32.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.820+0000 7f7920aae640 1 -- 172.21.15.153:0/165033521 wait complete. 2024-04-05T03:29:32.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:32 smithi153 ceph-mon[106102]: pgmap v717: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:32.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:32 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2547129875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:32.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:32 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3484036726' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T03:29:32.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:32 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3808726955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:32.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T03:29:32.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 -- 172.21.15.153:0/1900285006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 msgr2=0x7f08c81014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 --2- 172.21.15.153:0/1900285006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c81014a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f08b80048d0 tx=0x7f08b8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 -- 172.21.15.153:0/1900285006 shutdown_connections 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 --2- 172.21.15.153:0/1900285006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 0x7f08c80fb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 --2- 172.21.15.153:0/1900285006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c81014a0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 -- 172.21.15.153:0/1900285006 >> 172.21.15.153:0/1900285006 conn(0x7f08c80f6ca0 msgr2=0x7f08c80f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 -- 172.21.15.153:0/1900285006 shutdown_connections 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.944+0000 7f08cde6e640 1 -- 172.21.15.153:0/1900285006 wait complete. 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 Processor -- start 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 -- start start 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 0x7f08c8113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c81138d0 con 0x7f08c81010d0 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08cde6e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c810ce90 con 0x7f08c8101a70 2024-04-05T03:29:32.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.945+0000 7f08c77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43358/0 (socket says 172.21.15.153:43358) 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c77fe640 1 -- 172.21.15.153:0/3098806107 learned_addr learned my addr 172.21.15.153:0/3098806107 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c6ffd640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 0x7f08c8113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c77fe640 1 -- 172.21.15.153:0/3098806107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 msgr2=0x7f08c8113240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c77fe640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 0x7f08c8113240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c77fe640 1 -- 172.21.15.153:0/3098806107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b8005040 con 0x7f08c81010d0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08c77fe640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f08b8004930 tx=0x7f08b8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b80094c0 con 0x7f08c81010d0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c810d110 con 0x7f08c81010d0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08b8032e50 con 0x7f08c81010d0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08b80433b0 con 0x7f08c81010d0 2024-04-05T03:29:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.946+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c810d5d0 con 0x7f08c81010d0 2024-04-05T03:29:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.948+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c80fc840 con 0x7f08c81010d0 2024-04-05T03:29:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.948+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f08b802f070 con 0x7f08c81010d0 2024-04-05T03:29:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.948+0000 7f08affff640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 0x7f08b00700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.948+0000 7f08c6ffd640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 0x7f08b00700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.948+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f08b80af500 con 0x7f08c81010d0 2024-04-05T03:29:32.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.949+0000 7f08c6ffd640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 0x7f08b00700c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:32.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:32.953+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f08b8078a80 con 0x7f08c81010d0 2024-04-05T03:29:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:32 smithi203 ceph-mon[95988]: pgmap v717: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:32 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2547129875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:32 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3484036726' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T03:29:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:32 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3808726955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T03:29:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.143+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f08c804bcc0 con 0x7f08c81010d0 2024-04-05T03:29:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.143+0000 7f08affff640 1 -- 172.21.15.153:0/3098806107 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f08b807c140 con 0x7f08c81010d0 2024-04-05T03:29:33.141 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 msgr2=0x7f08b00700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 0x7f08b00700c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc008040 comp rx=0 tx=0).stop 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 msgr2=0x7f08c8112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f08b8004930 tx=0x7f08b8009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 shutdown_connections 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f08b006dc10 0x7f08b00700c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08c8101a70 0x7f08c8113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 --2- 172.21.15.153:0/3098806107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08c81010d0 0x7f08c8112d00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 >> 172.21.15.153:0/3098806107 conn(0x7f08c80f6ca0 msgr2=0x7f08c8065020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:33.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 shutdown_connections 2024-04-05T03:29:33.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.146+0000 7f08cde6e640 1 -- 172.21.15.153:0/3098806107 wait complete. 2024-04-05T03:29:33.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T03:29:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- 172.21.15.153:0/2057415161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940fb110 msgr2=0x7f88940fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 --2- 172.21.15.153:0/2057415161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940fb110 0x7f88940fb560 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f887c0048d0 tx=0x7f887c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- 172.21.15.153:0/2057415161 shutdown_connections 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 --2- 172.21.15.153:0/2057415161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88940fb110 0x7f88940fb560 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 --2- 172.21.15.153:0/2057415161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8894101140 0x7f8894101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- 172.21.15.153:0/2057415161 >> 172.21.15.153:0/2057415161 conn(0x7f88940710b0 msgr2=0x7f88940714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- 172.21.15.153:0/2057415161 shutdown_connections 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- 172.21.15.153:0/2057415161 wait complete. 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 Processor -- start 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.261+0000 7f889a35d640 1 -- start start 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88940fb110 0x7f8894111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8894112740 con 0x7f8894101140 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889410bd90 con 0x7f88940fb110 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43382/0 (socket says 172.21.15.153:43382) 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 -- 172.21.15.153:0/2801806060 learned_addr learned my addr 172.21.15.153:0/2801806060 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f8893fff640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88940fb110 0x7f8894111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 -- 172.21.15.153:0/2801806060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88940fb110 msgr2=0x7f8894111c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88940fb110 0x7f8894111c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 -- 172.21.15.153:0/2801806060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c005040 con 0x7f8894101140 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f888bfff640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f887c0041d0 tx=0x7f887c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c0092e0 con 0x7f8894101140 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f887c009440 con 0x7f8894101140 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c043640 con 0x7f8894101140 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889410c010 con 0x7f8894101140 2024-04-05T03:29:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.262+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889410c4e0 con 0x7f8894101140 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.264+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88940fc800 con 0x7f8894101140 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.264+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f887c03b9f0 con 0x7f8894101140 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.264+0000 7f88917fa640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 0x7f88800700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.264+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f887c0b0360 con 0x7f8894101140 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.264+0000 7f8893fff640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 0x7f88800700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.265+0000 7f8893fff640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 0x7f88800700c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f88840041d0 tx=0x7f8884009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:33.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.269+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f887c079960 con 0x7f8894101140 2024-04-05T03:29:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.491+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8894108220 con 0x7f8894101140 2024-04-05T03:29:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.491+0000 7f88917fa640 1 -- 172.21.15.153:0/2801806060 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f887c07d020 con 0x7f8894101140 2024-04-05T03:29:33.489 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 msgr2=0x7f88800700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 0x7f88800700c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f88840041d0 tx=0x7f8884009040 comp rx=0 tx=0).stop 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 msgr2=0x7f8894112140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f887c0041d0 tx=0x7f887c009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 shutdown_connections 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f888006dc10 0x7f88800700c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8894101140 0x7f8894112140 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 --2- 172.21.15.153:0/2801806060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88940fb110 0x7f8894111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.493+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 >> 172.21.15.153:0/2801806060 conn(0x7f88940710b0 msgr2=0x7f88940f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.494+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 shutdown_connections 2024-04-05T03:29:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.494+0000 7f889a35d640 1 -- 172.21.15.153:0/2801806060 wait complete. 2024-04-05T03:29:33.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.616+0000 7f3c2b95d640 1 -- 172.21.15.153:0/2528325644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c24101140 msgr2=0x7f3c24101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.616+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/2528325644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c24101140 0x7f3c24101510 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3c200048d0 tx=0x7f3c20009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- 172.21.15.153:0/2528325644 shutdown_connections 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/2528325644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c240fb110 0x7f3c240fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/2528325644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c24101140 0x7f3c24101510 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- 172.21.15.153:0/2528325644 >> 172.21.15.153:0/2528325644 conn(0x7f3c240710b0 msgr2=0x7f3c240714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- 172.21.15.153:0/2528325644 shutdown_connections 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- 172.21.15.153:0/2528325644 wait complete. 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 Processor -- start 2024-04-05T03:29:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- start start 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 0x7f3c2410e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2410e8a0 con 0x7f3c240fb110 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c2b95d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2410ea10 con 0x7f3c24101140 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.617+0000 7f3c296f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:43396/0 (socket says 172.21.15.153:43396) 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 -- 172.21.15.153:0/1249339233 learned_addr learned my addr 172.21.15.153:0/1249339233 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c28ef8640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 0x7f3c2410e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 -- 172.21.15.153:0/1249339233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 msgr2=0x7f3c2410e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 0x7f3c2410e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 -- 172.21.15.153:0/1249339233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20005040 con 0x7f3c240fb110 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c28ef8640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 0x7f3c2410e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T03:29:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c296f9640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c20030640 tx=0x7f3c20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c20030820 con 0x7f3c240fb110 2024-04-05T03:29:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c200092e0 con 0x7f3c240fb110 2024-04-05T03:29:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2410ec90 con 0x7f3c240fb110 2024-04-05T03:29:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.618+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c20043400 con 0x7f3c240fb110 2024-04-05T03:29:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.619+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c24117140 con 0x7f3c240fb110 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.620+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c240fc800 con 0x7f3c240fb110 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.620+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 40) v1 ==== 91861+0+0 (secure 0 0 0) 0x7f3c2003b420 con 0x7f3c240fb110 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.621+0000 7f3c19ffb640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 0x7f3c100700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.621+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f3c200b0240 con 0x7f3c240fb110 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.621+0000 7f3c28ef8640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 0x7f3c100700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T03:29:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.621+0000 7f3c28ef8640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 0x7f3c100700c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3c24064bc0 tx=0x7f3c1400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T03:29:33.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.627+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3c20079900 con 0x7f3c240fb110 2024-04-05T03:29:33.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:33 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/165033521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:33.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:33 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/3098806107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:33.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:33 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/2801806060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:33.844 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.846+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3c2410ec30 con 0x7f3c240fb110 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.847+0000 7f3c19ffb640 1 -- 172.21.15.153:0/1249339233 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v84) v1 ==== 81+0+2347 (secure 0 0 0) 0x7f3c2007cfc0 con 0x7f3c240fb110 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6829/1509596739 2024-04-06T03:11:23.582651+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3454911926 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1531431225 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1352217920 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6826/2246788375 2024-04-06T02:59:57.435445+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/437915692 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/663556540 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1804478369 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/458993476 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:33.845 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2873889549 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/2095582209 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6825/1771218289 2024-04-06T03:11:23.582665+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6827/2246788375 2024-04-06T02:59:57.435445+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/458993476 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/1792434239 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2293398789 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/1535236796 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3744404984 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/4110399623 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.846 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/1535236796 2024-04-06T02:52:55.613050+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6824/1771218289 2024-04-06T03:11:23.582665+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2060103876 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3064925142 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6828/2106885355 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6828/1509596739 2024-04-06T03:11:23.582651+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/971996676 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:6829/2106885355 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/499404754 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/4288916206 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/2121051366 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6801/2121051366 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3429410039 2024-04-06T02:52:24.549449+0000 2024-04-05T03:29:33.847 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3393095164 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.203:0/3658559261 2024-04-06T03:04:07.806216+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/663556540 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/3278817285 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/220940347 2024-04-06T03:02:35.649226+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:6800/4288916206 2024-04-06T02:55:13.379556+0000 2024-04-05T03:29:33.848 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153:0/2297491169 2024-04-06T03:06:57.410816+0000 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:listed 39 entries 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 msgr2=0x7f3c100700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 0x7f3c100700c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3c24064bc0 tx=0x7f3c1400b040 comp rx=0 tx=0).stop 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 msgr2=0x7f3c2410de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c20030640 tx=0x7f3c20009da0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 shutdown_connections 2024-04-05T03:29:33.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:6800/3734499116,v1:172.21.15.153:6801/3734499116] conn(0x7f3c1006dc10 0x7f3c100700c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c24101140 0x7f3c2410e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 --2- 172.21.15.153:0/1249339233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c240fb110 0x7f3c2410de20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T03:29:33.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 >> 172.21.15.153:0/1249339233 conn(0x7f3c240710b0 msgr2=0x7f3c240f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T03:29:33.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 shutdown_connections 2024-04-05T03:29:33.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-05T03:29:33.849+0000 7f3c2b95d640 1 -- 172.21.15.153:0/1249339233 wait complete. 2024-04-05T03:29:33.859 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi153.front.sepia.ceph.com... 2024-04-05T03:29:33.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T03:29:33.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-05T03:29:33.888 INFO:teuthology.orchestra.run:waiting for 300 2024-04-05T03:29:34.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:33 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/165033521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T03:29:34.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:33 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/3098806107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T03:29:34.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:33 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/2801806060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T03:29:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:34 smithi153 ceph-mon[106102]: pgmap v718: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:34 smithi153 ceph-mon[106102]: from='client.? 172.21.15.153:0/1249339233' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T03:29:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:34 smithi203 ceph-mon[95988]: pgmap v718: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:34 smithi203 ceph-mon[95988]: from='client.? 172.21.15.153:0/1249339233' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T03:29:36.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:36 smithi153 ceph-mon[106102]: pgmap v719: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:37.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:36 smithi203 ceph-mon[95988]: pgmap v719: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:38.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:38 smithi153 ceph-mon[106102]: pgmap v720: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:39.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:38 smithi203 ceph-mon[95988]: pgmap v720: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:40.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:40 smithi153 ceph-mon[106102]: pgmap v721: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:41.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:40 smithi203 ceph-mon[95988]: pgmap v721: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:42 smithi153 ceph-mon[106102]: pgmap v722: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:43.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:42 smithi203 ceph-mon[95988]: pgmap v722: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:44 smithi153 ceph-mon[106102]: pgmap v723: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:44.847 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:44 smithi203 ceph-mon[95988]: pgmap v723: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:46.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:46 smithi153 ceph-mon[106102]: pgmap v724: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:47.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:46 smithi203 ceph-mon[95988]: pgmap v724: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:48.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:48 smithi153 ceph-mon[106102]: pgmap v725: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:49.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:48 smithi203 ceph-mon[95988]: pgmap v725: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:50.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:50 smithi153 ceph-mon[106102]: pgmap v726: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:51.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:50 smithi203 ceph-mon[95988]: pgmap v726: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:53.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:52 smithi203 ceph-mon[95988]: pgmap v727: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:52 smithi153 ceph-mon[106102]: pgmap v727: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:29:55.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:54 smithi203 ceph-mon[95988]: pgmap v728: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:54 smithi153 ceph-mon[106102]: pgmap v728: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:29:57.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:56 smithi203 ceph-mon[95988]: pgmap v729: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:57.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:56 smithi153 ceph-mon[106102]: pgmap v729: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:29:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:58.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:29:59.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:29:58 smithi203 ceph-mon[95988]: pgmap v730: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:29:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:29:58 smithi153 ceph-mon[106102]: pgmap v730: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:00.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:00 smithi153 ceph-mon[106102]: pgmap v731: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:00.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:00 smithi153 ceph-mon[106102]: overall HEALTH_OK 2024-04-05T03:30:01.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:00 smithi203 ceph-mon[95988]: pgmap v731: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:01.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:00 smithi203 ceph-mon[95988]: overall HEALTH_OK 2024-04-05T03:30:03.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:02 smithi203 ceph-mon[95988]: pgmap v732: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:02 smithi153 ceph-mon[106102]: pgmap v732: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:05.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:04 smithi203 ceph-mon[95988]: pgmap v733: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:04 smithi153 ceph-mon[106102]: pgmap v733: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:06 smithi203 ceph-mon[95988]: pgmap v734: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:06 smithi153 ceph-mon[106102]: pgmap v734: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:09.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:08 smithi203 ceph-mon[95988]: pgmap v735: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:08 smithi153 ceph-mon[106102]: pgmap v735: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:10.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:10 smithi153 ceph-mon[106102]: pgmap v736: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:11.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:10 smithi203 ceph-mon[95988]: pgmap v736: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:12 smithi203 ceph-mon[95988]: pgmap v737: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:12 smithi153 ceph-mon[106102]: pgmap v737: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:15.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:14 smithi203 ceph-mon[95988]: pgmap v738: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:14 smithi153 ceph-mon[106102]: pgmap v738: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:17.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:16 smithi203 ceph-mon[95988]: pgmap v739: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:17.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:16 smithi153 ceph-mon[106102]: pgmap v739: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:18.871 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:18 smithi153 ceph-mon[106102]: pgmap v740: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:19.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:18 smithi203 ceph-mon[95988]: pgmap v740: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:20.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:20 smithi153 ceph-mon[106102]: pgmap v741: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:21.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:20 smithi203 ceph-mon[95988]: pgmap v741: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:23.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:22 smithi203 ceph-mon[95988]: pgmap v742: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:22 smithi153 ceph-mon[106102]: pgmap v742: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:24 smithi203 ceph-mon[95988]: pgmap v743: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:24 smithi153 ceph-mon[106102]: pgmap v743: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:26 smithi203 ceph-mon[95988]: pgmap v744: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:26 smithi153 ceph-mon[106102]: pgmap v744: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:28.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:28 smithi203 ceph-mon[95988]: pgmap v745: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:28 smithi153 ceph-mon[106102]: pgmap v745: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:30.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:30 smithi153 ceph-mon[106102]: pgmap v746: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:31.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:30 smithi203 ceph-mon[95988]: pgmap v746: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:32.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:31 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:30:32.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:31 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:30:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:32 smithi203 ceph-mon[95988]: pgmap v747: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:30:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:30:33.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:30:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:32 smithi153 ceph-mon[106102]: pgmap v747: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:30:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:30:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:30:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:34 smithi203 ceph-mon[95988]: pgmap v748: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:34 smithi153 ceph-mon[106102]: pgmap v748: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:37.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:36 smithi203 ceph-mon[95988]: pgmap v749: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:37.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:36 smithi153 ceph-mon[106102]: pgmap v749: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:39.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:38 smithi203 ceph-mon[95988]: pgmap v750: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:39.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:38 smithi153 ceph-mon[106102]: pgmap v750: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:41.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:40 smithi203 ceph-mon[95988]: pgmap v751: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:41.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:40 smithi153 ceph-mon[106102]: pgmap v751: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:43.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:42 smithi203 ceph-mon[95988]: pgmap v752: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:42 smithi153 ceph-mon[106102]: pgmap v752: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:45.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:44 smithi203 ceph-mon[95988]: pgmap v753: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:45.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:44 smithi153 ceph-mon[106102]: pgmap v753: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:47.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:46 smithi203 ceph-mon[95988]: pgmap v754: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:46 smithi153 ceph-mon[106102]: pgmap v754: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:49.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:48 smithi203 ceph-mon[95988]: pgmap v755: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:49.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:48 smithi153 ceph-mon[106102]: pgmap v755: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:51.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:50 smithi203 ceph-mon[95988]: pgmap v756: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:50 smithi153 ceph-mon[106102]: pgmap v756: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:53.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:52 smithi203 ceph-mon[95988]: pgmap v757: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:52 smithi153 ceph-mon[106102]: pgmap v757: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:30:55.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:54 smithi203 ceph-mon[95988]: pgmap v758: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:54 smithi153 ceph-mon[106102]: pgmap v758: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:30:57.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:56 smithi203 ceph-mon[95988]: pgmap v759: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:57.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:56 smithi153 ceph-mon[106102]: pgmap v759: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:30:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:58.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:30:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:30:58 smithi203 ceph-mon[95988]: pgmap v760: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:30:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:30:58 smithi153 ceph-mon[106102]: pgmap v760: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:01.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:00 smithi203 ceph-mon[95988]: pgmap v761: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:01.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:00 smithi153 ceph-mon[106102]: pgmap v761: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:03.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:02 smithi203 ceph-mon[95988]: pgmap v762: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:02 smithi153 ceph-mon[106102]: pgmap v762: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:05.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:04 smithi203 ceph-mon[95988]: pgmap v763: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:04 smithi153 ceph-mon[106102]: pgmap v763: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:07.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:06 smithi203 ceph-mon[95988]: pgmap v764: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:06 smithi153 ceph-mon[106102]: pgmap v764: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:09.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:08 smithi203 ceph-mon[95988]: pgmap v765: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:08 smithi153 ceph-mon[106102]: pgmap v765: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:11.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:10 smithi203 ceph-mon[95988]: pgmap v766: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:11.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:10 smithi153 ceph-mon[106102]: pgmap v766: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:12 smithi203 ceph-mon[95988]: pgmap v767: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:12 smithi153 ceph-mon[106102]: pgmap v767: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:15.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:14 smithi203 ceph-mon[95988]: pgmap v768: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:14 smithi153 ceph-mon[106102]: pgmap v768: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:17.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:16 smithi203 ceph-mon[95988]: pgmap v769: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:17.072 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:16 smithi153 ceph-mon[106102]: pgmap v769: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:19.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:18 smithi203 ceph-mon[95988]: pgmap v770: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:18 smithi153 ceph-mon[106102]: pgmap v770: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:21.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:20 smithi203 ceph-mon[95988]: pgmap v771: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:21.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:20 smithi153 ceph-mon[106102]: pgmap v771: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:23.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:22 smithi203 ceph-mon[95988]: pgmap v772: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:22 smithi153 ceph-mon[106102]: pgmap v772: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:24 smithi203 ceph-mon[95988]: pgmap v773: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:24 smithi153 ceph-mon[106102]: pgmap v773: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:26 smithi203 ceph-mon[95988]: pgmap v774: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:26 smithi153 ceph-mon[106102]: pgmap v774: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:28.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:28 smithi203 ceph-mon[95988]: pgmap v775: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:28 smithi153 ceph-mon[106102]: pgmap v775: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:31.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:30 smithi203 ceph-mon[95988]: pgmap v776: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:30 smithi153 ceph-mon[106102]: pgmap v776: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:32 smithi203 ceph-mon[95988]: pgmap v777: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:32 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:31:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:32 smithi153 ceph-mon[106102]: pgmap v777: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:32 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:31:34.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:31:34.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:31:34.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:31:34.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:31:34.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:31:34.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:31:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:34 smithi203 ceph-mon[95988]: pgmap v778: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:34 smithi153 ceph-mon[106102]: pgmap v778: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:37.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:36 smithi203 ceph-mon[95988]: pgmap v779: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:37.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:36 smithi153 ceph-mon[106102]: pgmap v779: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:39.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:38 smithi203 ceph-mon[95988]: pgmap v780: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:39.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:38 smithi153 ceph-mon[106102]: pgmap v780: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:41.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:40 smithi203 ceph-mon[95988]: pgmap v781: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:41.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:40 smithi153 ceph-mon[106102]: pgmap v781: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:42 smithi153 ceph-mon[106102]: pgmap v782: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:43.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:42 smithi203 ceph-mon[95988]: pgmap v782: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:43.295 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:45.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:44 smithi153 ceph-mon[106102]: pgmap v783: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:45.097 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:44 smithi203 ceph-mon[95988]: pgmap v783: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:46 smithi153 ceph-mon[106102]: pgmap v784: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:47.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:46 smithi203 ceph-mon[95988]: pgmap v784: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:49.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:48 smithi153 ceph-mon[106102]: pgmap v785: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:49.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:48 smithi203 ceph-mon[95988]: pgmap v785: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:50 smithi153 ceph-mon[106102]: pgmap v786: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:51.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:50 smithi203 ceph-mon[95988]: pgmap v786: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:52.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:51 smithi153 ceph-mon[106102]: pgmap v787: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:52.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:51 smithi203 ceph-mon[95988]: pgmap v787: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:31:54.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:54 smithi203 ceph-mon[95988]: pgmap v788: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:54.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:54 smithi153 ceph-mon[106102]: pgmap v788: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:31:56.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:56 smithi203 ceph-mon[95988]: pgmap v789: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:56.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:56 smithi153 ceph-mon[106102]: pgmap v789: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:31:58.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:58 smithi203 ceph-mon[95988]: pgmap v790: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:58.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:31:58 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:31:58.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:58 smithi153 ceph-mon[106102]: pgmap v790: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:31:58.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:31:58 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:00.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:00 smithi203 ceph-mon[95988]: pgmap v791: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:00.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:00 smithi153 ceph-mon[106102]: pgmap v791: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:02.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:02 smithi203 ceph-mon[95988]: pgmap v792: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:02.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:02 smithi153 ceph-mon[106102]: pgmap v792: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:04.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:04 smithi203 ceph-mon[95988]: pgmap v793: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:04.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:04 smithi153 ceph-mon[106102]: pgmap v793: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:06.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:06 smithi203 ceph-mon[95988]: pgmap v794: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:06.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:06 smithi153 ceph-mon[106102]: pgmap v794: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:08.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:08 smithi203 ceph-mon[95988]: pgmap v795: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:08.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:08 smithi153 ceph-mon[106102]: pgmap v795: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:10.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:10 smithi203 ceph-mon[95988]: pgmap v796: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:10.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:10 smithi153 ceph-mon[106102]: pgmap v796: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:12.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:12 smithi203 ceph-mon[95988]: pgmap v797: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T03:32:12.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:12 smithi153 ceph-mon[106102]: pgmap v797: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T03:32:13.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:13 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:13.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:13 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:14.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:14 smithi203 ceph-mon[95988]: pgmap v798: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T03:32:14.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:14 smithi153 ceph-mon[106102]: pgmap v798: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T03:32:16.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:16 smithi203 ceph-mon[95988]: pgmap v799: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:32:16.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:16 smithi153 ceph-mon[106102]: pgmap v799: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:32:18.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:18 smithi203 ceph-mon[95988]: pgmap v800: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:32:18.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:18 smithi153 ceph-mon[106102]: pgmap v800: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T03:32:20.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:20 smithi203 ceph-mon[95988]: pgmap v801: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-05T03:32:20.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:20 smithi153 ceph-mon[106102]: pgmap v801: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-05T03:32:22.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:22 smithi203 ceph-mon[95988]: pgmap v802: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T03:32:22.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:22 smithi153 ceph-mon[106102]: pgmap v802: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T03:32:24.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:24 smithi203 ceph-mon[95988]: pgmap v803: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T03:32:24.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:24 smithi153 ceph-mon[106102]: pgmap v803: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T03:32:26.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:26 smithi203 ceph-mon[95988]: pgmap v804: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:32:26.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:26 smithi153 ceph-mon[106102]: pgmap v804: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T03:32:27.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:27.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:28.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:28 smithi203 ceph-mon[95988]: pgmap v805: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:32:28.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:28 smithi153 ceph-mon[106102]: pgmap v805: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T03:32:30.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:30 smithi203 ceph-mon[95988]: pgmap v806: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:30.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:30 smithi153 ceph-mon[106102]: pgmap v806: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:32.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:32 smithi203 ceph-mon[95988]: pgmap v807: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:32.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:32 smithi153 ceph-mon[106102]: pgmap v807: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:33.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:33 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:32:33.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:33 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:32:34.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:34 smithi203 ceph-mon[95988]: pgmap v808: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:34.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:32:34.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:32:34.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:32:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:34 smithi153 ceph-mon[106102]: pgmap v808: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:32:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:32:34.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:32:36.795 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:36 smithi203 ceph-mon[95988]: pgmap v809: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:36.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:36 smithi153 ceph-mon[106102]: pgmap v809: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:38.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:38 smithi153 ceph-mon[106102]: pgmap v810: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:39.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:38 smithi203 ceph-mon[95988]: pgmap v810: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:40.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:40 smithi153 ceph-mon[106102]: pgmap v811: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:41.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:40 smithi203 ceph-mon[95988]: pgmap v811: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:42 smithi153 ceph-mon[106102]: pgmap v812: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:42.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:43.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:42 smithi203 ceph-mon[95988]: pgmap v812: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:43.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:44.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:44 smithi153 ceph-mon[106102]: pgmap v813: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:45.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:44 smithi203 ceph-mon[95988]: pgmap v813: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:46.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:46 smithi153 ceph-mon[106102]: pgmap v814: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:46.846 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:46 smithi203 ceph-mon[95988]: pgmap v814: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:48.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:48 smithi153 ceph-mon[106102]: pgmap v815: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:49.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:48 smithi203 ceph-mon[95988]: pgmap v815: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:50.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:50 smithi153 ceph-mon[106102]: pgmap v816: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:51.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:50 smithi203 ceph-mon[95988]: pgmap v816: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:52.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:52 smithi153 ceph-mon[106102]: pgmap v817: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:53.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:52 smithi203 ceph-mon[95988]: pgmap v817: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:32:54.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:54 smithi153 ceph-mon[106102]: pgmap v818: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:55.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:54 smithi203 ceph-mon[95988]: pgmap v818: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:32:56.820 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:56 smithi153 ceph-mon[106102]: pgmap v819: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:57.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:56 smithi203 ceph-mon[95988]: pgmap v819: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:32:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:58.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:32:59.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:32:58 smithi203 ceph-mon[95988]: pgmap v820: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:32:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:32:58 smithi153 ceph-mon[106102]: pgmap v820: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:00.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:00 smithi153 ceph-mon[106102]: pgmap v821: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:01.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:00 smithi203 ceph-mon[95988]: pgmap v821: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:03.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:02 smithi203 ceph-mon[95988]: pgmap v822: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:02 smithi153 ceph-mon[106102]: pgmap v822: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:05.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:04 smithi203 ceph-mon[95988]: pgmap v823: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:04 smithi153 ceph-mon[106102]: pgmap v823: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:06 smithi203 ceph-mon[95988]: pgmap v824: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:06 smithi153 ceph-mon[106102]: pgmap v824: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:09.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:08 smithi203 ceph-mon[95988]: pgmap v825: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:08 smithi153 ceph-mon[106102]: pgmap v825: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:10.883 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:10 smithi153 ceph-mon[106102]: pgmap v826: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:11.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:10 smithi203 ceph-mon[95988]: pgmap v826: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:12 smithi203 ceph-mon[95988]: pgmap v827: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:12 smithi153 ceph-mon[106102]: pgmap v827: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:15.046 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:14 smithi203 ceph-mon[95988]: pgmap v828: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:14 smithi153 ceph-mon[106102]: pgmap v828: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:16.846 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:16 smithi203 ceph-mon[95988]: pgmap v829: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:17.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:16 smithi153 ceph-mon[106102]: pgmap v829: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:18.872 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:18 smithi153 ceph-mon[106102]: pgmap v830: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:19.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:18 smithi203 ceph-mon[95988]: pgmap v830: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:20.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:20 smithi153 ceph-mon[106102]: pgmap v831: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:21.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:20 smithi203 ceph-mon[95988]: pgmap v831: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:23.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:22 smithi203 ceph-mon[95988]: pgmap v832: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:22 smithi153 ceph-mon[106102]: pgmap v832: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:24 smithi203 ceph-mon[95988]: pgmap v833: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:24 smithi153 ceph-mon[106102]: pgmap v833: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:26 smithi203 ceph-mon[95988]: pgmap v834: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:26 smithi153 ceph-mon[106102]: pgmap v834: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:28.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:28 smithi203 ceph-mon[95988]: pgmap v835: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:28 smithi153 ceph-mon[106102]: pgmap v835: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:30.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:30 smithi153 ceph-mon[106102]: pgmap v836: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:31.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:30 smithi203 ceph-mon[95988]: pgmap v836: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:33.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:32 smithi203 ceph-mon[95988]: pgmap v837: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:33.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:32 smithi153 ceph-mon[106102]: pgmap v837: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:34 smithi203 ceph-mon[95988]: pgmap v838: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:35.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:33:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:33:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:33:35.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:34 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:33:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:34 smithi153 ceph-mon[106102]: pgmap v838: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T03:33:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T03:33:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T03:33:35.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:34 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' 2024-04-05T03:33:37.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:36 smithi203 ceph-mon[95988]: pgmap v839: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:37.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:36 smithi153 ceph-mon[106102]: pgmap v839: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:39.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:38 smithi203 ceph-mon[95988]: pgmap v840: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:39.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:38 smithi153 ceph-mon[106102]: pgmap v840: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:40.884 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:40 smithi153 ceph-mon[106102]: pgmap v841: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:41.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:40 smithi203 ceph-mon[95988]: pgmap v841: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:43.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:42 smithi203 ceph-mon[95988]: pgmap v842: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:43.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:42 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:42 smithi153 ceph-mon[106102]: pgmap v842: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:43.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:42 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:45.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:44 smithi203 ceph-mon[95988]: pgmap v843: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:45.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:44 smithi153 ceph-mon[106102]: pgmap v843: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:47.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:46 smithi203 ceph-mon[95988]: pgmap v844: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:47.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:46 smithi153 ceph-mon[106102]: pgmap v844: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:49.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:48 smithi203 ceph-mon[95988]: pgmap v845: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:49.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:48 smithi153 ceph-mon[106102]: pgmap v845: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:51.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:50 smithi203 ceph-mon[95988]: pgmap v846: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:51.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:50 smithi153 ceph-mon[106102]: pgmap v846: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:53.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:52 smithi203 ceph-mon[95988]: pgmap v847: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:53.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:52 smithi153 ceph-mon[106102]: pgmap v847: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:33:55.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:54 smithi203 ceph-mon[95988]: pgmap v848: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:55.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:54 smithi153 ceph-mon[106102]: pgmap v848: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:33:57.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:56 smithi203 ceph-mon[95988]: pgmap v849: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:57.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:56 smithi153 ceph-mon[106102]: pgmap v849: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:33:58.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:57 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:58.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:57 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:33:59.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:33:58 smithi203 ceph-mon[95988]: pgmap v850: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:33:59.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:33:58 smithi153 ceph-mon[106102]: pgmap v850: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:01.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:00 smithi203 ceph-mon[95988]: pgmap v851: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:01.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:00 smithi153 ceph-mon[106102]: pgmap v851: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:03.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:02 smithi203 ceph-mon[95988]: pgmap v852: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:03.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:02 smithi153 ceph-mon[106102]: pgmap v852: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:05.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:04 smithi203 ceph-mon[95988]: pgmap v853: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:05.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:04 smithi153 ceph-mon[106102]: pgmap v853: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:07.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:06 smithi203 ceph-mon[95988]: pgmap v854: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:07.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:06 smithi153 ceph-mon[106102]: pgmap v854: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:09.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:08 smithi203 ceph-mon[95988]: pgmap v855: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:09.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:08 smithi153 ceph-mon[106102]: pgmap v855: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:11.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:10 smithi203 ceph-mon[95988]: pgmap v856: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:11.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:10 smithi153 ceph-mon[106102]: pgmap v856: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:13.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:12 smithi203 ceph-mon[95988]: pgmap v857: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:13.045 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:12 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:34:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:12 smithi153 ceph-mon[106102]: pgmap v857: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:13.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:12 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:34:15.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:14 smithi203 ceph-mon[95988]: pgmap v858: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:15.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:14 smithi153 ceph-mon[106102]: pgmap v858: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:17.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:16 smithi203 ceph-mon[95988]: pgmap v859: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:17.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:16 smithi153 ceph-mon[106102]: pgmap v859: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:19.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:18 smithi203 ceph-mon[95988]: pgmap v860: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:19.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:18 smithi153 ceph-mon[106102]: pgmap v860: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:21.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:20 smithi203 ceph-mon[95988]: pgmap v861: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:21.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:20 smithi153 ceph-mon[106102]: pgmap v861: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:23.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:22 smithi203 ceph-mon[95988]: pgmap v862: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:23.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:22 smithi153 ceph-mon[106102]: pgmap v862: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:25.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:24 smithi203 ceph-mon[95988]: pgmap v863: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:25.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:24 smithi153 ceph-mon[106102]: pgmap v863: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:27.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:26 smithi203 ceph-mon[95988]: pgmap v864: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:27.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:26 smithi153 ceph-mon[106102]: pgmap v864: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T03:34:28.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:27 smithi203 ceph-mon[95988]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:34:28.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:27 smithi153 ceph-mon[106102]: from='mgr.34104 172.21.15.153:0/3267981976' entity='mgr.smithi153.npzyqy' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T03:34:29.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:28 smithi203 ceph-mon[95988]: pgmap v865: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:29.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:28 smithi153 ceph-mon[106102]: pgmap v865: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T03:34:31.044 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:30 smithi203 ceph-mon[95988]: pgmap v866: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:31.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:30 smithi153 ceph-mon[106102]: pgmap v866: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T03:34:32.960 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-05T03:34:32.961 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T03:34:32.962 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-05T03:34:32.975 INFO:tasks.cephadm:Teardown begin 2024-04-05T03:34:32.975 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T03:34:32.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T03:34:33.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:32 smithi153 ceph-mon[106102]: pgmap v867: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:33.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T03:34:33.038 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:32 smithi203 ceph-mon[95988]: pgmap v867: 65 pgs: 65 active+clean; 252 MiB data, 967 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T03:34:33.043 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-05T03:34:33.044 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T03:34:33.062 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T03:34:33.101 INFO:tasks.cephadm:Stopping all daemons... 2024-04-05T03:34:33.101 INFO:tasks.cephadm.mon.smithi153:Stopping mon.smithi153... 2024-04-05T03:34:33.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153 2024-04-05T03:34:33.320 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:34:33.764 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:34:33.432+0000 7f9e362bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:34:33.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153[106098]: 2024-04-05T03:34:33.432+0000 7f9e362bb700 -1 mon.smithi153@0(leader) e3 *** Got Signal Terminated *** 2024-04-05T03:34:33.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 podman[151996]: 2024-04-05 03:34:33.504445168 +0000 UTC m=+0.187748422 container died 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:34.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 podman[151996]: 2024-04-05 03:34:33.966585961 +0000 UTC m=+0.649889229 container cleanup 26862dff0163017db4bd97b3719de8f49d9ff098b0e756930d07b6ff46f939c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-05T03:34:34.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Apr 05 03:34:33 smithi153 bash[151996]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi153 2024-04-05T03:34:34.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi153.service' 2024-04-05T03:34:34.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:34:34.472 INFO:tasks.cephadm.mon.smithi153:Stopped mon.smithi153 2024-04-05T03:34:34.472 INFO:tasks.cephadm.mon.smithi203:Stopping mon.smithi203... 2024-04-05T03:34:34.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi203 2024-04-05T03:34:34.794 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:34 smithi203 systemd[1]: Stopping Ceph mon.smithi203 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:34:35.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:34 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203[95984]: 2024-04-05T03:34:34.797+0000 7f6329243700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi203 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:34:35.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:34 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203[95984]: 2024-04-05T03:34:34.797+0000 7f6329243700 -1 mon.smithi203@1(peon) e3 *** Got Signal Terminated *** 2024-04-05T03:34:35.294 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:35 smithi203 podman[130498]: 2024-04-05 03:34:35.002503 +0000 UTC m=+0.330191585 container died 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:34:35.957 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:35 smithi203 podman[130498]: 2024-04-05 03:34:35.606799973 +0000 UTC m=+0.934488561 container cleanup 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-05T03:34:35.957 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:35 smithi203 bash[130498]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203 2024-04-05T03:34:35.957 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:Apr 05 03:34:35 smithi203 podman[130512]: 2024-04-05 03:34:35.841886186 +0000 UTC m=+0.831977656 container remove 1cf9e00346436ad20171e32279920961588a179b28ebc7c8adbcc9c6243843b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-mon-smithi203, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-05T03:34:35.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@mon.smithi203.service' 2024-04-05T03:34:36.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:34:36.016 INFO:tasks.cephadm.mon.smithi203:Stopped mon.smithi203 2024-04-05T03:34:36.016 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-05T03:34:36.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.0 2024-04-05T03:34:36.318 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:36 smithi153 systemd[1]: Stopping Ceph osd.0 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:34:36.570 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:36 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:34:36.320+0000 7f6f33fbe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:34:36.570 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:36 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:34:36.320+0000 7f6f33fbe700 -1 osd.0 84 *** Got signal Terminated *** 2024-04-05T03:34:36.570 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:36 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0[117223]: 2024-04-05T03:34:36.320+0000 7f6f33fbe700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:34:42.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:41 smithi153 podman[152129]: 2024-04-05 03:34:41.81381427 +0000 UTC m=+5.592147747 container died 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-05T03:34:42.427 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:42 smithi153 podman[152129]: 2024-04-05 03:34:42.195789588 +0000 UTC m=+5.974123059 container cleanup 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:34:42.427 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:42 smithi153 bash[152129]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0 2024-04-05T03:34:42.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:42 smithi153 podman[152147]: 2024-04-05 03:34:42.430431413 +0000 UTC m=+0.614534683 container remove 81dada4d04a038516f6e458450b7c1deea7b089ca9fb4e86573c36cc03316242 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:34:43.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:42 smithi153 podman[152233]: 2024-04-05 03:34:42.672606651 +0000 UTC m=+0.020909740 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:34:43.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:42 smithi153 podman[152233]: 2024-04-05 03:34:42.789710754 +0000 UTC m=+0.138013833 container create 121c763e657edc7e9262a32d96e6cd99084cf4236048e835a44ba578c0d62b9e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-0028525, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-05T03:34:43.071 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:43 smithi153 podman[152233]: 2024-04-05 03:34:43.000979594 +0000 UTC m=+0.349282672 container init 121c763e657edc7e9262a32d96e6cd99084cf4236048e835a44ba578c0d62b9e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:43.071 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:43 smithi153 podman[152233]: 2024-04-05 03:34:43.005395664 +0000 UTC m=+0.353698736 container start 121c763e657edc7e9262a32d96e6cd99084cf4236048e835a44ba578c0d62b9e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-05T03:34:43.330 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:43 smithi153 podman[152233]: 2024-04-05 03:34:43.092488306 +0000 UTC m=+0.440791381 container attach 121c763e657edc7e9262a32d96e6cd99084cf4236048e835a44ba578c0d62b9e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:34:43.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Apr 05 03:34:43 smithi153 podman[152233]: 2024-04-05 03:34:43.333785235 +0000 UTC m=+0.682088310 container died 121c763e657edc7e9262a32d96e6cd99084cf4236048e835a44ba578c0d62b9e (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-0028525, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-05T03:34:43.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-05T03:34:44.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:34:44.040 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-05T03:34:44.041 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-05T03:34:44.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.1 2024-04-05T03:34:44.320 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:44 smithi153 systemd[1]: Stopping Ceph osd.1 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:34:44.820 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:44 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:34:44.363+0000 7f2550a4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:34:44.820 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:44 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:34:44.363+0000 7f2550a4b700 -1 osd.1 84 *** Got signal Terminated *** 2024-04-05T03:34:44.820 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:44 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1[122763]: 2024-04-05T03:34:44.363+0000 7f2550a4b700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:34:50.070 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:49 smithi153 podman[152341]: 2024-04-05 03:34:49.815277801 +0000 UTC m=+5.578404691 container died 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-0028525, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-05T03:34:50.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:50 smithi153 podman[152341]: 2024-04-05 03:34:50.207413891 +0000 UTC m=+5.970540771 container cleanup 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:34:50.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:50 smithi153 bash[152341]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1 2024-04-05T03:34:50.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:50 smithi153 podman[152370]: 2024-04-05 03:34:50.44232615 +0000 UTC m=+0.625260064 container remove 4ca694768d37a8aa2e44598ce2844971c2ec1b6823370b626c49701cbded686d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-0028525, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:34:51.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:50 smithi153 podman[152463]: 2024-04-05 03:34:50.668122932 +0000 UTC m=+0.017488134 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:34:51.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:50 smithi153 podman[152463]: 2024-04-05 03:34:50.793468097 +0000 UTC m=+0.142833282 container create 0f636d380d998320ef85e36fc503346828d9d1a78cdd49ada8f23f9874d14448 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:51.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:51 smithi153 podman[152463]: 2024-04-05 03:34:51.021257219 +0000 UTC m=+0.370622417 container init 0f636d380d998320ef85e36fc503346828d9d1a78cdd49ada8f23f9874d14448 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, GIT_BRANCH=HEAD) 2024-04-05T03:34:51.320 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:51 smithi153 podman[152463]: 2024-04-05 03:34:51.024652453 +0000 UTC m=+0.374017643 container start 0f636d380d998320ef85e36fc503346828d9d1a78cdd49ada8f23f9874d14448 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=squid-0028525, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:51.320 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:51 smithi153 podman[152463]: 2024-04-05 03:34:51.104189287 +0000 UTC m=+0.453554491 container attach 0f636d380d998320ef85e36fc503346828d9d1a78cdd49ada8f23f9874d14448 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-0028525, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-05T03:34:51.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 05 03:34:51 smithi153 podman[152463]: 2024-04-05 03:34:51.347440981 +0000 UTC m=+0.696806168 container died 0f636d380d998320ef85e36fc503346828d9d1a78cdd49ada8f23f9874d14448 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-0028525, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-05T03:34:51.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-05T03:34:51.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:34:51.970 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-05T03:34:51.971 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-05T03:34:51.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.2 2024-04-05T03:34:52.317 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:52 smithi153 systemd[1]: Stopping Ceph osd.2 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:34:52.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:52 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:34:52.319+0000 7f5e1434b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:34:52.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:52 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:34:52.319+0000 7f5e1434b700 -1 osd.2 84 *** Got signal Terminated *** 2024-04-05T03:34:52.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:52 smithi153 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2[128411]: 2024-04-05T03:34:52.319+0000 7f5e1434b700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:34:57.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:57 smithi153 podman[152570]: 2024-04-05 03:34:57.70256528 +0000 UTC m=+5.497064094 container died fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-0028525, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:58.320 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152570]: 2024-04-05 03:34:58.052308553 +0000 UTC m=+5.846807385 container cleanup fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:34:58.320 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 bash[152570]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2 2024-04-05T03:34:58.668 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152589]: 2024-04-05 03:34:58.328567302 +0000 UTC m=+0.629829873 container remove fc7ea381564e0982ff1bd397acadb353a5d63b6953419e60f259bfb4c03e8094 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:34:58.937 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152681]: 2024-04-05 03:34:58.574474407 +0000 UTC m=+0.022755097 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:34:58.937 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152681]: 2024-04-05 03:34:58.698712765 +0000 UTC m=+0.146993454 container create 8bc641bc52e1f9262a89f41196931211e0112c72140cf80d0491b5a4729d82ed (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-05T03:34:58.938 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152681]: 2024-04-05 03:34:58.940527525 +0000 UTC m=+0.388808225 container init 8bc641bc52e1f9262a89f41196931211e0112c72140cf80d0491b5a4729d82ed (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-0028525, io.buildah.version=1.33.5) 2024-04-05T03:34:59.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:58 smithi153 podman[152681]: 2024-04-05 03:34:58.94473697 +0000 UTC m=+0.393017667 container start 8bc641bc52e1f9262a89f41196931211e0112c72140cf80d0491b5a4729d82ed (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-0028525, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-05T03:34:59.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:59 smithi153 podman[152681]: 2024-04-05 03:34:59.015484574 +0000 UTC m=+0.463765266 container attach 8bc641bc52e1f9262a89f41196931211e0112c72140cf80d0491b5a4729d82ed (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-05T03:34:59.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 05 03:34:59 smithi153 podman[152681]: 2024-04-05 03:34:59.261208884 +0000 UTC m=+0.709489572 container died 8bc641bc52e1f9262a89f41196931211e0112c72140cf80d0491b5a4729d82ed (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5) 2024-04-05T03:34:59.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-05T03:34:59.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:34:59.932 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-05T03:34:59.932 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-05T03:34:59.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.3 2024-04-05T03:35:00.241 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:34:59 smithi203 systemd[1]: Stopping Ceph osd.3 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:35:00.500 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:00 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:35:00.243+0000 7f985b9e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:35:00.501 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:00 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:35:00.243+0000 7f985b9e7700 -1 osd.3 84 *** Got signal Terminated *** 2024-04-05T03:35:00.501 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:00 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:35:00.243+0000 7f985b9e7700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:35:03.044 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:02 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:02.735+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:03.975 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:03.714+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:04.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:03 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:03.977+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:04.949 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:04 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:04.698+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:04.950 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:04 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3[105438]: 2024-04-05T03:35:04.849+0000 7f98547d4700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:40.226412+0000 front 2024-04-05T03:34:40.226527+0000 (oldest deadline 2024-04-05T03:35:04.326591+0000) 2024-04-05T03:35:05.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:04 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:04.951+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:05.834 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:05 smithi203 podman[130622]: 2024-04-05 03:35:05.567191202 +0000 UTC m=+5.439190406 container died 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-05T03:35:05.835 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:05 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:05.711+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:06.209 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:05 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:05.934+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:06.209 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:05 smithi203 podman[130622]: 2024-04-05 03:35:05.927285313 +0000 UTC m=+5.799284520 container cleanup 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-0028525, ceph=True, org.label-schema.schema-version=1.0) 2024-04-05T03:35:06.209 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:05 smithi203 bash[130622]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3 2024-04-05T03:35:06.538 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130653]: 2024-04-05 03:35:06.211995258 +0000 UTC m=+0.639912749 container remove 3c8b7968d2cb31e98c8e74d72b7cb6a1cdc4c99ab597a2a8faee2cb31d1bd901 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3, org.label-schema.build-date=20240403, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-05T03:35:06.794 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:06 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:06.695+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:06.795 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130748]: 2024-04-05 03:35:06.443611105 +0000 UTC m=+0.018004997 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:35:06.795 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130748]: 2024-04-05 03:35:06.580007062 +0000 UTC m=+0.154400944 container create 4bf88676b273c5b354fdd5abbb11b12a8decbc7ff6e5e887d2c974de56bba13d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-0028525, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-05T03:35:07.133 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:06 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:06.917+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:07.133 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130748]: 2024-04-05 03:35:06.835795956 +0000 UTC m=+0.410189842 container init 4bf88676b273c5b354fdd5abbb11b12a8decbc7ff6e5e887d2c974de56bba13d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-05T03:35:07.134 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130748]: 2024-04-05 03:35:06.838986549 +0000 UTC m=+0.413380437 container start 4bf88676b273c5b354fdd5abbb11b12a8decbc7ff6e5e887d2c974de56bba13d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-05T03:35:07.134 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:06 smithi203 podman[130748]: 2024-04-05 03:35:06.92691559 +0000 UTC m=+0.501309482 container attach 4bf88676b273c5b354fdd5abbb11b12a8decbc7ff6e5e887d2c974de56bba13d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-0028525, org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:35:07.495 INFO:journalctl@ceph.osd.3.smithi203.stdout:Apr 05 03:35:07 smithi203 podman[130748]: 2024-04-05 03:35:07.136580351 +0000 UTC m=+0.710974230 container died 4bf88676b273c5b354fdd5abbb11b12a8decbc7ff6e5e887d2c974de56bba13d (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240403, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-0028525) 2024-04-05T03:35:07.786 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:07 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:07.654+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:07.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-05T03:35:07.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:35:07.856 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-05T03:35:07.857 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-05T03:35:07.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.4 2024-04-05T03:35:08.044 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:07 smithi203 systemd[1]: Stopping Ceph osd.4 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:35:08.045 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:07 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:07.925+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:08.544 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:08.159+0000 7f99ed7de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:35:08.544 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:08.159+0000 7f99ed7de700 -1 osd.4 84 *** Got signal Terminated *** 2024-04-05T03:35:08.544 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:08.159+0000 7f99ed7de700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:35:08.921 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:08.630+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:09.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:08 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:08.923+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:09.891 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:09 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:09.633+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:10.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:09 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:09.893+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:10.934 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:10 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:10.656+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:10.935 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:10 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:10.844+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:11.794 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:11.687+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:11.794 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:11.687+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:47.533484+0000 front 2024-04-05T03:34:47.533427+0000 (oldest deadline 2024-04-05T03:35:11.633393+0000) 2024-04-05T03:35:12.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:11.797+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:12.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:11 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:11.797+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:13.044 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:12.681+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:41.132782+0000 front 2024-04-05T03:34:41.132800+0000 (oldest deadline 2024-04-05T03:35:02.232709+0000) 2024-04-05T03:35:13.044 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4[110834]: 2024-04-05T03:35:12.681+0000 7f99ea5d3700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:47.533484+0000 front 2024-04-05T03:34:47.533427+0000 (oldest deadline 2024-04-05T03:35:11.633393+0000) 2024-04-05T03:35:13.045 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:12.818+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:13.045 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:12 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:12.818+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:13.925 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:13 smithi203 podman[130855]: 2024-04-05 03:35:13.593767847 +0000 UTC m=+5.550907985 container died 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-0028525, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-05T03:35:13.925 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:13 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:13.837+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:13.925 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:13 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:13.837+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:14.216 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130855]: 2024-04-05 03:35:14.01790811 +0000 UTC m=+5.975048247 container cleanup 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-0028525, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-05T03:35:14.216 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 bash[130855]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4 2024-04-05T03:35:14.544 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130873]: 2024-04-05 03:35:14.219406238 +0000 UTC m=+0.615252990 container remove 57d855c137b526b29567ba3d523fa724bfd973611c275ffbbce3d89bb0bb8dce (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-0028525, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, ceph=True) 2024-04-05T03:35:14.912 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:14 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:14.879+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:14.912 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:14 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:14.879+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:14.913 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130964]: 2024-04-05 03:35:14.538008926 +0000 UTC m=+0.017759343 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:35:14.913 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130964]: 2024-04-05 03:35:14.662144571 +0000 UTC m=+0.141894987 container create 164d39d51a8616c078530d370ca1bd12084ddcb9f946b21e9e885108fa45edbd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-0028525, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-05T03:35:14.913 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130964]: 2024-04-05 03:35:14.914844622 +0000 UTC m=+0.394595052 container init 164d39d51a8616c078530d370ca1bd12084ddcb9f946b21e9e885108fa45edbd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, RELEASE=squid-0028525, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:35:15.220 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:14 smithi203 podman[130964]: 2024-04-05 03:35:14.918734667 +0000 UTC m=+0.398485084 container start 164d39d51a8616c078530d370ca1bd12084ddcb9f946b21e9e885108fa45edbd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, RELEASE=squid-0028525, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-05T03:35:15.220 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:15 smithi203 podman[130964]: 2024-04-05 03:35:15.031948691 +0000 UTC m=+0.511699116 container attach 164d39d51a8616c078530d370ca1bd12084ddcb9f946b21e9e885108fa45edbd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, RELEASE=squid-0028525, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-04-05T03:35:15.507 INFO:journalctl@ceph.osd.4.smithi203.stdout:Apr 05 03:35:15 smithi203 podman[130964]: 2024-04-05 03:35:15.223800734 +0000 UTC m=+0.703551147 container died 164d39d51a8616c078530d370ca1bd12084ddcb9f946b21e9e885108fa45edbd (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:35:15.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-05T03:35:15.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:35:15.826 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-05T03:35:15.826 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-05T03:35:15.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.5 2024-04-05T03:35:16.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:15 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:15.867+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:16.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:15 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:15.867+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:16.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:15 smithi203 systemd[1]: Stopping Ceph osd.5 for 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4... 2024-04-05T03:35:16.544 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:16 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:16.146+0000 7f4b0b1be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T03:35:16.544 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:16 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:16.146+0000 7f4b0b1be700 -1 osd.5 84 *** Got signal Terminated *** 2024-04-05T03:35:16.544 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:16 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:16.146+0000 7f4b0b1be700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T03:35:17.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:16 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:16.881+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:17.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:16 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:16.881+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:18.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:17 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:17.851+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:18.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:17 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:17.851+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:19.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:18 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:18.824+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:19.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:18 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:18.824+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:20.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:19 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:19.778+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:20.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:19 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:19.778+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:21.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:20 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:20.795+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-04-05T03:34:39.091448+0000 front 2024-04-05T03:34:39.091585+0000 (oldest deadline 2024-04-05T03:35:03.191592+0000) 2024-04-05T03:35:21.044 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:20 smithi203 ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5[115820]: 2024-04-05T03:35:20.795+0000 7f4b03fab700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-04-05T03:34:45.492418+0000 front 2024-04-05T03:34:45.492511+0000 (oldest deadline 2024-04-05T03:35:11.292903+0000) 2024-04-05T03:35:21.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:21 smithi203 podman[131070]: 2024-04-05 03:35:21.703171961 +0000 UTC m=+5.656842279 container died 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-0028525, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-05T03:35:22.254 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131070]: 2024-04-05 03:35:22.046714889 +0000 UTC m=+6.000385209 container cleanup 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-0028525, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-05T03:35:22.254 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 bash[131070]: ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5 2024-04-05T03:35:22.544 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131086]: 2024-04-05 03:35:22.25676807 +0000 UTC m=+0.549913690 container remove 750dd619bf46067d25171ff2af42cde52b72c9d6fbec1e9287ec583929403ac0 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-0028525, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-05T03:35:22.982 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131179]: 2024-04-05 03:35:22.563646916 +0000 UTC m=+0.018059434 image pull d8c35eaf3c25b624514c7b18d9f1d2cedc1bbdfe6c87a522d661faa4038ccad8 quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca 2024-04-05T03:35:22.982 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131179]: 2024-04-05 03:35:22.682629877 +0000 UTC m=+0.137042380 container create da794d6c5299c06674bd6761dcc25e94e010bbd1ec4bfdb7ff259bb482a56f10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-05T03:35:22.982 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131179]: 2024-04-05 03:35:22.902136426 +0000 UTC m=+0.356548932 container init da794d6c5299c06674bd6761dcc25e94e010bbd1ec4bfdb7ff259bb482a56f10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-0028525) 2024-04-05T03:35:22.983 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131179]: 2024-04-05 03:35:22.90528479 +0000 UTC m=+0.359697296 container start da794d6c5299c06674bd6761dcc25e94e010bbd1ec4bfdb7ff259bb482a56f10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-0028525, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-05T03:35:22.983 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:22 smithi203 podman[131179]: 2024-04-05 03:35:22.985341938 +0000 UTC m=+0.439754458 container attach da794d6c5299c06674bd6761dcc25e94e010bbd1ec4bfdb7ff259bb482a56f10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-0028525) 2024-04-05T03:35:23.294 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 05 03:35:23 smithi203 podman[131179]: 2024-04-05 03:35:23.197358362 +0000 UTC m=+0.651770869 container died da794d6c5299c06674bd6761dcc25e94e010bbd1ec4bfdb7ff259bb482a56f10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b1ac2def9baea35d725102f90134220b12af7a73c65adaf7747c4c8e1408c7ca, name=ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-0028525, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-05T03:35:23.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa69b9e-f2f7-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-05T03:35:23.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T03:35:23.850 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-05T03:35:23.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T03:35:27.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi153.stderr:ceph-fuse[75258]: fuse finished with error 0 and tester_r 0 2024-04-05T03:35:46.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T03:36:02.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T03:36:02.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T03:36:02.896 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-05T03:36:02.899 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi153/crash 2024-04-05T03:36:02.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T03:36:02.937 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T03:36:02.937 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-05T03:36:02.939 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi203/crash 2024-04-05T03:36:02.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T03:36:02.966 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T03:36:02.966 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-04-05T03:36:02.967 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-05T03:36:02.967 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-04-05T03:36:03.019 INFO:teuthology.orchestra.run.smithi153.stdout:1712285706.7406883 mon.smithi153 (mon.0) 170 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-05T03:36:03.019 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-05T03:36:03.019 INFO:tasks.cephadm:Compressing logs... 2024-04-05T03:36:03.019 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-05T03:36:03.062 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi153.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.vicaky.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi153.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-05T03:36:03.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.vicaky.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi153.npzyqy.log 2024-04-05T03:36:03.091 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.fmpmvd.log 2024-04-05T03:36:03.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi153.npzyqy.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.fmpmvd.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:: No such file or directory 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi203.iakodz.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi203.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi203.iakodz.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi203.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.xpteyc.log 2024-04-05T03:36:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.3.log: 91.1% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-05T03:36:03.098 INFO:teuthology.orchestra.run.smithi153.stderr: 91.0% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-05T03:36:03.098 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-05T03:36:03.098 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-05T03:36:03.099 INFO:teuthology.orchestra.run.smithi203.stderr: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.hgbifk.log 2024-04-05T03:36:03.099 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.xpteyc.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-05T03:36:03.108 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log 2024-04-05T03:36:03.110 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.hgbifk.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log 2024-04-05T03:36:03.110 INFO:teuthology.orchestra.run.smithi203.stderr: 85.1% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-05T03:36:03.110 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-05T03:36:03.117 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-05T03:36:03.117 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-05T03:36:03.118 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-05T03:36:03.122 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log: 88.9% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-05T03:36:03.122 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-05T03:36:03.123 INFO:teuthology.orchestra.run.smithi203.stderr: 89.0% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-05T03:36:03.127 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-05T03:36:03.161 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log: /var/log/ceph/ceph-client.1.log: 90.1% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi203.iakodz.log.gz 2024-04-05T03:36:03.162 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-05T03:36:03.184 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.2.log: /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-05T03:36:03.610 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi153.npzyqy.log.gz 2024-04-05T03:36:03.737 INFO:teuthology.orchestra.run.smithi203.stderr: 92.5% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi203.log.gz 2024-04-05T03:36:04.869 INFO:teuthology.orchestra.run.smithi153.stderr: 90.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mon.smithi153.log.gz 2024-04-05T03:36:08.574 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-05T03:36:08.751 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-05T03:36:08.946 INFO:teuthology.orchestra.run.smithi203.stderr: 93.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-05T03:36:09.273 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-05T03:36:09.853 INFO:teuthology.orchestra.run.smithi203.stderr: 93.5% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-05T03:36:11.789 INFO:teuthology.orchestra.run.smithi203.stderr: 93.4% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-05T03:36:12.109 INFO:teuthology.orchestra.run.smithi203.stderr: 95.2% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.hgbifk.log.gz 2024-04-05T03:36:12.179 INFO:teuthology.orchestra.run.smithi153.stderr: 95.2% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.vicaky.log.gz 2024-04-05T03:36:19.052 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-05T03:36:19.052 INFO:teuthology.orchestra.run.smithi203.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-05T03:36:19.198 INFO:teuthology.orchestra.run.smithi153.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-05T03:36:46.815 INFO:teuthology.orchestra.run.smithi203.stderr: 93.4% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi203.xpteyc.log.gz 2024-04-05T03:36:46.817 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-05T03:36:46.817 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m43.741s 2024-04-05T03:36:46.817 INFO:teuthology.orchestra.run.smithi203.stderr:user 1m27.860s 2024-04-05T03:36:46.817 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m2.680s 2024-04-05T03:36:50.757 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/3fa69b9e-f2f7-11ee-b649-cb9ed24678a4/ceph-mds.cephfs.smithi153.fmpmvd.log.gz 2024-04-05T03:36:50.759 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-05T03:36:50.759 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m47.685s 2024-04-05T03:36:50.760 INFO:teuthology.orchestra.run.smithi153.stderr:user 1m29.160s 2024-04-05T03:36:50.760 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m3.161s 2024-04-05T03:36:50.760 INFO:tasks.cephadm:Archiving logs... 2024-04-05T03:36:50.760 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi153/log 2024-04-05T03:36:50.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T03:37:03.696 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi203/log 2024-04-05T03:37:03.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T03:37:15.961 INFO:tasks.cephadm:Removing cluster... 2024-04-05T03:37:15.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 --force 2024-04-05T03:37:16.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa69b9e-f2f7-11ee-b649-cb9ed24678a4 --force 2024-04-05T03:37:17.163 INFO:tasks.cephadm:Removing cephadm ... 2024-04-05T03:37:17.164 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T03:37:17.181 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T03:37:17.196 INFO:tasks.cephadm:Teardown complete 2024-04-05T03:37:17.196 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-05T03:37:17.213 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T03:37:17.214 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-05T03:37:17.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T03:37:17.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T03:37:17.315 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-05T03:37:17.328 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-05T03:37:17.328 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T03:37:17.331 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T03:37:17.343 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-05T03:37:17.345 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 -708us[ -708us] +/- 97ms 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 65 +54us[ +55us] +/- 60ms 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +409us[ +409us] +/- 65ms 2024-04-05T03:37:17.348 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T03:37:17.349 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T03:37:17.349 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-05T03:37:17.350 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 -712us[ -732us] +/- 97ms 2024-04-05T03:37:17.350 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 +63us[ +43us] +/- 60ms 2024-04-05T03:37:17.350 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +419us[ +399us] +/- 65ms 2024-04-05T03:37:17.350 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T03:37:17.350 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-05T03:37:17.363 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-05T03:37:17.364 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-05T03:37:17.378 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T03:37:17.475 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T03:37:17.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-05T03:37:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:37:17.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-05T03:37:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:37:17.617 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-05T03:37:17.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-05T03:37:17.693 INFO:teuthology.task.internal:Duration was 3243.843219 seconds 2024-04-05T03:37:17.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-05T03:37:17.705 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-05T03:37:17.705 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T03:37:17.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T03:37:17.748 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T03:37:17.752 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T03:37:18.107 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-05T03:37:18.107 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-05T03:37:18.108 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T03:37:18.167 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-05T03:37:18.167 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T03:37:18.197 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-05T03:37:18.197 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T03:37:18.210 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T03:37:18.267 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-05T03:37:18.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T03:37:18.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T03:37:19.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-05T03:37:19.134 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-05T03:37:19.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T03:37:19.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T03:37:19.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-05T03:37:19.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T03:37:19.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T03:37:19.230 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-05T03:37:19.260 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-05T03:37:19.577 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T03:37:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:37:19.591 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T03:37:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:37:19.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-05T03:37:19.647 INFO:teuthology.task.internal:Transferring archived files... 2024-04-05T03:37:19.648 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi153 2024-04-05T03:37:19.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T03:37:19.705 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241/remote/smithi203 2024-04-05T03:37:19.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T03:37:19.752 INFO:teuthology.task.internal:Removing archive directory... 2024-04-05T03:37:19.752 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T03:37:19.755 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T03:37:19.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-05T03:37:19.815 INFO:teuthology.task.internal:Not uploading archives. 2024-04-05T03:37:19.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-05T03:37:19.827 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-05T03:37:19.828 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T03:37:19.830 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T03:37:19.843 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 5 03:37 /home/ubuntu/cephtest 2024-04-05T03:37:19.843 INFO:teuthology.orchestra.run.smithi153.stdout: 666185 4 d--------- 2 ubuntu ubuntu 4096 Apr 5 03:00 /home/ubuntu/cephtest/mnt.0 2024-04-05T03:37:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-05T03:37:19.844 INFO:teuthology.orchestra.run.smithi153.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-05T03:37:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T03:37:19.844 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph_9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-05T03:37:19.846 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-05T03:37:19.860 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-05T03:37:19.873 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T03:37:19.873 INFO:teuthology.nuke:Checking targets against current locks 2024-04-05T03:37:19.900 DEBUG:teuthology.nuke:shortname: smithi153 2024-04-05T03:37:19.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T03:37:19.923 DEBUG:teuthology.nuke:shortname: smithi203 2024-04-05T03:37:19.924 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T03:37:19.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 02:34:04.049151', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIqvfTydc8eNoNP7JJcUTv+zFEoz6LliN1iHyZNDmR6Z56IJThqPrsYzps+muW4nmbyg/TQe1ZvQ12NWLLvh3pU='} 2024-04-05T03:37:19.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_02:08:13-upgrade-squid-distro-default-smithi/7641241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 02:34:04.047720', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWj/nYva+qMAnCZelK9sstMdPhQ3++tTCxh4dzIMfL5e/3TDnp+FxE8FQghwVMwdb5hxv70bFCvu3OQ0HjZ/Qc='} 2024-04-05T03:37:19.980 INFO:teuthology.orchestra.console.smithi153:Power off 2024-04-05T03:37:19.980 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T03:37:20.009 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-04-05T03:37:20.009 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:20.025 INFO:teuthology.orchestra.console.smithi203:Power off 2024-04-05T03:37:20.026 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T03:37:20.035 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-05T03:37:20.048 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-04-05T03:37:20.049 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:20.071 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-05T03:37:24.037 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:24.072 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:24.146 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-05T03:37:24.182 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-05T03:37:28.150 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:28.183 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:28.260 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-04-05T03:37:28.293 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-05T03:37:28.362 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-04-05T03:37:32.294 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T03:37:32.404 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-04-05T03:37:32.505 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-04-05T03:37:32.610 INFO:teuthology.run:Summary data: description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3243.8432188034058 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-05T03:37:32.610 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T03:37:32.694 INFO:teuthology.run:FAIL